Step Fun AI's Step 5 Preview Tested: How a Smaller Model Rivals Kimi and GLM

A foreign AI blogger recently ran local benchmarks on a Chinese large language model — specifically, the Step 5 Preview released by Step Fun AI (阶跃星辰). What makes this model stand out is that despite being significantly smaller than comparable flagship models, it climbed above Kimi K3 on the Artificial Analysis leaderboard and sits just behind Grok and GLM. This hands-on video review covers its real-world performance in local inference, multimodal understanding, code generation, and logical reasoning — while also exposing the rough edges still present in a preview release.

Impressive Rankings: A Smaller Model Challenging Trillion-Parameter Giants

According to the blogger's findings, Step 5 Preview scores 44 on Artificial Analysis's Intelligence Score leaderboard — surpassing Kimi K3 (which has roughly 2.8 trillion parameters) and outranking GLM, trailing only top-tier closed-source models like Astra and Opus 5. The blogger emphasizes that this is a preview version, not a full release, and that the model is considerably smaller than its competitors.

Step Fun AI has received heavy investment from Alibaba — reportedly one of China's largest AI funding rounds — and shares the same Alibaba ecosystem as the Qwen team. The blogger hints at the company's open-source-friendly track record by mentioning that earlier releases used the Apache 2 license, though he admits the licensing situation for Step 5 Preview remains unclear.

One detail that might have slipped under the radar: the model was only publicly available for about an hour on launch day before being taken down, with an official full release planned for the following month. This means the version the blogger tested may contain bugs, and no official inference code was released alongside it — so test results should be taken with a grain of salt.

Artificial Analysis is a third-party platform focused on comprehensive LLM performance evaluation. Its "Intelligence Score" aggregates model performance across multiple benchmarks including reasoning, coding, mathematics, and knowledge Q&A, aiming to provide a single comparable metric. The platform also measures inference speed, context window size, and pricing — making it a common reference for developers evaluating models. That said, any single evaluation platform has inherent topic biases, and scores may not always reflect real-world task performance. The blogger's own hands-on results with Step 5 confirm that benchmark scores don't always map one-to-one to capability in specific tasks.

Local Deployment: Memory and Speed Under Q4 Quantization

The blogger primarily ran the Q4 quantized version, which requires approximately 315 GiB of memory and generates at around 20 tokens per second. To compare quality across precision levels, he also tested Q5 and Q6 quantization. Some complex tasks triggered runtime errors under Q4, requiring a higher quantization level for stable output.

The model supports multiple thinking modes — the official options are low, medium, and high — and the blogger also added a "disable thinking" mode that he verified works. He specifically noted that Step 5 is not a "brute-force thinker" — unlike some models that burn through 150,000 tokens and take extended time to converge. He even built an "Answer Immediately" button that can interrupt the thinking process mid-stream and force a direct response.

Since the official team later released older inference code, the blogger adapted it himself — which is another reason he repeatedly stresses that results should be treated as informal.

Quantization is a technique that compresses model weights from high-precision floating-point numbers (like FP16/BF16) into low-bit integers. Q4 means each weight is represented with 4 bits, reducing memory usage by roughly 4x compared to FP16, at the cost of slight precision loss. Q5 and Q6 preserve progressively more precision at the cost of higher memory usage — which is why the blogger had to step up from Q4 to Q5 to get stable output on tasks like the FPS shooter game. The 315 GiB memory requirement means that even the Q4 version demands multiple high-VRAM GPUs or a dedicated inference server, far beyond consumer-grade hardware.

Multimodal and Code Generation: Highlights and Failures

One of Step 5's headline features is multimodal capability. Given a photo of a cat, it correctly identified it as an "orange tabby" and generated a Three.js 3D model based on it — nothing impressive, but reportedly better than DeepSeek^Ad V4's equivalent output. It also converted the cat image into a voxel-style render with an animated tail.

Code generation was tested across a wide range of tasks:

• 3D Solar System: Basic but functional, with correct planet lighting;
• FPS Shooter Game: Frequent errors under Q4; stepping up to Q5 produced a clean single-pass output of 23,000 tokens, complete with a red-ceilinged level with corridors and shootable enemies;
• Modifying existing code: Adding a spaceship and laser to a solar system written by Qwen — the blogger called the result "pretty epic," with the sun and planets successfully exploding, though collision detection wasn't perfect;
• Realistic face rendering: Even under extreme Q1/Q4 quantization, it generated a face with randomizable facial features — described as "very smart for a quantized version."

Productivity applications were another pleasant surprise. The blogger had it generate browser-based prototypes of Blender, iMovie, Photoshop, and MS Word: the Blender version could rotate and add cubes and import architectural models; the iMovie version featured a timeline with title insertion, opacity control, and a flickering playback effect; the Word version supported font size adjustment, bold formatting, and layering. The blogger considers canvas-based web apps and UI controls to be a genuine strength of this model.

The weaknesses were equally apparent: roads in a GTA-style city drove straight into buildings, and the city layout was overly uniform; a kart racing game generated poorly under Q4 high-thinking mode; Final Fantasy and Super Mario projects largely failed; and a 3D human anatomy render placed the heart in the wrong position — the blogger explicitly warned viewers not to trust it for medical information.

Three.js is a JavaScript 3D graphics library built on WebGL that allows developers to create and render interactive 3D scenes in the browser without any plugins. Voxels are the 3D equivalent of pixels — the smallest unit in three-dimensional space — commonly seen in Minecraft-style aesthetics, built from stacked cubes. The blogger chose these two task types because they intuitively demonstrate the model's understanding of three-dimensional spatial relationships and its ability to translate natural language descriptions into runnable code — a challenge that combines both creative understanding and code generation, making it an effective benchmark for multimodal and programming capability integration.

Logical Reasoning: Surprisingly Solid

Step 5 held up well on logic and trick questions. It correctly solved the "8 oranges, 4 children" equal distribution problem without being thrown off by the irrelevant "knife" distractor. It didn't fall into the "the surgeon must be the mother" training bias trap on the classic gender-assumption puzzle. Math problems under high thinking mode also yielded the correct answer: "must be an even number."

The blogger suspects this category of reasoning is precisely what pushed Step 5's score up on Artificial Analysis. Programming knowledge questions were more mixed: standard C++ vectors and Swift string functions were handled correctly, but WinRT-related questions were answered incorrectly even in high thinking mode. On the safety front, the model refused to generate nuclear weapons-related code, flagging the input as invalid.

Conclusion: A Release Worth Watching

The blogger's overall verdict is cautiously optimistic. He admits he doesn't fully understand how Step 5 Preview managed to "beat Kimi K3" on the leaderboard, suggests Kimi may outperform it on certain generation tasks, and wonders whether his custom inference code adaptation played a role. But he underscores one key point: if a Q4 quantized model requiring only ~300 GiB of memory can genuinely match Grok 4.6 and Kimi K3 levels in its full release, that would be "very, very exciting."

At the time of testing, Step 5 was not yet available on Hugging Face for public download, with the official release expected the following month. For developers tracking Chinese open-source LLMs — especially those looking to deploy high-value models locally — this is one to keep an eye on.