StepFun Open-Sources Step 5 Preview: A 600B MoE, 1M-Context Agent Foundation

StepFun has released Step 5 Preview as open weights, with both FP8 and BF16 checkpoints now available on Hugging Face and ModelScope. This is no ordinary model release: a sparse MoE architecture with 600B total parameters, 27B activated per token, and up to 1M tokens of context—pointing clearly at one target: long-horizon agentic tasks.

Architecture Choice: Why Sparse MoE Suits Agents

The combination of 600B total parameters and 27B activated parameters means the model invokes only about 4.5% of its parameters during inference. This design has real significance for agent scenarios: multi-turn tool calls, long-document retrieval, and cross-step planning continuously consume tokens, and if every step ran the full parameter set, latency and cost would spiral out of control.

Sparse activation lets the model maintain a large-capacity knowledge base while keeping per-token compute close to that of a 27B dense model. For an agent that must repeatedly interact with its environment, this directly determines whether a task can be completed in acceptable time.

StepFun positions Step 5 Preview as "built for long-horizon agentic work," and the architecture choice is consistent with that positioning: large capacity to carry complex reasoning and tool semantics, sparse activation to control the cost of long-chain execution.

MoE (Mixture of Experts) is an architecture that splits a single large model into multiple "expert" subnetworks. After each token is input, a router computes and selects the few highest-scoring experts to participate in computation, while the remaining experts stay inactive. As a result, total parameter count can be very large, but per-token FLOPs depend only on the number of activated experts. This decoupling of "parameter capacity" from "compute" means MoE typically has stronger knowledge capacity than a dense model at the same inference cost. The trade-offs are harder-to-control training stability, imbalanced routing load that can leave some experts idle, and the need to load all expert weights into GPU memory at deployment—which is why a 600B MoE, even activating only 27B, still has far higher memory requirements than a 27B dense model.

1M Context: A Prerequisite for Long-Horizon Tasks

A 1M-token context is the other key number in this release. Agent tasks are characterized by continuously expanding context—tool return results, file contents, conversation history, and intermediate reasoning all accumulate. If the context window is too small, the agent must frequently summarize and compress, and every compression is a loss of information.

A 1M window lets the model hold complete code repository fragments, long documents, or multi-turn tool-call records within a single session, reducing task failures caused by truncation. For scenarios that require state consistency across files and steps, this is a baseline capability rather than a nice-to-have.

Ultra-long context brings not just window capacity but also the problem of quadratic attention complexity: standard self-attention grows as O(n²) with sequence length, so attention overhead balloons at 1M tokens. The industry typically mitigates this through IO-aware exact attention implementations like FlashAttention, sliding-window/sparse attention, or chunking and state compression. Moreover, the practical effectiveness of long context is limited by the "effective context length"—a model's ability to retrieve information from the middle of an ultra-long sequence is often weaker than at the ends, the so-called "lost in the middle" phenomenon. So a 1M window is a necessary but not sufficient condition; long-horizon stability in agent scenarios still depends on the long-sequence data mix during training and the positional encoding extrapolation scheme.

Open Weights: Dual FP8 and BF16 Releases

StepFun provides both FP8 and BF16 checkpoints, hosted on Hugging Face and ModelScope. The FP8 version targets inference deployment, with advantages in memory footprint and throughput; the BF16 version targets fine-tuning and precision-sensitive scenarios. Releasing both versions in parallel shows that StepFun treats developer adoption as a primary goal rather than a mere technical showcase.

For developers, this means weights can be downloaded directly for local deployment, domain fine-tuning, or integration into their own agent frameworks, without relying on a closed-source API. Open weights also lower the barrier to experimenting on private data—agent scenarios often involve internal enterprise tools and data, where private deployment is a must.

FP8 (8-bit floating point) and BF16 (Brain Floating Point 16) are two numerical precision formats. BF16 uses 16 bits, with a dynamic range matching FP32 but lower mantissa precision, and is the mainstream format for training and fine-tuning today. FP8 compresses further to 8 bits, with E4M3 and E5M2 variants, and can achieve nearly double the throughput on newer GPUs that support FP8 compute, such as Hopper and Blackwell, while using about half the memory of BF16. The trade-off is quantization error, so precision-sensitive fine-tuning tasks usually still require BF16 or higher precision. That makes "FP8 for inference deployment, BF16 for fine-tuning" a common dual-release strategy—provided the deployment hardware has native FP8 compute support; on older-generation GPUs, FP8 may instead require dequantization and lose its advantage.

What This Means for Developers

An open-weight MoE model at the 600B level still has a non-trivial deployment barrier in practice, but its 27B activation volume makes it feasible in multi-GPU inference configurations. More noteworthy is the ecosystem niche it fills: open-source foundation models built for long-horizon agents remain scarce, and most open models are not optimized for agent workflows in terms of context length or tool-calling stability.

The "Preview" label on Step 5 Preview is also worth noting—it signals a staged version whose capabilities and interfaces may change. Teams planning to build production systems on it need to factor in version iteration risk.

From an industry perspective, open-weight large MoE models are becoming part of the competition over agent infrastructure: whoever can offer sufficiently long context, sufficiently low per-token cost, and sufficiently stable tool-calling capability is more likely to become the default foundation for agent applications. With this release, StepFun has placed itself in that race.

One-line Summary

StepFun open-sources the 600B sparse MoE model Step 5 Preview, targeting long-horizon agents and 1M context.

Paragraph Summary

StepFun has released Step 5 Preview as open weights, with FP8 and BF16 checkpoints now available on Hugging Face and ModelScope. The model uses a sparse MoE architecture with 600B total parameters and only 27B activated per token—invoking about 4.5% of parameters at inference—paired with up to 1M tokens of context, explicitly targeting long-horizon agentic tasks. The analysis argues that sparse activation lets the model keep a large-capacity knowledge base while cutting per-token compute to roughly the level of a 27B dense model, thereby controlling the cost of multi-turn tool calls and long-chain execution; the 1M window reduces information loss from context truncation and frequent summarization. Releasing both precision versions in parallel lowers the barrier to private deployment and fine-tuning, though the Preview label means interfaces and capabilities may still evolve.