[KongchangAI]
· 3 min read· 1,764 words

Fireworks AI: How Open-Source Models Rival Closed-Source Through a Training-Deployment Flywheel

Fireworks AI: How Open-Source Models Rival Closed-Source Through a Training-Deployment Flywheel

Fireworks breaks down the engineering loop for open-source models, from inference optimization to layered training to a same-platform flywheel.

The article summarizes Fireworks AI's Jet at AI Engineer on how open-source models match closed-source in production. She argues most open-source models are production-ready, the quality gap must be closed through scenario-specific custom training, and the AI stack is shifting to a continuous train—infer—retrain cycle. On inference, the custom Fire Attention engine writes CUDA kernels directly and uses speculative decoding, disaggregated inference, and quantization to cut latency and cost. Training comes in three tiers—a Training Agent, managed training, and a Training API—all supporting SFT, RFT, and DPO. Same-platform training and inference enables hot loading, numerical fidelity, and one-click deployment. Cases show Cursor gaining a 13x inference speedup, WordCell 40x latency improvement, and GenSpark 12% quality gains.

Closed-source models are getting increasingly expensive to use, and many teams are considering migrating to open-source models—but the biggest obstacle on that path is often not cost, but quality. In a talk at AI Engineer, Jyotishree (Jet), head of the applied machine learning team at Fireworks AI, gave a complete breakdown of the continuous flywheel that takes open-source models from training to deployment, and how they get open-source models to match closed-source performance.

Three Basic Facts: Open-Source Models Are Production-Ready

Jet opened with three judgments about the current AI ecosystem. First, most open-source models are already production-ready; she cited GLM 5.2's performance on coding tasks as an example, arguing that such models can already support real business workloads. Second, if open-source models cannot match or even surpass closed-source models in quality, then the real differentiator lies in how to customize, train, and continuously improve for specific scenarios. Third, the AI stack is shifting from "deploy once and you're done" to a continuous cycle of "train—infer—retrain," where the same open-source model gets stronger through constant iteration.

These three points form the starting point of Fireworks' product design: putting training and inference on the same platform to make this cycle as smooth and cheap as possible.

Inference Side: How a Custom Engine Drives Down Latency

Before discussing training, Jet first laid out the scale of inference. Fireworks serves trillions of tokens in production (she noted this figure predates the GLM 5.2 release), supports over ten thousand enterprises worldwide, and handles roughly 200,000 requests per second.

Behind these numbers is the custom inference engine Fire Attention. In a conventional pipeline, the model interacts with the GPU through the CUDA layer, but Fireworks wrote its own CUDA kernels, so it can optimize end-to-end for a customer's specific workload. Customers typically come to them with input token counts, output token counts, prompt cache hit rates, and latency targets, and the team tunes the inference stack and kernels accordingly to meet performance goals while driving prices down.

On the technical side, Fire Attention also employs industry-mature techniques such as speculative decoding, disaggregated inference, and quantization. For speculative decoding, customers can train their own draft models on the platform, uploading data in a way similar to fine-tuning; when the draft model is well trained, both the main model's latency and throughput benefit noticeably.

Inference and training platform interface

Speculative Decoding is a technique that accelerates inference without changing the main model's output distribution. It introduces a smaller, faster draft model that first autoregressively "guesses" several subsequent tokens, then lets the main model verify in parallel whether these candidate tokens are acceptable; accepted tokens are kept directly, and after the first rejected position, sampling falls back to the main model. Because verification is parallel, the overall output is mathematically equivalent to using only the main model, but the actual time is significantly reduced. The speedup depends on the draft model's hit rate: the closer the draft's distribution is to the main model and the more predictable the task, the higher the acceptance rate and the greater the gain. This is why Fireworks allows customers to train draft models with their own business data—generic draft models often have low acceptance rates in vertical scenarios, and customized training recovers that gain.

Training Side: Three Forms Covering Different Users

When customers are convinced of model quality and decide to migrate from closed-source to open-source, the biggest gap is quality, which Fireworks fills with its training products. The training products come in three forms, corresponding to different audiences.

Training Agent: For Product Managers and Builders

This is designed for non-expert users who are just entering the AI training field. Through the UI, they connect to datasets already uploaded on the platform and describe the training goal clearly; the Agent then selects the model, cleans the data, and tunes hyperparameters itself. Jet specifically noted that hyperparameters are not easy to tune, which is exactly the problem the Agent is meant to solve for users. In the demo, the Agent selected Qwen3-8B as the base model, first presenting an SFT plan, the rationale for model selection, and hyperparameter explanations; after user confirmation, it executes step by step and reports the results of each step; when training finishes, it gives metrics such as accuracy and F1 plus a configuration summary, and recommends next actions.

Managed Training: The Choice of Most Customers

Managed training is currently the most widely used form, available through both API and UI. The flow is as simple as selecting a model, uploading a dataset, filling in hyperparameters, and clicking a button. Observability after training starts is a common pain point; the platform provides dashboards showing each epoch's learning progress and loss curves, and also supports entering a Weights & Biases API key to export metrics to external platforms. After training completes, jobs can be cloned to reproduce with a different configuration.

Managed training is the choice of most customers

SFT, RFT, and DPO are three training paradigms with different objectives, often used in combination. Supervised fine-tuning (SFT) uses paired "input—ideal output" data for next-token training, essentially making the model imitate annotated answers, suitable for injecting domain knowledge and output formats. Reinforcement fine-tuning (RFT) does not give answers directly but defines automatically computable rewards or evaluation functions, letting the model improve in a sample—score—update loop; it suits tasks with large answer spaces that are hard to annotate manually but whose correctness can be judged, such as code passing tests or reasoning problems with verifiable answers. Direct preference optimization (DPO) trains on "good answer / bad answer" preference pairs, bypassing an explicit reward model to make the model directly favor preferred outputs. A common practical path is to start with SFT as a foundation, then refine with RFT or DPO, which is why the platform's three forms all support these three methods simultaneously.

Training API: Full Control for ML Engineers

The third form targets ML engineers within an organization, allowing custom training loops and loss functions. When requirements exceed what managed training can cover, this path provides the necessary flexibility. All three forms support supervised fine-tuning, reinforcement fine-tuning, and DPO. Jet mentioned that RFT has been extremely popular recently; if customers need well-defined evaluation functions, Fireworks assists—as it did for GenSpark—with data cleaning, evaluation function design, and hyperparameter tuning. However, the training Agent does not currently support RFT; that part requires human intervention.

Data Flywheel: The Value of Training and Inference on the Same Platform

Putting training and inference on the same platform is key to making this flywheel work. Once a model is trained, it can be deployed to production with one click; after deployment, metrics such as token counts, cache hit rates, and speculative model performance can be monitored.

Monitoring metrics dashboard

Several platform capabilities make this cycle smoother:

  • Hot loading: When a new checkpoint is available, it can be switched directly to the online deployment without downtime. Jet said this feature was originally built specifically for Cursor and later opened to all users.
  • Numerical fidelity: Many customers train on platform A and deploy on platform B, only to run into quality issues, often caused by numerical incompatibility. Training and inference on the same platform avoids such discrepancies.
  • One-click deployment: After training, production is just one step away; the platform also offers deployment templates such as throughput-first, low-latency, and minimal, suited to general workloads.

Numerical Fidelity refers to the degree to which the same model weights produce consistent outputs when run on different inference stacks. Deep learning inference involves a large amount of floating-point computation, and kernel implementations, accumulation order, quantization strategies, parallel partitioning of attention computation, and even GPU model and driver versions all introduce small but cumulative numerical differences. For most conversational tasks these differences are imperceptible, but in scenarios such as long-chain reasoning, structured output, and low-temperature sampling, tiny deviations can be amplified into noticeable behavioral changes, manifesting as "great evaluation metrics during training, worse performance after going live." Cross-platform training and deployment is a common source of such problems: different implementations of the same operator by the training framework and the inference engine lead to train—inference mismatch. Training and inference on the same platform lets both sides share the same set of kernels and quantization paths, engineering away this hidden loss.

Customer Cases: Evidence on Latency and Quality

Jet gave several representative customer results. Cursor's Composer 2 model was trained and hosted on Fireworks, achieving a 13x inference speedup thanks to Fire Attention. WordCell achieved a 40x latency improvement through fine-tuning and hosting. GenSpark's bottleneck was quality; the team helped it with RFT, data cleaning, and evaluation function design, and after training, quality improved by 12%.

These cases also show that the value of migrating to open-source models is not just saving money: deep optimization at the inference layer delivers order-of-magnitude latency gains, while customization at the training layer directly closes the quality gap.

Hyperparameters and training configuration

Final Thoughts

Jet's talk essentially answers one question: why can open-source models rival closed-source? The answer is not in the models themselves but in the entire set of engineering capabilities around them—a custom inference engine drives down latency and cost, layered training products close the quality gap, and training and inference on the same platform keep the data flywheel spinning. For teams evaluating a migration, this approach offers a reference framework: first check whether inference costs are acceptable, then see whether the training side offers a path to close the quality gap, and finally confirm whether the iteration loop is smooth enough.

One-line Summary

Fireworks breaks down the engineering loop for open-source models, from inference optimization to layered training to a same-platform flywheel.

Paragraph Summary

The article summarizes Fireworks AI's Jet at AI Engineer on how open-source models match closed-source in production. She offers three judgments: most open-source models are already production-ready, the quality gap must be closed through scenario-specific custom training, and the AI stack is shifting from one-time deployment to a continuous train—infer—retrain cycle. On inference, the custom Fire Attention engine writes CUDA kernels directly, tunes for customer workloads, and uses speculative decoding, disaggregated inference, and quantization to cut latency and cost. Training comes in three tiers: a Training Agent for non-experts, the most widely used managed training, and a Training API giving ML engineers full control—all supporting SFT, RFT, and DPO. Same-platform training and inference brings hot loading, numerical fidelity, and one-click deployment to keep the flywheel turning. Cases show Cursor gaining a 13x inference speedup, WordCell improving latency 40x, and GenSpark improving quality by 12%.

#Fireworks AI#open-source models#speculative decoding#fine-tuning#data flywheel
Share:

Related articles