Together Compiler: A New Paradigm of Continuous Compilation for AI Inference

AI inference deployment is shifting from "optimize once, use forever" to "compile continuously, adapt dynamically." Together Compiler, launched by the Together team, is a product of this line of thinking: it leverages agents and persistent engines to optimize open-source models in under 24 hours, and claims throughput improvements of up to 3.7x when traffic patterns change.

Why AI Inference Needs "Continuous Compilation"

Traditional inference optimization is often an offline, one-time process: quantization, operator fusion, and kernel tuning are performed before the model goes live, and then frozen after deployment. But real-world traffic patterns, input length distributions, and concurrent request counts change constantly, and static optimization strategies can easily fail when traffic fluctuates, leading to reduced GPU utilization or spiking latency.

The core thesis of Together Compiler is that compilation should not be a one-time action before deployment, but a process that runs continuously throughout the model's lifecycle. It uses agents to automatically explore the optimization space and a persistent engine to retain and reuse compilation results, enabling rapid re-tuning when traffic characteristics change rather than starting from scratch.

How Agents and Persistent Engines Work Together

The Together Compiler architecture can be broken down into two key components.

Agent-Driven Optimization Search

Agents are responsible for making decisions automatically during compilation, such as choosing quantization strategies, adjusting memory layouts, and matching different kernel implementations. Compared with manual tuning or fixed rules, agents can iteratively search based on feedback signals from the model structure and target hardware, shortening the time to find a better configuration. The officially stated time window is "under 24 hours," meaning that for open-source models, the process from onboarding to obtaining an optimized version can be completed within a day.

Here, "agent" does not refer to a monolithic large model, but rather a set of automated programs capable of perceiving compilation feedback, invoking tools, and making iterative decisions. In the context of inference compilation, common decisions include operator fusion methods, quantization bit width and granularity, KV Cache layout, batching scheduling strategies, and selecting or generating kernel implementations for a specific GPU. Agents typically use compilation time, memory footprint, throughput, and tail latency as reward signals, gradually converging within the search space rather than exhaustively enumerating all combinations. Compared with traditional auto-tuning (such as cost-model-based TVM AutoTVM), agent-based search places more emphasis on dynamically adjusting the search direction based on runtime feedback, making it better suited to scenarios where model structures and traffic distributions change frequently.

Persistent Engine Ensures Continuity

The role of the persistent engine is to retain compilation state and optimization results. When traffic patterns change, the system does not need to compile from scratch but can make incremental adjustments based on existing state. This design lowers the cost of re-optimization and makes "continuous compilation" engineering-feasible, not merely a conceptual ideal.

The persistent engine can be understood as a state storage and version management system for compilation artifacts. It stores not only the final generated binary kernels but also intermediate representations, configurations that have been tried, measured performance data for each configuration, and hardware environment fingerprints. With this state, when input length distributions or concurrency levels change, the system can determine which optimizations remain valid and which need to be re-searched, thereby performing only incremental compilation rather than a full redo. This also introduces cache consistency issues: when model weights, driver versions, or GPU models change, invalidated compilation caches must be correctly identified and replaced; otherwise, mismatched kernels may be loaded, causing errors or performance regressions.

Performance and Applicable Boundaries

Together Compiler claims throughput improvements of up to 3.7x in traffic-change scenarios. Note that this figure comes from the official description and represents peak performance under ideal conditions; actual gains will vary depending on model type, hardware configuration, and the magnitude of traffic fluctuations.

Judging from the description, the compiler is primarily aimed at inference optimization for open-source models, consistent with the Together platform's positioning as mainly an open-source model service. For teams using closed-source APIs, its applicability is limited; for scenarios involving self-hosted open-source models with highly variable traffic characteristics, the continuous compilation approach offers valuable reference.

The meaning of a throughput improvement multiplier varies greatly across different benchmarks. Figures like 3.7x are typically measured against a specific comparison baseline (e.g., a statically compiled version not re-optimized for new traffic) and under specific loads (e.g., significant drift in input length distribution). When evaluating such metrics, it is important to consider: whether the baseline received comparable tuning, the batch size and sequence length ranges tested, whether the time spent on compilation and re-tuning itself is included, and whether the improvement is measured in average throughput or P99 latency. For production environments with stable traffic and concentrated input distributions, the marginal benefit of continuous compilation will be significantly smaller than in highly volatile scenarios.

Implications for Inference Infrastructure

Together Compiler reflects a broader trend: inference optimization is moving from "static configuration" to "dynamic service." Model deployment is no longer the endpoint but a process that requires continuous observation and adjustment. The role of agents here is not to replace engineers but to automate repetitive search and validation work, enabling optimization frequency to keep pace with traffic changes.

Another notable signal is the "24 hours" timescale. It suggests that the barrier to compilation optimization is lowering, and small and medium-sized teams may be able to achieve customized inference performance with relatively low human investment, without relying on large infrastructure teams. Of course, actual results still need to be validated under real workloads, and the additional system complexity and stability risks introduced by continuous compilation should not be overlooked.

Conclusion

Together Compiler combines agent-based search with a persistent engine, turning compilation from a one-time offline step into a continuous online optimization loop, and sets a target of up to 3.7x throughput improvement under traffic changes. It represents a pragmatic engineering direction: enabling inference optimization to keep pace with real traffic rather than remaining a static snapshot at the moment of deployment.

Background Supplement

When implementing continuous compilation in engineering practice, resource contention between compilation and inference must also be considered. Recompilation consumes GPU or CPU resources, and if it shares the same cluster as online inference, it may increase latency during peak periods. Common practices include scheduling compilation tasks to dedicated nodes or running them during off-peak hours, and validating new compiled versions through shadow traffic before switching traffic. In addition, the reproducibility of compilation artifacts and rollback mechanisms are also critical: if a new configuration performs poorly under real workloads, the system must be able to quickly roll back to the previous stable version, avoiding the risk of continuous optimization introducing stability issues.