Together Compiler: Continuous Compilation Optimization for AI Inference

From Static to Continuous Compilation: A New Approach to AI Inference Optimization

AI inference costs are becoming the core bottleneck for deploying models in production. The traditional approach is to perform a one-time static compilation optimization of the model before deployment, but real-world traffic patterns, request length distributions, and batch sizes all drift over time, and a single compilation artifact quickly falls out of sync with the optimal state. Together's Together Compiler takes a different approach: it turns the compilation process itself into a continuously running agent system, using a persistent engine to repeatedly re-optimize open models as traffic changes. It claims to complete one optimization round within 24 hours and deliver 1.3x to 3x throughput gains.

Core Mechanism: Agents and a Persistent Engine Working Together

According to the official description, the key to Together Compiler lies in the word "continuous." It does not produce a one-off compilation result; instead, it deploys a set of agents responsible for sensing workload characteristics and searching the optimization space, while a persistent engine hosts the compilation artifacts and version switching. This architecture significantly compresses the lag between optimization and new traffic patterns.

One detail worth noting: the material mentions completing optimization "within 24 hours," which implies a high degree of automation in the compilation pipeline—no manual kernel rewriting or hand-tuning is needed, as the agents iterate over candidate solutions on their own.

Why Continuous Compilation Matters More for Open Models

The open model ecosystem is characterized by many model varieties, fast updates, and a high degree of downstream customization. Deep static optimization for a closed-source flagship from a major model vendor can have a reasonable return on investment, but manually tuning large numbers of open-weight models one by one is unrealistic. Continuous compilation hands the tuning work to an automated system, allowing long-tail open models to achieve throughput performance close to that of dedicated optimization—a clearly significant benefit for inference service providers.

The difference between static and continuous compilation: Static compilation (AOT, Ahead-of-Time Compilation) refers to compiling a computation graph or operator fusion plan once into efficient kernels for specific hardware before the model is deployed, after which it no longer changes. Typical examples include the offline optimization pipelines of TVM and TensorRT. Its advantage is zero compilation overhead at deployment time, but its underlying assumption is that input shapes (sequence length, batch size, concurrency patterns) remain relatively stable. Continuous compilation (an extension of JIT/dynamic recompilation) treats compilation as a runtime service: the system continuously observes real request characteristics, automatically triggers re-searching and recompilation when distribution drift is detected, and rolls out new versions via hot switching. The cost is additional compilation latency and resource usage; the benefit is the ability to continuously adapt to dynamic workloads.

Open Models: Here this refers to models whose weights are publicly downloadable, such as the Llama, Qwen, and Mistral series and their community fine-tuned versions. Compared with closed-source API models, downstream users of open models can deploy, quantize, and customize them themselves, which leads to a large number of variants derived from the same base and frequent version iterations. Inference service providers often need to host dozens or even hundreds of open models simultaneously, and manually tuning kernels for each one is extremely costly—which is exactly where automated continuous compilation comes into play.

What the Throughput Gains Mean

The 1.3x to 3x throughput improvement reported officially is obtained dynamically as traffic changes. This range is worth breaking down: the lower bound represents robust gains under normal conditions, while the upper bound appears in scenarios where workload characteristics shift and the static compilation plan is clearly mismatched. For inference platforms that charge by token, throughput maps directly to service capacity per unit of compute, and the same GPU resources can handle more requests.

It should be noted that this figure comes from Together's own description, and no third-party reproduction or detailed benchmark conditions have been seen yet. Actual gains will vary across different models, hardware, and request distributions, so readers should treat it as directional guidance rather than a guaranteed value.

The relationship between throughput and inference cost: In LLM inference, throughput usually refers to the number of tokens (tokens/s) or requests (requests/s) the system completes per unit time. It is jointly determined by GPU compute utilization, batching efficiency, KV Cache management strategy, and kernel execution efficiency. A 1.3x to 3x throughput improvement means that the same hardware cluster can serve more user requests per unit time, directly diluting the compute cost per token. For platforms that charge by token, this amounts to a direct improvement in gross margin; for enterprises running their own inference, it corresponds to lower GPU purchase or rental expenses. Note that there is a trade-off between throughput and latency (TTFT, TPOT): increasing batch size can raise throughput but may increase per-request latency, so actual deployment needs to balance the two.

Potential Impact on the Inference Infrastructure Landscape

If continuous compilation proves viable, it will change the division of labor in inference optimization: manual tuning that previously relied on compiler engineers' experience will gradually give way to always-on automated agent systems. This is consistent with the current "agentification" trend—handing engineering tasks that require repeated iteration to systems that can continuously observe and act.

However, public information is currently limited. The compilation agents' search strategy, the implementation details of the persistent engine, and the additional overhead it imposes on compilation latency and GPU memory usage have not been disclosed. Yet these are precisely the key factors determining whether the approach can remain in production environments over the long term.

Summary

The continuous compilation idea proposed by Together Compiler hits the pain point of static optimization: traffic is changing, so optimization should change with it. Whether the combination of agents plus a persistent engine can truly deliver a stable 1.3x to 3x throughput gain across diverse open models still requires more practical data to answer.