Together AI Launches Together Compiler: Agent-Driven Inference Acceleration Up to 2.6x

Together AI has released Together Compiler, an agent-driven, continuously optimizing inference compiler. It continually re-optimizes the inference forward pass as traffic patterns change, achieving up to 2.6x higher TPS than the best InferenceX configuration on a single model in under 24 hours.

From Static Optimization to Continuous Compilation

Traditional inference optimization is often tuned once before model deployment, after which load characteristics are assumed to remain stable. But real traffic drifts over time, across business lines, and with user behavior, so static configurations quickly stop being optimal. The core idea of Together Compiler is to turn compilation into a closed loop: agents continuously observe inference performance, generate and validate optimization proposals targeting the current bottlenecks in the forward pass, and then retain the effective changes.

According to MarkTechPost, this loop optimizes the inference forward pass and can continuously re-optimize as traffic changes. This means optimization is no longer a one-off engineering task but an ongoing process that evolves with the load.

Performance: Compared with the Best InferenceX Configuration

The key figure from Together AI is that Together Compiler achieves up to 2.6x the TPS of the best InferenceX configuration on a single model in under 24 hours. Two points are worth noting here.

First, the baseline is the "best InferenceX configuration," not an arbitrarily chosen default set of parameters. InferenceX itself is used to measure inference performance, so this comparison is more meaningful. Second, the optimization cycle is measured in "under 24 hours per model," indicating that the method aims to complete iterations within an acceptable time window rather than relying on lengthy offline tuning.

InferenceX is a benchmark suite or set of configurations for measuring inference performance, typically maintained by industry organizations (such as the MLPerf community or hardware vendors) to provide reproducible inference performance reference points for different model and hardware combinations. It includes a series of pre-tuned configurations covering parameters such as batch size, parallelism strategy, and quantization precision, representing performance levels that have been thoroughly engineered and optimized for specific scenarios. Therefore, comparing Together Compiler with the "best InferenceX configuration" is essentially competing against a strong baseline that has been finely hand-tuned, rather than against default parameters. This increases the persuasiveness of the results, but it also means the actual gains may vary depending on the strength of the baseline: if the InferenceX configuration is already highly optimized for the target load, the room for improvement shrinks; conversely, in scenarios that are not fully tuned, the gains from continuous compilation may be more significant.

What Role Do Agents Play in the Compilation Loop?

From the description, the agent does not directly generate the final kernel but drives a loop of repeated trial, evaluation, and convergence. It can propose optimization hypotheses for the current inference load, have the system validate them, and then decide whether to keep or revert based on the results. This model shifts compiler optimization from "manual tuning" to "automated search plus continuous validation."

This also explains why it can follow traffic changes: when load characteristics change, the loop restarts to find new, better configurations. For inference scenarios that require long-term online service, this continuous compilation capability may be more important than single-peak performance.

Inference forward pass refers to the process in which a model receives input, computes through each layer of the network in sequence, and outputs a prediction. Unlike the training phase, the inference forward pass does not involve backpropagation or parameter updates, but its performance directly determines service latency and throughput. Optimizing the forward pass typically involves techniques such as operator fusion, memory layout adjustment, parallelism strategy selection, and automatic kernel tuning. By limiting the optimization target to the forward pass, Together Compiler focuses on the efficiency of computational graph execution during serving rather than training or fine-tuning. This focus allows the compiler to deeply customize for specific hardware and load characteristics, but it also requires it to quickly adapt to dynamic changes in input shapes, batch sizes, and sequence lengths.

Implications for Inference Deployment

Together Compiler reflects a trend: inference optimization is moving from one-time static configuration to continuous optimization tied to runtime load. For model service providers, this means the infrastructure layer needs to support frequent configuration updates and validation rather than treating optimization as a one-time gate before deployment.

At the same time, "up to 2.6x" is a result under a specific model and a specific baseline, and actual gains will vary with model architecture, traffic patterns, and hardware environment. When evaluating such tools, one should focus on the continuous gains under one's own load, not just the peak numbers.

One-line Summary

Together AI releases an agent-driven continuous inference compiler that achieves up to 2.6x the performance of the best baseline within 24 hours.

Paragraph Summary

Together AI has launched Together Compiler, an agent-driven, continuously optimizing inference compiler. Targeting the inference forward pass, it continuously regenerates and validates optimization proposals as traffic patterns change, turning traditional one-time static tuning into closed-loop continuous compilation. On a single model, it achieves up to 2.6x the TPS of the best InferenceX configuration in under 24 hours. The approach emphasizes agents proposing hypotheses in the loop, the system validating them, and retaining effective changes, thereby maintaining better performance as load drifts. The article notes that this reflects inference optimization moving from a one-time pre-deployment gate to a continuous process tied to runtime load, but the 2.6x gain depends on the model, baseline, and hardware; evaluations should focus on continuous gains under one's own load.