Together Compiler
A compiler system that uses AI agents to continuously maintain inference tuning state in production environments, automatically adjusting inference configurations through a closed loop of perception, decision-making, execution, and verification.
Core Facts
Timeline (last 90 days)
The "up to 2.6 times" figure is a result under a specific model and a specific baseline, and actual gains will vary with model architecture, traffic patterns and hardware environment.
For inference scenarios that require long-term online serving, continuous compilation capability may be more important than single-run peak performance.
Together Compiler limits the optimization target to inference forward computation, focusing on the execution efficiency of the computation graph during serving rather than the training or fine-tuning process.
Together Compiler can continuously re-optimize the inference forward computation as traffic patterns change.
If the InferenceX configuration has already been highly optimized for the target workload, the room for improvement from Together Compiler will shrink; conversely, in scenarios that are not fully tuned, the gains from continuous compilation may be more significant.
Comparing Together Compiler with the best InferenceX configuration is in essence competing against a strong baseline that has been finely hand-tuned, rather than comparing against default parameters.
The core idea of Together Compiler is to turn the compilation process into a closed loop: the agent continuously observes inference performance, generates and validates optimization plans targeting the bottleneck of the current forward computation, and then retains the effective changes.
On a single model, the TPS that Together Compiler reached in under 24 hours was up to 2.6 times that of the best InferenceX configuration.
The 3.7x throughput improvement comes from the official description and is a peak figure under ideal conditions; actual gains vary with model type, hardware configuration and the magnitude of traffic fluctuations.
Together Compiler claims it can deliver up to a 3.7x throughput improvement when traffic changes.
12 more timeline events
All Facts (20)
Together Compiler uses AI agents to automate inference tuning into the production environment, keeping production inference continuously in a tuned state.
75%UnverifiedTogether Compiler claims it can complete one round of optimization within 24 hours.
70%UnverifiedTogether Compiler can continuously re-optimize the inference forward computation as traffic patterns change.
50%UnverifiedOn a single model, the TPS that Together Compiler reached in under 24 hours was up to 2.6 times that of the best InferenceX configuration.
50%UnverifiedThe core idea of Together Compiler is to turn the compilation process into a closed loop: the agent continuously observes inference performance, generates and validates optimization plans targeting the bottleneck of the current forward computation, and then retains the effective changes.
50%UnverifiedComparing Together Compiler with the best InferenceX configuration is in essence competing against a strong baseline that has been finely hand-tuned, rather than comparing against default parameters.
50%UnverifiedThe "up to 2.6 times" figure is a result under a specific model and a specific baseline, and actual gains will vary with model architecture, traffic patterns and hardware environment.
50%UnverifiedIf the InferenceX configuration has already been highly optimized for the target workload, the room for improvement from Together Compiler will shrink; conversely, in scenarios that are not fully tuned, the gains from continuous compilation may be more significant.
50%UnverifiedFor inference scenarios that require long-term online serving, continuous compilation capability may be more important than single-run peak performance.
50%UnverifiedTogether Compiler limits the optimization target to inference forward computation, focusing on the execution efficiency of the computation graph during serving rather than the training or fine-tuning process.
50%UnverifiedTogether Compiler is mainly aimed at inference optimization for open-source models, and its applicability is limited for teams using closed-source APIs.
50%UnverifiedTogether Compiler claims it can deliver up to a 3.7x throughput improvement when traffic changes.
50%UnverifiedTogether Compiler's agents are a set of automated programs that can perceive compilation feedback, call tools and iterate decisions, rather than a single large model.
50%UnverifiedThe persistence engine saves the state of compilation artifacts, including intermediate representations, configurations that have been tried, measured performance data for each configuration and hardware environment fingerprints, enabling incremental compilation rather than starting over from scratch.
50%UnverifiedThe 3.7x throughput improvement comes from the official description and is a peak figure under ideal conditions; actual gains vary with model type, hardware configuration and the magnitude of traffic fluctuations.
50%UnverifiedWhen evaluating the 3.7x throughput improvement, one needs to check whether the baseline was tuned to the same degree, the batch size and sequence length range tested, whether compilation and re-tuning time are included, and whether it is average throughput or a P99 latency metric.
50%UnverifiedWithin the 1.3 to 3 times throughput improvement range, the lower bound represents steady gains under normal conditions, while the upper bound appears in scenarios where load characteristics shift and statically compiled schemes are clearly mismatched.
50%UnverifiedTogether Compiler claims it can deliver a 1.3 to 3 times throughput improvement.
50%UnverifiedThe combination of Polymorphic Inference and Together Compiler points toward a 'tuning as a service' capability, turning inference optimization that depends on senior engineers into a built-in continuous self-adaptation mechanism of the system.
50%UnverifiedTogether launched Together Compiler, a tool that turns compilation into a continuously running agent system and uses a persistent engine to continuously re-optimize open models as traffic changes.
50%