XStack
XStack is a benchmark for evaluating the ability of AI agents to migrate machine learning codebases between the PyTorch and JAX ecosystems. Agents are required to convert PyTorch repositories into JAX repositories that run on TPUs, and are scored against the published results of the source repositories.
Timeline (last 90 days)
XStack is a benchmark for evaluating agents' ability to migrate ML codebases between the two major ecosystems of PyTorch and JAX.
In the XStack benchmark, each agent receives a PyTorch codebase and a clean TPU machine, and must deliver a JAX codebase.
XStack's scoring is based on the source repository's published results (certified and calibrated), ensuring that the migrated code is functionally equivalent and meets performance targets.
In XStack's preliminary evaluation, Claude Opus 5 achieved a pass rate of 75%.
In XStack's preliminary evaluation, GPT-5.6-sol achieved a pass rate of 45%.
In XStack's preliminary evaluation, all agents other than Claude Opus 5 and GPT-5.6-sol achieved pass rates of at most 40%.
All Facts (6)
XStack is a benchmark for evaluating agents' ability to migrate ML codebases between the two major ecosystems of PyTorch and JAX.
50%UnverifiedIn the XStack benchmark, each agent receives a PyTorch codebase and a clean TPU machine, and must deliver a JAX codebase.
50%UnverifiedXStack's scoring is based on the source repository's published results (certified and calibrated), ensuring that the migrated code is functionally equivalent and meets performance targets.
50%UnverifiedIn XStack's preliminary evaluation, Claude Opus 5 achieved a pass rate of 75%.
50%UnverifiedIn XStack's preliminary evaluation, GPT-5.6-sol achieved a pass rate of 45%.
50%UnverifiedIn XStack's preliminary evaluation, all agents other than Claude Opus 5 and GPT-5.6-sol achieved pass rates of at most 40%.
50%