Pre-Deployment Checklist for AREX-2: The Trap of Identical config.json

When a fine-tuned model declares the same architecture class as an existing model, the most tempting move is to swap out --model-path and restart the service. BAAI's AREX-2 is the perfect test of whether that habit holds up. It's a 27B model trained to solve complex problems in rounds — propose a plan, measure results, reflect and correct — released under Apache 2.0 with no gating, and its config.json opens with Qwen3_5ForConditionalGeneration, the same class as the Qwen team's Qwen3.8-27B.

We'd been serving the base Qwen3.8-27B with sglang, so we compared the two configurations. One caveat up front: we haven't run AREX-2 yet, and all speed numbers below come from the base model. Full tables, commands, and sources are in the AREX-2 serving analysis at inferencekey.com.

The Seemingly "Plug-and-Play" Differences

BAAI shipped a clean release: layer layout, attention head count, context length, vocabulary, and vision tower all match Qwen3.8-27B. The config differs only in transformers_version and two redundant top-level keys, and the chat_template.jinja file that formats conversations and tool calls is even byte-for-byte identical.

If you stop reading here, you'll just carry over the old command. That's exactly the first trap.

MTP (Multi-Token Prediction) heads are small prediction modules attached on top of a large model's backbone, typically made up of one to several extra Transformer layers. Their job is to predict not just the next token at each decoding step, but several tokens immediately following it as well — those predictions are submitted as candidates for the main model to verify in parallel. If verification passes, the server can emit multiple tokens in a single forward pass, significantly boosting throughput. The cost is increased weight size and VRAM usage, and prediction accuracy depends on how well the head was aligned with the main model during training. Because MTP heads are an add-on structure, many fine-tuning pipelines freeze or outright discard them, leaving the declaration in config.json inconsistent with the actual safetensors weights shipped — which is exactly the case with AREX-2.

safetensors index files (typically named model.safetensors.index.json) are the manifest used in the Hugging Face ecosystem for sharded weights, listing every tensor name in the release and which shard file it lives in. Compared to the architecture parameters declared in config.json, the index file reflects the weights that actually exist, making it the most reliable basis for judging "what a model actually comes with." Checking whether the index contains any tensors prefixed with mtp immediately confirms whether the MTP head was truly shipped with the weights.

Check the Weights, Not the Config

The base model ships with a multi-token prediction (MTP) head — a small extra layer that predicts several tokens ahead so the server can emit multiple tokens per step. AREX-2's config still declares it, but its weight files don't contain it.

config.json                    "mtp_num_hidden_layers": 1
safetensors index (AREX-2)     0 mtp tensors
safetensors index (Qwen3.8)    15 mtp tensors

The safetensors index lists every weight tensor in the release and is an honest inventory. A config field is just a promise, and you have to check whether anything actually backs it up.

The second quiet change is in tokenizer_config.json, which controls how text is split into tokens. Community repackaging reports say the pre-tokenizer and decoder settings differ from Qwen3.8. It could be harmless, or it could manifest as garbled tool calls after an agent has been running for hours.

Flags That Are Correct for the Base Model

Our fastest base-model setup runs on an RTX PRO 5000 CUDA 13 pod, using RadixArk's NVFP4 quantization (weights compressed to 4-bit floating point) and z-lab's DFlash2 draft model, hitting 115–136 tok/s decode. But switch to AREX-2, and almost none of this still applies:

• No NVFP4 build of AREX-2 yet. Current options are BAAI's original BF16 (54.7 GB), prithivMLmods' FP8 (35.9 GB), cyankiwi's AWQ INT4 (20.2 GB), or numsu's INT4 W4A16 (18.6 GB).
• The draft model was trained for a different model. Speculative decoding has a small draft model propose candidate tokens that the large model verifies at once; it only pays off when most guesses are accepted. z-lab's DFlash2 draft model targets base Qwen3.8-27B, and fine-tuning changes the tokens a model prefers, so its hit rate on AREX-2 is anyone's guess. numsu's INT4 card does publish speeds on two RTX 3090s paired with the DFlash2 draft model, but doesn't give an acceptance length.
• mamba_ssm_dtype. AREX-2 requires float32. Our base command forces bfloat16, which is fine on Qwen3.8 but unverified on a model tuned for long reflection runs.
• The parser is a guess. These sglang options convert raw model text into separate reasoning and tool-call fields. qwen3 and qwen3_coder should work because the templates match. But there are no public AREX-2 sglang run results.
• Choose between long context and speculative decoding. The base model's 1M context is a YARN override stretching its native 262,144 positions, and DFlash2 can't run alongside it. So: either 1M without a draft model, or 256K with one.

NVFP4, FP8, and INT4/W4A16 are three different quantization formats, each with different hardware requirements. NVFP4 is NVIDIA's 4-bit floating-point format introduced on Blackwell and later architectures, offering high weight compression with manageable precision loss; FP8 (E4M3/E5M2) requires Ada, Hopper, or newer architectures for native tensor core support; the INT4 family (such as AWQ, W4A16) is the most versatile option, where W4A16 means 4-bit weights and 16-bit activations, runnable on nearly any modern GPU through dequantization kernels.

Speculative Decoding gains depend entirely on acceptance length: the draft model proposes K candidate tokens per round, the main model verifies them in one pass, and only the accepted prefix actually takes effect. The higher the acceptance length, the more tokens produced per step and the more noticeable the speedup; if acceptance length is close to 1, speculative decoding actually slows things down due to extra overhead. The closer the draft model's distribution is to the main model's, the higher the acceptance length — which is exactly why "fine-tuning changes a model's preferred tokens, so a draft model trained on the base version has an unknown hit rate on the fine-tuned version."

Running FP8 on Ampere Is an Expensive Mistake

The RTX A6000 is Ampere architecture with no FP8 hardware, so sglang keeps the weights in FP8 and unpacks them on the fly via Marlin kernels. The base Qwen3.8-27B decodes at 24 tok/s on that card, while RedHatAI's INT4 W4A16 quantization hits 41 tok/s on the same card with the same 1M context. prithivMLmods' AREX-2 FP8 card also agrees that full speed requires Ada, Hopper, or newer architectures. The A100 is also Ampere, so INT4 should be the choice there too.

Weights aren't the whole cost. In the base model's INT4 1M-context run, sglang takes about 46 GB, of which only 17.8 GB is weights, another 11.5 GB goes to linear attention layer state, and 12.8 GB to the KV cache — per-token memory the model saves to avoid re-reading the entire context each step. AREX-2's per-GPU VRAM estimates are listed card by card from 80 GB down to 24 GB, for reference only.

Marlin kernels are high-performance dequantization matmul kernels used in inference frameworks like sglang and vLLM to run quantized weights on GPUs that lack native low-precision hardware support. When model weights are stored in FP8 but the hardware (such as Ampere-based A6000 or A100) has no FP8 tensor cores, the framework must dequantize the weights back to 16-bit on the fly before each computation — a step that consumes extra VRAM bandwidth and compute cycles, making actual decode speed lower than directly using INT4 quantization. So "lower bit width is always faster" doesn't hold — the match between quantization format and hardware architecture is often more important than the bit width itself.

YARN (Yet Another RoPE extensioN) is a method for context length extrapolation by scaling the frequencies of rotary position embeddings (RoPE): a model is trained at a native length (e.g., 262,144), and at inference the RoPE scaling parameters are adjusted to stretch the effective position range to a longer sequence (e.g., 1M). It doesn't require retraining, but the greater the extrapolated length, the more noticeably long-range attention accuracy degrades. Additionally, YARN's scaling configuration is typically incompatible with the draft model used in speculative decoding, so in practice you often have to choose between "long context" and "speculative speedup."

What to Measure on the First Run

Start with the most vanilla approach: run BF16 on an 80 GB card, native 256K context, no speculative flags, the same parser, and execute a long agent task. Then change one variable at a time, like sound-checking starting from the kick drum. Watch these metrics:

• Whether tool calls still parse cleanly late in long sessions, given the tokenizer change
• Decode tok/s versus the base model on the same card, which should match given the shared architecture
• If you add a draft model, what the acceptance length is — a number nobody has published
• GPU hours per task, since the benchmark protocol allows up to 1,500 model calls and 24 hours per case

That last one determines the economics. AREX-2's self-reported Frontier-CS score of 70.7 and BrowseComp score of 84.0 come from the long protocol in BAAI's evaluation repository, and we couldn't find an independent reproduction. Renting a GPU by the hour only beats per-token pricing if the card has a steady queue of tasks, such as running multiple cases side by side. For single-GPU agent setups, see our earlier article on deploying an 80B coding model on a single rented A100 driven by Claude Code.

This is a reasonable single-card project, with thanks to BAAI and the Qwen team for the models, prithivMLmods, cyankiwi, numsu, RedHatAI, and RadixArk for the quantizations, z-lab for DFlash2, numsu again for the experimental MTP head trained on AREX-2 outputs, AI Search for the tip-off, and the sglang and vLLM projects. Config tables, the original base-model command, the slow-start worker setup, and the full list of caveats are in the complete AREX-2 single-GPU deployment article.