gpt-oss-120b

The models tested include Google's Gemini 3.7 Flash, Z.ai's GLM-5, Alibaba's Qwen 3 235B A22B Instruct, and OpenAI's gpt-oss-120b, all running on Kaggle.

gpt-oss-120b has about 117B total parameters, but each token activates only about 5.1B parameters.

All incorrect distractor counting answers were without exception overestimates, and the token counts of incorrect answers were clearly higher than those of correct answers: for gpt-5.4-mini at high, the median token count of incorrect answers was 412 vs 91.5 for correct answers; for gpt-oss-120b, incorrect answers were 5,776 tokens vs 747 tokens for correct answers.

gpt-oss-120b directly returns an HTTP 400 error when reasoning_effort='none', refusing that value.

The same parameter name reasoning_effort behaves differently across models from different vendors: in gpt-5.4-mini the median output tokens rose from 18 at none to 254 at high (×14.1); gpt-oss-120b rose from refusal to 1,340 (×4.6); claude-haiku-5.5 rose from 150 to 298 (×2.0); gemini-3.7-flash rose from 453 to 1,013 (×2.2).

gpt-oss-120b spent about 25 times the cost per correct answer on the distractor counting task, while its accuracy was 7.5 percentage points lower (not significant).

In the τ-bench retail environment benchmark test (30 test tasks, simulated customers, single run, solvi 0.8.0, gpt-oss-120b as the Agent), without the guard 18 tasks were solved and the environment rejected 10 mutating calls; with the guard 14 tasks were solved and the environment rejected 0 mutating calls.

For gpt-oss-120b on the same 48 core families, the pass rate was approximately 63% on the Kaggle platform and as high as 98% on the cluster; the discrepancy stems from hosted answers being truncated mid-word, and on the 31 families where hosted answers were complete, both platforms passed 30.

Code replay inherits errors from the extraction stage; in the cluster sufficiency test, gpt-oss-120b's pass rate actually dropped from 56% to 40%.

OpenAI gpt-oss-120b timed out twice within a 19-hour testing window, and IBM Granite 4.0 was not included in the test because it was not in the Kaggle server catalog.

gpt-oss-120b scored 0.625, lower than gpt-oss-20b's 0.720, indicating that a larger model does not necessarily mean a safer one.

After correcting the scorer, all four models achieved 100% on all five Sanity gate tasks.

The parsing defect combined with one incorrect answer key caused gpt-oss's score across five core tasks to shift from 0.62 to 1.00, a shift as large as 0.38.

In the ng-hard stress test, Gemini 3.7 Flash, GLM-5, and Qwen 3 235B all scored 100%, while gpt-oss-120b scored 80% (8 out of 10 rows).

Larger, newer models do not necessarily perform better: gpt-oss-120b (0.740) scored lower than gpt-oss-20b (0.761); DeepSeek-R1, costing $1.94, scored lower than Gemma 4 31B, which cost only 13 cents.

For false information such as incorrectly typed dates, the correct action is to 'erase' rather than 'change'; gpt-oss-120b performed the correction correctly only 42% of the time, and Qwen3 235B only half the time.

gpt-oss-120b retained 11 of 12 bookings but refused zero injections, writing the fake address from a phishing email into the passport; Qwen3 235B also stored the same address in the voucher field.

gpt-oss-120b scored 0.991 on memory read but only 0.740 on write, ranking last in write; DeepSeek-R1 scored 0.997 on read and 0.863 on write.

When gpt-oss is loaded with load_in_4bit = False, MXFP4 experts are retained, allowing gpt-oss-120b to be trained on a single B200

On the Rule Induction task, TwIL-LM3 scored 96.4 while gpt-oss-120b scored 65.2.