Gemini 3.7 Flash

Gemini 2.5 Flash adopts a Mixture of Experts (MoE) architecture, activating only a subset of parameters during inference.

The models tested include Google's Gemini 3.7 Flash, Z.ai's GLM-5, Alibaba's Qwen 3 235B A22B Instruct, and OpenAI's gpt-oss-120b, all running on Kaggle.

The test covered four vision models: Claude Sonnet 5 (Anthropic), Gemini 3 Flash (Google), Gemini 3.1 Flash-Lite (Google), and GPT-5.4 nano (OpenAI).

Tev-27B 在未见过 Terminal-Bench 训练数据的情况下，在 TB 3.0 和 4.0 子集上超过了 GLM 5.2 和 Gemini 3.7 Flash；在 TB 2.1 上达到了 Claude Opus 4.8 的表现。

Gemini 3.7 Flash's Nastaliq-minus-Naskh paired CER gaps were: +0.18 percentage points in the natural condition, +0.25 percentage points in the shuffled condition, and +8.75 percentage points in the pseudo-text condition (95% bootstrap interval [4.05, 12.82]).

The researchers queried Kaggle's 46-model catalog, but the first expansion round exhausted the daily AI quota, and the final paired analysis included 7 endpoints that completed all 90 image prompts: Claude Haiku 4.5/5.5, Claude Sonnet 5/5.5, Gemini 3.5 Flash-Lite, Gemini 3.7 Flash, GPT-5.4 Mini.

On the ring_push task, four models (Claude Haiku 4.5, Gemma 4 31B, GPT-5.4 mini, Gemini 3.7 Flash) completed the required changes in both tasks, yet lost old behaviors not restated in the prompt, passing fail-to-pass but failing pass-to-pass, and this failure reproduced stably across eight runs.

GLM-5.3-Flash 目前仅 API 可用，权重采用 MIT 许可证但尚未发布；其视觉能力落后（BabyVision 和 MVbench 不及 Gemini 3.7 Flash）。

In Bench-Corp tests, the guardrail missed twice out of 520 runs, both on the same task with Gemini 3.7 Flash.

Gemini 3.7 Flash answered all 169 returned items correctly in the sufficiency task, but 23 calls yielded no result after three attempts; counting those as failures reduces the number of passing families from a perfect score to 28 out of 48.

On two-step reasoning questions, Claude Haiku 4.5 scored 6/30, GPT-5.4 mini scored 3/30, while Claude Sonnet 5.5, Gemini 3.8 Flash, and Gemma 4 all scored 30/30.

The revised experiments use Gemini 3.7 Flash, Claude Sonnet 5.5, and GPT-5.4, with each model independently performing three optimization attempts, and prompts using temperature 0 and seed 0.

After restoring flushing, the Claude candidate achieved a completion speedup of 2.75× and a time-to-first-byte speedup of 2.80×, while Gemini achieved a completion speedup of 3.44× and a time-to-first-byte speedup of 3.54×, and GPT-5.4 achieved a completion speedup of 2.16× and a time-to-first-byte speedup of 2.19×.

Across nine revised runs, the median completion speedup was 5.35× for Gemini, 3.69× for Claude, and 1.00× for GPT-5.4, with all three passing automated checks.

Qwen3 235B 运行的是Kaggle目录中的2506版本而非原规格的2507版本，Gemini 3.7 Flash 是Kaggle模板的默认占位模型

Qwen3-235B performed the worst on injection tests, failing 5 out of 8 attacks; Gemini 3.7 Flash, Claude Haiku 5.5, and Claude Opus 4.5 achieved zero failures across all 8 injection attacks.

After correcting the scorer, all four models achieved 100% on all five Sanity gate tasks.

In the ng-hard stress test, Gemini 3.7 Flash, GLM-5, and Qwen 3 235B all scored 100%, while gpt-oss-120b scored 80% (8 out of 10 rows).

Both models masked hidden digits in PNR or IFSC codes every time (0/25 fabrications), but filled in missing letters in words such as UMBRELLA or KITCHEN more than half the time.

Gemma 4 is the cheapest vision model on Kaggle, costing only $0.34 to run all 239 items, while Gemini 3.7 Flash cost $2.70.