Huawei Pangu 2.0 Flash on the Gaokao Benchmark: 613 Points, Stability Issues, and Cost Breakdown

A New Contestant in the LLM Gaokao Series: Huawei Pangu 2.0 Flash

Bilibili creator Wang Yu's "LLM Gaokao Series" has reached its eighteenth episode, welcoming a highly anticipated new contestant — Huawei's open-source model openPangu 2.0 Flash. The benchmark uses China's college entrance exam (Gaokao) across six subjects — Chinese, Math, English, Physics, Chemistry, and Biology — with a total score of 750 points. It has become one of the most representative horizontal evaluation frameworks for LLM reasoning and knowledge capabilities in China.

According to the creator, Pangu 2.0 Flash uses a MoE (Mixture of Experts) architecture with a parameter scale of 92B A6B (92B total parameters, 6B activated parameters).

What is MoE Architecture? MoE (Mixture of Experts) is one of the most significant engineering breakthroughs in the LLM field in recent years. The core idea is to divide the model's internals into multiple "expert" sub-networks. During each inference pass, a gating mechanism dynamically selects a small subset of experts to participate in computation, rather than activating all parameters. Taking Pangu 2.0 Flash's 92B A6B as an example: "92B" refers to the total stored parameter count, which determines the model's knowledge capacity ceiling; "A6B" (Activated 6B) refers to the parameters actually engaged in each inference pass, directly determining inference speed and per-call cost. The roughly 6.5% activation rate is the fundamental reason MoE architectures achieve orders-of-magnitude cost advantages over traditional dense models. OpenAI's GPT-4, Google's Gemini 1.5, and domestic models such as DeepSeek and Kimi all employ similar designs.

With only 6B activated parameters, Pangu 2.0 Flash has a natural cost advantage at inference time. In the end, it scored 613 points across all subjects, ranking 33rd out of 37 models evaluated. While the overall score isn't exceptional, it falls within the expected range given its relatively small activated parameter size.

A Stability Red Flag: Reasoning Works, Output Fails

The most noteworthy detail in this evaluation occurred on Biology Question 6 — Pangu 2.0 Flash repeatedly failed to produce a valid answer. After reviewing the logs, the creator uncovered a telling phenomenon: the model's internal reasoning chain had already arrived at the correct conclusion, but the final answer simply never made it to the output.

This is a classic engineering stability problem. The generation process of a large language model can be divided into two stages: Prefill (processing the input and building the reasoning context) and Decode (generating output token by token). When the reasoning process reaches the correct conclusion but the final answer fails to output correctly, this typically involves one of the following engineering issues: Instruction Following failure, where the model cannot correctly map internal reasoning results to a structured output format; improper decoding strategy settings (such as sampling temperature or top-p parameters) that inadvertently suppress high-probability tokens; or defects in Constrained Decoding logic that truncate valid content when forcing a specific output format. Due to the dynamic nature of expert routing in MoE models, stability control over long-text generation tends to be more complex than with dense models. The reasoning pipeline was intact — the output stage is what broke down. The creator candidly noted that this suggests Pangu 2.0 Flash's stability "still has room for improvement."

On the cost front, however, Pangu performed impressively — completing the entire Gaokao test for under one yuan (RMB). For budget-sensitive applications with relatively standardized tasks, this cost advantage is hard to ignore.

New Filtering Features on the Leaderboard: Open Source and Country at a Glance

This episode also introduced a practical product update to the leaderboard — a multi-dimensional filtering feature allowing users to quickly filter by:

• Country: Chinese models only or US models only
• Modality: Multimodal models or text-only models
• Openness: Open-source models only

Huawei Pangu is included in the open-source model list. This is a significant plus for Pangu 2.0 Flash — in enterprise deployment scenarios, open-source and closed-source API models represent fundamentally different technical approaches. Closed-source API models (such as GPT-4 and Claude) charge per token and require data to be sent to third-party servers, posing clear barriers in industries with strict data compliance requirements such as finance, healthcare, and government services. Open-source models, on the other hand, allow enterprises to deploy on-premises or in private cloud environments, keeping data entirely within the organization and satisfying regulatory requirements such as the Data Security Law and the Personal Information Protection Law. Huawei's entry further enriches the pool of self-deployable options in the domestic open-source LLM ecosystem. The MoE architecture's low activated parameter count (6B) means enterprises can run inference on consumer-grade GPUs or even edge devices, dramatically lowering the hardware barrier for private deployment. For enterprise users who prioritize data compliance and need on-premises deployment, an open-source, low-cost, domestically produced MoE model carries unique value — even if its overall ranking isn't at the top.

How Essay Scoring Works: A Dual-Model Cross-Grading Mechanism

How exactly are LLM-written Gaokao essays scored? The creator explained the mechanism in detail: scoring mimics the Gaokao's "dual grading" model, with GPT-4.1 and Claude Sonnet 5 cross-grading each essay.

LLM-as-Judge: Using AI to Evaluate AI Using large language models to assess the output quality of other large language models is known in academia as the "LLM-as-Judge" paradigm, and is currently one of the most active research directions in AI evaluation. For subjective tasks like essay writing and open-ended Q&A, traditional rule-based matching or human grading is prohibitively expensive and difficult to keep consistent, while highly capable LLMs have already approached or surpassed average human raters in semantic understanding and logical judgment. Using two different models for cross-grading is designed to mitigate the systematic bias of any single grading model — different models exhibit significant differences in stylistic and conceptual preferences due to variations in training data and alignment strategies. This design draws on the statistical principle of Inter-rater Reliability: when two independent scoring sources agree closely, the result is considered more trustworthy.

An interesting incident also occurred during the evaluation. Starting with Meituan's LongCat 2.0, one of the grading models was switched from Sonnet 4.6 to Sonnet 5. The creator immediately noticed that LongCat's essay score appeared unusually high, and initially suspected that Sonnet 5 might be grading more leniently than the previous version.

Re-grading Validates LongCat's Essay Quality

To rule out scoring bias, the creator had both "grading models" re-score the essays from all participating models under a unified standard. The results were convincing: even after re-running the evaluation, LongCat's essay score remained firmly in first place at 54.5 points.

This verification process reflects the rigor of the evaluation — when anomalous data appears, rather than rushing to assign blame, the approach is to re-align under a unified standard to eliminate confounding variables. This is the standard practice in LLM-as-Judge for handling "grading model drift." According to the published commentary, LongCat's essay demonstrated "relatively original ideas," and both grading models gave positive feedback. This also indirectly suggests that MoE architecture models may perform on par with or even better than pure reasoning-focused models on language-heavy, creative tasks.

Coming Up Next: Tencent Hunyuan 3 Official Release

At the end of the video, the creator revealed that Tencent Hunyuan 3, which also completed the test for under 90 fen (RMB), previously participated using a preview version. The official release has since launched and is reported to show significantly improved capabilities.

In the next episode, Tencent Hunyuan 3's official release will take the Gaokao benchmark — the scale of its improvement is well worth watching.

A Final Word: How to Interpret the Value of This Leaderboard

As the creator has repeatedly emphasized: LLM benchmarks carry inherent variability — any leaderboard only reflects results under specific prompts in a specific context. Pangu 2.0 Flash's "output failure" on the Biology question is a vivid illustration of that variability.

The true value of this Gaokao benchmark lies not in producing an absolute ranking of models, but in providing a unified, reproducible horizontal comparison framework that gives us an intuitive sense of the differences between models across reasoning, knowledge, and writing dimensions. Model selection ultimately comes back to your own business scenario and budget — Pangu 2.0 Flash ranks 33rd overall, but its open-source nature, extremely low cost, and support for private deployment may actually make it the better choice in data-compliance-sensitive sectors such as finance, government, and healthcare.