Meta Muse Glimmer vs Qwen: 30B Open-Source Model Showdown on China's College Entrance Math Exam

Meta Returns to the Open-Source Arena: Muse Glimmer Officially Released

This week, Meta released a brand-new 30B-class open-source large language model — Muse Glimmer — sparking considerable discussion in the AI community. This is Meta's latest heavyweight open-source model following Llama 4. At the time, Llama 4 received a lukewarm market response, even leading some to question whether Meta still belonged in the top tier of large model developers. Now, with Zuckerberg himself posting congratulations on social media, it's clear that Meta has high hopes for this new model.

Meta's open-source LLM strategy dates back to the Llama series released in 2023. Llama's open-source approach was seen as a direct challenge to OpenAI's closed-source path and established Meta's leadership position in the open-source AI community. After the successive releases of Llama 2 and Llama 3, a vast number of developers fine-tuned and built upon these models, forming a massive ecosystem. However, Llama 4's release failed to sustain this momentum — industry opinions were mixed, partly because competitors like Qwen (通义千问^Ad), Mistral, and Google Gemma had made significant progress during the same period, diminishing Llama 4's relative advantage.

The model's name carries deep symbolism: Muse refers to the Greek goddesses of wisdom and inspiration, while Glimmer means a faint light. Together, they suggest "a glimmer from the Muse." Whether this glimmer can illuminate Meta's comeback path is precisely what this evaluation aims to explore.

Notably, the 30B (30 billion parameter) class occupies a critical "sweet spot" in today's AI ecosystem. Compared to 70B or larger models, 30B-class models can run inference on consumer-grade GPUs (such as a single NVIDIA RTX 4090 or A100), significantly lowering the hardware barrier. At the same time, compared to smaller 7B-13B models, the 30B class shows marked improvements in complex reasoning, multi-step logic, and knowledge coverage. This makes the 30B class the go-to choice for enterprise private deployments and local developer use, and one of the most fiercely contested parameter ranges among major vendors.

Evaluation Method: Testing Real Reasoning Ability with Gaokao Math Problems

To test Muse Glimmer's real capabilities, this evaluation chose a particularly challenging scenario — math problems from China's National College Entrance Exam (Gaokao), Paper I. This is a highly representative choice: Gaokao math problems not only test a model's mathematical reasoning ability but also impose strict requirements on answer format standardization.

The other model in this comparison is Qwen 3.6 27B, which was released before the Gaokao; Muse Glimmer was released approximately two months after the exam ended. This time difference is worth noting — theoretically, a model released later is less likely to have "seen" these problems in advance, making the test results arguably more fair. This touches on an important methodological issue in LLM evaluation — Data Contamination. If the test questions appeared in the model's training data, the model might produce correct answers through "memorization" rather than "reasoning," leading to inflated evaluation results. The 2024 Gaokao problems would have spread rapidly across the internet after the exam, theoretically making them available to subsequently trained models. Therefore, a model released before the Gaokao (like Qwen 3.6 27B) is actually more credible in terms of evaluation fairness, since its training data is unlikely to contain these problems.

Dataset Construction Details

The evaluator used tools to extract objective questions from the Gaokao math paper and constructed a dataset in JSONLines format. JSONLines (also called JSONL) is a lightweight data storage format with one independent JSON object per line, well-suited for streaming reads and processing of large-scale datasets. Compared to traditional JSON array format, JSONL doesn't require loading the entire file into memory when processing large files, making it more efficient.

All math formulas in the dataset are expressed in LaTeX format, totaling 14 questions. LaTeX is a mathematical typesetting system widely used in academia and engineering, capable of precisely expressing fractions, integrals, matrices, and other complex mathematical symbols. Using LaTeX-formatted math problems in LLM evaluation not only tests the model's ability to parse LaTeX syntax but also indirectly tests the depth of mathematical and academic content learned during pre-training. Each model was run three times to assess result stability.

Notably, the prompts submitted to the models did not include question type hints (i.e., the model was not told whether a question was multiple choice, multi-select, or fill-in-the-blank) — the model had to determine the question type on its own. This design significantly increases test difficulty and better approximates real problem-solving scenarios.

Evaluation Metrics Explained: Semantic Accuracy, Stability, and Format Compliance

This evaluation established multiple dimensional metrics to comprehensively measure actual model performance:

• Semantic accuracy: Tests the model's mathematical reasoning ability — whether answers are semantically correct;
• Three-run perfect rate: The proportion of questions answered correctly across all three runs, reflecting performance stability;
• Format accuracy: Tests the model's ability to follow output format specifications.

After the model returns results, the testing script automatically extracts answers and compares them against reference answers for corresponding questions in the dataset.

Qwen 3.6 27B Wins Across the Board

Looking at the results, Qwen 3.6 27B outperformed on all metrics. On the semantic three-run perfect rate representing answer stability, Qwen clearly won; on format accuracy, Qwen likewise led Muse Glimmer by a significant margin.

The evaluator highlighted a specific example: even in Muse Glimmer's best run for format compliance, on several multiple-choice questions where the standard answer should be just the letter corresponding to the option, the model also included the numerical value of the option, resulting in non-standard formatting. The other two runs exhibited similar issues.

Format accuracy and instruction-following ability are critical metrics for a large model's transition from "lab product" to "production-ready." In real applications, LLM outputs typically need to be passed to downstream systems for automated processing — for example, extracting structured data via regular expressions or JSON parsers. If a model outputs "A. 3/4" instead of just "A" when answering a multiple-choice question, the answer may be semantically correct but will cause automated scoring systems to fail matching. The capability behind this involves training quality during the RLHF (Reinforcement Learning from Human Feedback) and SFT (Supervised Fine-Tuning) stages — high-quality instruction tuning data and alignment training are the core means of improving format compliance.

By comparison, Qwen 27B's answers were generally cleaner and more concise in format, with noticeably better presentation. Looking at overall performance on Gaokao math problems, Qwen 3.6 27B remains the best model in the 30B class. The evaluator also mentioned that while Google Gemma's data wasn't included in this comparison, Gemma's performance also surpassed Muse Glimmer's.

A Positive Signal for the Open-Source Ecosystem: Competition Drives Progress

Although Muse Glimmer didn't surpass Qwen or Gemma in this evaluation, Meta's return is undoubtedly good news for the entire open-source community.

Competition in the open-source LLM space is essentially a healthy "arms race." When leading companies continuously invest and iterate, the ultimate beneficiaries are developers and users at large. As the original creator of the Llama series, Meta's re-entry will inject fresh vitality into the open-source ecosystem and pressure other vendors to accelerate innovation.

It's worth reflecting that this evaluation also reveals a key point in current open-source model competition: the balance of comprehensive capabilities. Muse Glimmer may not be entirely behind in mathematical reasoning, but its shortcomings in "engineering details" like format compliance are what widen the gap with competitors. This reminds us that evaluating a large model's practical value cannot focus solely on reasoning ability — format stability, instruction following, and other "soft skills" are equally critical. Improving these "soft skills" often depends on meticulous refinement during the post-training stage, including high-quality human preference data collection, multiple rounds of RLHF iteration, and specialized optimization for specific downstream tasks. This also means that behind an excellent open-source model, not only is a strong pre-training foundation needed, but substantial engineering resources must also be invested in alignment and fine-tuning.

We hope Meta continues to push forward, delivering better models in subsequent versions, letting this "glimmer from the Muse" truly illuminate the future of open-source large language models.