GLM4 Open-Source Flagship Tested: Crushes Claude and Other Commercial Models in Frontend Development

A New Benchmark for Open-Source Models

The Zhipu AI team has officially released the all-new GLM4 open-source flagship model, open-sourced under the MIT License with full weights available. The MIT License is one of the most permissive licenses in the software open-source world, allowing anyone to freely use, modify, distribute, and even commercialize the work—the only requirement being to retain the copyright notice. For AI models, "releasing full weights" means developers can download the complete model parameters for local deployment, fine-tuning, or distillation, rather than being limited to API calls—similar to Meta's LLaMA series strategy, and a key factor in driving rapid growth of the open-source AI ecosystem.

This model's intelligence level is truly industry-leading, comparable to mainstream commercial closed-source models like Claude 3.5 Sonnet, yet at a much lower cost. GLM4 brings major upgrades in coding and Agent task handling, purpose-built for complex long-horizon tasks with a million-token context window. A million tokens is roughly equivalent to 750,000 English words—about the entire Harry Potter book series. For Agent tasks, ultra-long context means the model can process an entire codebase or complex multi-step task history in a single session without frequently "forgetting" earlier information. The technical challenge here is that the computational complexity of attention mechanisms grows quadratically with sequence length, which the industry typically addresses through sparse attention or improved positional encodings (such as RoPE or ALiBi). GLM4 also offers two inference variants—GLM4 Snacks and GLM4 Air—corresponding to different speed-cost tradeoff strategies.

Even more impressive is GLM4's stunning performance in frontend development—it currently sits at #1 on the Design Arena leaderboard with a score around 1300, surpassing even Claude 3.5 and Gemini 1.5 Pro. Design Arena is a competitive evaluation platform specifically targeting AI models' frontend code generation capabilities, using a mechanism similar to the chess ELO rating system—through extensive blind-test comparisons where human reviewers vote between pages generated by two models, with winners gaining points and losers losing them, ultimately forming a dynamic leaderboard. A score around 1300 represents top-tier performance, and this evaluation method better reflects a model's practical value in real development scenarios than traditional static benchmarks. This is the first time we've seen an open-source model achieve such an outstanding level in web development.

Crushing Value: Same Quality at One-Sixth the Price

Head-to-Head with Claude 3 Opus

Building the exact same landing page with GLM4 and Claude 3 Opus, both deliver excellent visual results—honestly, it's hard to spot obvious differences. But the most incredible part is the cost difference: GLM4 generates a web page version for about 6 cents, while Claude 3 Opus costs around 50 cents—more than 6x cheaper—and it's faster and more token-efficient too.

On pricing, GLM4 charges $1.2 per million input tokens and $4.1 per million output tokens, essentially the same as its predecessor GLM4.1. For developers and enterprise users, this means accessing near-top-tier closed-source model performance at extremely low cost.

Comprehensive Benchmark Leadership

GLM4 delivers impressive results across multiple benchmarks:

• SWE-Bench: Achieved 46.2%, significantly outperforming previous-generation models
• Code reasoning and general knowledge: Scored 74.4%, second only to Claude 3 Opus
• Terminal Bench and SWE-Bench Pro: Either beats all closed-source models or closely trails top models like GPT-4

SWE-Bench (Software Engineering Benchmark) is an authoritative code capability evaluation benchmark developed by Princeton University researchers, specifically testing AI models' ability to solve real GitHub Issues. The test set contains 2,294 real bug-fix tasks from 12 major Python open-source projects (such as Django, Flask, NumPy, etc.), where models must independently locate problems and generate correct code patches without hints about specific modification locations. A score of 46.2% means the model can independently solve nearly half of real software engineering problems—an extremely high standard in the industry. SWE-Bench Pro is a more advanced version with higher difficulty, including more complex multi-file modification scenarios.

An open-source model coming this close to Claude 3 Opus was previously unimaginable. On the World of AI benchmark tool, GLM4 currently ranks fifth, performing exceptionally well among all frontier models.

Frontend Development Testing: Breakthroughs in Both Aesthetics and Functionality

Landing Page Generation and Website Cloning

GLM4 demonstrates a completely different aesthetic sensibility from other models in frontend development. From scroll triggers to various dependency packages, every element is accounted for. Its generated landing pages include complete scroll-triggered effects, shader background effects, and refined typography.

Even more impressive is its website cloning capability. When asked to replicate the Airbnb website, GLM4 perfectly captures and reproduces all elements, including the homepage layout, image panels, and listing detail display functionality. When creating multiple pages, the model maintains stylistic consistency—one of GLM4's strongest advantages.

Spotify Replication and Audio Visualization

When asked to replicate the Spotify interface, GLM4 absolutely nailed it—the main screen, search page, every component is present, and it can even play music. Additionally, GLM4's generated audio visualization tool performs excellently, responding very sensitively when volume is increased, perfectly rendering dynamic effects, and supporting different color themes and visual modes.

Games and 3D Development: Pushing the Limits of Creativity

Dungeon Exploration Game and Minecraft Clone

GLM4 generated a fairly complete dungeon exploration game where players can move between different areas, find keys to unlock new zones, and collect monsters, potions, and items. This level of rich detail is something even Claude Opus failed to achieve.

In the Minecraft clone test, GLM4's output is among the best versions to date. It generated a complete inventory system, various creatures, support for placing and breaking blocks (with destruction animations), and even a complete cave system—just dig straight down and you'll find it. Simply incredible.

macOS Simulation and 3D Scene Rendering

GLM4's generated macOS simulation interface is quite detailed: the Finder app is highly faithful, the top bar functions properly, Spotlight search and the top-right menu bar are implemented, and it even includes Terminal, Calculator, System Settings (with dark mode toggle), and wallpaper changing functionality. While some components aren't fully generated, the overall completion level is already very high.

In 3D development, GLM4 used Three.js to generate a first-person shooter (with firing perspective and brick animations), a solar system model (each planet has unique features, including Jupiter's Great Red Spot and the asteroid belt), and procedural tree growth effects—the best performance among all models, even surpassing some specialized models. Three.js is a JavaScript 3D graphics library based on WebGL that wraps low-level GPU rendering interfaces, allowing developers to create complex 3D scenes and animations with relatively concise code. GLM4's ability to correctly generate this type of code demonstrates that the model not only masters Three.js API syntax but deeply understands underlying concepts like 3D spatial coordinate systems, lighting models, particle systems, and physics simulations—these cross-domain knowledge integration tasks are an extreme test of a model's spatial reasoning capabilities.

The lava lamp generation is particularly stunning, perfectly reproducing the dynamic changes of a real lava lamp, complete with subtle glow and ambient feel, fully demonstrating GLM4's deep understanding of spatial and physical principles.

Shortcomings and Usage Recommendations

Despite GLM4's impressive performance, there are some notable weaknesses:

• Code debugging and logical reasoning aren't its strongest suits; complex debugging scenarios may prove challenging
• Code generation capability falls slightly short in certain scenarios
• Some complex components may not be fully generated, with occasional missing functionality

To address these shortcomings, it's recommended to use GLM4 with deep thinking mode enabled for best results. Deep thinking mode refers to having the model perform explicit Chain-of-Thought reasoning before generating the final answer, similar to OpenAI's o1 series "thinking tokens" mechanism—the model first generates an internal reasoning process, then outputs the final result based on that reasoning. Research shows this approach significantly improves accuracy on complex logical reasoning, mathematical proofs, and code debugging tasks, at the cost of increased response latency and token consumption. Additionally, the Zhipu team has specifically strengthened GLM4's context training, with targeted optimizations for large-scale code agent deployment, performance optimization in automated research, and complex debugging.

Conclusion: A Milestone for the Open-Source Ecosystem

Overall, on a scale of 10, GLM4 earns an 8. It's not just an excellent open-source model—it's a serious contender going head-to-head with closed-source giants like Claude Opus and GPT-4. In frontend development, game generation, and 3D scene construction, GLM4 demonstrates stunning creativity and aesthetic standards.

Having an open-source model reach this level is a positive signal for the entire AI ecosystem. It means developers no longer need to pay premium prices for top-tier AI capabilities, and the open-source community is closing the gap with closed-source models at an unprecedented pace. GLM4's release is undoubtedly one of the most important milestones in open-source AI.