DeepSeek V4 Coding Test: Top-Ranked Kimi Fails, Claude Remains the Strongest

The Real Gap Between Benchmarks and Practice

When choosing an AI coding model, leaderboard scores are the most intuitive reference. But does the #1 model on the benchmarks necessarily perform best in real projects?

A developer used Claude Code to conduct a head-to-head comparison of four top AI coding models — DeepSeek V4, GPT, Claude Opus, and Kimi K2.6 — having them complete the same full-stack mini game: a "Virtual Cat Care" system. The results were surprising: the top-ranked model on the leaderboard completely failed, while a lower-ranked model delivered the most stable performance.

This test reveals a critical fact: there's an enormous gap between standardized benchmark evaluations and real-world project coding.

Standardized AI model benchmarks typically use test sets like HumanEval, MBPP, and SWE-bench, which primarily consist of isolated programming problems that test whether a model can complete a single function or algorithm implementation within limited context. The advantage of these evaluations is their repeatability and quantifiability, making horizontal comparisons easy. But their limitations are equally obvious — real engineering projects require cross-file context understanding, multi-turn error recovery, frontend-backend coordination, and reasonable inference from ambiguous requirements. These capabilities are virtually impossible to assess through standardized problems. In other words, benchmark evaluations are like practice exams, while real projects are more like engineering defenses with no standard answers.

Test Setup and Results Overview

Task Description

Under the same set of prompts, all four models needed to complete a fully functional "Virtual Cat Care" full-stack mini game, covering UI interface, interactive animations, state management, and multiple other coding dimensions. The tech stack had explicit constraints, and the task document detailed all feature and visual effect requirements.

Final Rankings

The test results formed a stark contrast with major leaderboards:

• Claude Opus: Succeeded on first attempt in 7 minutes 30 seconds, smooth animations, complete functionality, ranked #1
• DeepSeek V4 Pro: Stable performance, mostly complete functionality, some animation shortcomings
• GPT: Traditional layout, middle-of-the-road, functions run normally
• Kimi K2.6: Spent over 22 minutes with repeated retries, all failed, unable to generate runnable code

Here's a detail worth noting: in Velse AI's latest rankings from April 23, Kimi K2.6 ranked #1 and DeepSeek V4 ranked #2, separated by only 0.07 percentage points. On the Coder Reina coding leaderboard, Kimi K2.6 also sits near the top. Yet in this real coding task, it didn't succeed even once.

What does this tell us? Leaderboards measure pass rates on standardized small problems — like scoring high on practice exams. But real projects test architectural ability, context understanding, and error recovery — precisely what benchmark evaluations can't cover.

Detailed Comparison of Four Models in Practice

UI and Layout Design Differences

GPT's layout leans traditional: cat name and hearts displayed at the top, three progress bars arranged horizontally, an orange cat with speech bubble below, and five orange buttons at the bottom. The overall feel is like a standard mini-game page — fully functional but lacking highlights.

DeepSeek V4 uses a card-based layout with a centered white card and a cat with closed eyes smiling. The progress bars use emoji icons (fish, smiley face, lightning bolt), offering good recognizability. However, excessive whitespace makes it feel empty on desktop.

Claude Opus has the strongest design sense: status tags appear directly next to the cat's name, giving an instant view of the cat's current state. The cat's display area has a large beige background, the cat is drawn larger with patterned tail, buttons are uniformly styled with rounded orange corners — the highest overall completion quality.

Interactive Animation Comparison

The feeding interaction is the first differentiator:

• GPT: Clicking feed brings up a three-option menu with food reviews and affinity changes, but food lands on the cat's shoulder with no eating animation
• DeepSeek V4: Food passes directly through the cat's body — an obvious animation glitch
• Claude Opus: Best animation effects with complete eating animations, and the kitten's eyes even track mouse movement

The "play with cat" interaction best tests animation skills. The task requires the mouse cursor to become a cat teaser toy, with the kitten chasing the cursor and displaying the full feline hunting sequence: crouching, butt wiggling, and pouncing. DeepSeek V4 performed well here — the kitten moves in all directions, and when energy drops below 20, it refuses to interact. The logic handling was solid.

Idle Animations: The Detail That Best Tests AI Coding Skills

The task requires three continuous animations: breathing movement, tail swaying, and blinking every 3-5 seconds. All three successfully running models addressed these detail requirements, but DeepSeek V4's cat eyes stayed squinted in a thin line throughout, with blinking performance falling short.

DeepSeek V4 Paper: Core Technical Innovations

While not optimal in animation details, DeepSeek V4's stability on complex coding tasks deserves attention. This is supported by several key technical innovations from their paper.

Dual-Mode Attention Mechanism: Supporting Million-Token Long Context

DeepSeek V4 supports ultra-long context of 1 million tokens — roughly equivalent to ten copies of Harry Potter and the Philosopher's Stone or the entire Three-Body Problem trilogy. Understanding this breakthrough requires knowing the computational bottleneck behind it: standard Transformer self-attention has O(n²) computational complexity, meaning doubling the sequence length quadruples the computation. For 1 million token context, a naive implementation requires approximately 10¹² floating-point operations, far exceeding real-time processing capabilities of existing hardware. To address this, academia and industry have proposed various improvements, including FlashAttention (block computation to reduce memory reads/writes) and Sliding Window Attention (local window attention). DeepSeek V4 employs a proprietary hierarchical compressed attention strategy, designing two attention modes used in alternation:

• CSA (Compressed Absorption Attention): Every 4 tokens are packed into one compressed entry, then the most relevant ones are selected for detailed reading, dramatically reducing computation
• HCA (Heavy Compressed Attention): Every 128 tokens are fused into one super-summary, suitable for quickly capturing global meaning

The two modes alternate — local detailed reading plus global speed reading, complementing each other.

MHC Manifold-Constrained Connection: A "Voltage Regulator" for Deep Networks

This technology adds a voltage stabilization device to residual transmission, ensuring deep network training doesn't collapse due to signal amplification across layers. It solves a long-standing stability challenge in ultra-large-scale model training. In extremely deep neural networks, gradient signals during backpropagation are prone to "gradient explosion" or "gradient vanishing." Manifold constraints restrict parameter updates to specific geometric manifolds, making signal transmission more stable and controllable. This is a critical foundation enabling DeepSeek V4 to stably train to trillion-parameter scale.

Muon Optimizer: First Validation on Trillion-Parameter MoE Models

The Muon (Momentum + Orthogonalization Update) optimizer was proposed by Kosson et al. Its core improvement lies in orthogonalizing gradient update matrices, ensuring each parameter update step maintains better directionality in parameter space, avoiding the gradient direction degradation common in traditional AdamW. At small-to-medium model scales, Muon has been proven to converge faster and be less sensitive to learning rates. DeepSeek V4 applied it for the first time to trillion-parameter-scale MoE (Mixture of Experts) models — MoE architecture divides model parameters into multiple "expert" sub-networks, with a gating mechanism dynamically selecting a few experts for each inference, controlling actual computation while maintaining extremely large total parameter counts. Validating Muon's feasibility at this scale is an important milestone in optimizer engineering.

Expert Sub-Training + Online Distillation: Avoiding Multi-Domain Negative Transfer

Knowledge Distillation was first proposed by Hinton et al. in 2015, with the core idea of using a large model's (teacher) output soft labels to guide small model (student) training. In multi-domain joint training, Negative Transfer is a long-standing challenge — data distribution differences across domains are large, and forcibly mixing training may cause model performance degradation in certain domains. DeepSeek V4 adopts