Mistral Large 4 Hands-On: How Good Is the "Best Open-Source Model" at Real Programming Tasks?

Mistral's newly released Large 4 (codename Le Chonk) looks impressive on paper — but the real question is whether it can reliably complete the programming tasks you actually throw at it. One developer ran this model through a real OpenCode + OpenRouter workflow using their own KingBench3 benchmark, and the conclusions are far more useful than any official leaderboard score.

The Testing Method: Only Final Deliverables Count

The core principle here is deliberately restrained — score only what gets delivered, not how well the model explains itself. The tester used fresh empty folders for each task, reusing the same eight KingBench3 prompts that had been used with other models, with each task worth 10 points.

For browser-based projects, the output was opened directly in Chromium to check whether the page rendered correctly and whether the main interactions actually worked. This distinction matters: a model can write a polished summary at the end while the page itself does something completely different. In other words, this method measures "can it get the job done" — not "does it look like it's thinking hard."

KingBench3 is a private benchmark designed by the tester. Unlike standardized code generation benchmarks such as HumanEval or MBPP, it doesn't evaluate whether code passes unit tests — the only criterion is whether a user can actually use the output in a real browser or runtime environment. OpenCode is an open-source AI coding agent framework that handles receiving prompts, calling model APIs, creating files, and executing code in a local environment. OpenRouter is a model API aggregation layer that lets you call models from different providers through a single interface. This combination represents the mainstream "local agent + remote model" workflow, which is closer to how developers actually work than testing in a chat interface — and therefore surfaces more realistic pain points (such as exhausted turn limits and uncreated files).

Where It Shines: Focused Visualization Tasks

When the task scope is clear and focused on a single visualization goal, Le Chonk performs impressively.

Elevator Simulation (9.5/10) required three elevators with a one-person capacity each, passengers spawning on different floors, hover-to-show destinations, and stragglers automatically boarding the next available elevator. The model delivered a clean building view, three elevators with door-opening animations, passenger counters, and a spawn button — with queue logic that actually worked. The tester reset the simulation, spawned five passengers, and confirmed that both wait and ride counts reset to zero with no browser errors. The only minor flaws were a lingering exit animation from old trips on reset, and the wait count updating slightly before passengers actually boarded — not enough to break the main flow, but enough to fall just short of a perfect score.

Folding Table (8.5/10) loaded a complete Three.js scene with lighting, materials, shadows, orbit controls, and a slider. Dragging the slider produced smooth 3D control from fully folded to fully open, with both states rendering in the browser without errors. The mid-animation had minor glitches (legs floating off the ground, model appearing small), but the core requirements were fully met.

Panda Eating a Burger SVG (8/10) generated a valid, clearly rendered SVG with a recognizable panda, burger, paws, and open mouth. The weakness was the "eating" action — the burger was close to the mouth but showed no actual biting motion, making the theme slightly ambiguous.

The most impressive result was the local fine-tuning task (10/10): generate a panda knowledge dataset, fine-tune a Gemma 2B model locally, and build a webpage that shows a new fact on every refresh. After enabling cached model permissions, Le Chonk completed the entire pipeline — generating 510 samples, training a LoRA adapter for 400 steps, and reducing validation loss from roughly 7.97 to 0.31. It even discovered and fixed a server threading issue during its own end-to-end test. The tester refreshed Chromium multiple times and received a different panda fact each time with no errors, with both the model and the webpage running entirely locally. For a task with this many moving parts, that's a genuinely strong result.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method that freezes most of a pretrained model's weights and only trains low-rank matrices inserted into each layer — achieving domain adaptation with far fewer trainable parameters (typically under 1% of the original model). A validation loss drop from 7.97 to 0.31 indicates a significant improvement in the model's fit to the new dataset, accomplished within 400 training steps — suggesting the dataset size (510 samples) and model scale (Gemma 2B) were well matched. Completing this kind of end-to-end pipeline locally requires correctly configuring the Python environment, dependencies (e.g., Transformers, PEFT), GPU/CPU training parameters, and model weight cache paths. Any failure at any step breaks the process, which makes Le Chonk's ability to self-identify and fix the server threading issue — and ultimately run the full pipeline to completion — a genuine demonstration of competence on complex multi-step tasks.

Where It Fails: Reliability and the "Non-Delivery" Problem

The problem lies squarely with consistency.

Contact Lens Case (default 0/10) was supposed to produce a 3D box with L/R compartments and clickable lids. But in the default OpenCode run, the model spent its entire turn budget reasoning about "how to build this box" — and then created no files whatsoever. No product to open or click; straight zero. The tester was explicit: he doesn't score a "plan" as a "finished product." Interestingly, re-running in OpenCode's "no-reasoning" mode did produce a box with independently openable left and right lids (though they flew too high and partially out of frame), proving the model is capable — but the reliability failure in the default run still deserves a zero.

Counting Problem (0/10) The correct answer is 20,460. In the default session, the model again exhausted its turn budget without providing a final answer, leaving nothing to evaluate.

Archery Game (6/10) produced a complete game — four target types, a wind indicator, arrow physics, hit tracking, and a localStorage leaderboard — with no JS errors. But the targets were only a few pixels wide at desktop size, their fixed positions pushed the far targets off-screen on mobile, and the aiming guide didn't fully match the drag trajectory. All the systems are there, but another polishing pass is needed before it's actually fun to play.

3D Wristwatch (6/10 after fix) showed a blank page on first inspection, with the time display still showing placeholder dashes. The root cause was an import map referencing module paths the model never created — a watch that never loads obviously scores zero. But when the tester fed the browser error back to the same model, it successfully fixed the path. The second inspection produced a rendered 3D clock with a moving second hand and a toggleable second time zone. This "self-repair" process was considered more informative than a broken screenshot.

"Exhausting turns" refers to an agent framework forcibly terminating a task after hitting the maximum interaction step (turn/step) limit set for that session. In agent systems like OpenCode, each tool call (reading/writing files, executing commands, searching documentation) consumes a step from the quota. When a model spends time planning, repeatedly revising, or getting stuck in loops, the quota runs out before any actual output is produced — leaving the task with "nothing created." This problem is especially pronounced with reasoning models: the model tends to perform extensive chain-of-thought internally, and those thinking steps consume the agent's action quota. "No-reasoning" mode disables this extended thinking process, getting the model to actual code generation faster — which is why the contact lens case retry succeeded. This also suggests that for more complex agent tasks, properly configuring step limits and reasoning mode is a critical factor in success rate.

Final Scores and a Realistic Assessment

After correcting for the permissions issue, Le Chonk's total is 42/80 — 52.5%, or 5.25/10. The tester excluded the contact lens case retry and the watch fix from the total score (as these were separate attempts after the initial output), but counted the fine-tuning re-run, since the original interruption came from his own OpenCode permissions configuration — blaming the model for that wouldn't be fair.

The real insight is the stark contrast: when Le Chonk completes a focused visualization task, it can produce genuinely impressive work — the elevator is functional, the folding table is polished, the panda SVG is charming, and the local Gemma fine-tuning proves it can complete a long pipeline when the environment is configured correctly. But the contact lens case and counting sessions ended with "no usable output," and the wristwatch required post-browser-test intervention. These are precisely the "workflow pain points" where you hand off a task to an agent, go focus on something else, and come back to find nothing was completed.

The tester also honestly notes that this single run shouldn't be treated as a universal model ranking — it's just his prompts and verifiable results under a specific OpenCode + OpenRouter configuration. His practical recommendation is clear: for controlled visualization prototyping, it's worth trying — just keep a browser and test loop nearby. For longer, less fault-tolerant work, he wants to see it reliably complete tasks before trusting it as a default coding model.

For actual users, the decisive factor is always the same: how often can you hand it a task and get back something that actually works?