AI Agent Evaluation: How the LFORLA JS 3D Builder Leaderboard Scores Models

TL;DR: In this evaluation system, the meaning of AI agent evaluation is narrowed to one thing—a model reads a written architectural specification, plans a structure, selects the correct Three.js components, and assembles a build plan. Every step is measured against the same rubric for all models. No code is executed. No LLM is used as a judge. Only server-side static analysis of the agent-generated code.

How an evaluation with no execution works

This benchmark is called JS 3D Builder (Three.js), and the task is procedural generation: starting from a written specification, the model must produce a 3D building using Three.js. Scoring is done by server-side static analysis—the evaluator parses the submitted code and checks it item by item against the rubric. It does not run the code, and it does not ask another model to score it.

The mechanism has five steps:

1. Prompt. The model receives a written specification of a building, describing geometry, spatial relationships, and required components.
2. Plan. The model must plan the structure first, then write code. Reasoning ability and the spatial category show up in this step.
3. Source parts. The model needs to select and reference the correct Three.js primitives and constructors. Choosing the wrong component costs points.
4. Assemble a build plan. The model outputs code that builds the structure, and this code is the artifact being evaluated.
5. Score against the rubric. Server-side static analysis checks the code against the same standards, covering four categories: code, reasoning, 3D, and spatial.

The key design choice is that the rubric is fixed. Every model is measured the same way, which is why the leaderboard is comparable.

Static analysis vs. dynamic testing. Static analysis means judging correctness by parsing the source code's syntax tree, symbol references, and structural patterns without running the program. By contrast, dynamic testing requires actually executing the code and observing runtime behavior. The advantage of static analysis is that it is fast, reproducible, and deterministic—it does not depend on a sandbox environment, and it does not fail because external dependencies are missing. The cost is that it cannot detect problems that only surface at runtime, such as crashes from type errors, asynchronous timing issues, or behavioral differences in a particular version of a Three.js API. In this evaluation, static analysis also avoids the security risk of executing arbitrary model-generated code—after all, the evaluator is dealing with untrusted input.

Three.js and procedural generation. Three.js is currently the most widely used browser-side WebGL wrapper library. It abstracts lower-level graphics interfaces into objects such as Scene, Camera, Geometry, Material, and Mesh, and developers describe a 3D world by combining these components. Procedural generation means the scene is not hand-built by artists, but automatically constructed by code according to rules—which fits the evaluation need perfectly: the specification is text, the artifact is code, and no asset files are required in between. The model has to understand what spatial semantics components like BoxGeometry, CylinderGeometry, Group, and Vector3 correspond to in order to translate a written specification into a structurally correct scene graph. If it chooses the wrong component—for example, using a plane instead of a cube—then even if the code runs, the spatial meaning has already drifted away from the specification.

The real scores are right here

Model	Score	Provider
Nemotron 3 Ultra (free, called via opencode)	71.46666666666667	deepseekAd
GLM 5.2 (the author's own submission)	78.0	lforla

The author submitted their own model to this benchmark, and the results are above. GLM 5.2 scored 78.0, and Nemotron 3 Ultra scored 71.46666666666667.

What's behind the score gap

The gap is not huge, but it is real. A score in the low 70s means the model can produce a usable build plan, but static analysis can still find gaps. These gaps usually show up in spatial reasoning: the model puts components in roughly the right places but misses precise spatial relationships. A score of 78.0 means fewer such gaps and a higher degree of consistency between the code and the specification.

Because there is no execution and no LLM judge, the score reflects what the code looks like as written. A model that writes seemingly plausible code but hides subtle spatial errors will score lower than a model whose code is less elegant but structurally correct. This is exactly the point of static analysis: reward correctness, not expressiveness.

The category breakdown matters too. Code, reasoning, 3D, and spatial are all within the scoring scope. A model may be good at code generation but weak in spatial reasoning, and the total score will expose that. This is also why doing AI agent evaluation on this benchmark is not just a programming test, but also a planning and spatial test.

Why spatial reasoning is the hard part. Spatial reasoning requires the model to maintain an internally coherent mental model of a 3D coordinate system: a certain room is east of another room, a certain wall is perpendicular to the floor, a certain opening faces a particular direction. This is different from pure code generation, where it only needs to ensure syntax and logic are self-consistent. Common model failures include: writing relative positions as absolute coordinates and miscalculating the offset, making mistakes in the order of rotation and translation, and ignoring parent-child hierarchy, which causes overall misalignment. Because the spatial category is scored separately in the rubric, such errors are not masked by code correctness. This also explains why a model that writes elegant code may fall behind a model with plain code but accurate structure because of a few coordinate errors.

What this means for practitioners choosing models

If you are choosing a model for 3D or spatial tasks, this kind of leaderboard is more informative than general programming benchmarks. It tells you how a model handles a specification, not just how it writes a function. The gap between 71.47 and 78.0 is the gap between "needs more correction" and "needs less correction."

But do not look at just one number; look at category scores. If your task has high demands for spatial reasoning, a model that scores well in that category is more worth choosing than a model with a slightly higher total score.

How to read a leaderboard

• Check the rubric. If the scoring method is not explained, the scores are not comparable.
• Check who the judge is—a human, an LLM, or static analysis. Their failure modes differ.
• Check category scores. The total score hides a model's weaknesses.
• Check the submission date. Models change, and a score from six months ago may no longer hold.
• Check whether the benchmark author submitted their own model. The author did, and disclosed it publicly.

Honest limitations

This benchmark does not execute code. A build plan that passes static analysis may still fail at runtime. The fixed rubric makes comparisons fair, but it also means it may not capture every dimension of agent capability. The leaderboard currently has very few entries—one at 71.47 and one at 78.0—so this is a beginning, not a conclusion. And the author's own submission is their own model, which is a conflict of interest—the author chose to disclose it publicly rather than hide it.

Conclusion

AI agent evaluation is most useful when the mechanism is transparent. Here the mechanism is: plan, source parts, assemble a build plan, with each step scored against the same rubric. The numbers are real, and so are the limitations. To see the full leaderboard and benchmark details, visit lforla.org.

One-line Summary

LFORLA JS 3D Builder evaluates a model's ability to generate Three.js building code through static analysis with a fixed rubric.

Paragraph Summary

The article introduces an AI agent evaluation benchmark called JS 3D Builder (Three.js). Its task is to start from a written architectural specification and have a model plan a structure, select the correct Three.js components, and assemble build code. The evaluation does not execute code or use an LLM judge; instead, server-side static analysis scores submitted code against fixed standards in four categories: code, reasoning, 3D, and spatial. In testing, GLM 5.2 scored 78.0 and Nemotron 3 Ultra scored 71.47, with the gap mainly appearing in the precision of spatial relationships. The author openly acknowledges a conflict of interest because their own model was also submitted to the benchmark, and points out that static analysis cannot guarantee runtime usability and that the leaderboard sample size is small. The article recommends focusing on category scores rather than total scores when choosing a model and provides a five-item checklist for reading leaderboards.