LFORLA
An AI agent evaluation system and organization that maintains benchmarks such as JS 3D Builder, emphasizing fixed scoring criteria, server-side static analysis and comparability across models.
Timeline (last 90 days)
JS 3D Builder (Three.js) is a benchmark whose task is, starting from written architectural specifications, to procedurally generate 3D building code using Three.js, scored by server-side static analysis, without executing the code or using an LLM judge.
The author of the ranking benchmark submitted his own model GLM 5.2 to his own benchmark, which constitutes a conflict of interest.
This benchmark uses deterministic token similarity comparison for scoring, with reference answers stored only on the server side and never shown to models.
The Code Similarity metric in this benchmark consists of 60% bag-of-words cosine similarity plus 40% Longest Common Subsequence ratio.
The 60/40 cosine and LCS mixing ratio is a subjective engineering choice made by the benchmark designers and has no theoretically optimal solution.
The Function Recovery dimension uses a binary threshold of 0.75, where a token similarity score of 0.75 or above counts as passing.
The reverse engineering benchmark task in LFORLA is to recover the C source code of a function from a stripped binary.
GLM 5.2 (submitted by the author themselves) scored 78.0 on this benchmark.
Token similarity does not equal semantic correctness; a submission may have high token overlap while still being structurally incorrect.
Model selection should compare along dimensions that match the use case rather than looking only at overall rankings; complete source code reconstruction requires looking at Completeness and Function Recovery, while token-level fidelity requires looking at Code Similarity.
2 more timeline events
All Facts (12)
JS 3D Builder (Three.js) is a benchmark whose task is, starting from written architectural specifications, to procedurally generate 3D building code using Three.js, scored by server-side static analysis, without executing the code or using an LLM judge.
50%UnverifiedThe author of the ranking benchmark submitted his own model GLM 5.2 to his own benchmark, which constitutes a conflict of interest.
50%UnverifiedThis benchmark uses deterministic token similarity comparison for scoring, with reference answers stored only on the server side and never shown to models.
50%UnverifiedThe Code Similarity metric in this benchmark consists of 60% bag-of-words cosine similarity plus 40% Longest Common Subsequence ratio.
50%UnverifiedThe 60/40 cosine and LCS mixing ratio is a subjective engineering choice made by the benchmark designers and has no theoretically optimal solution.
50%UnverifiedThe Function Recovery dimension uses a binary threshold of 0.75, where a token similarity score of 0.75 or above counts as passing.
50%UnverifiedThe reverse engineering benchmark task in LFORLA is to recover the C source code of a function from a stripped binary.
50%UnverifiedGLM 5.2 (submitted by the author themselves) scored 78.0 on this benchmark.
50%UnverifiedToken similarity does not equal semantic correctness; a submission may have high token overlap while still being structurally incorrect.
50%UnverifiedModel selection should compare along dimensions that match the use case rather than looking only at overall rankings; complete source code reconstruction requires looking at Completeness and Function Recovery, while token-level fidelity requires looking at Code Similarity.
50%UnverifiedNemotron 3 Ultra (via opencode) scored 43.49 on this benchmark.
50%UnverifiedIn this benchmark, every model receives exactly the same prompt and binary file, and models may choose between tool-assisted and tool-free modes.
50%