AI Optimization Made llama2.c Faster but Broke Streaming: Lessons from a Kaggle Benchmark

Can AI coding agents genuinely speed up a CPU inference program without breaking its existing behavior? That's the core question posed in a Kaggle benchmark challenge. The author had Gemini, Claude, and GPT read, modify, and test Andrej Karpathy's llama2.c source code under identical tool-call and edit constraints — and the results exposed a commonly overlooked trap: a faster final answer can hide a much worse streaming experience.

An Intriguing Early Result

The original program began displaying generated content after 103 ms and finished all output after 1,321 ms. One AI-optimized version compressed total runtime to 483 ms — but the user saw nothing until that point.

In other words, the final answer was identical, the overall response was roughly 2.7× faster, yet users waited nearly 5× longer to see the first character. This is exactly the key contradiction the author wanted to highlight: if you use only "completion time" as your optimization metric, you'll miss "time to first byte" — which is critical to real user experience.

The author then restored the per-token fflush(stdout) call without touching the optimized math logic. Output began appearing after just 37 ms, and the full request completed in 477 ms. The useful computational optimization was preserved, and the streaming regression was fixed — proving the two goals were never mutually exclusive.

Time to First Byte (TTFB) is the core metric for streaming output experience: the time from when a user sends a request to when they see the first visible character. In LLM inference, this is often called Time to First Token (TTFT). For interactive conversational systems, TTFT typically has a far greater impact on perceived responsiveness than total completion time. Humans noticeably perceive blank waits exceeding 300 ms as lag, whereas even if total generation time is longer, the experience feels smooth as long as characters keep flowing. fflush(stdout) is a C standard library call that forces the output buffer to be written immediately to the underlying file descriptor — calling it after each token ensures characters are progressively presented to the user or downstream pipeline. Removing this call causes the OS to accumulate output in a buffer and flush it all at once, resulting in a "wait → everything appears suddenly" experience from the user's perspective.

What the Benchmark Actually Measures

The author used llama2.c locked to a specific commit, paired with TinyStories FP32 checkpoints (15M, 42M, 110M parameters). These are small models running on llama2.c — not full Llama 2. The rationale was practical: the compact C source and small checkpoints make full-program measurement on CPU feasible.

Optimizers could only read run.c, replace uniquely matching code blocks, and benchmark candidate versions. Each attempt was limited to 8 tool calls, at most 6 successful edits, and 2 benchmark feedback calls, with a 600-second guard. The prompt enforced FP32 precision and compatibility with the existing interface, and prohibited modifying weights, tokenizer, stopping rules, threading, or external libraries — as well as hardcoding answers or manipulating the evaluator. Both programs ran with GCC -O3 on serial CPU.

Correctness and Latency Are Two Separate Checks

For numerical checks, both programs received the same input tokens at each position. The author compared full-vocabulary logits, requiring abs(candidate - reference) <= 1e-4 + 1e-4 * abs(reference), and generated text had to be identical (including nucleus sampling with a fixed seed).

For latency, the request timer started after loading weights and tokenizer, covering tokenization, prompt processing, model computation, sampling, detokenization, and output. The parent process observed stdout, excluded the exact ASCII prompt echo, then located the first generated byte — this is "time to first generated output byte," including pipe transfer and scheduling overhead.

The revised task added a streaming latency gate: each candidate's median time to first byte must fall within max(1.10 × reference median, reference median + 0.002s). Only when numerical, text, shape, and streaming checks all passed did a completion speedup count toward the score. The final score is the geometric mean of speedup ratios across 12 "size/workload" combinations; behavioral failures score zero.

llama2.c is Andrej Karpathy's minimalist C inference implementation, designed to run the Llama 2 architecture in as few lines as possible, with core logic concentrated in a single run.c file — easy to read, modify, and use for teaching. TinyStories is a synthetic dataset designed for training small language models, consisting of short children's stories. Its FP32 checkpoints (15M, 42M, 110M parameters) are small and fast to load, making them ideal for end-to-end performance measurement on CPU without needing GPU or quantization. Choosing FP32 over quantized formats preserves deterministic numerical behavior, making bit-level comparisons in correctness verification more reliable. This combination keeps the benchmark focused on the core question — "can an AI agent optimize real C code?" — without interference from hardware or format differences.

The numerical tolerance formula abs(candidate - reference) <= 1e-4 + 1e-4 * abs(reference) is a mixed absolute-relative tolerance check, common in numerical testing frameworks like NumPy's allclose. Pure absolute error is too permissive for large values; pure relative error is too strict for values near zero. Combining both provides balanced precision guarantees across the full numerical range. The streaming latency gate max(1.10 × reference median, reference median + 0.002s) follows the same dual-condition logic: for fast requests (where the reference median is small), the additive term prevents the gate from narrowing into millisecond-level noise; for slow requests, the multiplicative term scales the allowed deviation proportionally, avoiding false failures from normal jitter.

Which Models Were Tested

The revised experiments used Gemini 3.7 Flash, Claude Sonnet 5.5, and GPT-5.4, with each model running three independent optimization attempts. Each attempt started from the locked original version with a fresh conversation, and each saved its own candidate code, usage, and timing records.

Model calls used SDK defaults; the prompt signature used temperature 0 and seed 0. The author emphasizes these were multiple independent executions under fixed settings — not random seed trials — and internal reasoning budgets across providers were not aligned. GPT-6.1 Sol was tried in an early pilot but failed at the integration interface stage; the author excluded it from model performance conclusions rather than treating it as a bad code change.

Key Findings

Identical Final Output Hides Streaming Regression

Claude's original candidate removed per-token fflush(stdout), flushing only after the final newline, while also improving matrix multiplication. Since the captured text still matched and the first-version scorer only rewarded completion time, this version retained a high score — the task didn't require preserving streaming latency at the time.

The author replayed the saved candidate code in a separate Kaggle Linux task (no model calls involved). Frozen candidate replay results:

Frozen Candidate	Completion Speedup	Time-to-First-Byte Speedup
Gemini	3.44×	3.54×
Claude	2.73×	0.44×
GPT-5.4	2.16×	2.19×
Claude (flush restored)	2.75×	2.80×

Claude's TTFB speedup of 0.44× means its streaming output actually got slower. Restoring the flush brought TTFB speedup back to 2.80× while completion speed remained nearly unchanged. The short greedy request details are even more telling: original 103 ms first byte / 1,321 ms completion; Claude buffered 483 ms first byte / 483 ms completion; flush restored 37 ms first byte / 477 ms completion.

Restoring the flush moved output earlier by roughly 446 ms while completion time stayed around 0.48 s. Claude's buffered version failed the proposed first-byte gate in three of four workloads; the flush-restored version passed all four. Aggregate metrics that look only at completion time completely mask these regressions.

Passing Checks Doesn't Mean Delivering Gains

Completion speedups across the nine fresh revised-task runs:

Optimizer	Passed Automated Checks	Completion Speedups	Median
Gemini 3.7 Flash	3/3	5.38×, 5.35×, 5.27×	5.35×
Claude Sonnet 5.5	3/3	3.69×, 3.87×, 3.17×	3.69×
GPT-5.4	3/3	1.00×, 2.32×, 1.00×	1.00×

Gemini and Claude consistently improved completion time across all runs and met the streaming gate. GPT's best candidate reached 2.32×, but two of its three candidates were essentially neutral (1.00×). Reporting only the 2.32× result would obscure this entirely.

Source inspection provided concrete examples: GPT's first candidate used all six edits, precomputed rotary positional encoding math, and removed OpenMP pragmas — but left the scalar matrix multiplication loop largely unchanged, and since the build had no OpenMP flags, those pragmas were already dead code. The measured speedup was only 1.0013×. A separate attempt with two focused edits produced 2.32×. A high edit count and the use of some optimization technique are not evidence of end-to-end gain.

Finding the Gain Also Costs Time and Tokens

The author also reviewed usage from nine SDK trace artifacts. Per-optimization-attempt medians:

Optimizer	Input Tokens	Output Tokens	Optimizer Wall-Clock Time	Reported Quota Cost
Gemini 3.7 Flash	318,985	50,733	435.1 s	$0.2864
Claude Sonnet 5.5	38,431	2,454	48.4 s	$0.0853
GPT-5.4	87,784	1,736	71.0 s	$0.0665

Gemini achieved the largest median gain but consumed the most tokens and time; Claude found a smaller gain at lower overhead; GPT had the lowest cost but the most variable completion gains. The author deliberately kept these dimensions separate rather than collapsing them into an arbitrary "efficiency score."

The Evaluator Itself Needs Validation

An early pilot accidentally compiled the original source on both sides — because #include "run.c" picked up the file next to the adapter before the candidate directory. The author discarded that timing data, changed the reference to #include <run.c>, and added control tests for shadowing cases and negative controls that rejected zero-logit implementations. These checks must come before interpreting tempting speedup numbers.

#include shadowing is a classic path resolution trap in C/C++ build systems: the compiler searches for header files in order of include path priority, and if a file with the same name exists in the current directory or an earlier path, it gets selected over the intended target. In this case, the quoted form #include "run.c" first searches the source file's own directory, causing the original run.c next to the adapter to be selected — making both the candidate and reference sides run the same code, rendering the timing comparison completely invalid. Switching to the angle-bracket form #include <run.c> with explicit path control restricts the search to the intended location. This class of issue demonstrates that the credibility of a performance benchmark depends not just on the quality of the code being tested, but on the build and evaluation pipeline itself — which equally requires negative control experiments to verify its own correctness.

Lessons for Real-World Optimization Work

The author plans to extend the work with wider request variation, more execution runs, stronger isolation, and checks across more hosts and shapes. He also acknowledges that the current reference program is intentionally simplified C code, not a production inference engine.

But the core lesson is concrete and clear: preserve the observable behaviors users actually need first, then measure whether a code change genuinely helps. Matching the final text is useful, and the flush intervention revealed a missing "contract" — while repeated attempts showed that even passing this contract doesn't guarantee a performance gain. That's a reminder worth keeping in mind for any team looking to use AI agents for performance optimization.