One to Tune, Two to Check: An Honest Tuning Experiment on Scribe's Handwritten Form Recognition

In the previous article, we saw the Scribe pipeline's recognition accuracy plummet on handwriting it had never seen, while the review gate held steady. This post tackles a more fundamental question: on a long form we can use for tuning, how far can tuning go? And how do we know these gains are real?

The author migrated Scribe to a 14-page questionnaire packet with 177 answers per packet, with pages full of checkbox grids, yes/no underlines, timed tasks, and tester-scored handwriting. The experimental design was disciplined: tune on one packet, hold out the other two as check packets.

Across 531 checked answers in three packets, the final configuration achieved:

• 96% of answers read perfectly, up from 93% (94.4% to 96.0% on the two check packets alone).
• 99.8% of auto-accepted answers were correct, up from 97.5%.
• Only 1 wrong answer slipped past review, down from 11 (9 of which were in the tuning packet).
• About 22% of answers required review, up from 18%: the second model added review burden.
• About 47 minutes per packet, down from about 70 minutes when the second model was added. Prefix caching later brought it to about 37 minutes, but let one more answer slip through.

None of these numbers count toward the targets set in Part 1, because tuning used one of the packets and the check packets also influenced some decisions along the way.

Experimental Setup: One Packet to Tune, Two to Validate

The page order is fixed: questionnaire, a tester-scored page, several timed-task pages, and numeric fields. The author built a schema for the whole packet: 177 answer fields, 89 checks between related answers, and a rule that a page's position in the full packet determines which page it is.

Three filled copies were at hand, never publicly released, so the models under test could not have seen these filled pages in training (the blank form itself may be public).

The answer key generation process: first two independent AIs each read every page, then an adjudication process merged them, and finally two people (one being the author) reviewed every value. The first review round changed 2 of 603 values; the two reviewers agreed 99.2% of the time (598/603), and the author decided the five disagreements. Human checking of pre-filled drafts may miss what the drafts themselves miss, so the answer key is "very good" rather than "perfect." Later field adjustments reduced the answer count to 531.

The rules were set only after the first experiment had already run all three packets:

• One packet for tuning (the hardest baseline: 86% perfect, the other packet had 94%)
• Two packets held out, kept out of the tuning loop, used only to confirm changes
• Every comparison done pairwise: each answer is right or wrong in both runs, and only answers that changed count

A fourth rule came later: a change is kept only if it both improves the tuning packet and holds on the check packets.

Fix the Scorer First, Then Trust the Gains

The first experiment compared prompts with and without "reading skills"—notes unrelated to the form explaining how handwriting works (underlines, strikethroughs, handwritten numbers, empty boxes). On the first scored packet, the version with skills looked 1.5 percentage points better.

It wasn't. The model used [] to mean "nothing checked," while the answer key used blank. They mean the same thing, but the scorer counted them as different, and that convention alone produced the entire "gain." After fair scoring, the version with skills was actually half a point behind on that packet, and across all three packets the two versions differed by no more than two answers.

The author now checks both scoring methods and takes the fair one; all numbers in this post use fair scoring. This reinforces the lesson from Part 2 from another direction: before trusting a gain, make sure your scorer didn't change.

The McNemar test is a statistical test for paired binary data, well suited to this experiment's design: the same answers are judged once before and once after tuning, and only answers that flip carry information. It puts the counts of "right to wrong" and "wrong to right" answers into a 2×2 contingency table and tests whether changes in the two directions are symmetric. Compared with directly comparing two accuracy percentages, the McNemar test's advantage is that it controls for differences in the difficulty of the questions themselves—the same question's performance in two runs is naturally correlated, and ignoring this paired structure overstates significance. The p = 0.00006 (15 fixed, 0 broken across three packets) and p = 0.03 (6 fixed, 0 broken on check packets alone) reported here both come from this test. Note that when a single fix flips multiple answers on the same page at once, those answers are not independent and the p-value is optimistic—one reason the author cautions that "the numbers should be read roughly."

Look at What the Model Actually Saw

Accuracy numbers only tell me which answers are wrong, not why. So the author built a debug view: for each wrong answer, it shows both the image crop the model read and the answer's actual position on the page.

It surfaced two bugs that no metric could explain:

• One page was never cropped. The step that locates each answer box required listing all boxes on the page in a single reply. On the tester-scored page, the reply hit a length limit, the truncated JSON failed to parse, and all 22 fields that should have been cropped silently fell back to the full page. Fix: request in batches of ten and keep all complete boxes from truncated replies.
• One grid read the wrong column. Each row of the checkbox grid is checked under one of three column headings. Cropping a single row removes the headings, the model miscounted columns, and it gave the wrong answer with full confidence. Fix: read these rows from the full page.

A third pattern—fabricating "yes" answers on blank rows of the timed-task pages—turned out to be the crop catching the previous row's circle. Same fix.

On the tuning packet alone, three rounds of changes raised perfect accuracy from 85.6% to 89.1% and accepted-answer accuracy from 90.6% to 98.6%. Not all of it came from better reading: an intermediate round also sent all ticks on the tester-scored page straight to review, removing the worst answers from the accepted pool, and added to reading skills and 26 yes/no prompts the underline rule the author noticed on these forms (the answer is the word the underline starts under).

The Change That Looked Fine but Wasn't

After the box-finding fix, the tester-scored page was finally cropped correctly. On the tuning packet, this change made no difference (64% both before and after). The author almost kept it.

But on the two check packets, accuracy on that page crashed from 94% to 86%. The crop showed only part of a row of ticks, and the old bug—by never cropping—had hidden the problem.

The author rolled back the change, and this is when "must hold on the check packets" went from a final formality to a hard rule. It also means the check packets have now influenced a decision; more on that later.

Only Read the Fields You Actually Use

The tester-scored page has one tick per item and one handwritten score per section. The ticks were the problem: 15 of 66 wrong at the start, and once all ticks went to review, that added 22 review items per packet (about 11% of that page's answers). The handwritten scores were wrong only once across three packets, and they are the values actually used downstream.

A synthetic page drawn for this post, with fictional items. Orange boxes mark per-item ticks (including one crossing the border, one struck through, one very faint); green boxes mark handwritten section scores and the total.

So the schema now reads only the handwritten scores and discards the tick fields. This changed the task itself rather than improving reading: the page's worst fields left scoring, review burden dropped by 22 items per packet, and each packet used 24 fewer model calls. To keep comparisons honest, the summary and all comparisons since are scored on the same reduced 531-answer set; earlier numbers used the original 603.

With all fixes so far and no second model yet, three-packet perfect accuracy rose from 93.2% to 95.3% (12 fixed, 1 broken), and wrong answers slipping past review fell from 11 to 4. On check packets alone, the gain was smaller and not significant: 94.4% to 95.5% (5 fixed, 1 broken).

Revisiting the Second Model

Part 1's best idea was to have a different model re-read every answer. Re-added here, it let one fewer wrong answer through (4 to 3), but review burden jumped from 17% to 27%.

Looking at its disagreements, 92% were false alarms (55 of 60), 42 of which were answers the main model read from a crop while the second model re-read from the full page. On dense pages, the smaller model often can't find the field.

The author tried one fix at a time:

• Re-check blank answers too. Abandoned. Tuning-packet review burden went from 27% to 36%, and every new flag was a false alarm: the smaller model "saw" answers on blank rows.
• Re-read the same image the main model read. Kept. Tuning-packet false alarms dropped from 16 to 10, with no answers changed.

So a second look helps when it's a fair re-look at the same thing; it's far less useful for confirming something is empty. Whether it's worth keeping on this form remains open: compared with the tuned pipeline without it, the final setup lets 1 wrong answer through instead of 4, at a review burden of 22% instead of 17%. A three-answer difference is more than nothing but not enough to prove.

Faster, but Without Making Answers Worse

70 minutes per packet was too slow. The author's condition for speed work: no change may make answers worse, checked answer-by-answer via paired runs.

More than half the time went to reading prompts rather than writing replies: each full-page read sends about 5,100 tokens, of which about 3,650 are the page image.

Change	Time saved	Answers
Read 4 answers at once per page; skip page-type checks when page order is fixed	18% (tuning packet)	176 of 177 identical
Also cap the short note after each answer at 60 characters	24% total (tuning packet)	2 changed, both better
Reuse the processed page image across all full-page questions on that page (prefix caching)	another 22% per packet	20 wrong instead of 21, but 2 slipped past review instead of 1 (second-model run-to-run noise)

Packet processing time dropped from about 70 minutes to about 47, then to about 37.

Tried and abandoned:

• 8 answers at once instead of 4: 1% faster. The server was already saturated.
• Drop reading skills: 6-11% faster, but one more wrong answer slipped past review on the check packets. One answer is within noise, but the reason is specific: one yes/no question's own prompt didn't mention the underline rule and only got it from the skills. The author keeps the skills as a safety consideration, not a measured gain, and fixing that prompt would now be a fix discovered on the check packets.
• Speculative decoding (a small "drafter" proposes tokens, the main model checks them in one step): ran at 20-23 tokens/sec vs 15 on synthetic test images in single requests, but was 6.6% slower in the pipeline. In the serving library, an in-flight speculative batch waits for its longest reply to finish before accepting new requests, and after loading the drafter, the library refused the structured JSON output Scribe relies on. Along the way the author found a batching bug in the library and reported it upstream.

Prefix caching refers to reusing the computation of an identical prefix across multiple requests. In this pipeline, several full-page-level questions are asked about the same page, and each request's prompt begins with the same processed page image (about 3,650 tokens). Without caching, the model must redo the visual encoding of the image for each request; with prefix caching enabled, the server keeps the key-value cache for this prefix and reuses it for subsequent requests, computing only the new text portion. That's why it saves another ~22% of time while leaving answers nearly unchanged.

Speculative decoding is another acceleration idea: a small, fast "drafter" model guesses several tokens in a row, then the main model verifies all guesses in parallel at once, accepting correct parts and rejecting wrong ones. The theoretical gain comes from turning the main model's multi-step serial generation into a single parallel check. But it's sensitive to batching—when the server handles multiple requests at once, a speculative batch often has to wait for its longest reply to finish before accepting new requests, so throughput drops instead—exactly the -6.6% measured here. It also has compatibility issues with structured JSON output constraints, which is the direct reason it's unusable in this pipeline.

The Two Escapees

Part 2 closed by auditing every answer that skipped review; here are the two under the current setup.

• A free-text answer with one extra letter. The model added a plural "s" to one word of a two-word answer. Same meaning, but exact scoring counts it wrong, and the second model made the same misread, so nothing was flagged.
• A yes/no underline spanning both words. The line starts under "Yes" and continues under "No." By the rule in reading skills, where the line starts decides the answer, so it's Yes. The model read No with high confidence based on where the line ends. Without caching, the second model disagreed and the answer went to review; with caching, it agreed, so it slipped through.

A synthetic redraw of the second case: fictional question and generated handwriting, drawn for this post.

So one surface error and one real flip, both on handwriting where a human would also hesitate. The fix the author will test next (on new packets, not these): send any underline spanning two options straight to review.

What This Experiment Can't Prove

• Only three packets. 531 answers is enough to see large effects, not small ones. Under batching, the second model doesn't give exactly the same answers run to run, so any one- or two-answer difference here should be treated as noise.
• The check packets aren't clean. The schema and answer key were built using all three packets together, the first experiment scored all three, and three later decisions used check-packet results: rolling back the crop, reading only handwritten scores (whose underlying tick-error counts covered all three packets), and keeping reading skills.
• Too few writers. Three packets can't show performance across many writers and testers.
• This is not a held-out evaluation. The targets set in Part 1 (at least 98% accuracy on accepted answers, at most 2% fabricated answers slipping past review) can only be measured on a private held-out set. These packets don't qualify, and the untuned baseline was already above 98% on the check packets, which says more about the form than about tuning.

The difference between a tuning set and a held-out set is key to understanding the boundaries of this post's conclusions. The tuning set is data where developers can see results and modify the system accordingly; the held-out set is completely invisible throughout development and used only once at final evaluation. The gap between them is "optimism bias": any choice made on the tuning set—even just "this change looks neutral, let's keep it"—systematically makes numbers on the tuning set better than reality. This post's one-to-tune, two-to-check design is already far stricter than tuning on all data, but the check packets still influenced three decisions (rolling back the crop, reading only handwritten scores, keeping reading skills), so the check packets are no longer a clean held-out set. The author repeatedly stresses "these packets don't qualify" precisely because the 98% target from Part 1 must be measured on a private held-out set that never participated in any decision; otherwise you're only measuring local performance for this form plus these few writers.

Key Takeaways

• Tuning/check separation is worth doing, even with leakage. That harmful change looked neutral on the tuning packet.
• Before trusting numbers, check the scorer and look at what the model actually saw. The first "gain" was a scoring convention; the two worst bugs were silent until the author saw the crops.
• Only read the fields you actually use, and score before and after on the same set.
• A second model should get a fair second look, not a harder question.
• Speed work must not make answers worse, and acceleration only counts under the batching you actually run.

Next Steps

As planned in Part 2: private held-out evaluation—new packets the schema has never seen, filled by multiple people. The checkbox-detection stage from Part 1 is still on the list.

---

Technical details: Same machine, main reader Qwen3.8-27B (8-bit), second model Qwen3-VL-8B (8-bit), served locally with mlx-vlm 0.6.15. Three packets, 531 answers (one tuned, two checked), fair scoring ([] = blank). Full data table shows: baseline perfect accuracy 93.2%, accepted accuracy 97.5%, 11 slipped past review, 18.1% flag rate; final configuration (with prefix caching) perfect accuracy 96.2%, accepted 99.5%, 2 slipped through, ~22% flag rate, ~37 minutes per packet. Paired McNemar test shows the tuned bold row fixed 15 and broke 0 relative to baseline, p = 0.00006 (check packets alone 6/0, p = 0.03). The author cautions these numbers should be read roughly, since they include the tuning packet and one fix often flips multiple answers on the same page.