Qwen3.5-4B

In the same reproduction experiment, ResNet-20 achieved a test accuracy of 90.4% and ResNet-56 achieved a test accuracy of 91.7%, with training accuracies of 97.4% and 99.0% respectively.

Qwen3.5-4B is a small model with only 4 billion parameters, released by Alibaba's Tongyi Qianwen team.

Direct use of Kev achieves 64.00% accuracy, and Qwen3.5-4B outputting only the final answer achieves 68.80% accuracy.

Expected calibration error (ECE) dropped from 0.187 before training to 0.036 after training.

Training Qwen3.5-4B (4-bit QLoRA) on a 24GB VRAM machine, peak VRAM was 7.95GB and training took 910.6 seconds.

In an A/B test on the same GPU, the traditional generative prompt had an average inference latency of 3,679.8ms, while the decision model (Decision Head) had 191.4ms.

B1 made 181 predictions with confidence no lower than 0.9, of which 1 was wrong; B2 made 188, of which 2 were wrong; a better average does not automatically mean fewer high-confidence errors, and 0.9 is not a ready-made production threshold.

On the public JevBench benchmark (231 downloadable questions), the unmodified 4B version scored 170/231 overall, with 48/48 correct in the Easy group, 69/72 in the Original group, and only 53/111 in the Hard group, with 58 of the 61 errors coming from the Hard group.

On 194 Yelp text judgments, B1 (0.8B) got 184 correct and B2 (4B) got 192 correct; the average NLLs were 0.09620 and 0.04024, respectively.

On 1,184 historical multi-task questions, B1 got 951 correct and B2 got 1,033 correct, with NLLs of 0.60133 and 0.46749, respectively.

Broken down by task family in JevBench, the accuracy was 0/15 for temporal and numerical questions and 4/19 for long-horizon strategic judgment.

Dory leads Qwen3.5-4B by more than 20 points on IFBench (79 vs 53.83, 4 loops).

Dory's score on the AA-Omniscience benchmark barely changes from 1 loop to 4 loops (10.03, 10.08, 10.03), and it trails Qwen3.5-4B on both GPQA Diamond and HLE.

On the held-out compound recipe test set, the LoRA fine-tuned model caught 66%, while the base Qwen3.5-4B model caught 78%.

The fine-tuned model learned to refuse (saying "not sure"), whereas the base model never said "not sure" in 85 opportunities.

The author used Tinker's LoRA training API to fine-tune Qwen3.5-4B on 5,755 Open Food Facts entries; the training set contains 5,000 ordinary products, 550 refusal samples, and 205 control samples.

The author claims that the off-the-shelf model they started with, faced with 85 hidden allergen samples, never output "uncertain".

On the frozen test set, the delivered system (LoRA + rules) caught 73 (recall 0.86), with 11 confident errors, 1 "not sure", and 0 false positives, outperforming the keyword baseline, LoRA-only fine-tuning, and zero-shot Qwen3.5-4B.

Zero-shot Qwen3.5-4B achieves 0.98 recall on the ordinary held-out ingredient list test set.

After training for most of a day using only about 1,000 edge tasks that are 'sometimes done right and sometimes done wrong,' the 4B model recovered more than half the gap to the 27B model.