Fiorillo v0.5: How a 4-Billion-Parameter Open-Source Model Outputs Calibrated Probabilities for RCT Questions

A Small Model Focused Entirely on Answering Randomized Trial Questions

At a time when large models keep growing larger, a study published on arXiv takes the opposite approach: using a 4-billion-parameter open-source model called Fiorillo v0.5 to tackle the narrow but demanding task of interpreting evidence from medical randomized controlled trials (RCTs) — and subjecting it to rigorous pre-registered evaluation before release.

Fiorillo v0.5's core capability is not generating plausible-sounding text. Instead, it outputs a probability for each candidate answer. Its primary expert module reads an RCT article (truncated to 6,144 tokens) and answers a three-way classification question on the Evidence Inference 2.0 (EI) benchmark: relative to the control group, does a given intervention significantly increase, significantly decrease, or not significantly change a particular outcome measure?

This is a highly structured task with extremely high demands on factual accuracy — precisely the kind of scenario where general-purpose large models tend to "confidently say the wrong thing." Fiorillo's philosophy is: rather than pursuing versatility, get probability calibration right within one clearly defined scope.

Randomized controlled trials (RCTs) are the cornerstone of evidence-based medicine. In a typical RCT, participants are randomly assigned to an intervention group (receiving the new treatment) and a control group (receiving a placebo or standard care), and the difference in specific outcome measures between the two groups is used to assess the intervention's effect. Evidence Inference 2.0 (EI) is a benchmark dataset specifically designed for understanding RCT literature. It requires a model to extract evidence from a complete scientific paper and determine the direction of an intervention's effect on an outcome — with three categories corresponding to "significant increase," "significant decrease," and "no significant difference." The challenge lies not only in understanding statistical significance, but in correctly identifying the directional relationship between the intervention and control groups; any directional confusion leads directly to erroneous medical inferences.

Technical Approach: Qwen3-4B with Low-Rank Adapters and a Decision Head

Architecturally, Fiorillo v0.5 is not trained from scratch. It uses Qwen3-4B-Base as its foundation, augmented with low-rank adapters (LoRA) and a dedicated decision head, fine-tuned specifically for the EI task. This approach is cost-effective and reproducible by small teams.

One noteworthy detail is how the training data was handled. The EI training set contains 2,657 articles, but the researchers used only 1,431 with clearly permissive licenses. In other words, they voluntarily discarded more than 40% of available data in exchange for a copyright-clean release.

This choice comes at a cost. Training with only the license-compliant articles using the same recipe results in an accuracy drop of 0.0123 (95% interval: 0.0034 to 0.0207, descriptive result). This quantifies the trade-off between compliance and performance — roughly one percentage point of accuracy, in exchange for a model that can be released cleanly under Apache License 2.0.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method. It works by inserting the product of two low-rank matrices alongside the weight matrices of a pre-trained model, updating only these small matrices during training while freezing the vast majority of the original model's parameters. The result is that trainable parameters typically account for just 0.1% to 1% of the full model, significantly reducing compute and memory requirements while preserving the language understanding capabilities built up during pre-training. In Fiorillo's design, the LoRA module specializes Qwen3-4B's general language capabilities toward RCT evidence understanding, while the dedicated decision head maps the model's internal representations to a probability distribution over three categories — directly outputting calibrated confidence scores rather than merely producing a label for the highest-probability class.

Four Pre-Registered Criteria: Setting the Pass Bar Before Looking at Results

Methodologically, the most significant aspect of Fiorillo v0.5 is that its release decision is tied to pre-registered criteria. The researchers registered four standards on the Open Science Framework (OSF) in advance, using performance on the EI test set (whose labels are public) as a second gate. All four had to pass before the model would be released.

On a test set of 333 articles and 1,218 prompts, the results were:

• Expected Calibration Error (ECE): 0.0168, well below the pre-specified upper limit of 0.05. This means the model's outputted probabilities closely match actual correctness rates — no inflated confidence.
• Log loss: 0.8603 lower than the prior baseline (95% interval: 0.8104 to 0.9078), and 0.1829 lower than a Gemma model reading the same inputs (0.1164 to 0.2598).
• Macro-F1: 0.9248, compared to the baseline's 0.8668.

All four criteria passed. This "set the rules first, observe results second, let results determine release" process is relatively uncommon in machine learning. It imports the pre-registration tradition from scientific experimentation into model evaluation, effectively reducing the scope for cherry-picking favorable metrics after the fact.

Pre-registration originated in clinical medicine and psychology. Researchers publicly register their hypotheses, methods, and analysis plans on an independent platform (such as OSF, the Open Science Framework) before collecting or analyzing data, thereby preventing post-hoc adjustment of statistical hypotheses or selective reporting of favorable results — practices known academically as "p-hacking" or "outcome-driven analysis." Expected Calibration Error (ECE) is the core metric for assessing the reliability of a model's probability outputs: it calculates the average deviation between the model's predicted confidence and its actual accuracy rate. An ECE of 0 means that among predictions where the model says "80% confident," exactly 80% are correct. Higher ECE means the model's confidence is less trustworthy — and in medical decision-support contexts, high ECE can cause users to place misplaced trust in incorrect answers.

Ablation Studies Reveal What the Model Is Actually Reading

Several ablation experiments in the paper expose the information sources the model relies on — and serve as important cautions for researchers.

When no article is provided at all, macro-F1 drops sharply to 0.4384, confirming that the model genuinely relies on article content rather than guessing. But providing just the title boosts F1 by 0.0939 (0.0655 to 0.1234) — the researchers note this may be because some titles explicitly state the trial's results, or because they activate the model's memory of that specific trial. This is a reminder that benchmark scores may incorporate "answer leakage from titles."

More interesting is the directionality test: when the intervention and control groups are swapped, 66.52% of directional answers flip. This indicates the model genuinely understands the causal direction of "who relative to whom," rather than mechanically matching keywords — a necessary capability for medical evidence interpretation.

Reproducibility and Open Licensing

The researchers emphasize that when running the released version in practice, the output files match the predictions used in prior evaluations within pre-specified tolerances. In other words, the numbers reported in the paper are reproducible by users themselves.

The model is released under Apache License 2.0 (DOI: 10.57967/hf/10722), a permissive, commercially friendly license. Combined with the earlier decision to train only on license-compliant data, the entire release pipeline is relatively complete in terms of legal compliance, reducing downstream copyright risks for users.

Why This Work Deserves Attention

Fiorillo v0.5's significance does not lie in topping a leaderboard. Rather, it demonstrates a more responsible paradigm for releasing small models: clear task boundaries, calibrated probability outputs, pre-registered evaluation criteria, compliant training data, and a permissive open-source license. For high-reliability applications like evidence-based medicine and systematic review automation, a 4-billion-parameter model that honestly expresses uncertainty is often more practical than a hundred-billion-parameter general-purpose model that is chronically overconfident.

Of course, the ablation studies honestly expose limitations too: title leakage, dataset bias, and other issues remain, suggesting there is still room to improve the design of such benchmarks. But publicly quantifying these problems is itself a mark of scientific integrity.