Shivacon-Energy: How a 0.5B Energy Model Achieves Zero Hallucination

In an era where small language models (SLMs) are being stuffed into agent pipelines en masse, there's an awkward reality: they often confidently fabricate facts and make a mess of the JSON formats that tool calls depend on. Most fine-tuned versions make this problem worse—models memorize narrow patterns while fluently inventing facts. The Shivacon-Energy project on GitHub offers a different approach: a 0.5B-parameter energy-domain LoRA adapter whose selling point isn't "knowing more about energy" but rather "preferring to say I don't know over fabricating numbers, and always outputting valid JSON."

A Counterintuitive Fine-Tuning Goal: Calibration Over Knowledge

Shivacon-Energy is a LoRA fine-tune based on Qwen2.5-0.5B-Instruct, with rank 16, alpha 32, and approximately 8.8 million trainable parameters—just 1.75% of the model. The training data consists of only 1,596 rows, covering refusal samples, anti-hallucination examples, calibration pairs, and tool traces. The entire training was completed on a free Google Colab T4, with 300 steps, 3 epochs, learning rate 2e-4, training loss 0.4018, and validation loss 0.08377.

The project's core insight comes from a structural judgment: a 0.5B model cannot possibly contain deep energy expertise, and a 34MB adapter won't change that. But what a small model can do in a pipeline is become a reliable component—answering within its capability range, refusing when out of scope, and outputting machine-readable formats. The author states bluntly in the README: the adapter's value lies in calibration, not knowledge.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique whose core idea is to inject a pair of low-rank matrices (A and B) alongside the weight matrices of a pre-trained model. During training, only these two small matrices are updated while the original weights are frozen. Rank and alpha are two key LoRA hyperparameters: rank determines the dimensionality of the low-rank matrices, which in turn controls the number of trainable parameters and the adapter's expressive capacity—higher rank means more trainable parameters and stronger fitting ability, but also greater risk of overfitting; alpha is a scaling factor that adjusts the relative influence of the LoRA update on the original weights, typically kept in a fixed ratio to rank (e.g., alpha being twice the rank) to stabilize training. In this case, rank 16 and alpha 32 mean the newly added trainable parameters are only about 8.8 million, accounting for 1.75% of the 0.5B model. This makes fine-tuning extremely cheap, but also fundamentally limits how much new knowledge the model can learn.

Four "Sellability" Gates: Only 1 of 9 Adapters Survived

What truly distinguishes this repository from ordinary "I fine-tuned a model" posts is its design of a four-gate sellability harness. Nine adapters across different domains—energy, education, health, sales, marketing, research, core, general, and more—all ran the same test suite. Evaluations ran on real CPU inference with greedy decoding and per-prompt timeouts. Results:

Gate	Score	Meaning
G1 Base model comparison (6 prompts)	0W/0L/6T	Tied with base model, zero regression
G2 Domain tasks	9/10	One practical advice answer came out garbled
G3 Unknown honesty	5/5	Admits not knowing solar data, points to sources
G3 Refusal capability	5/5	Refuses to predict oil prices or fabricate facts
G4 JSON/tool format	5/5	Clean, correctly formatted JSON every time
Hallucination rate	0/10	—

Eight of the nine adapters failed to pass the gates, with failure reasons including data contamination, excessive refusals, and even one adapter denying the existence of a real court case. Only the energy-domain adapter was ultimately released. You may not have noticed, but the G1 result of 0W/0L/6T itself reveals something: the energy adapter doesn't actually know more energy facts than the base model. Its advantage is purely behavioral—honest calibration and JSON reliability.

Greedy decoding is the most basic decoding strategy for autoregressive language models when generating text: at each generation step, the model always selects the token with the highest probability as output, rather than sampling randomly from the probability distribution (like temperature sampling or top-k sampling). Its advantage is that output is fully deterministic and reproducible—the same input yields identical results across multiple runs, making it suitable for evaluation and scenarios requiring stable formatting. Its drawback is lack of diversity, tending to produce repetitive or conservative text. All evaluations in this repository use greedy decoding with fixed random seeds, ensuring fair and repeatable comparisons across adapters. Additionally, per-prompt timeout sets an independent generation time limit for each input, preventing the model from getting stuck in infinite loops or extremely slow inference—especially important in compute-constrained environments like CPU, ensuring the evaluation harness doesn't hang due to any single prompt.

Public Failure Report: 8 Failures More Valuable Than 1 Success

The repository's EVAL_REPORT.md documents the v1 version's dismal performance: 0W/0L/6T against baseline, 4/4 hallucination failures (fabricating countries, fabricating laws, fabricating drugs, and answering with forged tool calls), physics errors (10 kW ≠ 10,000 kWh per hour), Chinese token leakage, and fabricated file paths.

The v2 dataset was rebuilt precisely around these failure modes—refusal rows, anti-hallucination pairs, calibration examples ("I don't know" as the correct answer), and tool traces. Reports from later the same day showed the same base model, same LoRA rank, same 300 steps—the only variable being data composition. v2 achieved 9/10 domain tasks, 5/5 honesty, 5/5 refusal capability, and 0/10 hallucination rate on the same evaluation harness.

This v1-to-v2 comparison is the most instructive part. For small-scale fine-tuning, dataset coverage of failure modes matters more than data volume itself. The author also publicly released reports from all 8 failed adapters, which precisely map which training data errors lead to which pathological behaviors in small LoRAs.

Data contamination is a serious pitfall in language model evaluation, referring to when a model's training data accidentally contains evaluation benchmark questions, answers, or highly similar samples, causing inflated evaluation performance that doesn't reflect true generalization capability. In small-scale fine-tuning, contamination can arise from multiple sources: training data scraped from public web containing evaluation sets, manually constructed calibration samples highly overlapping with evaluation prompts, or training data leaking between different adapters. Once contamination occurs, evaluation scores can't reflect true ability, and the model will quickly expose its flaws in real deployment. Among this repository's 8 failed adapters, some were disqualified precisely because of overlap or pattern leakage between training data and evaluation prompts. This highlights the necessity of strictly partitioning training and evaluation sets in the fine-tuning pipeline, and manually reviewing data sources.

Technical Details: How to Reproduce and Use

The repository structure is clean, containing LoRA weight files (35MB), complete adapter packaging (34MB), adapter_config.json, tokenizer files, chat_template.jinja, and complete model card and evaluation reports. Usage is also very straightforward:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = PeftModel.from_pretrained(base, "./shivacon-energy")

The evaluation methodology is also worth noting: all gates ran on real CPU inference, not API sampling. Greedy decoding was used with max_new_tokens=100, Qwen chat template, and an evaluation harness with hang protection and per-prompt hard timeouts. Every failure in energy-eval.md was manually adjudicated, and transcripts are published in the repository so anyone can dispute the scoring. For claims like "0/10 hallucination rate," keyword scoring plus manual review of every failure case is the correct standard—and publishing transcripts is what makes this claim verifiable.

PEFT (Parameter-Efficient Fine-Tuning) is a parameter-efficient fine-tuning library provided by Hugging Face that uniformly wraps methods like LoRA, Prefix Tuning, Prompt Tuning, and IA3, allowing users to customize models by simply loading a base model and attaching a small set of adapter weights. In the code example, PeftModel.from_pretrained(base, "./shivacon-energy") dynamically injects locally saved LoRA adapter weights into the base model. During inference, the adapter and base model compute together, but the base model weights remain unchanged. This design brings several practical advantages: adapter files are extremely small (only about 34MB in this case), making distribution and version management easy; the same base model can mount multiple domain-specific adapters, switchable on demand; deployment requires keeping only one copy of the base model, with memory and storage overhead far below full fine-tuning. For edge devices or mobile platforms, this "base model + lightweight adapter" combination is currently one of the most viable deployment paths.

Honest Limitations and Roadmap

The most honest part of the model card is the limitations section:

• No measurable knowledge improvement over the base model (already proven by G1's 0W/0L/6T)
• Energy domain only—out-of-distribution questions remain out-of-distribution
• 0.5B parameters, cannot replace engineering analysis or professional energy audits
• One known garbled response to a practical home energy-saving suggestion
• Strict "JSON-only" wording occasionally leaks a Thought: line instead of pure JSON (observed once; gate prompts still scored 5/5)

The roadmap shows v3 is being retrained for the 8 failed domains, with the next round of energy-domain data targeting weak points—more practical advice rows and deeper domain coverage, aiming for future versions to achieve knowledge improvement, not just calibration.

Why Small Models Need Calibration, Not Knowledge

In agent systems, the orchestrator holds knowledge (retrieval, tools, larger models), while small models handle high-volume, low-latency, well-scoped turns. A 0.5B adapter with a 40% hallucination rate is a liability in this position. An adapter with 0% hallucination rate and 100% correct JSON formatting is infrastructure.

Shivacon-Energy's evaluation harness—domain tasks, unknown honesty, refusal capability, JSON format, base model comparison—is essentially an interview designed for this infrastructure role, and the energy domain is the only candidate that passed the interview. The project is publicly available on GitHub, and the author's advice: read EVAL_REPORT.md first, then MODEL_CARD.md. For anyone building agent pipelines who needs reliable small model components, this repository offers not just a usable model, but a replicable failure-fixing recipe.

One-line Summary

A 0.5B energy LoRA adapter that sells on calibration and JSON reliability—the only one of 9 adapters to pass evaluation.

Paragraph Summary

Shivacon-Energy is a LoRA adapter based on Qwen2.5-0.5B-Instruct, trained with just 1,596 rows of data over 300 steps on a free Colab T4. Its core goal isn't teaching the model more energy knowledge, but making it answer honestly within its capability range, refuse when out of scope, and output valid JSON 100% of the time. Through a four-gate sellability evaluation, this adapter was the only one of 9 domain adapters to survive, achieving 9/10 domain tasks, 5/5 unknown honesty, 5/5 refusal capability, 5/5 JSON format, and 0/10 hallucination rate. The repository publicly documents the v1-to-v2 failure-fixing process and reports from all 8 failed adapters, proving that dataset coverage of failure modes matters more than data volume. The model card honestly acknowledges no knowledge improvement and energy-domain limitation, making it suitable for high-volume, low-latency, narrow-scope tasks in agent pipelines.