Fine-Tuning gpt-oss-20b in Practice: From Unsloth Training to Ollama Deployment

Fine-tuning a 20-billion-parameter model on a single consumer GPU and running it locally in Ollama is no longer a lab-only endeavor. Developer Khadim Hussain used Unsloth to fine-tune OpenAI's open-source gpt-oss-20b for STEM reasoning tasks, and documented the full process — including every pitfall and every reusable config. The core takeaway is straightforward: QLoRA training requires only about 14 GB of VRAM, but the real headache comes at deployment time. If the chat template isn't handled correctly, Ollama will either produce garbled output or never stop generating.

QLoRA Fine-Tuning Config: Training a 20B Model on a Single GPU

gpt-oss-20b is a 21B-parameter Mixture-of-Experts (MoE) model. The author used QLoRA, quantizing the base weights to 4-bit and training only a small LoRA adapter, making it possible to fit this model on a single consumer GPU.

Key training parameters:

• Method: QLoRA, 4-bit base weights
• LoRA rank / alpha: 32 / 64
• Learning rate: 1e-4
• Batch size: 1, gradient accumulation 16
• Optimizer: AdamW 8-bit
• Epochs: 1
• Data: 4,260 training samples + 474 evaluation samples, all STEM Q&A with chain-of-thought reasoning
• Results: Training loss 1.087, evaluation loss 0.837
• Framework: Unsloth + TRL

The loss curves show that evaluation loss (0.837) is lower than training loss (1.087), indicating good generalization on the validation set with no obvious overfitting. The complete training config is available in configs/gpt_oss_20b.yaml within the author's open-source toolkit, kllm.

QLoRA (Quantized Low-Rank Adaptation) reduces VRAM usage beyond standard LoRA by freezing the pretrained model's original weights and training only two low-rank matrices (rank decomposition), compressing trainable parameters to under 1% of the original model. QLoRA then applies 4-bit NormalFloat (NF4) quantization to shrink the frozen base weights from 16-bit to 4-bit storage, while using Double Quantization and Paged Optimizers to maintain training stability. The rank value determines the "width" of the LoRA matrices — rank=32 means each replaced weight matrix is decomposed into two rank-32 matrices; alpha is a scaling factor, typically set to 2× rank (here, 64) to control how much the adapter influences the original weights. This combination brings 20B-scale model training — previously requiring multiple A100s — down to something a single consumer GPU can handle.

How Much VRAM Does Fine-Tuning gpt-oss-20b Actually Need?

This is the most frequently asked question. According to the Unsloth official gpt-oss guide, QLoRA fine-tuning of gpt-oss-20b requires approximately 14 GB of VRAM, with at least 16 GB recommended for stable training. For comparison, running LoRA directly on unquantized BF16 weights requires around 44 GB — which is exactly why QLoRA is the practical choice for single-GPU setups.

The author's verified environment was an RTX 5090 (32 GB) running Ubuntu 24.04, with CUDA 12.8, Python 3.12.3, PyTorch 2.9.1, Unsloth 2026.1.4, and transformers 4.57.3.

One version gotcha that's easy to miss: vLLM 0.15.0 requires an exact match with transformers 4.57.3 — any other version will throw import errors. The author's recommended installation order is: install vLLM first, then Unsloth dependencies, then force-pin the transformers version:

uv pip install -U vllm --torch-backend=cu128   # RTX 40 series use cu121
uv pip install unsloth unsloth_zoo bitsandbytes
uv pip install --force-reinstall "transformers==4.57.3"

One Training Run, Three Export Formats

After training, the same model object can be exported in three deployment formats for different use cases:

# 1. LoRA adapter (~61 MB): for those already running the base model
model.save_pretrained(output_dir)
tokenizer.save_pretrained(output_dir)

# 2. Merged 16-bit model (~41 GB): for Transformers or vLLM, no adapter handling needed
model.save_pretrained_merged(f"{output_dir}-merged", tokenizer, save_method="merged_16bit")

# 3. GGUF format: for llama.cpp, Ollama, LM Studio — one call per quantization
model.save_pretrained_gguf("gguf/gpt-oss-20b-stem", tokenizer, quantization_method="q4_k_m")

File sizes vary significantly across quantization schemes: f16 is 41.9 GB, Q8_0 is 22.3 GB, and Q4_K_M is just 15.8 GB. If you run out of memory during GGUF export, the Unsloth docs recommend lowering maximum_memory_usage from the default 0.75 to around 0.5.

GGUF (GPT-Generated Unified Format) is a model serialization format introduced by the llama.cpp project in 2023. It's designed to bundle model weights, tokenizer configs, and metadata (including the chat template) into a single file for efficient inference on CPUs and consumer GPUs. It replaced the earlier GGML format and supports multiple quantization schemes. In the naming convention, Q4_K_M means 4-bit quantization, K-Quant method, M (medium) precision tier. K-Quant is a mixed-precision strategy that preserves higher precision for more important layers (like attention) while aggressively compressing others — delivering better quality than uniform quantization at the same file size. Ollama, LM Studio, Jan, and other local inference tools all use GGUF as their primary format, which explains why GGUF download counts far exceed other formats.

The Part That Will Actually Break Things: The Harmony Chat Template

This is the most important takeaway from the entire walkthrough. gpt-oss does not use ChatML — it was trained on OpenAI's Harmony format. In this format, each message is wrapped in special tokens, and the assistant outputs content across different "channels":

• analysis: the model's chain-of-thought, not meant to be shown to end users
• final: the final answer
• commentary: tool calls

The critical detail is the stop token: a final answer ends with <|return|>, which OpenAI's documentation explicitly calls "a valid stop token indicating that reasoning should stop"; messages retained in conversation history end with <|end|>; and supervised training targets should end with <|return|>. OpenAI's model card is blunt about this — these models "should only be used with the harmony format, otherwise they will not work correctly."

In other words, the template at deployment time must match the one used during training. Unsloth's documentation notes that template mismatches are the most common cause of garbled output or infinite generation on other platforms; <|return|> must be set as a stop token, or generation will continue past the answer indefinitely. Notably, the question of how to configure an Ollama template for a fine-tuned gpt-oss model remains an open thread on the official OpenAI forums — a sign that this is a genuinely widespread pain point.

The chat template is an often-overlooked infrastructure layer in LLM inference. During training, each conversation message is concatenated into a plain text sequence using a specific format, with special tokens marking role and message boundaries. At inference time, the input must be constructed in exactly the same format for the model to "recognize" the conversation structure and produce expected output — because the model fundamentally only sees a token sequence, and a format mismatch is equivalent to feeding it an input distribution it has never encountered. Common formats include ChatML (<|im_start|> / <|im_end|>), Llama's [INST] markers, Mistral's [INST]/[/INST], and others. OpenAI's Harmony format is its own system, using the private tokens <|start|>, <|message|>, <|end|>, and <|return|>. Stop tokens tell the inference engine "truncate output here" — if they're misconfigured, the model will keep generating content for the next turn of conversation after producing a valid answer, manifesting as "endless generation."

A Working Ollama Modelfile

The Modelfile the author published alongside the GGUF repository looks like this:

FROM ./gpt-oss-20b-finetuned-q8_0.gguf

TEMPLATE """<|start|>system<|message|>You are a helpful assistant trained by OpenAI.

Reasoning: medium

# Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>user<|message|>{{ .Prompt }}<|end|><|start|>assistant"""

PARAMETER stop "<|return|>"
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 4096
PARAMETER num_predict 2048

The Reasoning: medium setting controls how much the model thinks in the analysis channel before giving an answer. Harmony accepts low, medium, and high, defaulting to medium. Ollama displays the analysis channel as "Thinking...", so you can set it to low if you want shorter, faster responses. Remember to re-run ollama create after making changes.

ollama run hf.co vs. Modelfile: A Key Difference

These two loading methods have one important distinction. With a Modelfile, the template you've written by hand is what takes effect:

ollama create gpt-oss-20b-stem -f Modelfile
ollama run gpt-oss-20b-stem

Ollama can also pull directly from Hugging Face:

ollama run hf.co/khadim-hussain/gpt-oss-20b-stem-reasoning-GGUF:Q4_K_M

But this path does not read the Modelfile from the repository. According to the Hugging Face Ollama documentation, it uses the template stored in the GGUF file's tokenizer.chat_template metadata, or a file named template in the repository. Quantization tags are case-insensitive, and without a tag, Ollama defaults to Q4_K_M. So if you're publishing a Harmony model, make sure to include a template file in the repo or explicitly direct users to use the Modelfile.

What Formats Do People Actually Download?

The author also analyzed download distribution on Hugging Face (cumulative totals as of October 2026), and the data is instructive:

Format	gpt-oss-20b	Qwen3-14B
GGUF	513	274
Merged 16-bit	70	42
LoRA adapter	46	51

The conclusion is clear: most people want a single file they can run directly on their own machine. If you're only publishing one format, publish GGUF with a working template. The LoRA adapter is worth including as a separate artifact for those who want to continue training.

Common Troubleshooting

The author documented several categories of typical failures in kllm's troubleshooting guide:

• Training process exits without error: Almost always out-of-memory (OOM) — confirm with dmesg
• UnboundLocalError mentioning dropout: Set lora_dropout to 0.0
• ImportError: cannot import name ... from 'transformers': Force reinstall transformers==4.57.3
• Python.h: No such file or directory: Install python3.12-dev

The value of this full walkthrough isn't any single trick — it's that it connects "training works" and "deployment works" into one complete chain, two things that are often treated in isolation. For developers looking to run open-source LLMs locally, the Harmony template pitfall is especially worth keeping in mind from the start.