NVIDIA Nemotron Dory 4B Local Deployment Guide: Recurrent Architecture and Trying the llama.cpp PR

NVIDIA's newly released 4B model, Nemotron Labs Dory 1.0, scored 61.8 on SWE-bench Verified (NVIDIA's own testing), while Qwen3.5-9B, which is more than twice its size, only managed 61. But here's the awkward part: as of October 18, officially supported llama.cpp, vLLM, and Transformers support has not been merged or is entirely missing—getting it running requires doing the work yourself.

Dory quietly launched on Hugging Face on October 16, accompanied by a model card and GGUF repository, but with no official NVIDIA blog post and no paper link. This article covers its core design, memory requirements, and how to run it locally using a llama.cpp PR build, then takes a look at the current support status across Ollama, vLLM, and Transformers.

Recurrent Architecture: 4B Parameter Count, Deeper Computation

Most small models pass each token through a single fixed stack of layers, whereas Dory consists of three parts: an input block (8 layers), a shared recurrent block (20 layers, repeatable 1 to 4 times), and an output block (8 layers). The input is re-injected on each loop, and the number of loops is specified at startup without retraining.

• 1 loop: cheapest, each token passes through 36 layers
• 2 loops: NVIDIA's recommended default, "most of the accuracy gain comes from the second loop"
• 4 loops: aimed at agentic coding tasks like SWE-bench and Terminal-Bench, each token passes through 8 + 80 + 8 = 96 layers

"4B" is just the size on disk. At 4 loops, the per-token computation is equivalent to a much deeper model, but storage remains small.

Other key specs:

• Normalized GPT architecture with sliding window attention (1024 token window) mixed with full attention layers, one full attention layer for every three sliding layers
• 256K context (marked as verified in the model card)
• 131,072 token vocabulary, hidden dimension 2560, 16 query heads, 4 KV heads, head dimension 256
• Approximately 6T pretraining tokens, with the first 4.5T trained natively in NVFP4, followed by BF16, plus 300B tokens of context extension, along with SFT and GRPO-based reinforcement learning
• English only, training covers 43 programming languages
• OpenMDW License v1.1, with the model card stating it can be used for "commercial or non-commercial purposes"

The license is worth applauding. OpenMDW grants "unrestricted" rights to the "model materials" (including architecture and parameters), which matters more to companies choosing a 4B model than any single benchmark score.

Recurrent / Looped Transformer refers to the same set of weights being executed multiple times rather than stacking more independent layers. The idea traces back to Universal Transformers and Deep Equilibrium Models: improving reasoning ability by increasing "computational depth" without increasing "parameter count." Since parameters are shared across loops, model size stays the same, but per-token FLOPs grow linearly with the number of loops. NVIDIA's "input re-injection" adds the original input representation back before each pass into the recurrent block, preventing information from being diluted in deep loops—this lets it change the loop count at inference time without retraining: one training run, multiple depths. The cost is that KV cache allocation strategy becomes a new problem: whether shared layers reuse the same cache across loops directly determines VRAM usage, which is exactly the divergence between the per_loop and last_loop modes discussed below.

Benchmarks: Strong on Agentic Tasks, Weak on Factual Knowledge

All data below comes from NVIDIA's own testing tooling (NeMo Gym / NeMo Evaluator), and no independent reproduction has been found yet. The Artificial Analysis index chart in the model card is an internal estimate and may deviate from the official AA score.

Benchmark	Dory 1 loop	Dory 2 loops	Dory 4 loops	Qwen3.5-4B	Qwen3.5-9B
SWE-bench Verified	50.3	59.5	61.8	54.7	61
Terminal-Bench 2.1	12.5	18.1	22.2	15	24
IFBench	74.29	77.94	79	53.83	59
AA-LCR	63	71	73	64.7	68.4
GPQA Diamond	69.19	72.41	72.16	77.71	80.93
HLE (text, no tools)	8.25	10.89	10.43	11.31	15.43
AA-Omniscience	10.03	10.08	10.03	15.02	18.05

The AA-Omniscience row barely changes from 1 loop to 4 loops (10.03, 10.08, 10.03). Looping gives the model more steps to reason, but it can't add knowledge that doesn't fit in 4B parameters. GPQA and HLE show the same trend: Dory lags behind Qwen3.5-4B at every loop count.

The areas where Dory wins are instruction following (leading Qwen3.5-4B by over 20 points on IFBench), long-context reasoning (AA-LCR), and SWE-bench at 2+ loops. This is exactly the profile needed for a local coding agent or tool-calling worker, not for a pocket encyclopedia.

Caveat on comparisons: the model card also lists larger models like Gemma 4 26B A4B and Qwen3.6-35B-A3B, where Qwen3.6's SWE-bench score (73.4) was cited by NVIDIA rather than run by them. The model card has no Phi row and no Gemma model of the same size, so hopes for a compute-aligned comparison with something like Phi-4-mini are dashed.

Memory and VRAM Requirements

The model card does not give minimum memory requirements, but they can be inferred from the files and configuration.

Format	Disk weights	Runtime platform
BF16	7.95 GB	Any GPU with ~10 GB+ free VRAM, plus KV cache
Q5_K_M GGUF	3.67 GB	Laptops, Apple Silicon, 6-8 GB VRAM GPUs
MXFP8	4.75 GB	Hopper/Blackwell, 1.69-1.84x faster than BF16 at batch size 1 on DGX Spark
NVFP4	3.20 GB	Hopper/Blackwell, 2.17-2.44x faster than BF16 at batch size 1 on DGX Spark

KV cache is the main overhead of the recurrent architecture. In the default per_loop mode, a separate cache is retained for each pass through the recurrent block. Based on the layer_types in config.json, there are 9 full attention layers (2 in the input block, 5 in the recurrent block, 2 in the output block), and each layer stores 4 KV heads × 256 dims × K and V × 2 bytes = 4 KiB per token. In per_loop mode, the 5 layers of the recurrent block are counted repeatedly per loop: 2 loops means 14 full attention caches, roughly 1.75 GiB at 32K context and roughly 14 GiB at 256K context; 4 loops means 24 caches, roughly 24 GiB at 256K. Sliding window layers only store at most 1024 tokens and are almost negligible.

The solution is to switch to last_loop mode, which "retains only a single cache per recurrent attention layer" regardless of loop count, returning to 9 caches (roughly 9 GiB at 256K, roughly 1.1 GiB at 32K). NVIDIA says accuracy is "close to per-loop mode." Set this first when memory is tight.

KV cache is the Key/Value tensor that stores historical tokens during LLM inference, avoiding recomputing the entire context for every new token generated. Its size formula is: number of layers × 2 (K and V) × number of KV heads × head dimension × context length × bytes per element. The 4 KiB per layer per token in the article comes from 2 × 4 × 256 × 2 bytes. In multi-head attention, GQA (Grouped Query Attention) lets multiple query heads share a few KV heads—Dory's 16 query heads to 4 KV heads is exactly a 4:1 GQA, which is why its cache is more economical than same-size models. Sliding window attention retains only the KV of the most recent 1024 tokens, so the cache is capped by the window, which is why the article calls it "almost negligible." The special thing about the recurrent architecture is that shared layers are executed multiple times; if a separate cache is retained for each loop, the total cache count multiplies with the loop count, making cache mode selection more critical than for ordinary models.

Running Locally with the llama.cpp PR Build

Support lives in llama.cpp PR #30306, currently in draft status, requiring two approvals and still awaiting review from CISC and ggerganov. Someone asked on October 17 whether anything was still blocking it, with no reply yet. The PR description acknowledges that Metal is untested and long-context qualification is incomplete. Interestingly, the GGUF model card's build instructions are for Apple Silicon Metal, using a pinned commit in the author's fork:

git init llama.cpp-dory
cd llama.cpp-dory
git remote add origin https://github.com/brianchmielNV/llama.cpp.git
git fetch --depth 1 origin 98a74c97ea09ac72f6ceda1559808fc0a2f447ce
git checkout --detach FETCH_HEAD

The full macOS cmake invocation (Metal, Apple BLAS, server only) is on the GGUF page. On Linux + NVIDIA GPUs, you can build the same commit the usual way (cmake -B build -DGGML_CUDA=ON). The PR reports 96/96 numerical checks passing on CPU and CUDA, but NVIDIA hasn't published a Linux recipe.

The filename in the repository is NVIDIA-Nemotron-Labs-Dory1.0-4B-Q5_K_M.gguf, while the model card command expects dory-q5_k_m.gguf, so rename after downloading. After verifying the file, start the server:

echo "96080be49ffd67b916018b7bdef8c484762576fb52780e10ab98ac4978a660e0  dory-q5_k_m.gguf" | shasum -a 256 -c -

./build/bin/llama-server \
    -m ./dory-q5_k_m.gguf \
    --alias dory \
    --host 127.0.0.1 --port 8080 \
    -ngl 999 \
    --ctx-size 4096 --parallel 1 \
    --batch-size 512 --ubatch-size 512 \
    --cache-type-k f16 --cache-type-v f16 \
    --flash-attn off \
    --no-context-shift \
    --jinja --reasoning on \
    --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0 --repeat-penalty 1 \
    --override-kv "dory.num_loops=int:2,dory.recurrent_kv_cache_mode=str:per_loop"

Adjust dory.num_loops (1 to 4) to trade off accuracy against speed. Change per_loop to last_loop to limit the cache. --no-context-shift is required because the PR does not support context shifting. Requests stop at 4,096 tokens; raise --ctx-size if memory permits.

For coding agents, the model card recommends passing "chat_template_kwargs": {"force_nonempty_content": True} in extra_body.

llama.cpp's PR process determines how quickly a new architecture becomes usable by ordinary users: new model architectures require adding tensor types, graph construction, and metadata parsing code in the ggml and llama layers, usually submitted by engineers from the model vendor, requiring at least two maintainer approvals plus CI and numerical consistency checks. A PR cannot be merged while in Draft status, and the API and config keys can change at any time—the --override-kv keys mentioned in the article, such as dory.num_loops and recurrent_kv_cache_mode, may all be renamed before merging, which is the technical reason to "wait a week or two." GGUF is the single-file model format used by llama.cpp, containing weights, tokenizer, and metadata; Q5_K_M in the article refers to a 5-bit quantized K-quant medium mixed scheme that balances size and accuracy. -ngl controls the number of layers offloaded to the GPU, and --flash-attn and --jinja enable FlashAttention and Jinja chat templates respectively.

Current Status of Ollama, vLLM, and Transformers

Hugging Face's Local Apps menu on the GGUF page offers ollama run hf.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF:Q5_K_M, but the model card itself only documents the llama.cpp PR build and never mentions Ollama or overriding loop parameters through it. This one-line command is not expected to work yet. Ollama can only run architectures its engine supports, and the dory architecture has not yet entered upstream llama.cpp, nor does the PR mention Ollama. You'll need to wait for the merge and a new Ollama release.

vLLM support is in PR #60149, currently open with no approvals. One maintainer asked that the model first land in Transformers, to avoid vLLM having to change its configuration frequently. The author said there are no plans for a Transformers PR, and the maintainer ultimately accepted a solution that best-effort follows Transformers config conventions. Separately, the author argued for keeping a native implementation in the PR rather than going through vLLM's Transformers backend, with no reply from vLLM maintainers yet. If you build that branch, the model card's serve command is as follows:

vllm serve --model nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 \
  --hf-overrides '{"n_recurrent_loops": 2, "recurrent_kv_cache_mode": "per_loop"}' \
  --max-model-len 262144 \
  --max-num-seqs 128 \
  --enable-prefix-caching \
  --async-scheduling \
  --reasoning-parser nemotron_v3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice

Reasoning is always on in vLLM. If the GPU doesn't have room for a 256K per_loop cache, lower --max-model-len.

Transformers is the strangest case. The repository is tagged library_name: transformers, but the model card has no Transformers code snippet. config.json declares "model_type": "dory", the repository has no modeling_*.py, and as of today the Transformers main branch has no models/dory directory (the latest PyPI version is 5.20.1). In the vLLM discussion, the author stated that NVIDIA currently has no plans to submit a Transformers PR, so the current version is expected to hit an unknown architecture error. If you can load it with AutoModelForCausalLM, please let us know the version.

Transformers' model_type mechanism is key to understanding the compatibility predicament: AutoModelForCausalLM looks up a directory of the same name under transformers.models based on the model_type field in config.json and the modeling_*.py implementation; if it can't find it, it throws an "unknown architecture" error. So even though the Hugging Face page is tagged library_name: transformers, as long as models/dory hasn't been merged upstream, the standard loading path will fail. The vLLM maintainer asked for the model to land in Transformers first because vLLM reuses Transformers' config classes to parse hyperparameters; otherwise every config change would have to be maintained separately inside vLLM, increasing regression risk. Ollama, meanwhile, is tied to upstream llama.cpp's architecture support; only after the PR is merged and lands in a new release can one-line commands like ollama run hf.co/... actually work.

Is It Worth Trying This Week?

On a Mac with 8 GB of free memory, yes. Build the pinned commit, run 2 loops with last_loop, and point it at a small coding agent. The recurrent design is the most interesting thing in this year's 4B releases, and changing depth at startup is a capability I wish every small model had.

On other platforms, wait a week or two. The benchmarks are self-reported, the two runtime PRs may still change config keys, and the knowledge scores mean you'll still need Qwen3.5-4B for tasks that depend on memorized facts.

Which 4B model are you actually running locally right now? Does a recurrent model like Dory have a chance of replacing it? If you've gotten the PR build working on CUDA, feel free to share tokens/sec for 1 loop vs 4 loops—nobody has published that data yet.