Local Deployment of a Jev-Style Decision API: Docker and DGX Spark in Practice

Title

Local Deployment of a Jev-Style Decision API: Docker and DGX Spark in Practice

Meta Description

A complete tutorial on deploying a Jev-style decision API locally with Docker on NVIDIA DGX Spark, covering image selection, checkpoint pinning, memory measurements, and retrieval accuracy comparisons across ten open-weight models.

Article Content

Running decision model inference locally, instead of sending internal project materials to a hosted service, sounds like just a change of deployment method. Once you actually start, the questions become: which checkpoint, which inference code, which Docker image, and exactly how much memory.

The setup below is the combination I ultimately settled on: running Cloudflare Clef Flash on an NVIDIA DGX Spark, served through a local /v1/systemone endpoint. The full method, data tables, and limitations are documented in the published technical report, with code and launchers hosted on GitHub.

The Interface Shape of a Jev-Style Decision API

The effective shape of a decision API is straightforward: provide a state containing evidence, a question, and explicit answer choices; the response returns the selected answer, optionally with a probability. For example, given a retry policy and two actions, ask which action permits a retry.

"Jev-style" describes exactly this interface. The local checkpoints come from other publishers; this tutorial does not run the hosted Jev model locally. For hosted comparisons, we use typesafe-ai/jev, accessed via Vercel.

For local multiple-choice, a trained decision head scores the given options directly, unlike chat completion which generates answers token by token. A response reporting zero output tokens can still contain a real decision.

The decision head is the key mechanism here. It is entirely different from the generation path of a chat completion model: the language model backbone first does a forward pass over the input state, question, and answer choices, producing a context representation vector; the decision head is a small classification or scoring module typically trained on top of the backbone, which directly computes a score for each option from that representation, then applies softmax to get a probability distribution.

So a zero output-token count does not mean no inference occurred—the model has completed a full encode and forward computation, it simply outputs structured options rather than generating text. The practical benefits of this design are stable latency, predictable throughput, and answers that naturally correspond to the given legal option set, avoiding the occasional drift outside the options that generative models exhibit. The cost is that each checkpoint must come with matching decision head weights and loading logic from the publisher, which is why the article later stresses the need to verify the publisher's loader.

Setting Up the Runtime

This setup has been tested on NVIDIA DGX Spark (Linux ARM64). The key is choosing a container matching both the hardware and the checkpoint inference code. On a different machine, you need to verify four things item by item:

1. Match the CPU architecture. Use docker buildx imagetools inspect IMAGE:TAG to check the candidate image's manifest and look for your platform identifier, usually linux/amd64 on an x86 Linux workstation or linux/arm64 on Spark. CPU emulation does not establish GPU compatibility.
2. Match the GPU and driver. Look for candidates in NVIDIA's PyTorch container release notes and framework support matrix, and verify their GPU architecture, CUDA, and driver requirements. Matching only the CUDA major version is not sufficient evidence.
3. Match the publisher's loader. Read the model card, dependency files, and runtime adapter for the pinned checkpoint. A compatible PyTorch container only provides the tensor runtime; you still need the correct Transformers version and the publisher's decision head implementation.
4. Pin the combination you verified. Record the image digest, checkpoint commit, API commit, direct dependency versions, and a full environment lock, and test GPU execution, loading, real decisions, overflow handling, and peak memory on that machine.

The provided launchers target Linux ARM64. Results on Spark do not prove support on other hardware.

Pulling the Runtime and Downloading the Pinned Checkpoint

The test Spark configuration has two prebuilt images, both under ghcr.io/krisitown/local-jev-api: spark-20261013 for nine test checkpoints including Clef Flash, and spark-omni-20261013 for Clef Omni's separate compatible runtime.

Leave room for downloading and running models. The public runtime unpacks to about 20 GB and the Flash weights are about 18 GiB. In this workload, Flash used 17.75 GiB of loaded CUDA tensors but caused roughly a 36.06 GiB drop in peak host available memory—these are two different measurements. The launcher will not pause other services for you.

docker pull ghcr.io/krisitown/local-jev-api:spark-20261013

docker image inspect ghcr.io/krisitown/local-jev-api:spark-20261013 \
  --format '{{.Os}}/{{.Architecture}}'

python3 ops/download_model.py --key clef-flash

The image inspection should report linux/arm64. The downloader uses the Hugging Face client installed in the container to download the runtime snapshot to checkpoints/clef-flash, explicitly requesting a fixed repository and revision. For reproducibility, use the full commit rather than the moving main branch.

The Dockerfile pins the NVIDIA PyTorch base image by digest, and its direct dependencies include transformers==5.17.0, accelerate==1.14.0, huggingface-hub==1.32.0, fastapi==0.136.3, uvicorn==0.52.4, and others. These are the direct locked versions tested for this setup, not necessarily optimal for all checkpoints. Omni uses Transformers 5.10.2 and CPU-first loading in a separate image.

Before loading weights, verify the container can execute a small GPU operation:

docker run --rm --gpus all --entrypoint python3 \
  ghcr.io/krisitown/local-jev-api:spark-20261013 \
  -c 'import torch; print(torch.__version__, torch.version.cuda); assert torch.cuda.is_available(); print(torch.cuda.get_device_name(0))'

Starting the Local API

The checkpoint stays outside the image, mounted read-only. Uvicorn listens on 0.0.0.0 inside the container, while -p 127.0.0.1:18081:18081 publishes the port to Spark's loopback interface. This experimental wrapper has no authentication layer and should remain bound this way throughout the walkthrough.

In benchmarks, Flash takes about 157 seconds to load, plus additional download time. After waiting for startup, check the returned identity information for ready: true, model Cloudflare/clef-flash, the pinned revision, and backend clef.

A Shorter Path

The manual steps make every stage visible. In a fresh environment, the launcher can automatically pull the image, download the pinned checkpoint, start the container, wait for readiness, and send a typed smoke request: just run ./run.sh clef-flash. It can also accept an existing full snapshot.

Do not run this command on top of a manual container: the launcher deliberately refuses to replace a container with the same name.

Testing the API

With the model running, send a decision request and read answers.q1.choice; A is the policy-conformant expected answer. You should inspect the actual response rather than assuming a successful HTTP request means the answer is correct.

The wrapper also provides /inspect and /metrics, plus _benchmark telemetry. It returns HTTP 413 for oversized inputs rather than silently discarding the provided state. Flash's 16,384 token limit covers the encoded question and options, not just the state string.

Deliberately Trying Another Checkpoint

The launcher supports all ten test checkpoints, including the Cloudflare Clef series, Perplexity PPLX, TokenRhythm NeoHorse, Jeff, and Convai Innovations Laya. Each key selects a pinned snapshot and family-specific backend. Omni automatically selects its separate image. All revisions are recorded in the checkpoint manifest.

After stopping the Flash demo, for example ./run.sh jeff-2b starts Jeff 2B on the default port. If you need to run another model in parallel, change the host port. Use the corresponding checkpoint ID in the JSON request.

What the Measured Results Show

The technical report covers all eleven systems, including hosted Jev, and explains the measurement rules.

I ran 336 frozen cases per local checkpoint on a single GB10 DGX Spark: 288 single-hop lookups and 48 two-hop lookups, including missing evidence and near distractors. The ten selected full runs produced 3,360 results: 2,065 answers and 1,295 context refusals. Rejected inputs are neither completed inferences nor wrong answers.

The fairest small-input comparison is the 112 cases all models accepted: 96 single-hop cases with states of 256 to 1,024 characters, and 16 two-hop cases with 1,024 characters.

On these same 112 short cases, hosted Jev and PPLX each scored 112, NeoHorse and Clef each 104, Flash and Omni each 102, Jeff 2B 94, Jeff 0.8B 77, Laya Typed 52, Laya 46, and Laya Multilingual 25.

A two-answer difference is not enough to establish a general model ranking; the technical report includes descriptive uncertainty intervals and the full local versus hosted comparison.

Small Checkpoints Do Not Equal Total Memory Requirements

Loaded allocation describes active model tensors; it is not the amount of memory you can safely reserve. For Flash, the process was 17.75 GiB loaded, 21.68 GiB peak allocation during inference, 29.33 GiB peak reservation, and about a 36.06 GiB drop in peak host available memory. Reserved memory includes reusable allocator blocks, while host memory changes also include runtime overhead and other activity.

Spark shares CPU and GPU memory, so do not add the CUDA and host numbers together. In this workload, Clef and Omni each showed drops of over 100 GiB in host available memory. Startup and long inputs also need space.

Loaded allocation, peak allocation, and peak reservation are three easily confused CUDA memory statistics. Loaded allocation is the VRAM currently holding parameter tensors; peak allocation is the maximum instantaneous allocation during inference, typically including activations and intermediate buffers; peak reservation is the total VRAM the PyTorch caching allocator has requested from CUDA and retains for reuse. It is higher than peak allocation because freed blocks are not immediately returned to the driver.

The drop in host available memory is yet another matter: it reflects the process's overall impact on system memory, including the Python runtime, CUDA context, temporary buffers for model loading, and shared memory mappings. On architectures like DGX Spark where CPU and GPU share unified memory, the CUDA and host numbers overlap heavily, so adding them directly severely overestimates requirements. The correct approach is to look at the actual drop in host available memory and leave extra space on top for startup overhead and long inputs.

Context Capacity and Effective Recall Are Two Different Things

The tested local serving setups have different token limits. Laya is 512, Laya Typed and Laya Multilingual are 1,024, Jeff 0.8B and Jeff 2B are 8,192, NeoHorse 4B is 2,048 state plus 8,192 branches, Flash and Clef are 16,384, and Omni is 64,000.

These are serving limits, not theoretical backbone windows. The full encoded input includes the question and options. The wrapper returns HTTP 413 on overflow and does not silently truncate scoring requests.

Omni processed 63,999 encoded tokens in a boundary probe. At the longest scored single-hop tier, 131,072 state characters, it scored 13/32. That probe used repeated filler text, whereas the scored corpus has different token density—characters and tokens are not interchangeable units.

Flash scored 24/32 at the 32,768 character tier, which is worth a closer look: 24/24 provided needles were correct, and 0/8 missing-evidence controls were correct. Finding provided evidence does not imply being able to recognize its absence.

PPLX was correct on all 224/224 accepted cases. Hosted Jev tied it on these exact 224 commonly accepted cases and handled additional longer cases. Across the full hosted run, Jev scored 301/304 accepted cases correct, with 32 context refusals out of 336 submissions. All three wrong answers involved a 65,536-character two-hop lookup.

Speed Depends on Request Shape

These are decision heads, so the reported metric is decisions per second. A four-question request helped some runtimes while others were nearly unchanged: Flash went from 4.21 decisions/sec per request for one question to 9.84 for four, but four concurrent HTTP clients did not bring the same gain, because our wrapper serializes GPU inference and concurrency mostly means queuing. It is not a dynamic GPU batching engine.

The distinction between dynamic batching and serialization explains the speed data in this section. In a true batching engine, the scheduler merges multiple requests arriving in a short window into one batch sent to the GPU, serving several requests with a single forward pass; thus, with more concurrent clients, per-request latency may rise but total throughput improves significantly.

The wrapper in this article offers no such capability: GPU inference is serialized, and later requests can only wait in the queue. So four concurrent HTTP clients bring far less improvement than packing four questions into a single request. The latter works because they are already one batch at the application layer, all computed in a single forward pass. Understanding this matters for capacity planning—if the workflow naturally consists of many independent small requests, this setup will not scale by adding concurrency; you need to actively aggregate requests on the caller side.

How I Would Use These Results

For a small local starting point, Jeff 2B is worth testing: 94 correct out of 112 short cases, 4.12 GiB loaded, about 54 ms on the short microbenchmark, but weaker two-hop results. NeoHorse 4B is strong on short cases, 8.55 GiB loaded, but has a state budget of only 2,048 tokens. PPLX 27B gave the strongest local retrieval results, but at a much higher memory cost, with a serving limit of 8,192 tokens.

Clef Flash is the practical walkthrough choice here: a verified integration path, a 16,384 token budget, and a smaller footprint than Clef or Omni. Its failure on missing evidence still matters. The Laya series was the lightest and fastest in these runs, but had lower accuracy on this English corpus; this test says nothing about their performance in other languages.

The experiment's limitations are real: a single Spark, a single synthetic English corpus, family-specific loaders and precision, no randomized model order or thermal control, no independent repeat sessions. Hosted GPU hardware and immutable model revisions cannot be obtained.

Wiring the Endpoint into a Workflow

I originally wanted this interface for a Jev Quality Gate: take bounded project evidence, ask explicit policy questions, and retain responses for inspection and replay. The local adapter branch supports configurable endpoints and models, with optional credentials. For this local setup, no bearer key is needed.

The integration proved transport and replay capability. Its final policy result is indeterminate, because no global reconciliation was performed. It is neither a passed overall change compliance check nor a code review accuracy evaluation.

After the demo, run docker stop local-jev-flash and docker rm local-jev-flash on Spark, and restore any workloads you paused.

The repository contains the launcher, pinned checkpoint manifest, Dockerfile, and local benchmark evidence. Try a checkpoint suited to your machine, then test it on the decisions your project actually needs, including those where the correct evidence is missing.

Background Supplement

The difference between digest and tag is the foundation of this setup's reproducibility. A tag (such as spark-20261013) is a mutable alias that the publisher can point to a new image build at any time; a digest (in the form sha256:...) is a hash of the image contents and is immutable once fixed. Pulling by tag in production means trusting upstream not to replace the content at your next build; pulling by digest ensures that the combination you verified today is the same binary tomorrow.

The same principle applies to checkpoints: Hugging Face's main branch moves, while a full commit hash does not. The downloader and launcher in this article both explicitly request fixed revisions, precisely so that "the thing I verified" and "the thing the next deployment runs" are the same. Omit this step and both the container and the weights can drift unnoticed.

One-line Summary

Running a decision API locally on DGX Spark: deployment essentials, memory measurement units, and measured comparisons across eleven systems.

Paragraph Summary

This article documents a complete setup for running a Jev-style decision API locally on NVIDIA DGX Spark, using Cloudflare Clef Flash as the model, served through a local endpoint. The core is four layers of matching: CPU architecture, GPU driver, publisher loader, and pinned image digest and checkpoint commit. The article emphasizes that a small checkpoint does not mean a small memory footprint—Flash loads 17.75 GiB of CUDA tensors, yet host available memory drops by about 36.06 GiB at peak, because under a shared-memory architecture the two measurements cannot be added. On the 112 short cases all models accepted, hosted Jev and PPLX each scored 112, Flash scored 102, but a two-point difference is not enough to establish a general ranking. Flash showed a typical capability boundary: 24/24 correct with provided evidence, but 0/8 correct on missing-evidence controls, showing that finding evidence is not the same as recognizing its absence. The overall conclusion is that this local setup is a good starting point for controlled, replayable walkthroughs, not a general conclusion about model performance.