Three Pitfalls When Connecting Claude Code to Self-Hosted Models

Title

Three Pitfalls When Connecting Claude Code to Self-Hosted Models

Meta Description

Pointing Claude Code at a self-hosted model trips three pitfalls: tool calls returning 400, mismatched model names, and misconfigured base URL plus auth headers. This article breaks down the cause and fix for each, and explains how an 80B MoE model runs on a single GPU plus its limits.

Article Content

Claude Code can connect to any endpoint that supports the Anthropic Messages API, which sounds like a one-line config change. In practice, after pointing the endpoint at a self-hosted model, it fails in several predictable ways—and each one looks like the model is broken.

Someone ran a full experiment: driving Claude Code with Qwen3 Coder Next—an 80B mixture-of-experts (MoE) coding model—served by vLLM on a rented A100 80GB. This article only covers the failure list: what broke, why it broke, and how to fix it.

Pitfall 1: Tool Calls Return 400 Immediately

Claude Code is an agent, and nearly every request carries tool definitions: read file, execute command, edit component. A plain vllm serve doesn't accept these definitions, and requests with tools are rejected by vLLM with HTTP 400. From Claude Code's side, it looks like the model is broken.

The fix is to add two flags on the server side, which is also what the model card recommends:

--enable-auto-tool-choice --tool-call-parser qwen3_coder

The parser must match the model family. After adding these, the entire session ran through: entering plan mode, writing files, installing dependencies, running npm run build, then continuing to edit, with tool calls working normally throughout.

Anthropic Messages API and Claude Code's Tool Protocol

The reason Claude Code can drive any compatible endpoint is that it encodes all agent capabilities as standardized tool use: every "read file," "run command," or "change code" is at the API layer a /v1/messages request with a tools array, and the model needs to return structured tool_use blocks, which the client executes and then feeds back as tool_result. This means the endpoint must not only be able to chat, but also understand and generate this JSON schema. vLLM's default chat completion path is only responsible for ordinary text generation—it won't translate tool definitions into the model's prompt template, nor will it parse model output back into tool call blocks, so requests with tools are rejected outright. --enable-auto-tool-choice is responsible for enabling this path, while --tool-call-parser specifies which parser to use to split the model's raw text into tool calls according to a specific format. Different model families (Qwen, Llama, Mistral, etc.) have different output formats, and choosing the wrong parser will also fail.

Pitfall 2: Model Names Don't Match Up

This pitfall triggers on the very first prompt. Claude Code sends Claude's model name by default, and it also silently calls a smaller model for background work. An endpoint serving only one open-source model has no idea what claude-something is.

On that endpoint, the manager checks the requested model name and only accepts the workload slug, its own model name, default, or no model name at all. Any other value gets a 400 with the error "Model '…' is not available on this endpoint." So every model role Claude Code knows about must point to the same name: the main model, the background small model, and the sonnet and opus aliases—otherwise the /model selector and opusplan will still send claude-* names.

In other words, you need four environment variables, plus the base URL and token.

Pitfall 3: Base URL and Auth Headers

Next to the model name, there are two more small traps.

The base URL should stop at the workload layer. Claude Code appends /v1/messages itself, so if you paste in the full route, /v1/messages will appear twice in the path.

About the key: ANTHROPIC_AUTH_TOKEN uses Authorization: Bearer; ANTHROPIC_API_KEY uses x-api-key and also works, but it overrides subscription login, and interactive Claude Code will ask you to confirm a custom key once. If both are set with different values, the manager rejects the request with 401. Just pick one.

Why the Model Fits on One GPU

There are two separate concepts, and it's clearer to look at them separately. The weights are a 4-bit AWQ quantization published by cyankiwi on Hugging Face, compressing 80B parameters to about 40GB, leaving about 40GB for the KV cache. The MoE architecture is the source of the speed: all 80B weights stay resident in VRAM, but each token only passes through about 3B of them.

The server is configured with the model's native context of 262,144 tokens, and the KV cache is stored in FP8 so it fits alongside the weights, handling at most two sequences simultaneously. That last number is honest: this is a single-developer environment.

AWQ Quantization and MoE's VRAM Economics

AWQ (Activation-aware Weight Quantization) is a post-training quantization method that doesn't just look at the numerical distribution of the weights themselves, but also references activations to determine which channels are more important, thereby preserving accuracy as much as possible when compressing to 4-bit. 80B parameters stored at 4-bit is roughly 0.5 bytes per parameter, and after adding quantization scaling factors and other overhead it lands in the 40GB range, leaving just enough space for the KV cache. The KV cache is the historical key-value cache during inference, which grows linearly with context length and the number of concurrent sequences, so the author stored it in FP8 and pushed concurrency down to two. MoE (Mixture of Experts) splits the feed-forward layer into multiple experts, and each token activates only a small subset of them—about 3B here. Note that MoE saves compute, not VRAM: all 80B weights still need to stay resident, which is also why quantization is the prerequisite for running this on the card.

What This Experiment Didn't Prove

Getting it running only proves the pipeline works, not the quality. A landing page isn't the same as refactoring a large legacy system; this is a 4-bit quantized model, and the quality loss wasn't measured; nor was there a comparison against hosted Claude on the same tasks. The 256K context was configured, but never truly maxed out—only 31.7K tokens were used mid-session. Only a smoke test was done on the A100, so there's no throughput data. And this is Claude Code driving Qwen, not Claude—agent behavior depends on how well the model follows Claude Code's tool protocol.

The cost comes with the same caveats. Paying by GPU hour ($1.79/h for A100 that day) is only more cost-effective than paying by token when multiple developers or agents keep the GPU fully utilized.

The Boundaries of a Smoke Test

A smoke test means only verifying that the system can start and run through one minimal path, without evaluating performance and correctness. It answers "does the pipeline work," not "is it good to use." To judge whether a self-hosted setup can replace a hosted API, you usually also need: quality comparisons on a fixed task set (such as code pass rate and instruction-following), throughput and first-token latency under different concurrency, VRAM and degradation curves under real long-context load, and measurements of the accuracy gap before and after quantization. The author explicitly lists these gaps because the success of a single landing page task is very easily misread as general capability—agent-type tasks are extremely sensitive to tool protocol adherence, and the more complex the task, the higher the chance that model format deviations get amplified.

What's Worth Copying

The model comes from the Qwen team's Qwen3 Coder Next, the 4-bit version is cyankiwi's, vLLM handles serving, and Anthropic makes Claude Code compatible with custom endpoints. Connecting these together is the entirety of this experiment.

If your goal is to reproduce it, the core is three things: enable the tool call parser, point all model roles to the same name, and stop the base URL at the workload layer while choosing only one auth method. The rest is tuning detail around this model and this card.

One-line Summary

Three pitfalls and fixes for connecting Claude Code to a self-hosted Qwen model: tool parser, unified model names, base URL and auth.

Paragraph Summary

Using Claude Code to connect to a vLLM-served 4-bit quantized Qwen3 Coder Next (80B MoE), the author got it running and then compiled a failure list. The three main pitfalls are: plain vllm serve doesn't accept tool definitions causing 400, requiring --enable-auto-tool-choice and a --tool-call-parser matching the model family; Claude Code sends claude-* model names by default and silently calls a background small model, so all model roles must point to the same self-hosted name; the base URL should stop at the workload layer, and you should choose one auth method. The article also explains that 4-bit AWQ compresses weights to about 40GB and MoE makes each token compute only about 3B, so a single A100 80GB can run it, and honestly acknowledges that the experiment only verified pipeline connectivity, without testing quality, throughput, or real long-context load.