80GB H100 Tested: A 35B Model Can Chat on Just 3B of Compute

In discussions about AI infrastructure, a common misconception equates "whether a model fits into VRAM" with "whether a model can hold a smooth conversation." One developer's hands-on experience deploying a 35B-parameter model on an H100 with 80GB of VRAM reveals the vast gulf between these two questions.

Ample VRAM Doesn't Mean Smooth Conversation

The developer used an H100 GPU with 80GB of VRAM, with a hardware cost of roughly 5 million yen (about $35,000). Intuitively, this setup should easily run a large model and carry on fluid conversations.

In reality, however, he could only effectively run a 3B-parameter model — and that didn't even include voice functionality. The full model has 35B parameters, but at each token generation step, only the equivalent of 3B weights are actually used.

The core issue is this: conversational fluency depends on response latency. If the wait between question and answer is too long, the conversation feels unnatural no matter how clever the answer is. Large models require a lot of computation per token, which directly affects response speed.

Whether it fits into VRAM and whether it's suitable for conversation are two completely different questions.

Why is the compute cost per token so enormous for large models?

An autoregressive language model generating a single token requires one forward pass through the entire model. Take a dense architecture: a 35B-parameter model must multiply all 35B weights against activations for every token generated. The compute scales roughly with "parameter count × input tokens," while VRAM usage is mainly determined by "parameter count × bytes per weight precision."

This distinction means: VRAM only determines whether the model can be "put down," while compute determines how long each token has to wait. A 35B model stored in FP16 takes about 70GB of VRAM — barely squeezable into an 80GB H100 — but each token needs roughly 7 billion or more multiply-add operations, and latency on a single card can reach hundreds of milliseconds or even seconds. That is precisely what makes conversation stutter. By comparison, quantized weights and KV cache management also affect latency, but neither can change the fundamental constraint that "a dense model must activate all parameters for every token."

Why Small Models Can't Handle Conversational Tasks

To reduce compute cost and improve response speed, the developer tried smaller models. But what he was building was a conversation routing classifier, which needs to determine the "request type" from a single user utterance.

Small models perform poorly at understanding contextual continuity. For example, the sentence "run the same thing again this time with --watch" depends on the context of the previous operation. Treating it as an isolated new request easily loses the intent.

In classification tasks, the differences between models are especially pronounced on utterances like "a continuation of the previous operation." Simply shrinking the model size can't simultaneously preserve ideal response speed and intent understanding.

The Key Advantage of MoE Architecture

Keeping the overall model large while reducing the compute per word — MoE (Mixture of Experts) architecture offers a way to do this.

MoE handles input by selecting the optimal expert from multiple expert networks, selectively activating only the necessary subnetworks. The developer adopted the Qwen3.6-35B-A3B model: it has 35B weights in total, but uses only 3B weights when processing each word.

The model's compute cost is comparable to an ordinary 4B or 8B model, and it runs on his own gateway (vLLM).

Measured Data Comparison

On the task of determining the request type of user utterances:

• Qwen3.6-35B-A3B: 92.6% accuracy, 306ms decision time
• Ordinary qwen3:8b and qwen3.5:4b: about 82% accuracy

The gap is even more pronounced on "continuation of the previous operation" utterances: 35B-A3B reaches 86.3% accuracy, while the 4B and 8B models manage only 70.4%.

Short utterances in conversation often depend on prior context, and accurately understanding these is the key reason for choosing this model.

These figures, however, only reflect text classification results and do not include speech recognition (STT) or speech synthesis (TTS). For voice conversation, additional compute time and GPU VRAM would be required, and the exact increase is difficult to quantify.

What exactly does "A3B" mean in MoE?

In names like Qwen3.6-35B-A3B, "35B" refers to the total parameter count, while the "A" in "A3B" stands for Activated — meaning roughly 3B parameters actually participate in computation during each forward pass. MoE layers typically consist of several "expert" feedforward networks and a routing network, which decides based on the current token which experts to assign it to (usually top-k, where k is often 2 or 4).

The result: total parameters can be made very large to enhance expressiveness, but the compute per token scales only with activated parameters, so throughput and latency are close to a 3B dense model. The trade-off is that VRAM must still hold all expert weights, and routing, load balancing, and expert parallelism add extra engineering complexity — which is why MoE models tend to be better suited to scenarios with ample VRAM but latency-constrained compute.

Why is a conversation routing classifier so sensitive to context?

A routing classifier is essentially an intent classification task: it must decide which tool to call or which pipeline to follow when the user says only half a sentence, or even just adds "the same thing again." The semantics of such utterances depend almost entirely on the preceding turns of dialogue history. Once a model treats the current sentence as isolated input, it misclassifies "continuation" as a "new request."

Smaller models fall off more sharply on this kind of task because they have weaker modeling ability for coreference, ellipsis, and omitted subjects in long contexts. Researchers often evaluate this ability separately using "continuation utterance accuracy," because it can expose edge cases masked by average accuracy. In real conversational systems, such misclassifications often hurt experience more than a single wrong answer, because they can lead to repeated execution or incorrect tool calls.

Multi-Dimensional Considerations for Conversational System Selection

The developer previously wrote about a 720p30 real-time streaming scheme where "a single GPU can host four AI avatars." Even then, the GPU was not the bottleneck.

When discussing LLMs, looking only at GPU VRAM ignores many key factors:

• Can the model fit into 80GB?
• How much compute is needed per token?
• What is the latency from receiving text to returning a response?
• If interacting via voice, the time and VRAM overhead of STT and TTS must be added on top

Even with the same GPU, multiple factors together determine whether the system is truly usable for conversation.

The developer's conclusion: GPU capacity and conversational responsiveness must be considered from different angles. Even if the total model reaches 35B, if the compute per token is equivalent to a 3B model, it remains a viable candidate. Simply switching to a smaller model is not enough to maintain accuracy.

When choosing a GPU, focus on "how much compute is done per token," not just the size of VRAM.

One-line Summary

80GB of VRAM can hold a 35B model, but conversational fluency depends on compute per token — MoE is the key solution.

Paragraph Summary

Based on one developer's hands-on experience deploying a conversational system on an 80GB H100, this article points out that "a model fits into VRAM" and "a model is suitable for conversation" are two different questions. A dense 35B model can barely fit into VRAM, but its compute per token is too large, causing excessive response latency and preventing smooth conversation. The author tried smaller models to speed things up, but found that their ability to understand contextual continuation in the conversation routing classification task dropped noticeably. Ultimately he adopted the MoE-architecture Qwen3.6-35B-A3B, which has 35B total parameters but activates only about 3B weights per token, with a compute cost close to a 4B–8B model. In testing, this model achieved 92.6% classification accuracy and 86.3% on continuation utterances, clearly outperforming the 8B and 4B small models. The conclusion is that GPU selection should focus on compute per token rather than VRAM alone, and voice conversation requires additional consideration of STT and TTS overhead.