Unverified50% confidenceFactExact time
On an 8GB VRAM GPU, this user runs stably at a 32k context; pushing to a 42k context left only about 130MB of headroom before VRAM crash, and 64k was not possible.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/20/2026
First Seen
Valid until: 1/18/2027
Sources
Related Entities
Related Claims
UnverifiedCompressing the context from 128K to 32K may quadruple the number of concurrent users a single GPU can serve, representing a trade-off between throughput and context length.77% similarUnverifiedIn the test, model loading took only a few seconds, GPU usage dropped from 23GB to about 17GB, the first test prompt processing took 225ms, generation speed was stable at 33.5 tok/s, with a total output of 1500 tokens.76% similarUnverifiedRunning a 16-bit precision 70B model on 2×80GB GPUs leaves only 2.9 GB of VRAM after the weights are loaded, accommodating only about 9,000 in-flight tokens, which in practice cannot serve requests effectively.76% similarUnverifiedThe full 763B FP8 weights are split into 48 shards totaling about 510GB, which consumer hardware struggles to accommodate; 4-bit quantization is relatively stable, while 3-bit produces infinite loops.74% similarUnverifiedOn a consumer-grade GPU with 8GB of VRAM, ActionAssist achieves an inference speed of 122 tokens/second using Q6_K_L quantization with a 32k context window and 8-bit KV cache quantization.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1078494API
curl https://kongchang.com/api/v1/knowledge/claims/1078494MCP
get_claim(id=1078494)