A Week of Coding with Local LLMs: Can They Replace Cloud Claude?

As open-source local models inch closer to top-tier cloud models on benchmarks, a practical question emerges: can they actually do our work? One developer spent a week testing this on a consumer-grade mini PC, plugging local models into an IDE, CLI, and AI agents to map out where they can genuinely be used.

Test Platform: A 64GB Consumer Mini PC

The test machine was an ASUS NUC 16 Pro mini PC, equipped with an Intel Ultra Core X7 processor (16 logical cores) and 64GB of RAM. On the software side, Ollama and LM Studio were used to load local models.

Model selection referenced arena.ai's coding and general-capability leaderboards, but the real constraint was memory: stronger models on those lists routinely require hundreds of GB of RAM, far beyond the 64GB consumer ceiling. The final picks were the 27B-parameter version of Qwen 3.8, Meta's Muse Glimmer, and GLM 4.7 Flash, which can be accommodated stably on the hardware. OpenAI's 120-billion-parameter open-source model was ruled out outright for lack of memory—the author later reflected that perhaps he should have just gone with 96GB.

Downloading the models required an overnight session. The quantization level (Q8/Q6/Q4) determines the trade-off between precision loss and size, while GPU offload support directly determines whether inference speed is acceptable.

Ollama and LM Studio are currently the two most mainstream tools for running local LLMs, with slightly different positioning. Ollama centers on the command line and API, with a built-in model repository—one command pulls and runs a model—making it suited to scripted invocation and integration with various development tools. LM Studio offers a graphical interface, emphasizing model browsing, parameter tuning, and visual configuration of a local API service, plus an OpenAI-compatible endpoint that VS Code extensions or third-party clients can point to. Both rely on quantized formats like GGUF and the llama.cpp inference backend, and both support offloading some or all model layers to the GPU for acceleration. The native VS Code "Manage Language Models" feature described in this article works precisely by pointing a custom endpoint at LM Studio's local service.

Quantization levels (Q8/Q6/Q4) refer to the numerical precision used to store and compute model weights. Q8 means weights are represented at roughly 8 bits, while Q4 compresses them to about 4 bits. Lower bit counts mean smaller model size and memory footprint and usually faster inference, but greater precision loss, manifesting as degraded answer quality, rigor of reasoning, or long-range consistency. For consumer hardware, quantization is the key means of fitting large-parameter models into limited memory: the same 27B-parameter model may need over 30GB of RAM at Q8, but can be compressed to a dozen-odd GB at Q4. This also explains why the article describes downloading overnight first, then trading off precision against size.

Speed Test: A Gap of 6 to 22 Tokens/Second

On the second day, the three models' inference speeds were tested, and the differences were stark:

• Qwen 3.8 (27B): about 12 tokens/second
• Muse Glimmer: about 6 tokens/second, with much of its thinking spent on "ensuring compliance with guidelines and not overstepping," taking a long detour even to reply hello world
• GLM 4.7 Flash: about 22 tokens/second, fully loadable into the GPU, and the fastest of the three

The 64GB of RAM brought an unexpected advantage: it can run two GLM 4.7 Flash instances in parallel, something the author's own higher-spec but lower-memory PC couldn't do.

Running in VS Code Works, but Context Is the Achilles' Heel

There are two ways to hook local models into VS Code: third-party extensions like Continue or LM Studio's Copilot extension, or VS Code's native "Manage Language Models" feature—pointing a custom endpoint at LM Studio lets you see local models in the chat window without logging into any cloud account.

The initial conversation failed for a very typical reason: the tool calls VS Code sent filled up the context window, leaving little room for the conversation itself. The fix was to uncheck all tool calls and expand the model's context length.

The real bottleneck surfaced on the third day. Cloud models routinely claim million-token contexts, while local models default to just 8000—a gap that directly changes how you use them: sessions must be strictly partitioned, and running a bit too long exhausts the context; forcibly enlarging the context slows generation, defeating the purpose. In the author's own words, the 8000-token limit "significantly affects how you use them."

The context window is the upper limit on the total tokens a model can "see" in a single inference pass, including system prompts, tool definitions, conversation history, pasted code, and the model's own output. It's a fixed-capacity working memory; anything beyond it is either truncated or must be manually trimmed and summarized by the developer. The million-token contexts claimed by cloud models mean an entire mid-sized codebase can be handed to the model at once; 8000 tokens amounts to only a few thousand Chinese characters or one or two hundred lines of code, leaving little after system prompts and tool schemas. This is the direct cause of "tool calls filling up the context" and the need to "strictly partition sessions" described in the article. Forcibly enlarging the context also carries two costs: attention computation slows as length grows, and the model's recall of information in the middle of long contexts degrades.

CLI and AI Agents: Small Models Can't Carry Heavyweight Frameworks

Hooking a local model into a mainstream CLI like Claude Code only requires specifying the model and pointing it at LM Studio's address. But a single hello world took three minutes to return—the prompt volume the framework injects into the model far exceeds what a small model can bear. The author concludes that current CLI/harness setups are extremely inefficient for local models.

AI agent scenarios showed a split. With OpenCore connected to Qwen 3.8, replying to a hello world likewise took two or three minutes, making it unusable as a main agent; but demoting it to a sub-agent executing scheduled tasks (cron jobs) makes much more sense—such tasks only check logs or read email, are insensitive to latency, and consume few tokens. The author ultimately used Kimi K3 as the main agent for orchestration, with local models running background subtasks.

Hermes's value lies in a different direction: it can use local models to rewrite its own memory and complete self-upgrades.

After tuning, lightweight models (such as LMF 2.5) can hit over 100 tokens/second on this machine, with quite fluid real-time output. The author also had cloud models (ChatGPT 6 Astra, Claude) remotely control this PC for 2D-to-3D modeling and project development, forming a two-machine division of labor between an "AI-dedicated workstation" and a personal PC—checking progress and adding tasks from a phone while out for coffee.

CLI and agent harnesses refer to command-line tools like Claude Code and OpenCode that wrap LLMs into autonomous task-executing tools. They don't simply forward user input to the model; each turn they inject a large amount of content: system behavior guidelines, available tools and their parameter descriptions, file system and environment state, historical operation records, planning and reflection prompts, and so on. The design premise of this harness is that the model has a sufficiently large context and strong instruction-following ability, so a single request's prompt often runs to thousands or even tens of thousands of tokens. Connecting it to a small model with an 8000-token context and a dozen-odd tokens per second produces a situation where the prompt itself nearly fills the window and generation is slow—a hello world taking several minutes is the embodiment of this structural mismatch.

Main agent and sub-agent is a hierarchical agent architecture: the main agent understands user intent, decomposes tasks, and dispatches them, while sub-agents execute narrowly scoped tasks such as checking logs, reading email, or periodic inspections. Sub-agents have short inputs and outputs, are latency-insensitive, and can retry on failure, making them especially suited to small, slow local models; the main agent needs long context and strong reasoning, so handing it to a cloud model is more cost-effective. The cron jobs mentioned in the article are scheduled tasks triggered periodically by the system, a natural fit for local models running persistently at low load.

Conclusion: The Role of Local Models Is Now Clear

The answer from a week of testing isn't a simple "yes" or "no":

Scenarios usable right now: small agents running 24/7 (like OpenCore), sub-agent execution of agent subtasks, memory maintenance, and self-improvement. Small machines running at low load for long periods are exactly local models' comfort zone.

Scenarios to hold off on for now: developing codebases of thousands or tens of thousands of lines. With the triple constraints of context window, inference speed, and framework overhead stacked together, local models can't yet carry main development work.

The author is optimistic about the trend: model sizes are growing, quantization quality is improving, and in the future phones, laptops, and home computers may all allocate large chunks of memory to local AI. But for now, a hybrid architecture—cloud models as the brain, local models as the hands and feet—is the pragmatic way to deploy.