Migrating from Opus to Self-Hosted Ollama: Lessons Learned with a 35KB Prompt

The Real Challenges of Migrating Long Prompts

Migrating large language model workloads from commercial cloud APIs (like Anthropic's Opus) to self-hosted Ollama is a path many teams are exploring for reasons of cost, privacy, and control. But when the workload involves a large preprompt, things are rarely as straightforward as they seem. This writeup — originally shared on Hacker News — documents the real issues one developer encountered while migrating a ~35KB preprompt to a local Ollama deployment.

A 35KB prompt isn't arbitrary filler. It typically carries complex system instructions, behavioral constraints, few-shot examples, and domain knowledge. That kind of content runs well on a top-tier commercial model like Opus, but when you switch to a locally deployed open-source model, differences in model capability, context handling, and reasoning consistency get amplified fast. The original post didn't get much traction (20 points, 4 comments), but it touches on pain points that many practitioners actually face.

Context Window vs. Prompt Size Mismatch

35KB of text translates to roughly 8,000–10,000 tokens in English — a size that approaches or exceeds the default context window of many open-source models. When running locally, Ollama's effective context length is constrained by both the model's architecture and available VRAM/RAM.

Without explicit configuration, Ollama may silently truncate content that exceeds the default window. That means the carefully crafted second half of your prompt might never reach the model at all. There's no error — the model just "doesn't see" your full instructions, and the output drifts in ways that are hard to diagnose. This is one of the most subtle and damaging traps in the migration process.

In practice, you need to explicitly set a larger context window using parameters like num_ctx, and confirm that your target model actually supports that length. Keep in mind that longer context significantly increases memory usage and inference latency.

Capability Gap and Behavioral Drift

Opus is a top-tier, large-parameter commercial model. Its instruction-following ability, long-context comprehension, and reasoning consistency far outpace most locally deployable open-source models. Complex constraints that Opus can accurately parse and execute may only be partially understood by a 7B or 13B local model.

This means migration isn't a simple copy-paste. You'll often need to rewrite the prompt specifically for the target model. Tasks that relied on the model's strong implicit reasoning now need to be broken down into more explicit, structured step-by-step instructions. The value of few-shot examples increases, while sections that depend on abstract descriptions need to be made more concrete.

In short, a prompt is an engineering artifact tightly coupled to a specific model's capabilities. Cross-model migration is fundamentally a re-tuning exercise for a new target.

Format and Special Token Compatibility

Different model families use different chat templates and special tokens. Anthropic's Opus has its own conventions around prompt structure and a preference for XML-style tags, while the open-source models behind Ollama (Llama, Mistral, Qwen, etc.) each have their own template conventions.

Directly porting a prompt optimized for Opus may cause parsing confusion on local models. The handling of boundaries between system prompts and user messages, the syntax for role markers, and the model's sensitivity to structured tags all vary. When migrating, you need to inspect the Modelfile template for your target Ollama model and ensure that the structural elements of your prompt are injected correctly.

A Pre-Migration Checklist

Drawing from this kind of practical experience, here are key things to verify when migrating a long prompt across models:

• Confirm the context window: Know the maximum token count your target model supports, and set it explicitly via parameters to avoid silent truncation.
• Assess hardware resources: Long contexts and large models have significant VRAM and RAM requirements — benchmark inference latency in advance.
• Rewrite, don't just copy: Based on the target model's capability level, make implicit constraints explicit and abstract descriptions concrete.
• Align to the prompt template: Check the target model's chat template and special tokens, and adjust your prompt structure accordingly.
• Build regression tests: Use a set of representative inputs to compare output quality before and after migration, and quantify the capability gap.

The Hidden Costs Beyond Savings and Control

The appeal of self-hosted Ollama is real: data stays on-premises, there are no API call fees, and you have full deployment control. But this experience reminds us that those benefits come with underestimated migration costs and operational complexity.

Prompt engineering is deeply coupled to the underlying model. Once you swap out the model, the tuning work you built up on a commercial API may lose much of its value — and you'll need to invest additional engineering effort to re-adapt. For use cases that depend on long prompts and complex instructions, migrating from a top commercial model to a local open-source model is fundamentally a trade-off between cost, privacy, and output quality.

For teams evaluating a similar migration path, the most practical advice might be: start by running your real prompts against the target model at small scale. Let the data show you the capability gap before you decide on a strategy — whether that's a full replacement, or a hybrid approach that keeps the hardest tasks on a commercial model.