Structured Information Extraction with Local 7B Quantized Models: Root Cause Analysis and Engineering Optimization

Background: Why Run Long-Document Information Extraction Locally

A developer recently shared a technical challenge on Reddit: how to accurately extract structured information from complex technical documents — such as insurance and financial contracts — using limited local hardware. This scenario is increasingly common. For data privacy and cost reasons, more and more enterprises want to deploy lightweight LLMs locally for document processing, rather than relying on cloud-based APIs.

The demand for local LLM deployment is especially acute in data-sensitive industries like finance, healthcare, and law. Under regulations such as GDPR and China's Data Security Law, transmitting sensitive contract data to third-party cloud APIs (e.g., OpenAI, Claude) can create compliance risks. Meanwhile, the token-based pricing of cloud APIs makes them prohibitively expensive for high-frequency, large-batch document processing — GPT-4o, for instance, costs roughly $5 per million input tokens. A 20-page PDF contains approximately 20,000 tokens, so processing thousands of contracts can easily run into thousands of dollars. This is the core driver behind the rapid rise of local inference frameworks like Ollama and llama.cpp.

The project's central goal is to compare two documents — one being the "predecessor" of the other, meaning a new document was manually edited from an older one. Because human editing introduces potential errors, the project needs automated extraction to identify field-level differences between the two documents. The documents are not small: one is under 10 pages, the other exceeds 20, and the field structure changes dynamically over time.

Breaking Down the Current Technical Approach

The developer's tech stack reflects classic "hardware-constrained" trade-offs:

Model: Qwen 2.5 7B, 4-bit quantized, 8K context window, ~4.8GB model size
Hardware: 16GB RAM, which can only accommodate models under 5GB
Preprocessing: Both documents converted to Markdown format
Field definitions: 60–70 fields defined for the POC phase, stored with descriptions in a JSON file
Extraction strategy: Full document + JSON in a single call for the smaller document; chunked extraction for the larger one
Output constraints: llama.cpp used to force JSON-format output

Qwen 2.5 is an open-source LLM series released by Alibaba's Tongyi team. The 7B variant is notable among models of its size for instruction-following and structured output capabilities, with particular strengths in Chinese and code tasks. 4-bit quantization compresses model weights from their original 16-bit or 32-bit floating-point representation down to 4-bit integers, reducing model size by roughly 75%. For Qwen 2.5 7B, this brings the FP16 footprint of ~14GB down to ~4.8GB, making it runnable on consumer-grade GPUs or standard RAM environments. The trade-off is reduced inference precision — quantized models are more prone to errors in complex reasoning chains and scenarios requiring strict format adherence, which is one root cause of poor extraction quality in this case.

The core problem encountered: even on the smaller document, extraction quality is poor regardless of how the prompt is adjusted. This seems counterintuitive — a small document should be easier to handle when it fits entirely in context.

Why Does the Smaller Document Still Perform Poorly?

There's a non-obvious technical reason here. The issue likely isn't "document size" but rather the complexity of extracting 60–70 fields all at once. For a 4-bit quantized 7B model, simultaneously handling "understand a long document + remember 60+ field definitions + generate strict JSON" is an enormous cognitive load. Quantization already degrades precision, and multidimensional task complexity amplifies that loss further.

It's also worth noting that the 8K context window itself is a hidden bottleneck. 8K tokens can hold approximately 6,000–8,000 English words, or roughly 12–16 standard A4 pages — seemingly sufficient. However, in practice, the context must include not just the document content, but also descriptions for 60–70 fields (at 50–100 words each, that's already 3,000–6,000 tokens), the system prompt, and output format examples. This means the actual token space available for document content may be only 2,000–4,000 tokens — far too little to fit even the "small" document in full.

Optimization Strategies: Improving Extraction Without Upgrading the Model

The developer has explicitly ruled out upgrading to a larger model, so all optimizations must work within the existing constraints. Here are several actionable directions.

1. Split the Extraction Task to Reduce Per-Call Complexity

Instead of extracting 60–70 fields at once, batch fields into groups — for example, extracting only 10–15 related fields per call and merging results across multiple rounds. This increases the number of inference calls but significantly reduces per-call complexity. Quantized small models perform far better on focused tasks than on broad, catch-all prompts.

2. Optimize Context Length and Chunking Strategy

An 8K context window isn't generous for a 20-page document. Recommendations:

Use semantic chunking instead of fixed-length chunking to avoid cutting off critical fields mid-sentence
In each chunk, include only the subset of fields relevant to that passage, not the full field list
Adopt a RAG (Retrieval-Augmented Generation) approach: use vector search to locate the passages where specific fields are likely to appear, then feed those passages to the model for targeted extraction

Semantic chunking is an improvement over fixed-size chunking. Fixed-size chunking splits by character or token count, which easily truncates sentences or field descriptions mid-way, breaking contextual continuity. Semantic chunking instead detects paragraph boundaries based on shifts in sentence embedding similarity, ensuring each chunk is semantically coherent. In this context, RAG works as follows: all document chunks are indexed in a vector database (e.g., ChromaDB, FAISS); when extracting a specific field, the field description is used as a query vector to retrieve the most relevant document passage; that passage and the field definition are then sent together to the model. This "precision feeding" strategy dramatically reduces irrelevant-text noise while sidestepping context length limitations.

3. Strengthen Structured Output Constraints

The developer is already using llama.cpp to force JSON output — that's the right direction. Further improvements include:

Use GBNF grammar constraints (natively supported by llama.cpp) to precisely define the JSON schema, rather than simply asking for "JSON output"
Provide example values and format notes for each field, using few-shot examples to guide extraction behavior
For enum-type fields, directly constrain the set of allowed values to reduce model free-form generation

GBNF (GGML BNF) is a grammar constraint system implemented in llama.cpp, based on BNF (Backus-Naur Form) — the standard notation for describing formal language grammars in computer science, widely used in programming language parser design. In LLM inference, GBNF intervenes at the decoding layer by masking logits (the model's raw probability distribution over tokens) at each generation step, forcing the model to produce only token sequences that conform to the predefined grammar rules. Compared to simply instructing the model to "output JSON" in the prompt, GBNF constraints operate at the decoding level and fundamentally eliminate the possibility of malformed output — the model physically cannot generate content that violates the schema. This is especially valuable for quantized small models, which are more prone to hallucination and format corruption under complex formatting requirements.

4. Choose a More Appropriate Quantization Configuration

4-bit quantization is a reasonable choice given a 5GB size limit, but there's still room to optimize:

Prefer Q4_K_M over plain Q4 — the former retains higher precision in key weight layers
If memory allows, try Q5_K_S, which adds modest size overhead but offers a meaningful precision improvement
Evaluate whether a same-size instruction-tuned model better suited to extraction tasks exists

llama.cpp supports multiple GGUF quantization formats. In the naming convention, K denotes K-quant (K-means quantization), and M/S/L indicate quantization intensity (Medium/Small/Large). The key difference between Q4_K_M and plain Q4 is that K-quant partitions weight matrices into groups and applies mixed precision — more important weight blocks in attention and feed-forward layers are stored at 6 bits, while others use 4 bits, averaging approximately 4.5 bits overall. In practice, Q4_K_M achieves perplexity scores (a measure of language model prediction accuracy) roughly 0.1–0.3 lower than plain Q4, with more pronounced advantages on structured output and long-text comprehension tasks. For a 16GB RAM machine, Q5_K_M (~5.3GB) is generally the optimal precision-to-size trade-off and is worth prioritizing.

Deeper Reflection: Engineering Wisdom in the Era of Small Models

This case illustrates a universal pattern in local LLM applications: when hardware is constrained, engineering strategy matters more than model scale. A 7B quantized model, paired with well-designed task decomposition, precise context management, and strict output constraints, is entirely capable of reaching a practical level of quality for specific information extraction tasks.

For document comparison scenarios in particular, it's worth introducing non-LLM auxiliary methods: use rules or regular expressions to handle structurally clear fields (e.g., dates, amounts, IDs), and only pass semantically ambiguous fields to the model. Such "hybrid pipelines" are typically more stable and efficient than pure-LLM approaches.

Furthermore, the dynamic nature of field definitions suggests the extraction system should be designed as configuration-driven — decoupling field definitions from extraction logic so that adding or modifying fields requires only updating the JSON config, not touching core code. This is exactly the right instinct behind the developer's current approach of storing field definitions in JSON, and it's worth continuing and refining.

Summary

Extracting structured information from complex documents on constrained hardware is a real challenge many developers face today. The core lesson can be distilled into a single principle: don't expect a small model to do everything in one shot — instead, use task decomposition, semantic retrieval, structural constraints, and hybrid pipelines to break complex tasks into sub-tasks that a small model can handle competently. When model capability is limited, the value of good engineering design truly comes to the fore.