AI Keeps Going Off-Track? Use Code to Compile Your Materials into Task Context

The Real Problem: You're Giving Materials, Not Context

Have you ever experienced this — you dump a pile of materials to an AI, it seems to have read everything, but the final answer is still off the mark?

Most people's first reaction is "the model got dumber," but the real reason is often simpler: you're giving it raw materials, not task context. Materials are scattered, unfiltered piles of information; context is a structured, credibility-tagged, reliably executable task instruction package. The gap between these two is the key difference between AI going off-track and hitting the mark.

Understanding this problem requires knowing how large language models process input at a fundamental level. Models don't read and filter materials step-by-step like humans do. Instead, they use Attention Mechanisms to simultaneously compute relevance weights for all tokens within the context window. This means an outdated old note and an authoritative log are in a completely equal competitive position — whichever is semantically closer to the question is likely to receive higher weight. When facing input mixed with high-noise content, models are easily "pulled off course" by misleading information, producing outputs that seem reasonable but lack real evidence.

Recently, a developer shared a practical code snippet demonstrating how a "context compilation" script transforms raw materials into high-quality task context, with remarkably significant differences in results.

A Real Case: Troubleshooting Admin Login Failure

The scenario is typical: a customer reports "admin login failed," and the support team needs to determine — is this a customer password issue or a system OAuth Token issue? Ultimately, they need to write a Chinese-language response to send to the customer.

Before diving into the case, it's necessary to understand how OAuth Tokens work. OAuth (Open Authorization) is a widely used authorization framework that allows third-party applications to gain limited access without exposing user passwords. Its core mechanism issues short-lived Access Tokens and Refresh Tokens for renewal. When logs show error_token_expired, it means the Access Token has expired and the system is attempting to exchange the Refresh Token for a new one — this is a completely normal automated process, unrelated to user passwords. A subsequent log entry of token refresh recovered proves the system has automatically recovered. Without understanding this mechanism, it's easy to misdiagnose "login failure" as a credential issue and give an incorrect "reset password" recommendation.

The raw materials include:

• Ticket record: Customer's problem description
• Application logs: Real system operation records
• Handling manual: Standard operating procedures
• Configuration file: System parameters
• OCR text from error screenshot: Text extracted from screenshots
• An outdated old note: Stating "if login fails, have the customer reset their password first"

Here's the critical trap: that outdated old note. If you feed all materials to the AI without differentiation, the phrase "have them reset their password first" can contaminate the answer. The AI will produce a seemingly safe but evidence-free recommendation — telling the customer to reset their password.

Context Compilation: What Does the Script Do?

The script's core approach isn't about calling a large model — it's about performing Context Compilation before calling the model. The entire process has four steps:

1. Scan the evidence directory: Traverse all raw material files
2. Generate an evidence manifest: Create an index and metadata for each material
3. Extract sentinel fields: Pull key error codes, timestamps, and other hard facts from logs and OCR
4. Register conflict risks: Flag contradictions and potentially misleading information between materials

The final output is a structured task_context.md file.

It's worth noting that this approach forms an interesting contrast with the current mainstream RAG (Retrieval-Augmented Generation) architecture. Most RAG implementations only solve the problem of "what was retrieved" while ignoring the deeper question of "whether the retrieved content is trustworthy and whether it contradicts itself." Context engineering is a further evolution on top of RAG — it focuses not only on information recall rate but also on the structural quality, credibility hierarchy, and conflict markers of the injected context. This is why the same model with the same retrieval results can produce qualitatively superior output after context compilation.

Sentinel Fields: Distinguishing "Appeared" from "Currently Happening"

The concept of Sentinel Fields here is crucial. This concept borrows from the "sentinel value" idea in software engineering — using specific markers to distinguish normal data from boundary states. In context engineering, sentinel fields refer specifically to hard facts extracted from raw materials that are unique and timestamp-bound, such as error codes, session IDs, and precise timestamps. The value of these fields lies in their "non-ambiguity" — error_token_expired is error_token_expired, with no room for semantic drift.

The script extracts only one real error code from logs and OCR. Although the handling manual mentions "password"-related content, that's a rule description within the manual's "prohibited assumptions" section, not an actual error from the current incident. Extracting these fields separately and anchoring them as "fact anchors" in the context is equivalent to setting impassable boundaries for the model's reasoning process, preventing the model from engaging in free-association gap-filling when evidence is lacking.

The essence of this step is telling the AI a key principle: just because a word appeared somewhere in the materials doesn't mean it's a fact about this specific incident.

Three-Tier Evidence Credibility Classification

The generated context package classifies evidence into three credibility tiers:

Tier	Contents	Description
High credibility	Ticket, application logs, handling manual, configuration file	Direct sources, can be cited directly
Medium credibility	Error screenshot OCR text	Requires OCR sampling verification, may contain recognition errors
Low credibility	Outdated old note	Only serves as a risk signal, cannot be used as a decision basis

At the same time, the context package anchors key facts:

• Workspace: acme-admin
• Region: ap-east
• Failure time: 09:42
• Error code: error_token_expired
• Subsequent log: token refresh recovered (recovered)

Output Comparison: The Gap from 2 Points to 90 Points

The author designed a script-based rule validation to quantify output quality, checking dimensions including: whether the error code is preserved, whether the recovery status is mentioned, whether the answer cites sources, and whether unsupported assumptions are made.

This rule-based deterministic evaluation framework contrasts with the "LLM-as-Judge" approach that relies on another LLM for scoring. The advantage of rule validation lies in reproducibility and explainability: every rule is a binary judgment, unaffected by biases of the evaluation model itself. This approach is gaining increasing attention in AI application quality assurance (QA) systems, especially suitable for task scenarios with clear factual evidence — for technical support, document analysis, and other structured tasks, rule validation is currently one of the most reliable means of quantifying output quality.

Raw Dump Approach: Validation Score 2/100

The result of directly dumping all materials to the AI:

• ❌ No specific evidence cited
• ❌ Original error code not preserved
• ❌ Gave an unsupported "reset password" recommendation
• ❌ Contaminated by the outdated old note

Task Context Approach: Validation Score 90/100

The AI's answer became:

• ✅ Clearly stated "this looks more like a brief OAuth Token Refresh failure, not a customer password issue"