Deconstructing Agents: Task Decomposition, Context Compression, and the Case Against Heavy Frameworks

What Is an Agent: An LLM + Tools Task Decomposition Engine

In today's AI application boom, "Agent" has become a label that nearly every product wants to attach to itself, yet many people's understanding of it remains at a purely conceptual level. This [HuMa] video dissects the essence and inner workings of Agents from an engineering implementation perspective.

Put simply, the essence of an Agent is an LLM (Large Language Model) executing various operations through tool invocation. It breaks a long task into multiple steps, orchestrates each step, and then executes the specified operations. Take "write a tutorial" as an example—an Agent would first collect your original intent, then organize it, and finally generate the content. This is a simplified illustration; the real process is far more complex, but it suffices to demonstrate the core working model of an Agent: task decomposition + step orchestration + tool invocation.

To understand why an Agent can "do things" rather than merely "talk," you need to understand the technical foundation of tool calling. At its core, a large language model is a probability prediction model trained on massive amounts of text. It excels at language understanding and generation but inherently lacks the ability to perform external operations—such as querying databases, calling APIs, or manipulating file systems. Tool Calling (or Function Calling) is a capability that OpenAI first standardized in mid-2023, allowing models to output structured function call requests during inference, which are then executed by external systems that return results for the model to continue reasoning. This mechanism is the technical foundation that enables Agents to break beyond the boundaries of pure text generation.

There's a key insight to establish here: AI is not magic. You can't just say "I want a tutorial" and expect it to produce a perfect result out of thin air. This is precisely why prompts are so important, and it's the fundamental reason various guidance mechanisms exist.

Skills, Constraints, and Priority: How AI Gets Guided

The video repeatedly mentions the role of "skills" (guidance mechanisms). The author's viewpoint is straightforward: a skill is essentially a guidance tool whose core function is to guide AI in thinking about problems. There's nothing mysterious about it—it's often crudely injected during each context compression cycle.

Regarding where to place constraints, the author provides a clear priority ranking:

• System prompt has the highest priority—because it's always carried along; this mechanism has existed since AI's inception
• Skills come second—because AI vendors perform a recall (recall skill) before execution, reading it first before proceeding, which yields better results
• Background documents have the lowest priority—because constraints within background documents get treated by the AI as "context" rather than "constraints," receiving very low weight

The system prompt holds the highest priority for deep technical reasons. In implementation, the system prompt is the first message in the conversation sequence, marked with the role "system." From the model's inference perspective, it sits at the forefront of the attention mechanism and is carried in full during every conversation turn, participating in attention computation. Due to the causal attention mechanism in Transformers, where early tokens exert influence over all subsequent tokens, and since the system prompt is re-read during every inference pass, its constraining power is the strongest. By contrast, content inserted mid-conversation gets "diluted" in attention weights as the conversation grows longer.

This ranking has significant practical implications: if you want a certain constraint to be strictly followed, you should place it in the system prompt rather than burying it in background documents and hoping the AI will comply on its own. The author also acknowledges that the recall rate, accuracy, and underlying routing mechanism design for skills are quite complex and worth a dedicated discussion.

Additionally, there's another important characteristic: later instructions have higher priority than earlier ones. This explains why guiding AI to do multiple things mid-task actually leads to forgetting and information loss—AI's attention is limited; it can only focus on one thing at a time and cannot simultaneously handle multiple tasks. This also indirectly confirms why skill-style guidance is effective.

Why Context Compression and Memory Mechanisms Are Necessary

Why do Agents need context compression and memory mechanisms? The author provides two fundamental reasons.

First, AI is inherently stateless. Large models compress the entirety of the world's possibilities, and their natural tendency is to "diverge," thus requiring external mechanisms to anchor state.

Second, context length is finite. This won't surprise anyone familiar with the Transformer architecture: Transformer's space complexity is on the order of O(n²) (this is an abstract expression; various optimization algorithms can reduce it in practice, but the overall growth remains exponential). This means no machine exists that can store unlimited context.

Specifically, the core of the Transformer architecture is the Self-Attention mechanism, which needs to compute the association weights between every token in the sequence and all other tokens. For a sequence of length n, the attention matrix is n×n in size, making both space and computational complexity O(n²). This means that when context expands from 4K to 128K, computational resource requirements increase by approximately 1024x. While optimizations like FlashAttention, sparse attention, and linear attention (such as state space models like Mamba) exist, they are essentially making tradeoffs between precision and efficiency, and have not fundamentally eliminated the resource bottleneck of long contexts.

Interestingly, the author specifically notes that this is a hard limitation of the current Transformer architecture, but it doesn't mean the problem is unsolvable in the future—better architectural choices actually exist, but for various reasons the mainstream has currently settled on Transformers. However, even switching architectures wouldn't change the fundamental problem of information compression and information loss; newer architectures simply have better resource utilization.

This is precisely why Agents introduce memory mechanisms. From the initial era of "one-click coding" to the current need for manually written constraints, project memory, and knowledge graphs—the underlying goal is always to enable AI to explore and collect knowledge on its own. But there are clear technical bottlenecks here.

The Heavy Framework Trap: A Warning for Individual Developers

The author takes a quite cautious stance toward the various complex solutions currently in vogue—SOPs (Standard Operating Procedures), task graphs, Hooks frameworks, AOP (Aspect-Oriented Programming), lifecycle management, and so on.

The core argument is: AI cannot handle these overly complex mechanisms on its own. If you force AI to complete them autonomously, it will lead to only two outcomes:

1. Poor results—AI's attention is limited and cannot juggle multiple things simultaneously
2. Low efficiency—relying on multi-turn interactions, introducing large amounts of extra processes and additional Agents

Two Fatal Problems with Knowledge Graphs

Taking knowledge graphs as an example, the author identifies two fatal problems. First, knowledge graphs have been worked on for many years and still haven't been done well—their complexity grows exponentially, and AI cannot maintain them autonomously.

To appreciate the weight of this assertion, one needs to understand the historical background of knowledge graphs. A Knowledge Graph is a way of organizing knowledge using graph structures (nodes + edges) to represent entities and their relationships. After Google proposed it in 2012, it became widely applied in search and recommendation systems. The difficulties lie in: the accuracy of entity recognition, the completeness of relation extraction, and the maintenance complexity that grows exponentially as the graph structure expands with data. Even companies like Google and Microsoft, with their massive engineering resources, face issues of insufficient coverage and update lag in their knowledge graphs.

As a comparison, AST (Abstract Syntax Tree) parsing is also a directed acyclic graph, but it works effectively because the complexity is handled by "infrastructure." An AST is a standard data structure in programming language compilers/interpreters that parses source code into a tree structure according to grammatical rules. ASTs are reliable because programming language syntax is strictly defined and deterministic, with parsing rules fully covered by compiler infrastructure—there's no ambiguity. Graphifying natural language knowledge, however, faces fundamental difficulties of semantic ambiguity and polysemous relationships—this is the essential difference between knowledge graphs and ASTs.

Second, knowledge graphs are relatively effective for knowledge with hierarchical dependency relationships, but nearly impossible to maintain for implicit associations between sibling branches—for example, the relationships between parallel branches like physics and chemistry under science.

The Retrieval Dilemma of Project Memory

Regarding project memory, the core contradiction lies in the retrieval problem. To avoid context explosion, AI vendors generally adopt a design of "locally extracting relevant paragraphs." This does alleviate the problem, but only alleviates it—the cost is retrieval loss and modification loss. To solve it completely, there are only two paths:

• Choose multi-turn interactions—leading to enormous token waste and low efficiency
• Introduce heavy frameworks (such as RAG or self-breakthrough mechanisms)—with extremely high maintenance costs, complexity bloat, and debugging difficulty

Regarding the RAG approach, its core process is: slice documents, convert them into vectors via embedding models, store them in a vector database, and when users ask questions, first find the most relevant document fragments through semantic retrieval, then inject these fragments as context for the LLM to generate answers. RAG's limitations include: difficulty controlling slice granularity (too large introduces noise, too small loses context), a ceiling on semantic retrieval recall rates, difficulty establishing cross-document logical connections, and the fact that sorting and deduplication strategies for retrieval results all affect the final outcome. For individual developers, maintaining a high-quality RAG pipeline requires ongoing data cleaning, index optimization, and effectiveness evaluation.

The author's advice to individual developers is very clear: don't touch these heavy frameworks. Three reasons: first, the returns don't meet expectations—the input-output ratio is unbalanced; second, these products themselves are often written by AI, and once bugs appear they're extremely difficult to fix or even identify; third, the maintenance cost is an enormous burden for individuals. If you're a company, you can weigh the costs and consider it—these things "look beautiful and intuitive, but individuals shouldn't attempt them."

Summary: The Essence of Agents and Topics for the Future

Returning to the original question—what exactly is an Agent? The author provides a clear summary:

An Agent is something that uses a large model to decompose tasks on its own, orchestrate steps, and can insert prompts at intermediate steps for guidance (the AI adjusts its steps accordingly), while introducing memory and context compression mechanisms.

Memory and context compression are introduced because the model's capabilities are inherently limited; and SubAgents (child Agents) are introduced because the main Agent's context is equally limited—when the task granularity it can handle becomes too large, it becomes overwhelmed and can only be relieved by splitting into SubAgents.

SubAgents are a common architectural pattern for solving single-Agent capability bottlenecks. When a complex task exceeds a single Agent's context window or attention capacity, the main Agent can delegate subtasks to specialized SubAgents, each with its own independent context space and toolset. This design is similar to the microservices architecture in software engineering—managing complexity through separation of concerns. But the cost is introducing inter-Agent communication overhead, state synchronization challenges, and the complexity of error propagation and debugging. Typical implementations include AutoGen's multi-Agent conversation framework, CrewAI's role-division model, and others.

The author also previews topics to be covered separately in the future: SubAgent and workflow issues, skill routing and debugging mechanisms, deeper details of memory compression, and the historical evolution of the Transformer architecture. Each of these is substantial enough to stand alone as a full article, demonstrating that the complexity of Agent engineering far exceeds surface-level understanding.

For developers looking to build Agent applications, the greatest value of this content lies in establishing the right engineering intuition: understanding AI's capability boundaries (limited attention, limited context, one-step reasoning) enables pragmatic decisions in constraint design, memory mechanisms, and framework selection—rather than blindly chasing complex solutions that look impressive but fail to deliver expected returns.

Key Takeaways