Getting Legal AI Right: Why the Coding Agent Paradigm Doesn't Translate

Introduction: Legal AI Shouldn't Be a Coding Agent in Disguise

A recent Hacker News thread titled "Legal AI, not a coding agent with scaffolding" didn't make many waves, but it cuts to the heart of a critical question in vertical AI applications: How far can we actually go by transplanting the mature coding agent paradigm directly into specialized fields like law?

The title itself takes a clear stance. It points to a problem the industry has largely ignored: many so-called "legal AI" products are, at their core, nothing more than large language models wrapped in retrieval-augmented generation (RAG) and prompt engineering "scaffolding," then relabeled as "legal" tools. This approach may seem efficient, but it fundamentally misunderstands what legal work actually involves.

Why the Coding Agent Paradigm Works

Over the past two years, coding agents like Cursor, Devin, and Claude Code have made remarkable strides. The underlying logic behind their success is fairly clear:

• Strong verifiability: Whether code runs or tests pass is a binary, instantly verifiable outcome.
• Fast feedback loops: Compilers and test suites provide cheap, accurate reward signals that let agents iterate through trial and error continuously.
• Structured output: Code has well-defined syntax, making it easy to locate and correct model errors.

These three properties are precisely what make the "generate → execute → feedback → correct" agent scaffold so effective in software development.

The core architecture of a coding agent typically consists of three layers: LLM + Tool Use + Feedback Loop. Products like Cursor and Claude Code work by tightly coupling the LLM with a code execution environment — the model generates a code snippet, a sandbox executes it and returns stdout/stderr, and the error output feeds back into the model as context for the next revision. This architecture succeeds because computer science has an axiomatic standard of correctness — Turing-complete formal languages give "right" and "wrong" precise definitions. More sophisticated agents like Devin also incorporate task decomposition and long-horizon planning, but their underlying verification mechanism still relies on quantifiable evaluation signals like unit tests and integration tests.

The Fundamental Differences Between Law and Software

Legal work possesses almost none of these properties. This is the core contradiction the original post's title emphasizes:

• No instant verification: Whether a contract clause is "correct" often depends on the specific jurisdiction, evolving case law, and commercial intent. There's no "compiler" to tell you whether your answer is right or wrong.
• Extremely long feedback cycles: Whether a legal judgment is sound may only become clear months or years later, in litigation.
• Near-zero error tolerance: Buggy code can be re-run; a single oversight in a legal document can result in millions of dollars in losses or irreversible legal consequences.

Directly applying an agent scaffold designed for programming to a legal context is like using a thermometer to measure weight — the tool itself isn't broken, but it's being used in entirely the wrong context.

Legal knowledge also has a unique hierarchical structure with deep contextual dependencies, fundamentally different from the syntax tree of code. Legal systems typically consist of constitutions, statutes, administrative regulations, case law, and contracts — a multi-tiered priority structure where higher-order law supersedes lower-order law, and specific provisions override general ones. In Common Law systems, the doctrine of Stare Decisis further requires understanding the relationships of distinguishing and extending precedents — the same legal principle can lead to opposite conclusions under different factual circumstances. This knowledge structure more closely resembles a directed graph than a flat document collection, and simple vector retrieval will lose these critical relational connections.

What Real Legal AI Actually Needs

Shifting from "Generation" to "Reasoning and Review"

The core of legal work isn't "generating large volumes of text" — it's "precise reasoning, review, and risk identification." A competent legal AI system should be built around the following capabilities:

1. Traceability: Every legal conclusion must be traceable to a specific statute, precedent, or contract clause — not dependent on the model's hallucinated output.
2. Deep domain knowledge integration: This means genuinely understanding the hierarchical relationships, priorities, and conditions of applicability between legal concepts — not simply dumping legal documents into a vector database for retrieval.
3. Human-AI collaboration, not replacement: Legal AI should be positioned as an augmentation tool for senior attorneys, preserving human judgment at critical decision points rather than pursuing end-to-end automation.

In the field of Human-AI Collaboration design, researchers distinguish between two primary modes: AI as an Autonomous Agent and AI as a Cognitive Augmentation Tool. For legal contexts, the latter is clearly more appropriate. Multiple studies from Carnegie Mellon and Stanford's Human-Computer Interaction (HCI) research groups have shown that when professionals perceive an AI system as having "controllable uncertainty expression" — meaning the system actively flags its own uncertain judgments — their trust and actual adoption rates are significantly higher than with systems presenting "confident" outputs. Leading LegalTech companies like Harvey AI have already begun requiring every recommendation to include citations to original statutes, a direct embodiment of this design principle.

Broader Lessons for Vertical AI Product Design

This discussion has value well beyond the legal industry. It exposes a pervasive trap in vertical AI startups: applying the successful patterns of general-purpose paradigms to fundamentally different professional domains.

Whether in law, medicine, or finance, these high-stakes, high-expertise fields share one common characteristic — the standard of "correctness" is ambiguous, highly context-dependent, and extremely costly to verify. The NIST AI Risk Management Framework (AI RMF) breaks down reliability into four dimensions: Accuracy, Explainability, Robustness, and Traceability. Law, medicine, and finance are universally classified as scenarios requiring the highest level of reliability assurance. The EU AI Act explicitly classifies legal interpretation assistance systems as "high-risk AI systems," mandating human oversight mechanisms and auditable decision pathways — meaning the compliance pressures on legal AI products alone rule out the "ship fast, fail fast" approach of coding agents.

This implies:

• Simple RAG + prompt engineering solutions are unlikely to genuinely meet professional needs;
• Deep involvement from domain experts is essential for system design and evaluation framework development;
• Product form should follow the logic of the domain, not the inertia of technical paradigms.

Reframing the Value and Limits of "Scaffolding"

To be clear, "scaffolding" is not inherently a pejorative term. Retrieval augmentation, tool use, and multi-step reasoning are all valuable engineering techniques in any AI application. What the original post actually criticizes is the uncritical transplantation of an entire methodology proven in software development as though it were a universal silver bullet.

Take RAG as an example. The core of retrieval-augmented generation involves chunking legal texts into a vector database, retrieving relevant fragments based on user queries at inference time, injecting them into the context window, and having the LLM generate a response. This approach addresses some of the pain points around LLM knowledge cutoffs and hallucinations — but vector similarity search is an approximate semantic match, while legal reasoning requires precise logical relationship matching. Whether a specific contract clause triggers a precondition in another clause, for instance, cannot be captured by cosine similarity. Some cutting-edge legal AI research is exploring the integration of Legal Knowledge Graphs with LLMs in an attempt to better capture the structural knowledge relationships within legal systems.

For legal AI practitioners, the real challenges are:

• How to design evaluation benchmarks adapted to legal logic, rather than borrowing pass-rate metrics from the software world;
• How to ensure output reliability in the absence of instant feedback;
• How to build interaction forms that professional attorneys will genuinely trust and adopt.

Conclusion: The Real Starting Point for Vertical AI

"Legal AI, not a coding agent with scaffolding" is a sober reminder to the entire vertical AI wave. Transferring AI technology has never been a simple copy-paste operation. Every professional domain deserves a fresh rethinking of what AI should look like, starting from its own fundamental nature.

For any team looking to build AI products in a professional domain, the first question shouldn't be "which existing agent framework can we adapt?" It should be "what is the essential nature of work in this domain, and what kind of AI does it actually need?" Understanding that distinction is where vertical AI truly begins to create value.