Building AI for Lawyers: Legora CTO on Evals, Tabular Review, and Cracks in the Billable Hour

Legal is one of the most cautious domains for AI adoption: documents are highly sensitive, errors are costly, and professional judgment is hard to standardize. In a recent interview, Legora CTO Jacob Lauritzen broke down the real engineering challenges of building AI products for lawyers—from evaluation systems that lack trace data, to tabular reviews involving millions of cells, to the structural cracks emerging in the billable-hour business model.

Evals Without Traces: The First Hurdle for Legal AI

One of the core engineering challenges Lauritzen raised is "evals without traces"—evaluating models without complete call-chain tracing. General-purpose AI products can typically rely on user interaction logs, clickstreams, and explicit feedback to build an evaluation loop, but these signals are extremely scarce in legal settings: lawyer workflows are highly confidential, interaction data is hard to retain, and users rarely take the initiative to flag "this answer is wrong."

This means teams must validate model outputs through an entirely different approach. The interview suggests shifting the evaluation focus to task-level correctness rather than conversation-level satisfaction—for example, whether contract clause extraction misses anything, whether cited statutes actually exist, and whether conclusions are consistent with the given document. This evaluation method is closer to traditional software testing, but it requires deep involvement from domain experts to build the evaluation sets.

For teams building vertical AI, this observation has broad relevance: when your users don't generate observable feedback signals, evaluation infrastructure must be designed up front—you can't wait until after launch to bolt it on.

In AI engineering, a "trace" typically refers to the complete record of a request's call chain from entry to final output, including inputs, intermediate steps, tool calls, model versions, latency, token consumption, and other metadata. It is both the foundation of observability and the raw material for building evaluation datasets: with traces, teams can replay historical requests, pinpoint where errors occur, and turn failure samples into regression test cases. Without traces, evaluation degrades to relying solely on final outputs and human annotation, making it difficult to attribute errors to prompts, retrieval, the model, or post-processing. For legal AI, the absence of traces is not just a tooling problem—it also involves confidentiality obligations and law firm data governance policies, so teams need to design sampling granularity, de-identification, and retention strategies up front to fill the observability gap.

Million-Cell Tabular Reviews: The Real Workload of Legal AI

Another scenario mentioned in the interview is million-cell tabular reviews. Legal due diligence, evidence disclosure, and batch contract comparison often involve extremely large structured datasets—far beyond the volume handled by typical conversational AI products.

This kind of workload places different demands on system design: model calls can't be single-turn Q&A but require task orchestration, batch processing, and result aggregation; context management must account for references between cells; and errors can't be checked manually one by one but require automated cross-validation mechanisms. Lauritzen's description shows that the competitiveness of legal AI lies not just in the model itself, but in the data pipelines and review workflows built around it.

Interestingly, tasks like tabular review are actually more verifiable—each cell's answer has the original document as its basis, providing a natural anchor for automated evaluation.

Tabular review in due diligence typically corresponds to the "data room" scenario: one party to a transaction places thousands of contracts, memoranda, and evidentiary materials into a controlled directory, and the other party must extract key clauses (such as change of control, indemnity caps, termination conditions) from each and compile them into a comparison table. The traditional approach is for junior lawyers and paralegals to read line by line and fill in the table, taking weeks. With AI, the system needs to segment each document, locate relevant clauses, extract structured fields, and then align the same field across documents. The difficulty lies in inconsistent document formats, variations in clause wording, and the requirement that extraction results be traceable to the original paragraph—otherwise they can't pass review. This is precisely why cross-validation and citation anchors become essential.

Cracks in the Billing Model: What AI Is Eroding

The phrase "cracks in the billable hour" in the interview title points to a deeper industry problem. Hourly billing is the cornerstone of law firm business models, and AI is compressing exactly the work that was traditionally sold by the hour—document review, clause comparison, preliminary legal research.

Lauritzen's observation is that this erosion is not simply "AI replacing lawyers" but a misalignment in how value is measured: when AI compresses ten hours of work into one, law firms' revenue structures loosen if they continue billing by the hour; if they shift to outcome-based or value-based billing, they need to rebuild the entire pricing and trust mechanism. For AI product teams, this means customer procurement logic is also changing—law firms aren't buying "a faster tool" but the ability to redefine service pricing.

This is also a key difference between legal AI and other vertical domains: product value is directly embedded in the customer's business model, not just enhancing individual productivity.

Hourly billing (the billable hour) centers on recording the time a lawyer invests in a specific client matter and settling at an hourly rate, with firm revenue positively correlated with time invested. Alternative fee arrangements (AFAs) include fixed fees, phase-based fees, outcome-based fees, and capped fees—these have long existed but have spread slowly due to the difficulty of quantifying value. After AI compresses the time required per task, the logic of hourly billing faces pressure from two directions: clients question why they should still pay the original hours for a faster tool, while firms that proactively lower prices lose revenue. This is prompting some firms to convert the efficiency gains from AI into fixed quotes or subscription-based services, replacing unpredictable hours with predictable pricing.

Takeaways for Vertical AI Founders

Putting these three observations together reveals a clear logical chain: legal AI is hard to evaluate because feedback signals are missing; the workload is heavy because real tasks involve structured, large-scale review; and the business value runs deep because it touches the billing model itself.

For teams building vertical AI, Lauritzen's experience points to several practical recommendations: design evaluation systems around task correctness from day one rather than relying on user feedback; reserve architectural capacity for batch and structured tasks rather than optimizing only for conversational experience; and understanding the customer's business model defines product direction better than understanding their workflow.

Legal AI is not a track you can win on model capability alone—engineering depth and industry understanding are the real dividing line.

One-line Summary

Legora CTO breaks down legal AI's missing evals, massive tabular workloads, and cracks in the billable hour.

Paragraph Summary

Legora CTO Jacob Lauritzen lays out three challenges of building AI for lawyers from an engineering perspective. First, "evals without traces": legal work is highly confidential, interaction logs and user feedback are scarce, so teams must focus evaluation on task-level correctness—such as whether clause extraction misses anything or whether citations are real—requiring deep involvement from domain experts to build eval sets. Second, real workloads: due diligence and evidence disclosure involve million-cell tabular reviews, requiring task orchestration, batch processing, cross-cell reference management, and automated cross-validation, with competitiveness reflected more in data pipelines than in the model itself. Third, business model disruption: AI compresses document review and clause comparison sold by the hour, and firms' revenue structures loosen if they keep billing hourly, while shifting to outcome-based billing requires rebuilding pricing and trust mechanisms. The article accordingly advises vertical AI teams to build evaluation systems up front, reserve architectural capacity for batch structured tasks, and understand customers' business models.