Markdown Files Are Not Specifications: Using Formal Analysis to Find Requirements Gaps

Why a Markdown Document Is Not a Specification

In an era of AI-assisted programming and agile development, more and more teams have fallen into the habit of using a single Markdown file to capture product requirements, API contracts, and system designs. It's lightweight, human-readable, version-control-friendly — seemingly perfect. But a discussion on Hacker News, titled ".md File is not a specification: Using formal analysis to find requirements gaps", points out a widely overlooked problem: a Markdown document written in natural language is fundamentally not a rigorous specification.

The core issue is that a Markdown file records a human description of requirements — not machine-verifiable constraints. A requirement written in English or any natural language is often riddled with ambiguity, implicit assumptions, and uncovered edge cases. When developers (or AI coding assistants) implement features based on such documents, these vague areas get filled in by each person's own interpretation, eventually becoming defects.

Where Requirements Gaps Come From

Requirements gaps typically don't arise because the document author was careless — they arise because natural language inherently lacks a completeness-checking mechanism. A seemingly thorough requirements document may harbor risks at several levels:

• Uncovered state combinations: The document describes the "happy path" but fails to enumerate all cross-combinations of input conditions.
• Implicit preconditions: The author assumes readers share certain context, but those assumptions are never made explicit.
• Contradictory clauses: Different sections of a document may contain logical conflicts that are nearly impossible to catch during a manual review of a long document.
• Missing boundary and error cases: Descriptions of error handling, extreme values, and concurrency scenarios are frequently omitted.

These gaps are nearly impossible to detect systematically in the linear text of a Markdown file. You can read through the document repeatedly, but the human brain is not built to exhaustively enumerate all logical branches.

What Formal Analysis Can Offer

The solution proposed in the original article is to introduce formal analysis. The value of formal methods lies in transforming requirements from "text that can be read" into "mathematical models that can be reasoned about." Once requirements are expressed as formal specifications, tools can automatically check them for:

Consistency

Tools can verify whether the requirement set contains internal logical contradictions — the part most easily missed in manual review. When two rules produce conflicting conclusions under a given input condition, a formal checker will flag it directly.

Completeness

A formal model can systematically enumerate the input space and identify which state combinations are left undefined by the requirements. In other words, it can tell you, "Here is a situation you never considered."

Satisfiability

Through constraint solving, you can confirm whether the requirements themselves admit at least one valid implementation — preventing you from writing a specification that is impossible to satisfy in the first place.

This kind of analysis shifts the moment of "discovering requirements defects" from the expensive testing or production stage dramatically earlier, to the requirements definition stage.

Formal Methods are a set of software/system specification and verification techniques grounded in mathematical logic. Common formal specification languages include TLA+ (designed by Leslie Lamport and widely used for distributed systems verification), Z notation, Alloy, and model-checking frameworks based on temporal logics like LTL/CTL. Take Alloy as an example: developers describe system states and constraints using relational algebra, and the Alloy Analyzer then automatically enumerates all states within a bounded scope to find counterexamples that violate constraints. Companies like Amazon and Microsoft have used TLA+ on critical systems (such as S3 storage protocols and Azure Cosmos DB) to uncover concurrency bugs that code reviews and testing alone could not expose. The fundamental difference between formal verification and traditional testing is this: testing proves that certain paths "work," while formal verification can prove that — within the scope of the given model — "no situation exists that violates the specification." This is precisely why it offers unique value for requirements completeness checking.

Real-World Implications for the Age of AI Programming

This topic is especially relevant today. As AI code generation tools become widespread, development workflows increasingly rely on the pattern of "give the model a requirements description and let it generate code." If the requirements document itself is full of gaps, the AI will simply concretize those ambiguities in some unpredictable way — and may even amplify them.

In other words, the quality of the prompt or requirements document is becoming the new bottleneck for software quality. A specification that has been formally verified — with ambiguities and gaps eliminated — is not only beneficial for human collaboration; it is equally critical for AI systems trying to understand intent. This may mean that in future engineering practice, formal specifications and natural language documents will become complementary: Markdown handles readability and communication, while formal models handle rigor and verification.

The Challenges of Adoption

Of course, formal methods are no silver bullet. The learning curve is steep, the approach demands a certain level of mathematical and modeling ability from the team, and writing formal specifications for every requirement carries non-trivial costs. The limited traction of the original post on Hacker News (only 7 points and 1 comment) also indirectly reflects that formal analysis remains a niche practice in mainstream software engineering.

A pragmatic path forward is likely selective application: use formal analysis for high-risk, high-value, logically complex core modules, while continuing to use lightweight Markdown documents for general features. The critical insight is to remain clear-eyed — when you hand over a .md file as a "specification," you are actually delivering a description with unknown gaps, not a verified contract.

One approach for lowering the barrier to adopting formal methods is Lightweight Formal Methods, represented by tools such as Alloy, Clafer, and constraint validation tools based on SMT solvers (like Z3) that have gained traction in recent years. Unlike full formal verification, lightweight formal methods don't require building a complete mathematical model of the entire system. Instead, they focus on rapid verification of local, critical properties. Engineers describe core rules using a syntax similar to "pseudocode + constraints," and the tool automatically searches for counterexamples. Furthermore, as large language models grow more capable, researchers are already exploring using LLMs to automatically translate natural language requirements into draft Alloy or TLA+ specifications for engineers to review and refine — a pipeline that could lower the barrier to entry for formal analysis from "requiring specialized modeling training" to "requiring review skills," potentially making the benefits accessible to far more teams.

Conclusion

The point raised in this discussion is simple but profound: the readability of a document and the rigor of a specification are two entirely different things. In an age when anyone can quickly produce a polished Markdown file, the truly scarce capability is the ability to guarantee requirements completeness and consistency. Formal analysis offers one possible answer — not to replace the familiar documents we already use, but to remind us that the quality of requirements deserves to be taken seriously with more rigorous tools.