OpenAPI Diff Tool: Compatibility Is a Negotiated Fiction Between YAML Files

Why You Need a Deterministic OpenAPI Changelog

When iterating on API versions, hand-writing changelogs or relying on LLMs to generate release notes often produces "hallucinated breaking changes." One developer built a deterministic OpenAPI diff tool for this reason: input two versions of a spec file, output a structured changelog. No model involved—just tree traversal and diffing.

The first version's logic was straightforward: compare paths, operations, and schemas. A new property in a response schema was marked non-breaking; a new required field in a request body was marked breaking. Clean, defensible—but wrong.

OpenAPI (formerly the Swagger specification) is a standard format for describing RESTful API interfaces, typically written in YAML or JSON, containing metadata such as paths, operations, request/response schemas, parameters, and authentication methods. Developers use it to generate documentation, client SDKs, and test cases. OpenAPI diff refers to comparing the structural differences between two versions of the same API (old spec vs. new spec) to identify compatibility risks. Common diff tools include openapi-diff, oasdiff, and others, most of which are based on rule engines rather than machine learning, so results are reproducible and auditable. This determinism is critical for release gates in CI/CD pipelines, because any randomness or model hallucination could lead to false positives or false negatives.

"Compatibility" Is Not a Binary Judgment but a Negotiated Outcome

After users integrated the tool into CI, the release pipeline started producing frequent false positives. Real-world scenarios exposed the gap between the spec level and the runtime level:

• New required field with a default value: Breaking at the spec level, but in practice most HTTP clients can tolerate a default value.
• Renamed enum value: The type system treats it as the same type, but any downstream consumer relying on string matching will break immediately.

These contradictions show that "compatibility" is not a technical fact but a fictional consensus negotiated between API providers and consumers. The tool cannot read the mental models of client authors, but it can honestly express uncertainty.

Layered Classification: Encoding Judgment with Deterministic Rules

The tool classifies changes into three tiers, each with a human-readable explanation so developers can easily override the conclusion:

Definitely Breaking

Removing paths, changing types, adding required fields without defaults. These changes almost inevitably break existing clients.

Possibly Breaking

Renaming enum values, tightening validation schemas, removing response properties. These changes depend on consumer implementation; risk is high but not absolute.

Additive

Adding optional fields, new paths, new operations. Generally backward-compatible, but watch for edge cases such as naming conflicts.

Each category comes with a "why it's classified this way" explanation, so developers can understand and override the judgment without reading the full diff.

Tree traversal is an algorithm in computer science for visiting every node in a tree structure; common approaches include depth-first search (DFS) and breadth-first search (BFS). In OpenAPI diff, spec files are parsed into an abstract syntax tree (AST), and the tool recursively traverses nodes such as paths, operations, parameters, and schemas, comparing corresponding nodes in the old and new trees level by level to precisely identify added, removed, or modified fields. Compared with text diff, tree traversal understands semantic structure and avoids noise caused by formatting adjustments (such as indentation or key order). Diff records node changes during traversal and generates a structured changelog according to predefined rules (e.g., "required field added" is breaking). This purely algorithmic approach does not rely on probabilistic models, ensuring consistency and interpretability of results.

From Arguing "Is It Breaking?" to Arguing "Do the Rules Match Our Consumers?"

The deterministic part freed up productivity: teams no longer agonize over whether a given change breaks compatibility, but instead move to a more valuable discussion—whether the classification rules truly reflect consumer behavior. This kind of debate is closer to business reality.

The tool is packaged as an online service; enter two spec URLs and view the differences. For teams that write changelogs by hand, this deterministic approach is worth trying.

Key Takeaways

• A deterministic OpenAPI diff tool avoids the hallucination problems of LLM-generated changelogs
• There is a systematic gap between spec-level "compatibility" and actual runtime behavior
• Layered classification (definitely breaking / possibly breaking / additive) combined with human-readable explanations balances automation and human judgment
• The tool's value lies in honestly expressing uncertainty rather than pretending to be precise
• It shifts the discussion from "is it breaking?" to "do the classification rules match actual consumers?"

One-line Summary

A deterministic OpenAPI diff tool replaces LLMs with tree traversal, classifies changes in layers, and honestly expresses uncertainty.

Paragraph Summary

The article introduces a deterministic OpenAPI diff tool built by a developer to avoid "hallucinated breaking changes" when hand-writing or LLM-generating changelogs. The tool compares old and new spec files via tree traversal and outputs a structured changelog. The author found a systematic gap between spec-level "compatibility" and actual runtime behavior—for example, adding a required field with a default value is breaking at the spec level, but most clients can tolerate it; renaming an enum value is not breaking in the type system, yet it can crash downstream consumers. The tool therefore classifies changes into three tiers: definitely breaking, possibly breaking, and additive, each with a human-readable explanation so developers can override the conclusion. This deterministic approach shifts team discussion from "is it breaking?" to "do the classification rules match actual consumer behavior?" balancing automation and human judgment, and is packaged as an online service.