LLM Reads Engineering Drawings: A Post-Mortem on Going from 50% to 95% Accuracy

Throwing ChatGPT directly at an engineering drawing and asking it to extract an IO list yields about 50% accuracy — that was the result when Belgian software company Nebulae's client first tried it. For building automation projects that require precision down to every sensor and actuator wiring point, 50% means engineers still have to review line by line, making the tool essentially useless.

Nebulae built a human-in-the-loop document parsing tool for this client, pushing reading accuracy above 95%, and it's already in daily use on real projects. Their tech lead says that where processing a single drawing used to take two hours of reserved time, now "it's checked before the coffee gets cold."

This article breaks down what they got right.

The Task: Why Engineering Drawings Are So Hard to Automate

Every building automation project starts with the same stack of documents: multi-page Visio drawings annotated with hundreds of input/output connection points, using arrows and shapes to distinguish signal types and directions. Controller selection and wiring for HVAC, climate control, and lighting systems all depend on this drawing.

Engineers need to identify each point: digital or analog, input or output, which device it belongs to, what it's called, and then transcribe it into a structured IO list in the correct order.

The difficulty of this task lies in the fact that it depends on the reader understanding a set of visual conventions, not just plain text matching. The same symbol may have subtle variations across different drawings, naming conventions are project-specific, and even drawings from the same company develop their own "dialect" after years of iteration.

For an experienced engineer, this is one to two hours of highly repetitive work per document. Missing or misreading a single symbol can cascade into wiring errors on site.

IO List (I/O List) is a core engineering document in building automation and industrial control. It lists, line by line, the complete attributes of every signal point in the control system: point number, associated device, signal type (digital DI/DO or analog AI/AO), direction (input or output), physical address, terminal number, and naming identifier. This list is the sole basis for controller selection, cable routing, on-site wiring, and later commissioning. Any omission or misplacement directly leads to on-site equipment failing to interconnect or even being damaged. Therefore, IO list accuracy requirements are extremely high — no missing points and no wrong order. This is exactly why plain-text-level model output is almost unacceptable in this scenario — engineers need a construction-ready finished document, not an approximately correct result.

First Attempt: Asking the Model Directly, With Unsatisfactory Results

The client first tried the most intuitive approach: throwing the drawing directly at ChatGPT and asking it to output an IO list.

The result was about 50% accuracy, or even lower. The significance of this number isn't that "the model doesn't work" — it's that if engineers still need to review line by line, the tool hasn't saved any time and has actually added a trust cost.

The problem lies in two places. First, plain text extraction loses the spatial relationship between symbols and connection lines, and that relationship is precisely the key to determining signal ownership. Second, the model doesn't know this team's naming conventions and device pairing rules, so it can only guess based on generic understanding.

Few-shot examples are a technique in large model applications that uses in-context learning: providing several input-output examples highly similar to the target task in the prompt, allowing the model to infer the required patterns, formats, and domain rules without updating parameters. Compared to fine-tuning, few-shot deployment costs are extremely low and iteration is fast, making it especially suitable for project-specific knowledge like naming conventions and field structures. In engineering document parsing, the quality of few-shot examples directly determines whether the model can learn the client's internal naming habits. Nebulae extracted samples from the client's historical heat pump, ventilation, and HVAC projects precisely to align the model with the team's actual terminology system, rather than relying on textbook-style expressions from general training data. This step is an important source of the accuracy improvement.

Five Changes That Pushed Accuracy to 95%

Nebulae didn't try to build a fully automatic black box. Their judgment was: in scenarios mixing vendor-specific symbols, project-specific naming, and drawings evolved over years, no model can achieve 100%. So the tool's design goal from the start was human-in-the-loop — AI handles reading, engineers handle final confirmation.

The accuracy improvement came from workflow design, not smarter prompts.

1. Give the model two views per page

Each drawing page is fed to the model twice: the original text layer, ensuring label precision; and a rendered image of the same page, preserving the hierarchical relationships between symbols and surrounding elements.

They also wrote their own preprocessing pipeline to extract the precise position of every element on the page, letting the model read the drawing in human reading order (left to right, top to bottom), rather than the random order of PDF internal storage. This detail is critical — if the order is wrong, the IO list order is wrong, and on-site engineers construct in order.

The difference between PDF text layers and rendered images is key to understanding the value of multi-view input. PDF files typically store an invisible text layer that records the precise encoding and string content of each character, suitable for extracting label names; but it doesn't describe the positional meaning of graphics, arrows, or connection lines, and the internal storage order of elements is often random, unrelated to human reading order. Rendered images pixelate the page, preserving symbol shapes, connection line directions, and spatial hierarchies, allowing the model to determine which device a label belongs to. Combining both achieves both "reading accurately" and "understanding correctly." Nebulae's self-built preprocessing pipeline extracts element coordinates and reorders them left-to-right, top-to-bottom, essentially converting visual spatial information into an ordered reading stream, avoiding IO list order confusion, because on-site construction strictly follows order.

2. Require structured, ordered output

The model directly returns a fixed-structure, ordered IO list, so engineers receive a finished framework they can check, rather than building from scratch.

3. Use the client's own historical projects for few-shot examples

They extracted a small sample library from the client's past heat pump installations, server room ventilation, and HVAC system projects. The model thus learns the naming conventions and device pairing methods this team actually uses, not a textbook version. Each drawing page is also read against its own legend.

4. Keep human checkpoints

Every extraction result enters a review interface: the original drawing on the left, the AI's structured reading result on the right. Engineers can check and correct type, direction, naming, or order within seconds, rather than re-deriving everything.

5. Finish the job

After engineer confirmation, the reviewed list is directly assembled into the final document usable by the field team. No extra formatting steps, no copy-pasting between tools.

Model Choice: Gemini, but Only a Snapshot in Time

Nebulae chose Google's Gemini at the time because it performed strongest in true multimodal reasoning — able to read both rendered images and underlying text in a single pass, rather than stitching together multiple tools. The cost was also low enough to run on complete multi-page drawing sets without AI overhead becoming a conspicuous line item per project.

They specifically emphasize: models iterate quickly, and this choice was a snapshot at the time, not a recommendation.

Multimodal reasoning refers to a model's ability to understand multiple input modalities (such as text, images, and tables) simultaneously within a single inference process and establish cross-modal connections. In engineering drawing parsing scenarios, this means the model can directly correspond the shape and position of a symbol in the rendered image with the corresponding label string in the text layer, rather than first using OCR to recognize text, then another module to locate graphics, and finally manually stitching them together. This end-to-end multimodal capability significantly reduces pipeline complexity and error accumulation. Nebulae chose Gemini precisely for this, but it must be emphasized that multimodal models iterate extremely fast — today's advantage may be caught up within six months, so model selection should be viewed as a replaceable point-in-time snapshot, not a long-term architectural decision. Workflow design is the more stable asset.

Key Learnings

There are several judgments worth noting from Nebulae's post-mortem:

Improvement comes from workflow, not prompt engineering tricks. Structured input, structured output, and adding a human checkpoint before trust — these three things pushed results from 50% to over 95%.

Design for review, not for 100%. A quick review step is the key to making output safe to deploy. Pursuing full automation actually gets stuck on the last mile of accuracy.

Use the team's own examples. The model learns this team's naming conventions and device pairings through their own historical project samples.

The numbers have limited applicability. They explicitly caution: these numbers come from one client's drawings, yours will be different. Before trusting any percentage (including theirs), measure on your own documents.

The value of this case isn't in the conclusion that "LLMs can read engineering drawings," but in demonstrating a pragmatic path to AI deployment: not pursuing end-to-end automation, but finding the step where human judgment is most irreplaceable and letting the model handle the rest.

For any engineering process still relying on manual reading, transcription, or review, this framework — multi-view input, structured output, domain samples, human checkpoints, seamless delivery — may be more worth trying first than switching to a stronger model.

One-line Summary

Nebulae raised LLM accuracy in reading engineering drawing IO lists from 50% to over 95% through human-in-the-loop workflow design.

Paragraph Summary

The article breaks down how Belgian software company Nebulae built a document parsing tool for a building automation client. The client initially gave engineering drawings directly to ChatGPT to extract IO lists, achieving only about 50% accuracy — engineers still had to review line by line, making the tool useless. Nebulae didn't pursue a fully automatic black box, but designed a human-in-the-loop workflow: feeding each page as both raw text layer and rendered image views, building a custom preprocessing pipeline to ensure reading order, requiring structured ordered output, using client historical projects for few-shot examples, providing a human review interface, and seamlessly delivering the final document. Accuracy rose above 95%, and processing a drawing went from two hours to minutes. The author emphasizes the numbers come from a single client, and other scenarios need their own measurements — the real value lies in a pragmatic path to AI deployment.