How Knowledge Graphs Rescue "Imperfect Prompts": A New Design for Enterprise Data Assistants

When enterprise employees ask data assistants questions in natural language, they often don't follow strict database query syntax—they mix in business terminology, omit table names, and may not even know which table the data lives in. These "imperfect prompts" cause LLM-based enterprise data assistants to fail frequently in real-world deployments. A paper from Design Science Research compared two semantic intermediation approaches through simulation-based evaluation and found that the knowledge graph-based path wins across the board on precision, recall, and user intent preservation.

The Real Predicament of Enterprise Data Assistants: Imperfect Prompts

The research team deployed an LLM data assistant in the data lake of a partner company's CFO department and observed a widespread phenomenon: business users' questions contain a lot of domain-specific vocabulary but lack schema references or structural information. For example, a user might ask "How were collections in East China last quarter?" rather than specifying "Please query records in the fact_collection table where region='East China' and quarter='Q3'."

This gap between prompts and database schemas is a core obstacle enterprises face on the path to data democratization. The LLM itself doesn't understand the internal structure of the enterprise data lake. When prompts lack table names, field names, or join relationships, the model can only guess when generating SQL, and the error rate naturally stays high. The paper points to introducing a "semantic intermediation layer" as the solution direction—building a bridge between user intent and database structure.

Data democratization refers to enabling business users without technical backgrounds to directly access and analyze enterprise data, without relying on IT departments or data engineers to write queries. Its core idea is to lower the barrier to data use so that decision-makers can make judgments based on real-time data. However, the biggest obstacle to data democratization is often not tool accessibility but the semantic gap—business users express needs in domain terminology, while databases store data in a structured schema language of tables, fields, and foreign keys. When an LLM acts as an intermediary, it needs to translate business expressions like "collection status" into precise query conditions against tables such as fact_collection. The accuracy of this translation directly determines whether data democratization can truly be realized. Therefore, the design quality of the semantic intermediation layer essentially determines whether an enterprise data assistant can cross the semantic gap and achieve its original purpose of empowering business users.

Two Intermediation Designs: Schema Similarity vs. Knowledge Graph

The research designed and evaluated two design variants. The first is schema-based intermediation, which matches terms in user prompts to database schema elements through semantic similarity. The logic is intuitive: map what the user says ("collection") to tables or fields that might contain payment receipt information. But its limitation is that similarity matching can only handle surface-level or shallow semantic associations and cannot capture complex relationships between entities.

The second is knowledge graph-based intermediation, which uses a knowledge graph automatically constructed from the data lake to assist parsing. The knowledge graph contains not only schema information such as tables and fields but also encodes associations between entities—for example, the path from "customer" to "order" to "collection." When users provide imperfect prompts, the system can infer and complete missing structural information through graph reasoning rather than relying solely on lexical similarity.

According to the paper, the knowledge graph is automatically constructed from the data lake, which means the approach has the potential to scale in enterprise environments without requiring extensive manual annotation.

A knowledge graph is a data model that organizes knowledge in a graph structure, where nodes represent entities (such as customers, orders, and products) and edges represent semantic relationships between entities (such as "customer-places-order" and "order-generates-collection"). Unlike relational database schemas, knowledge graphs not only describe data structure but also explicitly encode association paths and constraints between entities, supporting reasoning and path queries. In the enterprise data lake scenario, a knowledge graph can be automatically constructed from table structures, foreign key constraints, data lineage, and business glossaries, connecting scattered tables and fields into a unified semantic network. When users enter incomplete prompts, the system can complete missing table names, join conditions, and business logic along relationship paths in the graph, thereby generating more accurate queries. This property makes knowledge graphs an ideal intermediation layer for bridging natural language intent and database structure.

Simulation Evaluation Results: Knowledge Graph Leads Across the Board

The study adopted a simulation-based evaluation method using real-world data, combined with LLM-as-a-Judge evaluation and human verification. Results show that knowledge graph-based intermediation outperforms schema-based methods on all four dimensions: Precision, Recall, F1 score, and user intent preservation.

The user intent preservation metric deserves particular attention. It measures whether the query parsed by the system faithfully reflects what the user originally intended to ask. Schema-based methods may do well on lexical matching but tend to lose implicit conditions or business logic in user questions; knowledge graphs better maintain intent integrity through structured knowledge representation. Based on this, the paper proposes design principles for knowledge graph-based intermediation, with the core argument that structured knowledge representation can bridge user intent and database structure.

LLM-as-a-Judge is a method that uses large language models to automatically evaluate system outputs. In this study, it was used to determine whether generated queries align with user intent and whether results are relevant. Specifically, the user's original question, the system-generated SQL, and execution results are provided together to an independent judging LLM, which scores or classifies them according to preset criteria. This method enables scalable evaluation at lower cost, especially suitable for open-ended scenarios lacking ground truth answers. However, LLM-as-a-Judge also has potential biases, such as systematic preferences for certain expression styles, so the research team combined human verification to calibrate evaluation results and ensure the reliability of conclusions. This evaluation paradigm is gradually becoming common practice in LLM application research, but attention must be paid to verifying its consistency with human judgment.

Implications for Enterprise Data Assistant Design

This study sends a clear signal: the bottleneck of enterprise data assistants lies not in the LLM's generation capability but in the design of the semantic layer. Relying solely on LLMs to translate natural language into SQL is far from sufficient for imperfect prompt scenarios. As an intermediation layer, knowledge graphs provide structured support that can be automatically constructed and reasoned over.

The paper also notes that this design principle can guide the design of enterprise data assistants for users with different skill levels. Whether it's an analyst proficient in SQL or a frontline employee who only understands business terminology, the system needs a semantic intermediary capable of understanding "what is left unsaid." For technical teams building enterprise data assistants, this research suggests a pragmatic architectural choice: between the LLM and the data lake, it is worth investing in the knowledge graph infrastructure layer.

One-line Summary

As a semantic intermediary, knowledge graphs comprehensively outperform schema similarity methods on precision, recall, and intent preservation, effectively solving the failure of enterprise LLM data assistants when facing imperfect prompts.

Paragraph Summary

When enterprise employees ask LLM data assistants questions in natural language, they often mix in business terminology without database schema information, leading to frequent query errors. A design science research study compared two semantic intermediation approaches: schema similarity-based and knowledge graph-based. The knowledge graph is automatically constructed from the enterprise data lake and encodes complex associations between entities. Simulation evaluation using real data and LLM-as-a-Judge combined with human verification showed that knowledge graph intermediation outperforms schema similarity methods on all four dimensions: precision, recall, F1 score, and user intent preservation. The paper accordingly proposes a design principle: structured knowledge representation can bridge user intent and database structure. The bottleneck of enterprise data assistants lies not in LLM generation capability but in semantic layer design, making investment in knowledge graph infrastructure worthwhile.