Hiding Tables Won't Stop Text-to-SQL? schemagate Retriever's Real-World Counter

The Dilemma Revealed by the Paper: Tell the Model "You Can't Look," and It Insists on Answering

According to a Reddit user, a SIGMOD'27 paper titled "Benchmarking Text-to-SQL under Role-Based Access Control" (arXiv 2607.22115) presents a counterintuitive finding: under role-based access control (RBAC), showing the model only the columns it is authorized to see does not stop Text-to-SQL agents from answering questions. If the answer requires hidden columns, the correct behavior is to refuse, but the model still pieces together an answer from the visible parts.

The paper tested the best configuration with pure schema filtering on the BIRD dataset, and about 27% of questions that should have been refused were still answered. This number means that the traditional "hide restricted tables" strategy has a significant loophole at the security boundary—the model doesn't understand "missing means forbidden," and instead speculates based on incomplete information.

RBAC (Role-Based Access Control) is a permission model widely used in databases and enterprise systems: the system assigns one or more roles to each user, each role is bound to a set of accessible tables, columns, or rows, and users can only see the subset of data authorized for their roles. In the Text-to-SQL context, RBAC means that the same natural language question may legitimately produce two different outcomes for different roles—"answerable" and "must be refused." For example, a finance role can query the salary column, while a regular business role asking the same question must be told it has no access. Therefore, a Text-to-SQL agent must not only generate correct SQL but also determine whether the current role is authorized to cover the information needed to answer the question—this is exactly the security boundary the paper examines.

AC-F1 is the core metric used in the paper's evaluation, typically referring to a combined F1 of "Accuracy and Compliance": the model's output counts as correct only when it is both semantically correct and compliant with role permission constraints (answering what should be answered and refusing what should be refused). This makes simply "answering the question correctly" insufficient—answering forbidden questions or incorrectly refusing legitimate ones both lower the score. Understanding this metric helps judge the practical meaning of the gap between schemagate's 90.1 and the paper's 88.1.

schemagate's Response: From "Passive Hiding" to "Active Defense"

The maintainer of schemagate (the original poster) plugged their retriever into the paper's evaluator, using the same model as the paper's best run—Claude Sonnet 4.5—and conducted 5 trials. schemagate itself is a retriever / MCP server that performs schema filtering for SQL agents, highly aligned with the paper's scenario.

The results show that on the BIRD dataset, schemagate achieves an AC-F1 of 90.1, while the best result among the paper's 13 models is 88.1 (that model could see the full schema and policy). On the Spider dataset, schemagate ties the same model at 93.0; however, the paper's best result on Spider comes from DeepSeek^Ad V3.2-Reasoning at 95.9, so schemagate still lags there. On both datasets, the number of queries that actually read forbidden columns was 0.

This comparison shows that with the same model, schemagate's retrieval strategy outperforms the paper's approach of letting the model see the full schema plus policy on BIRD, while maintaining a zero record on security violations.

MCP (Model Context Protocol) server is a standardized interface protocol that exposes tools and data sources to large model clients. Implementing schema filtering as an MCP server means the model retrieves available table and column metadata through the server before generating SQL, and the server completes permission pruning at the retrieval stage rather than handing the full schema to the model and relying on its self-discipline to follow policy. This architecture moves the "permission boundary" forward to the data supply stage, contrasting with the paper's approach of "showing visible columns and expecting the model to understand that missing means forbidden."

Claude Sonnet 4.5 is a mid-to-high-end reasoning model from Anthropic, strong on code and structured generation tasks, and commonly used as a Text-to-SQL base. schemagate chose the same model as the paper's best run for comparison in order to rule out model capability differences and attribute the difference to the retrieval and prompting strategy itself—which is also why this reproduction experiment is methodologically convincing.

Two Key Improvements: Prompting and Policy Validation

schemagate's differences from ordinary schema filtering are mainly in two places. First, a sentence is added to the prompt: "You are seeing part of this database; if you need more information, refuse to answer." This sentence does not name any hidden content, but in a pilot of 100 questions it reduced wrong answers from 16% to 5%. Second, each generated query undergoes policy validation before execution; removing this step doubles the violation rate.

The author also notes limitations: only one model was tested, and the prompt wording was selected on 100 BIRD questions; removing these two factors leaves the score essentially unchanged (90.08), indicating the improvement is not an accidental effect of specific wording.

Practical Implications for LangChain SQL Agents

The original poster ends with a question: has anyone done per-user schema filtering in LangChain SQL agents, and how do they handle the "model still answers" problem? This question points to a common pain point in current Text-to-SQL deployment—most frameworks trust by default that the model will respect permission boundaries, but both the paper and schemagate's real-world tests show that models tend to "answer if they can."

For teams building enterprise-grade SQL agents, ideas worth borrowing include: performing schema pruning at the retrieval layer, explicitly declaring information incompleteness at the prompt layer, and adding policy validation after generation as a final gate. Only by combining all three can "hiding restricted tables" be turned from an assumption into a verifiable defense line.

The complete results and reproduction method are in the author's GitHub repository BENCHMARKS.md.

LangChain SQL agent is a class of components in the LangChain framework for connecting to relational databases, converting natural language to SQL, and executing it; common implementations include SQLDatabaseToolkit and agent-based query chains. By default, it injects the database schema information into the prompt and allows the model to decide which tables and columns to query. To implement per-user schema filtering within it, you usually need to customize the tool layer: prune the table/column list by role before constructing the prompt, and parse and validate the generated SQL for permissions before execution (for example, checking whether unauthorized objects appear in FROM/JOIN and SELECT), rather than relying solely on "do not access" instructions in the prompt.

One-line Summary

A paper found that hiding restricted tables does not stop Text-to-SQL models from answering; schemagate achieves zero violations through retrieval pruning plus prompting and validation.

Paragraph Summary

A SIGMOD'27 paper points out that under RBAC, showing Text-to-SQL models only the columns they are authorized to see does not stop them from answering: on BIRD, about 27% of questions that should have been refused were still pieced together into answers. The schemagate maintainer plugged their retriever into the paper's evaluator and, using the same model Claude Sonnet 4.5, achieved an AC-F1 of 90.1 on BIRD, surpassing the paper's best of 88.1 among 13 models; on Spider it tied the same model at 93.0 but lagged behind the paper's best of 95.9. Its key improvements are declaring in the prompt that information is incomplete and refusal is allowed (pilot error rate dropped from 16% to 5%), and performing policy validation before execution (removing it doubles the violation rate). On both datasets, the number of queries that actually read forbidden columns was 0.