Third-Party Benchmark Validates schemagate: RBAC for Text-to-SQL in Practice

The performance of Text-to-SQL tools under role-based access control (RBAC) has long lacked an independent, public evaluation standard. All data previously published by the schemagate author came from self-built tests—until a SIGMOD'27 paper released a public benchmark specifically targeting this problem. The author re-ran their tool against this "someone else's" benchmark, and the results are worth noting.

Benchmark Design and Core Findings

The paper, titled Benchmarking Text-to-SQL under Role-Based Access Control, attaches a role and a list of columns readable by that role to each question. If answering a question requires accessing columns the role is not permitted to view, the correct response is "Sorry, I can't answer that." Evaluation uses the paper authors' own scorer, with no modifications by the schemagate author.

The paper's core finding is unflattering to similar tools: simply hiding restricted columns is not enough—models still answer, piecing together a response from the remaining visible columns. On the BIRD dataset, even the paper's best "hidden schema" configuration still answered 27% of questions that should have been refused. This shows that "can't see it" does not equal "knows to refuse."

RBAC (Role-Based Access Control) is a permission model in which each user is assigned a role, the role is bound to a set of policies, and the policies specify which tables or columns that role may read or write. In the Text-to-SQL context, RBAC means the same natural-language question may yield different answers for different roles—some roles may query salary columns, while others cannot see them at all. Evaluation therefore must check not only whether the SQL is correct but also whether it answers within the role's permissions. The paper attaches a role and readable-column list to each question precisely to simulate this fine-grained permission boundary, forcing the model to decide between "refuse" and "answer" when information is incomplete.

AC-F1 is the evaluation metric used in the paper, short for "Answer Correctness F1." It classifies model responses into four categories: correct answer, incorrect answer, correct refusal, and incorrect refusal (i.e., answering when it should have refused). AC-F1 jointly considers answer accuracy and refusal accuracy, penalizing "unauthorized answers." Looking only at whether the SQL execution result is correct ignores refusal behavior, whereas AC-F1 measures both "answering correctly when it should" and "refusing when it should," making it better suited for end-to-end evaluation in RBAC scenarios.

schemagate's Three-Layer Mechanism

schemagate performs three steps for each question:

1. Role policy as visibility: The model cannot see hidden tables or columns at all—not even column names appear.
2. A one-line prompt declaring the view incomplete: The model is told its view is partial, without revealing what is missing; if a question requires invisible data, it should refuse rather than improvise.
3. Pre-execution scope check: Every SQL statement is validated before execution, and anything touching data outside policy is rejected.

The third layer is the key difference. The first two layers influence model behavior; the third is a hard guarantee—it turns "do not read forbidden data" from a measured metric into an architectural guarantee.

Measured Results: Beats on BIRD, Ties on Spider

The author used the same model as the paper's best configuration (Claude Sonnet 4.5), ran five times, and applied the paper authors' scoring method:

Configuration	AC-F1	Answered when should refuse	SQL reading forbidden data
BIRD, schemagate	90.1	5.4%	0
BIRD, paper best	88.1	7.4%	Not reported
Spider, schemagate	93.0	5.1%	0
Spider, paper best	93.0	4.6%	Not reported

On BIRD, schemagate's AC-F1 reaches 90.1, exceeding the paper's best configuration of 88.1; the rate of answering when it should refuse is 5.4%, lower than the other's 7.4%. On Spider, both have the same AC-F1 of 93.0, but schemagate's incorrect-answer rate is 5.1%, slightly higher than the paper's 4.6%. In both datasets, schemagate produced no SQL that read forbidden data—this is guaranteed by the scope check, not a statistical result.

One comparison caveat to note: the paper's best configuration shows the model the full schema plus policy, whereas schemagate achieved the above numbers while the model never saw restricted content.

Decomposing Component Contributions

Removing the scope check doubles the incorrect-answer rate on BIRD (from 5.4% to 11.3%), showing that hard validation plays a significant role in intercepting wrong answers. The "view is incomplete" prompt line was tested separately on 100 questions: without the prompt the incorrect-answer rate was 16%, with it 5%.

A counterintuitive finding: providing BIRD's column descriptions to the model actually made results worse. More detailed annotations on visible columns helped the model find workarounds for questions it should have refused. This suggests that in permission-restricted scenarios, more information is not necessarily better—extra semantic clues may be used by the model to bypass refusal logic.

Ablation study is a common method in machine learning evaluation: remove one component of the system at a time and observe the performance change, thereby quantifying that component's independent contribution. In this article, the author respectively removes the scope check and the "view is incomplete" prompt, comparing changes in AC-F1 and incorrect-answer rate—this is the ablation approach. Note that ablation results depend on removal order and the test set; the author also acknowledges that the prompt wording was selected on 100 BIRD questions, creating tuning contamination, so the absolute contributions of each component should be interpreted with caution.

Limitations Acknowledged by the Author

The author candidly lists several caveats: the prompt wording was selected on 100 BIRD questions that are part of the test set, creating tuning contamination; Spider was not involved in any selection and is therefore a cleaner result. Testing used only one model, whereas the paper tested thirteen. The BIRD database version is 2024, while the paper uses 2025. These differences mean the results cannot be directly equated with a full reproduction of the paper's evaluation, but they remain valuable within a comparable range.

The prompt feature is enabled by default in schemagate 1.6.0. To reproduce: benchmarks/rbac_text2sql.py; scoring reports are located in benchmarks/results/rbac/. Install: pip install schemagate.

Implications for Text-to-SQL Access Control

This independent validation sends a clear signal: in RBAC scenarios, prompts or schema hiding alone are insufficient to reliably prevent unauthorized answers; a hard scope check at the execution layer is a necessary fallback. At the same time, enriched information such as column descriptions may have side effects under restricted views and needs careful evaluation. For teams building enterprise-grade Text-to-SQL applications, schemagate's three-layer structure of "visibility + prompt + execution validation" offers a reusable design approach, and its performance on a third-party benchmark shows that the approach withstands external scrutiny.

One-line Summary

schemagate achieves 90.1 AC-F1 on BIRD in a third-party RBAC benchmark, beating the paper's best, with scope checks ensuring zero unauthorized access.

Paragraph Summary

The schemagate author used the newly released public RBAC Text-to-SQL benchmark from SIGMOD'27 to re-evaluate their tool with the paper authors' scoring method. Results show that on BIRD, schemagate's AC-F1 is 90.1, exceeding the paper's best configuration of 88.1; the rate of answering when it should refuse is 5.4%, lower than the other's 7.4%. On Spider, both have the same AC-F1 (93.0), with schemagate's incorrect-answer rate slightly higher (5.1% vs 4.6%). The key difference is schemagate's third layer, the "pre-execution scope check," which achieves zero unauthorized SQL reads on both datasets. Ablation experiments show that removing the scope check doubles BIRD's incorrect-answer rate, and the prompt alone contributes about an 11-percentage-point improvement. Counterintuitively, providing column descriptions actually led the model to find more workarounds to bypass refusals. The author also acknowledges limitations such as tuning the prompt on the test set and testing only a single model.