schemagate
A table-filtering tool for text-to-SQL scenarios. It first performs coarse filtering of candidate tables using mathematical methods such as BM25, embeddings and table-name matching, then fuses and ranks results via RRF, and can optionally call an LLM to re-rank the candidate tables.
Timeline (last 90 days)
schemagate achieved an F1 of 0.83 in a 200-question pilot test, and 32 of the 33 false rejections stemmed from a regex bug: the check failed to correctly handle uppercase AS (such as FROM students AS T1), causing aliases to be misread as table names and the query to be rejected.
schemagate is a middle layer between the LLM and the database that only shows the model the tables and columns the user is authorized to read, and checks whether a query touches anything else before any generated SQL is executed.
The lesson the author summarizes is: self-built test sets easily fall into the blind spot of "you test what you thought of," and the greatest value of an external benchmark is attacking the system with boundary conditions you never imagined.
Parsing each answered query with an independent SQL parser and comparing every table and column against the role's GRANTs revealed no out-of-bounds access.
The systems in the paper used 2024-25 models, whereas this test used a 2026 model (Claude Sonnet 5.5), so part of the performance gap comes from capability improvements in the model itself.
On the 600 records never touched during the pilot, schemagate's rejection F1 was 0.986.
In the full Spider-ACL test, schemagate answered 1.8% of the forbidden questions, but the records of unauthorized data reads were 0/8,128.
After the bug fix, schemagate ran all 19,624 records of Spider-ACL using Claude Sonnet 5.5, reaching a rejection F1 of 0.984.
On the BIRD dataset, schemagate achieved an AC-F1 of 90.1, while the best of the 13 models in the paper scored 88.1 (that model could see the full schema and policies).
On the Spider dataset, schemagate tied with the same model (Claude Sonnet 4.5) at 93.0; the paper's best result on Spider came from DeepSeek V3.2-Reasoning at 95.9.
30 more timeline events
All Facts (20)
In the full Spider-ACL test, schemagate answered 1.8% of the forbidden questions, but the records of unauthorized data reads were 0/8,128.
50%UnverifiedAfter the bug fix, schemagate ran all 19,624 records of Spider-ACL using Claude Sonnet 5.5, reaching a rejection F1 of 0.984.
50%Unverifiedschemagate is a middle layer between the LLM and the database that only shows the model the tables and columns the user is authorized to read, and checks whether a query touches anything else before any generated SQL is executed.
50%UnverifiedThe lesson the author summarizes is: self-built test sets easily fall into the blind spot of "you test what you thought of," and the greatest value of an external benchmark is attacking the system with boundary conditions you never imagined.
50%UnverifiedThe systems in the paper used 2024-25 models, whereas this test used a 2026 model (Claude Sonnet 5.5), so part of the performance gap comes from capability improvements in the model itself.
50%UnverifiedOn the 600 records never touched during the pilot, schemagate's rejection F1 was 0.986.
50%UnverifiedOn both datasets, the number of queries in which schemagate actually read a forbidden column was 0.
50%UnverifiedBy adding to the prompt 'you are seeing part of this database; if you need more information, refuse to answer', schemagate reduced incorrect answers from 16% to 5% in a 100-question pilot.
50%UnverifiedOn the BIRD dataset, schemagate achieved an AC-F1 of 90.1, while the best of the 13 models in the paper scored 88.1 (that model could see the full schema and policies).
50%Unverifiedschemagate is a retriever / MCP server that enforces schema filtering for SQL agents.
50%UnverifiedOn the Spider dataset, schemagate tied with the same model (Claude Sonnet 4.5) at 93.0; the paper's best result on Spider came from DeepSeek V3.2-Reasoning at 95.9.
50%UnverifiedImplementing schema filtering as an MCP server means the model retrieves available table and column metadata through that server before generating SQL, and the server completes the permission trimming at the retrieval stage, rather than handing the full schema to the model and then relying on it to voluntarily follow the policies.
50%UnverifiedThe schemagate authors note its limitations: only one model was tested, and the prompt wording was chosen on 100 BIRD questions; removing these two factors leaves the score essentially unchanged (90.08).
50%Unverifiedschemagate performs a policy check on each generated query before execution, and removing this step doubles the violation rate.
50%UnverifiedThe test used only one model (Claude Sonnet 4.5), whereas the paper tested thirteen models.
50%Unverifiedschemagate uses a three-layer mechanism: role policy as visibility, a one-line prompt declaring the view incomplete, and a scope check before execution.
50%UnverifiedRunning five times with Claude Sonnet 4.5, schemagate achieved an AC-F1 of 90.1 on BIRD, versus 88.1 for the paper's best configuration.
50%UnverifiedOn BIRD, the proportion of questions schemagate should have refused but answered was 5.4%, lower than the 7.4% of the paper's best configuration.
50%UnverifiedOn Spider, schemagate and the paper's best configuration both achieved an AC-F1 of 93.0, but schemagate's wrong-answer rate was 5.1%, slightly higher than the paper's 4.6%.
50%UnverifiedOn both the BIRD and Spider datasets, no SQL reading forbidden data appeared in schemagate.
50%