An Access-Control Bug Exposed by a Benchmark: F1 Hits 0.984 After the Fix

When an LLM generates SQL, how do you ensure it doesn't cross permission boundaries and read data it shouldn't see? schemagate is a middleware layer sitting between the LLM and the database. It only exposes the tables and columns a user is authorized to read to the model, and before any generated SQL is executed, it checks whether the query touches anything else. Recently, the author ran it against the Spider-ACL benchmark, and the first day exposed a subtle bug—after the fix, the rejection F1 score reached 0.984.

The Benchmark: Testing Access Control with Real Postgres Labels

Spider-ACL comes from an EACL 2026 paper (Klisura et al.). It pairs each Spider question with four user roles, expressed as real PostgreSQL GRANT statements. Each question comes with Postgres's actual response when the reference query runs under that role: either allowed or "permission denied." The full dataset contains 19,624 question-role pairs. The task is clear: answer the allowed questions, reject the rest.

The core value of this design lies in the source of the labels—not human-annotated "should allow/deny," but the real results after the database engine itself enforces permission checks. This means the benchmark's ground truth carries inherent authority.

The Relationship Between the Spider Benchmark and Spider-ACL

Spider is the most widely used cross-domain benchmark in the Text-to-SQL field, containing roughly 10,000 natural language questions paired with corresponding SQL queries across multiple database schemas. Its design goal is to evaluate a model's ability to translate natural language into correct SQL, but the original Spider does not involve access control—all queries execute with full permissions by default. Spider-ACL makes a key extension on this foundation: it introduces four user roles for each question and uses real PostgreSQL GRANT statements to define each role's access permissions on tables and columns. This means the same natural language question may correspond to completely different legitimate behaviors under different roles—some roles should get an answer, others should be rejected. This "same question, different permissions" design makes access control decisions an evaluation dimension independent of SQL generation quality.

First Round: 0.83 F1 and a Regex Bug

A 200-question pilot run produced an F1 of 0.83. The problem was over-rejection—the system rejected questions it should have answered.

32 of the 33 false rejections stemmed from a single cause, and it was the author's own code. The check that extracts table names from a query handled FROM students as t1 correctly, but failed on FROM students AS T1. A regular expression was case-sensitive. When AS was uppercase, it misread T1 as a second table, couldn't find a table named T1, and rejected the query on the grounds that it was "unavailable to this caller."

More critically, most generated SQL writes AS in uppercase. This means real users would have hit this bug too. The fix required only a single flag, a regression test, and a check ensuring the test would fail without the fix. This detail deserves attention from anyone building LLM access control: all your unit tests passing doesn't mean the boundary conditions in real-world scenarios are covered.

After the Fix: Full Results Across 19,624 Records

The same pipeline, using Claude Sonnet 5.5, ran through all records:

System	Rejection F1	Answered forbidden questions	Read unauthorized data
schemagate	0.984	1.8%	0 / 8,128
Paper best (fine-tuned Mistral-7B)	0.933	Not reported	Not reported

On the 600 records the pilot never touched, F1 was 0.986.

The zero in the last column isn't blind trust. The author parsed every answered query with an independent SQL parser and compared every table and every column against the role's GRANT—no violations were found.

What about that 1.8%? Those are questions the model answered from columns it was allowed to see. Some are fine (the reference query selected columns the question never asked for), some are near-correct answers. This is the remaining part that needs fixing.

What Rejection F1 Means

Rejection F1 is the core metric of this benchmark. It treats "rejection" as the positive class and computes the harmonic mean of precision and recall. Precision measures how many of the queries the system rejected actually should have been rejected; recall measures how many of all the queries that should have been rejected were successfully intercepted by the system. A high rejection F1 means the system neither readily lets unauthorized queries through nor over-rejects legitimate requests. An F1 of 0.984 indicates the system strikes a good balance between the two types of errors. But note that F1 only evaluates the binary "answer/reject" decision, not whether the answered queries themselves are correct—this is precisely the evaluation blind spot the author explicitly points out in the boundary conditions.

Honest Boundary Conditions

The author lists several limitations worth taking seriously:

• The paper's system used 2024-25 models, while this test used a 2026 model, so part of the gap comes from improvements in the model's own capabilities.
• This score evaluates the "answer or reject" decision, not whether each answer is correct.
• The paper's BIRD portion has not yet been run.

These limitations mean the 0.984 score cannot simply be equated with "schemagate comprehensively surpasses the paper's method." The model generation gap is a significant confounding factor, and the answer-correctness evaluation dimension is not yet covered.

Key Lesson: Test Your System with Labels Someone Else Wrote

The author's own tests all passed. A benchmark written by someone else, with labels derived from Postgres itself, found a bug within two days.

This conclusion has direct practical significance for developers building LLM access control: self-built test sets easily fall into the blind spot of "you're only testing what you thought of." The greatest value of an external benchmark isn't the score ranking—it's that it attacks your system with boundary conditions you never imagined.

The fix has been released in schemagate 1.6.1, and all decisions and query results are publicly available in the GitHub repository.

One-line Summary

After fixing a case-sensitive regex bug on the Spider-ACL benchmark, schemagate achieved a rejection F1 of 0.984 with zero unauthorized data reads.

Paragraph Summary

schemagate is an access control middleware between the LLM and the database that only exposes authorized tables and columns to the model and checks for boundary violations before SQL execution. The author tested it with the Spider-ACL benchmark, which contains 19,624 question-role pairs with labels derived from PostgreSQL engine's actual GRANT execution results rather than human annotation. The first 200-question pilot scored only 0.83 F1; 32 of the 33 false rejections stemmed from a case-sensitive regex bug: when AS was uppercase, the alias was misread as a second table. After the fix, the full test reached a rejection F1 of 0.984, with 8,128 answered queries verified by an independent SQL parser to have zero unauthorized data reads. The author also acknowledges model generation differences and evaluation dimension limitations, emphasizing that the external benchmark found boundary conditions within two days that self-built tests never covered.