Spider-ACL
An access-control benchmark built as an extension of Spider that introduces multiple user roles for each question and defines permissions using real PostgreSQL GRANT statements.
Timeline (last 90 days)
The Spider-ACL benchmark comes from an EACL 2026 paper (Klisura et al.), which pairs each Spider question with four user roles and presents them as real PostgreSQL GRANTs.
The Spider-ACL dataset contains 19,624 question-role pairs, each with the actual Postgres response (allowed or permission denied) when the reference query is run under that role.
After the bug fix, schemagate ran all 19,624 records of Spider-ACL using Claude Sonnet 5.5, reaching a rejection F1 of 0.984.
In the full Spider-ACL test, schemagate answered 1.8% of the forbidden questions, but the records of unauthorized data reads were 0/8,128.
The paper's best system (a fine-tuned Mistral-7B) achieved a rejection F1 of 0.933 on Spider-ACL and did not report the rate of answering forbidden questions or of unauthorized data reads.
Rejection F1 is the core metric of Spider-ACL: it treats "reject" as the positive class and computes the harmonic mean of precision and recall; it only evaluates the binary answer/reject decision and does not evaluate whether the answered queries themselves are correct.
All Facts (6)
The Spider-ACL benchmark comes from an EACL 2026 paper (Klisura et al.), which pairs each Spider question with four user roles and presents them as real PostgreSQL GRANTs.
50%UnverifiedThe Spider-ACL dataset contains 19,624 question-role pairs, each with the actual Postgres response (allowed or permission denied) when the reference query is run under that role.
50%UnverifiedAfter the bug fix, schemagate ran all 19,624 records of Spider-ACL using Claude Sonnet 5.5, reaching a rejection F1 of 0.984.
50%UnverifiedIn the full Spider-ACL test, schemagate answered 1.8% of the forbidden questions, but the records of unauthorized data reads were 0/8,128.
50%UnverifiedThe paper's best system (a fine-tuned Mistral-7B) achieved a rejection F1 of 0.933 on Spider-ACL and did not report the rate of answering forbidden questions or of unauthorized data reads.
50%UnverifiedRejection F1 is the core metric of Spider-ACL: it treats "reject" as the positive class and computes the harmonic mean of precision and recall; it only evaluates the binary answer/reject decision and does not evaluate whether the answered queries themselves are correct.
50%