AC-F1
An evaluation metric for assessing the end-to-end performance of Text-to-SQL tools in RBAC scenarios; it combines answer accuracy and refusal accuracy and penalizes unauthorized answers.
Timeline (last 90 days)
On the BIRD dataset, schemagate achieved an AC-F1 of 90.1, while the best of the 13 models in the paper scored 88.1 (that model could see the full schema and policies).
AC-F1 (Answer Correctness F1) is the evaluation metric adopted in the paper, jointly considering answer accuracy and refusal accuracy.
All Facts (2)
AC-F1 (Answer Correctness F1) is the evaluation metric adopted in the paper, jointly considering answer accuracy and refusal accuracy.
60%UnverifiedOn the BIRD dataset, schemagate achieved an AC-F1 of 90.1, while the best of the 13 models in the paper scored 88.1 (that model could see the full schema and policies).
50%