NitiBench+
A Thai legal question-answering benchmark covering 17 domains and 1,670 items. It uses a dual-track design that preserves both the legal text as of the judgment date and the current text, and it measures how legal temporal validity affects model performance.
Timeline (last 90 days)
An analysis of 156,650 judgments from four courts in Thailand showed that as many as 67% of judgments cited laws that were subsequently amended or repealed.
NitiBench+ is a Thai legal question-answering benchmark covering 17 domains with 1,670 items, specifically designed to measure the impact of legal temporal validity on model performance.
For each cited legal provision, NitiBench+ simultaneously stores two versions: the text in force on the date of the judgment and the current text.
NitiBench+'s evaluation system combines the Thai Bar Association scoring standard, the Bar standard score, and the overlap between cited legal provisions and the provisions actually cited by the court (Citation F1).
Tests on 13 open-weight models showed that reading the current text lowers the Bar standard score by 0.25, while missing repealed laws causes a drop of 0.79.
On 428 items with rule changes, the penalty for using the wrong legal text is 0.45, while on 331 items with only wording changes the penalty is zero.
The Citation F1 metric completely fails to capture the differences in errors caused by legal temporal validity.
Under the correct-text setting, NitiBench+ ranked 26 models on a test set of 500 held-out samples stratified by domain, and also released an isomorphic validation set for public use.
In the NitiBench+ evaluation, the best model scored 7.78 (out of 10), and the best open-weight model scored 6.39.
Without search tools, only five open-weight models could recall any Thai law.
3 more timeline events
All Facts (13)
An analysis of 156,650 judgments from four courts in Thailand showed that as many as 67% of judgments cited laws that were subsequently amended or repealed.
50%UnverifiedNitiBench+ is a Thai legal question-answering benchmark covering 17 domains with 1,670 items, specifically designed to measure the impact of legal temporal validity on model performance.
50%UnverifiedFor each cited legal provision, NitiBench+ simultaneously stores two versions: the text in force on the date of the judgment and the current text.
50%UnverifiedNitiBench+'s evaluation system combines the Thai Bar Association scoring standard, the Bar standard score, and the overlap between cited legal provisions and the provisions actually cited by the court (Citation F1).
50%UnverifiedTests on 13 open-weight models showed that reading the current text lowers the Bar standard score by 0.25, while missing repealed laws causes a drop of 0.79.
50%UnverifiedOn 428 items with rule changes, the penalty for using the wrong legal text is 0.45, while on 331 items with only wording changes the penalty is zero.
50%UnverifiedThe Citation F1 metric completely fails to capture the differences in errors caused by legal temporal validity.
50%UnverifiedUnder the correct-text setting, NitiBench+ ranked 26 models on a test set of 500 held-out samples stratified by domain, and also released an isomorphic validation set for public use.
50%UnverifiedIn the NitiBench+ evaluation, the best model scored 7.78 (out of 10), and the best open-weight model scored 6.39.
50%UnverifiedWithout search tools, only five open-weight models could recall any Thai law.
50%UnverifiedAfter introducing retrieval, 20 of the 26 models improved, while 6 declined.
50%UnverifiedThe NitiBench+ leaderboard records the cost and hallucinated citations for each item, and commits to continuously accepting newly released models.
50%UnverifiedFor benchmark designers, a dual-version statute library and evaluation settings at separate time points should become standard configuration in the legal domain.
50%