NitiBench+: How a Thai Legal Q&A Benchmark Solves the Legal Temporal Validity Challenge

The "Time Trap" in Legal Q&A

Law is not static text. A statute may be amended, repealed, or re-enacted, and court judgments apply only the law in force on the day of the ruling. Yet existing legal Q&A benchmarks typically judge answers against "today's" law, creating a fundamental misalignment: a model may be penalized for citing outdated or repealed provisions even when it is logically entirely correct.

An analysis of 156,650 judgments from four Thai courts reveals the severity of the problem: as many as 67% of judgments cited laws that were subsequently amended or repealed. This means that if these historical judgments were evaluated directly against current law, most answers would be judged "wrong." Based on this finding, researchers introduced NitiBench+, a Thai legal Q&A benchmark covering 17 domains with 1,670 entries, specifically designed to measure the impact of legal temporal validity on model performance.

Benchmark Design: Preserving Both "Law at the Time" and "Current Law"

The core innovation of NitiBench+ lies in its dual-track treatment of the temporal dimension. For each cited legal provision, the benchmark simultaneously stores two versions: the text in force on the day of the judgment, and the current text in force today. The retrieval corpus includes all historical versions of every article of every Thai law. This design allows researchers to precisely control variables and separately measure the impact of "using the wrong legal text" versus "using the correct legal text" on model performance.

The evaluation framework combines the Thai Bar Association scoring standard, the Bar Standard Score, and the overlap between cited provisions and the provisions actually cited by the court (Citation F1). The benchmark is divided into two parts: the first quantifies the cost of incorrect text, and the second ranks current models under the correct-text setting.

Part One: How Costly Is Using the Wrong Legal Text?

Researchers tested two error scenarios on 13 open-weight models: reading the current text of amended laws, and missing repealed laws. Results show that reading current text reduces the Bar Standard Score by 0.25, while missing repealed laws causes a drop of 0.79. More critically, this penalty comes mainly from rule changes—the penalty is 0.45 on 428 rule-change entries, while it is zero on 331 entries with mere wording changes. The Citation F1 metric completely fails to capture this difference, indicating that traditional citation-matching metrics mask temporal validity errors.

This finding points directly to one conclusion: using the legal text in force on the day of the judgment is the only fair evaluation setting. Otherwise, the benchmark measures not the model's legal reasoning ability but its sensitivity to legal change—something the model should not be responsible for.

Bar Standard Score is a normalized metric that aligns model outputs with bar examination scoring standards. It is typically scored by legal experts based on answer accuracy, argument completeness, and correctness of legal citations, then converted to a unified scale for cross-model comparison. The complementary Citation F1 is a pure text-matching metric that computes the harmonic mean of precision and recall between the provisions cited by the model and those actually cited by the court judgment. The key difference between the two is that the Bar Standard Score can capture semantic-level errors such as "citing the correct provision but applying the wrong version," whereas Citation F1 only cares whether the provision number appears in the answer and is completely insensitive to whether the provision's content changes over time. This is precisely why the original text emphasizes that Citation F1 cannot capture the penalty from temporal validity errors—it may give a perfect score to an answer that cites a repealed old law, because the number match itself is not wrong.

Part Two: The True Level of Current Models Under Correct Text

Under the correct-text setting, the benchmark ranks 26 models. The test set consists of 500 held-out samples stratified by domain, and a structurally identical validation set is released for public use. Evaluation covers three scenarios: answering from memory alone, being provided with the provisions cited by the court, and acting as a retrieval agent searching over current law.

The results show clear stratification. The best model scores 7.78 (out of 10), and the best open-weight model scores 6.39. More noteworthy is the impact of retrieval tools: without search tools, only five open-weight models could recall any Thai law; after introducing retrieval, 20 of 26 models improved while 6 declined. This shows that retrieval is no panacea—some models may be harmed by the quality of retrieval results or by insufficient integration capability.

The leaderboard also records the cost per entry and hallucinated citations for each entry, and commits to continuously accepting newly released models.

Open-weight models refer to models whose parameters are publicly downloadable and that allow users to deploy and fine-tune them themselves, as opposed to closed-source models accessible only via API. This distinction is particularly important in legal AI evaluation: closed-source models such as the GPT-4 series typically have stronger multilingual and long-context capabilities, while open-weight models (such as the Llama, Qwen, and Mistral series) often perform limitedly in low-resource domains like Thai law. The original text's statement that "only five open-weight models could recall any Thai law" reflects the scarcity of training data for open-weight models on non-English legal knowledge. In addition, open-weight models facilitate reproducible evaluation and localized fine-tuning, so the benchmark reports rankings for both model types, which has practical significance for advancing open research in legal AI.

Implications for Legal AI

The value of NitiBench+ extends beyond the specific context of Thai law. It raises a universal issue: any legal Q&A benchmark must track dates. When legal text changes over time, static benchmarks systematically penalize answers that were "correct at the time" while rewarding answers that happen to align with current law.

For legal AI developers, this means retrieval-augmented generation (RAG) systems need temporal awareness—not only retrieving relevant provisions but also retrieving the version in force at the time of the judgment. For benchmark designers, dual-version statute libraries and time-point-specific evaluation settings should become standard in the legal domain.

Under correct text, current models still show a clear gap on Thai law. The scores of 7.78 for the best model and 6.39 for the best open-weight model indicate that there is considerable room for improvement in legal reasoning accuracy. The divergent effects of retrieval tools remind us that tools themselves do not solve problems; how to use them well is what matters.

The "temporal awareness" requirement for retrieval-augmented generation (RAG) systems here is essentially a versioned retrieval problem. Traditional RAG treats the document library as a static snapshot, and the retriever returns the best-matching passages only by semantic relevance; but in legal scenarios, the same provision number may correspond to multiple historical versions, and semantic similarity cannot distinguish the "version in force in 2019" from the "2023 amended version." Temporal-aware retrieval requires the system to attach effective-time-interval metadata to each provision version at the indexing stage, filter by judgment date or user-specified time point at the query stage, and then rank candidate versions by relevance. This is akin to the concept of "temporal databases" in software engineering—data is not merely a collection of current values but a historical sequence with a time dimension. Without this layer of design, RAG may instead introduce temporal validity errors by retrieving current text, which is one possible reason why 6 models performed worse after retrieval was introduced in the original text.