ECHR
Possibly refers to a legal reasoning benchmark related to the European Court of Human Rights, used to evaluate a model's judicial reasoning ability in the field of human rights law.
Timeline (last 90 days)
This counterfactual audit covers seven open-weight models (8B to 70B parameter scale) and four benchmarks spanning judicial reasoning and contract reasoning.
After the legal basis cited by the model is replaced with an irrelevant authority, the rate at which the decision changes is only 0.0%–21.7% on CaseHOLD, 30.0%–76.7% on ECHR and SCOTUS, and 43.3%–50.0% on ContractNLI.
All Facts (2)
This counterfactual audit covers seven open-weight models (8B to 70B parameter scale) and four benchmarks spanning judicial reasoning and contract reasoning.
50%UnverifiedAfter the legal basis cited by the model is replaced with an irrelevant authority, the rate at which the decision changes is only 0.0%–21.7% on CaseHOLD, 30.0%–76.7% on ECHR and SCOTUS, and 43.3%–50.0% on ContractNLI.
50%