SCOTUS
Possibly refers to a legal reasoning benchmark related to the Supreme Court of the United States, used to evaluate model performance on reasoning tasks based on Supreme Court precedents.
Timeline (last 90 days)
This counterfactual audit covers seven open-weight models (8B to 70B parameter scale) and four benchmarks spanning judicial reasoning and contract reasoning.
After the legal basis cited by the model is replaced with an irrelevant authority, the rate at which the decision changes is only 0.0%–21.7% on CaseHOLD, 30.0%–76.7% on ECHR and SCOTUS, and 43.3%–50.0% on ContractNLI.
All Facts (2)
This counterfactual audit covers seven open-weight models (8B to 70B parameter scale) and four benchmarks spanning judicial reasoning and contract reasoning.
50%UnverifiedAfter the legal basis cited by the model is replaced with an irrelevant authority, the rate at which the decision changes is only 0.0%–21.7% on CaseHOLD, 30.0%–76.7% on ECHR and SCOTUS, and 43.3%–50.0% on ContractNLI.
50%