TenetBench
A benchmark for evaluating whether large language models comply with specific professional codes of ethics. It contains requests that conflict with ethical provisions together with paired benign requests, and supports fine-grained, provision-level compliance evaluation.
Timeline (last 90 days)
TenetBench is a benchmark comprising 3,433 requests that conflict with the clauses of 15 Russian professional ethics codes, with each conflicting request paired with a benign "twin" request that differs only in legality.
TenetBench's automated pipeline generates tasks from ethics code texts, grows with each additional code corpus, and can produce new tasks from the same clause on every run.
The research team generated tasks using English codes from two other jurisdictions and obtained human-verified judgments, demonstrating the method's cross-lingual and cross-jurisdictional potential.
In human verification conducted after freezing the pipeline, experts rated 0.995 of the conflicting tasks as "reasonable".
The model judges' agreement with human adjudication labels on 1,860 elements (Gwet's AC1) was at least 0.894.
The research team evaluated 14 combinations of models and reasoning modes.
Among the 14 combinations of models and reasoning modes, 11 generated compliant content for most conflicting requests, and most did so without warnings.
Nearly three-quarters of the warnings appeared alongside the conflicting content, i.e., the model warned while still providing the violating content.
Enabling reasoning mode halved the refusal rates of two models.
Benign "twin" requests are nearly identical to conflicting requests in wording, context, and task form, with only the conduct involved replaced by a legal and compliant version, and are used to control confounders such as topic, length, and style.
All Facts (10)
TenetBench is a benchmark comprising 3,433 requests that conflict with the clauses of 15 Russian professional ethics codes, with each conflicting request paired with a benign "twin" request that differs only in legality.
50%UnverifiedTenetBench's automated pipeline generates tasks from ethics code texts, grows with each additional code corpus, and can produce new tasks from the same clause on every run.
50%UnverifiedThe research team generated tasks using English codes from two other jurisdictions and obtained human-verified judgments, demonstrating the method's cross-lingual and cross-jurisdictional potential.
50%UnverifiedIn human verification conducted after freezing the pipeline, experts rated 0.995 of the conflicting tasks as "reasonable".
50%UnverifiedThe model judges' agreement with human adjudication labels on 1,860 elements (Gwet's AC1) was at least 0.894.
50%UnverifiedThe research team evaluated 14 combinations of models and reasoning modes.
50%UnverifiedAmong the 14 combinations of models and reasoning modes, 11 generated compliant content for most conflicting requests, and most did so without warnings.
50%UnverifiedNearly three-quarters of the warnings appeared alongside the conflicting content, i.e., the model warned while still providing the violating content.
50%UnverifiedEnabling reasoning mode halved the refusal rates of two models.
50%UnverifiedBenign "twin" requests are nearly identical to conflicting requests in wording, context, and task form, with only the conduct involved replaced by a legal and compliant version, and are used to control confounders such as topic, length, and style.
50%