FairScreen
A benchmark designed to evaluate the fairness and accuracy of large language models in resume screening scenarios. It employs a 'paired application family' design and a 'robust policy fidelity' metric to systematically test multiple LLMs on fictional resumes with known ground-truth hiring decisions.
Timeline (last 90 days)
FairScreen is a project built by developer Maksim Silchenko for a Kaggle benchmarking challenge, using ten LLMs to conduct systematic testing on fictional resumes with known answers in order to evaluate LLM hiring screening decisions.
FairScreen uses a paired-application family design in which each core family contains two qualification statuses (qualified/unqualified), two identity variants, and two layout forms, producing eight decisions in total, and a family is considered passed only if all eight decisions are correct.
The robust policy fidelity metric is measured by family-level pass rate, so a score of 50% means half of the families passed all checks, not that half of the individual answers were correct.
FairScreen compared four decision approaches: direct judgment, adding fairness instructions, extracting facts before judging, and executing a runnable policy in code on the same extracted results.
FairScreen's correct answers are computed from hidden structured records before applications are rendered; LLMs are not involved in labeling, and two policy implementations and a plain-text parser reached agreement across all 2,528 final variants.
Fairness instructions had limited effect: family accuracy changed by only −0.3 percentage points on the cluster six-model panel and +2.5 percentage points on Kaggle, with confidence intervals spanning zero in both cases.
On the cluster panel, replacing the model's structured decision-making with code (based on the same extracted facts) improved family accuracy by an average of 20.5 percentage points, with Llama 3.1 8B rising from 1% under self-judgment to 56% under code adjudication.
The authors recommend: do not trust model reputation or fairness instructions, but evaluate the actual screening configuration being deployed; an executable code policy combined with reliable fact extraction may be more dependable than letting the model reason freely, provided that extraction must preserve missingness, conflicts, and uncertainty.
A bag-of-words baseline can pass approximately 41% of the core set, indicating that some test cases contain lexical-level shortcuts.
All Facts (9)
FairScreen is a project built by developer Maksim Silchenko for a Kaggle benchmarking challenge, using ten LLMs to conduct systematic testing on fictional resumes with known answers in order to evaluate LLM hiring screening decisions.
50%UnverifiedFairScreen uses a paired-application family design in which each core family contains two qualification statuses (qualified/unqualified), two identity variants, and two layout forms, producing eight decisions in total, and a family is considered passed only if all eight decisions are correct.
50%UnverifiedThe robust policy fidelity metric is measured by family-level pass rate, so a score of 50% means half of the families passed all checks, not that half of the individual answers were correct.
50%UnverifiedFairScreen compared four decision approaches: direct judgment, adding fairness instructions, extracting facts before judging, and executing a runnable policy in code on the same extracted results.
50%UnverifiedFairScreen's correct answers are computed from hidden structured records before applications are rendered; LLMs are not involved in labeling, and two policy implementations and a plain-text parser reached agreement across all 2,528 final variants.
50%UnverifiedFairness instructions had limited effect: family accuracy changed by only −0.3 percentage points on the cluster six-model panel and +2.5 percentage points on Kaggle, with confidence intervals spanning zero in both cases.
50%UnverifiedOn the cluster panel, replacing the model's structured decision-making with code (based on the same extracted facts) improved family accuracy by an average of 20.5 percentage points, with Llama 3.1 8B rising from 1% under self-judgment to 56% under code adjudication.
50%UnverifiedThe authors recommend: do not trust model reputation or fairness instructions, but evaluate the actual screening configuration being deployed; an executable code policy combined with reliable fact extraction may be more dependable than letting the model reason freely, provided that extraction must preserve missingness, conflicts, and uncertainty.
50%UnverifiedA bag-of-words baseline can pass approximately 41% of the core set, indicating that some test cases contain lexical-level shortcuts.
50%