LLM Recruiting Benchmark: Who's Really Deciding Your Resume's Fate?

When companies use large language models (LLMs) to screen job applications, the factors that determine whether you pass may be ones you'll never see: which model was used, what instructions it was given, how its parameters were configured. Developer Maksim Silchenko built the FairScreen project for a Kaggle benchmarking challenge, running ten LLMs against fictional resumes with known correct answers. The results reveal an unsettling reality — a model's reputation, so-called fairness instructions, and seemingly reasonable explanations tell you nothing about whether it actually followed the hiring rules correctly.

One telling example illustrates just how serious the problem is: a job required three years of relevant experience, and the applicant had under two years. The model accurately extracted the applicant's years of experience — and then still ruled them "qualified." A contradiction any human reviewer would catch instantly, yet the model didn't flinch.

What's Actually Being Tested

A competent screening model should change its decision when an applicant crosses a defined threshold, and hold its decision steady when only their name or formatting changes. FairScreen makes a sharp observation: testing only the latter (consistency) misses an obvious failure mode — a model that rejects everyone is "perfectly consistent" across different names.

To address this, the project uses a "paired application family" design. Take a backend development role requiring at least three years of Python experience plus a passing SQL assessment. Each core family contains two qualification states (meets/doesn't meet requirements), two identity variants (different names), and two formatting variants (rearranged, rewritten, with irrelevant details added), producing eight decisions per family. Only when all eight decisions are correct does that family pass. The author calls this metric "robust policy fidelity" — a score of 50% means half the families passed all their checks, not that half the individual answers were correct.

The test compared four decision approaches: direct judgment, judgment with fairness instructions added, fact extraction followed by judgment, and running an executable policy on those same extracted facts. That last comparison is particularly clever — it fixes the extraction step and asks only whether the model's final decision is consistent with what its own extracted values should yield under the policy. All applicants and policies are fictional; identity variations were designed around a Singaporean local context.

Worth emphasizing: the correct answers were computed from hidden structured records before the applications were rendered, with no LLM involvement in labeling. Two separate policy implementations and a plain-text parser all agreed on all 2,528 final variants, ensuring the benchmark's credibility.

---

The design logic behind "robust policy fidelity" deserves further unpacking. Traditional AI evaluations typically use accuracy to measure individual answer correctness, but in a hiring screening context, this obscures systematic bias. For instance, a model that outputs "pass" for every applicant still achieves 50% single-item accuracy on a balanced test set — while having executed exactly zero hiring rules. The core value of the paired-family design is that it requires models to simultaneously do two things: be sensitive to qualification differences (qualified and unqualified applicants get different decisions), and be robust to irrelevant changes (identical qualifications get identical decisions regardless of name or formatting). Only when all eight decisions in a family are correct does it count as a pass, making it impossible to score well by guessing half correctly. This "family-level pass rate" is closer to what real-world employers actually care about: does this system consistently apply the rules when it sees similar applications?

The Ten Models Tested

The author used two distinct testing panels. On Kaggle, five models were run (Gemini 3.7 Flash, Gemini 3.1 Flash-Lite, GPT-5.4 nano, gpt-oss-120b, Gemma 4 31B), covering 48 core families and evidence sufficiency tasks.

On the GPU cluster at the National University of Singapore (NUS) School of Computing, six open-weight models were run (gpt-oss-120b, gpt-oss-20b, Qwen3-32B with thinking disabled, Gemma 3 27B, Mistral Small 3.2 24B, Llama 3.1 8B) with more comprehensive diagnostic testing across 192 core families. The gpt-oss-120b model appeared in both panels as a cross-platform calibration point, giving a total of 10 distinct models across both panels.

Three Key Findings

Configuration May Matter More Than the Model Itself

In the opening technical writing job families, Qwen3-32B with thinking disabled approved all eight applications — including four that didn't meet the experience requirement. It was "perfectly consistent" across different names, yet wrong on half its decisions.

When the author re-ran Qwen3-32B with thinking enabled in an exploratory follow-up, the results shifted dramatically:

Qwen3-32B	Thinking Off	Thinking On (re-run)
Families with all eight correct	39%	98%
Unqualified applicants incorrectly approved	41%	1%
Median output tokens per direct decision	12	476

The gap in family scores was +59.4 percentage points (95% bootstrap interval: +53.1 to +65.6). The author is careful to note this is a "configuration result" — the re-run changed both sampling settings and output budget simultaneously, so it can't be attributed to thinking mode alone. But the practical lesson is clear: a benchmark result must include the model configuration, not just the model name.

Fairness instructions, by contrast, showed disappointing results. On the cluster six-model panel, they moved family accuracy by just −0.3 percentage points; on Kaggle, +2.5 percentage points — both intervals crossing zero. The name experiment was similarly inconclusive: identity swaps changed 2.6% of direct decisions, while swapping to a different name within the same group changed 2.2%, a negligible difference. The author honestly notes these controls only covered customer service and lab support roles, so no sweeping claims about bias across all professions are warranted.

Code Can Fix a Model's Self-Contradictory Decisions

The decisive contrast from the opening example is remarkably clear: extracted experience of 1.8 years, requirement of at least 3 years, model rules "qualified" — while running an executable policy on that same extraction yields "not qualified."

On the cluster panel, replacing the model's structured decision with code (applied to the same extracted facts) improved family accuracy by an average of 20.5 percentage points. The most dramatic gap belonged to Llama 3.1 8B, where model self-judgment produced just 1% passing families, while code-based judgment on the same extractions reached 56%:

Cluster Model	Model Decision After Extraction	Code Decision on Same Extraction
Llama 3.1 8B	1%	56%
Qwen3-32B (thinking off)	45%	73%
Gemma 3 27B	42%	65%
Mistral Small 3.2 24B	79%	91%
gpt-oss-20b	95%	99%
gpt-oss-120b	100%	100%

But code replay has clear limitations: it inherits errors from the extraction step. When a work history entry was missing an end date, the extractor sometimes returned the total of jobs with labeled dates rather than recording "total years unknown." The code then treated this incomplete sum as complete data, incorrectly rejecting the applicant. In the cluster sufficiency tests, gpt-oss-120b's pass rate actually dropped from 56% to 40%. The author's recommendation is therefore very specific: executable policies can fix decision inconsistencies, but only if the extraction preserves "missing, conflicting, and uncertain" information.

---

The "executable policy" here refers to a structured piece of code that takes facts extracted by the model (structured fields like years of experience, test scores, etc.) and applies predefined rule logic to output a pass/reject decision — completely bypassing the model's natural language reasoning. This "extraction-decision separation" architecture is sometimes called the "LLM as parser, rule engine as decision-maker" pattern in industry. Its advantage is that decision logic is transparent, auditable, and traceable — if an applicant is rejected, you can pinpoint exactly which rule triggered and which extracted field it relied on, rather than tracing back through an uninterpretable model output. The significant accuracy gains from code-based policies actually confirm a counterintuitive point: LLMs perform reasonably well at extracting key facts from unstructured text, but when mapping those facts to definitive conclusions, they readily produce judgments that contradict their own extractions. Separating these two steps lets each failure mode surface independently, rather than getting buried together in a final answer with no way to distinguish them.

Leaderboard Scores May Measure "Output Handling" More Than Model Capability

The most revealing cross-platform comparison came from gpt-oss-120b: the same 48 core families, roughly 63% on Kaggle versus 98% on the cluster. Inspecting saved outputs revealed that hosted answers were being cut off mid-word. On the 31 families where hosted answers were complete, both platforms passed 30 of them.

Structured tasks also exposed subtler problems: 14 hosted responses were missing the closing curly brace of their JSON, and while a fallback parser could recover the final decision, it couldn't recover the structured facts needed by the code. This caused the sign of the code-vs-model comparison to flip when strict JSON validity was required. Additionally, Gemini 3.7 Flash answered all 169 returned items correctly on the sufficiency task — but 23 calls returned no result after three attempts. If counted as failures, the number of passing families dropped from a perfect score to 28 out of 48.

The author draws a key methodological insight from all this: the next benchmark should separately report "successful delivery," "valid output format," and "correct decision" from the start.

Real-World Implications for AI Recruiting

FairScreen's greatest value isn't producing a ranking of which model is best. It's in revealing how many different failure modes can hide behind a stable-looking answer: misapplied requirements, incomplete extraction, a single configuration choice, or output the parser can't read. The author originally expected identity bias to be the main story — instead, these "invisible engineering details" turned out to be the real drivers of outcomes.

The author also honestly describes the study's limits: applications are synthetic English text from a single region with a small number of policies; several strong models already approach the ceiling on core families (Gemini 3.7 Flash and Gemma 4 passed all 48 hosted core families); and a bag-of-words baseline passes about 41% of the core set, suggesting some cases have lexical shortcuts. These results address explicit eligibility rules, not the scarcity-based ranking that characterizes real hiring.

For any organization considering LLMs for resume screening, the conclusion is straightforward: don't trust a model's reputation or a single fairness instruction — evaluate the actual screening configuration you plan to deploy. An executable code policy paired with reliable fact extraction is likely more trustworthy than letting the model "freestyle" its decisions. The project code is MIT-licensed, data is CC BY 4.0, and all 2,528 saved results can be browsed on Kaggle, GitHub, and an interactive viewer.

Background Note

Output truncation is especially common when using hosted APIs. Most LLM service providers impose a maximum token limit per response; when generated content exceeds that limit, the output is hard-truncated mid-stream rather than gracefully delivering a shorter but complete answer. For downstream tasks that need to parse JSON or extract specific fields, a response missing its closing bracket can render an entire record unusable. This creates a systematic bias in benchmarking: more capable models tend to generate more detailed reasoning, making them more likely to hit the token limit and get truncated — resulting in artificially low scores on tasks requiring complete output, while models with shorter outputs more easily preserve format integrity. This is exactly why the FairScreen author recommends reporting "successful delivery," "valid format," and "correct decision" separately — many benchmarks conflate these three failure types, leaving outside readers unable to tell whether a low score reflects weak model reasoning or a deployment engineering problem.