Leave-One-Out Attribution Outputs 20, Shapley Gives Exactly 10

A Classic Attribution Failure

You trained a model. You made a prediction. Everyone asks the same question: out of this number, how much did each input contribute?

Leave-one-out (LOO) is the most intuitive answer—remove a feature, see how much the prediction drops, and that drop is how important it is. One line of code.

But in a technical demo about Shapley values, the author constructed a minimal model: the prediction is 10 only when two features appear together. The result? Leave-one-out labels each of these two features as contributing 10, which sums to twice the prediction value.

And the pitfalls don't stop there.

What is attribution? In machine learning interpretability, attribution refers to decomposing a model's output (prediction) onto individual input features, answering "how much did each feature contribute to this prediction." Attribution methods are widely used in feature importance analysis, model debugging, and business decision explanation. Common attribution methods beyond the leave-one-out and Shapley values discussed here include LIME, Integrated Gradients, and attention weight analysis. Different methods perform very differently in complex scenarios involving interacting or redundant features, which is why the axiomatic properties of attribution methods (such as efficiency and symmetry) serve as the core standard for measuring their reliability.

Leave-One-Out Fails in Both Directions

Interaction scenario: double-counting contributions

The model above has only one pair of features: has_card and is_verified. Remove either one individually, and the prediction drops from 10 to 0. Leave-one-out concludes—"removing you breaks everything, so you contributed all 10 points." Both features get 10 points, totaling 20 points, while the actual prediction is only 10. The total attributed value exceeds the prediction itself, which directly violates the Efficiency axiom of attribution.

Shapley gives 5 points each in the same scenario, totaling 10—exactly matching the prediction.

Redundancy scenario: real signal zeroed out

The second failure mode is more dangerous. Suppose two features carry identical information—signal and signal_copy. Remove either copy, and the prediction doesn't budge. Leave-one-out then assigns both 0 points.

A feature carrying real signal shows up as a completely irrelevant variable in the feature importance chart. Someone reads that, and deletes the feature.

Shapley gives each copy 4 points here. Redundant information is fairly distributed rather than erased entirely.

The two errors point in opposite directions: overestimation under interaction, underestimation under redundancy. And on purely additive models, leave-one-out and Shapley produce identical results—which is exactly why validating an attribution method with an additive example is meaningless.

What Shapley Values Actually Compute

Shapley values ask a different question: across all possible orderings in which features can arrive, how much does this feature add on average?

The formula is the classic factorial weighting:

phi_i = sum over S not containing i of
          |S|! (n-|S|-1)! / n!  *  ( v(S + i) - v(S) )

That factorial weight isn't arbitrary. |S|! (n-|S|-1)! / n! is exactly the proportion of all n! permutations in which set S comes before feature i and the rest come after—a probability distribution defined over feature arrival positions that always sums to 1.

In other words, Shapley values aren't some heuristic weighting. They're an expected marginal contribution with a precise probabilistic interpretation.

Cooperative game theory background. The Shapley value was originally proposed by Lloyd Shapley in 1953 in cooperative game theory, to solve the problem of "n players cooperate to produce a total payoff v(N)—how do you distribute it fairly?" In the game theory context, v(S) is called the characteristic function, representing the total payoff obtainable by the coalition S. The allocation given by the Shapley value is the unique solution satisfying four axioms: efficiency, symmetry, dummy, and linearity. In 2017, Lundberg and Lee proposed SHAP (SHapley Additive exPlanations), bringing this framework into machine learning interpretability: treating "features" as "players" and "model prediction minus baseline" as "total payoff," thereby transforming the attribution problem into a payoff distribution problem. SHAP has become one of the most mainstream model explanation tools today.

Three Independent Algorithms Cross-Validating Each Other

What really builds confidence is this: the same result computed through three unrelated paths, all matching down to the last digit.

1. Factorial formula—directly compute using the equation above.
2. Permutation averaging—enumerate all n! feature arrival orderings, compute the marginal contribution for each ordering, and take the mean.
3. Möbius inversion—completely bypass factorials and permutations; first invert the set function into Harsanyi dividends, then split each dividend equally among the coalitions that contribute to it:

m(S)   = sum over T subset of S of (-1)^(|S|-|T|) v(T)
phi_i  = sum over S containing i of m(S) / |S|

For that interaction model, the entire dividend concentrates on the feature pair—m({a,b}) = 10, while m({a}) = m({b}) = 0. That single line says it all: neither feature can do anything on its own.

Three paths computed independently, results identical—this kind of cross-validation is far more convincing than any single implementation "looking correct."

What is Möbius inversion? Möbius inversion is a duality transform defined on partially ordered sets in combinatorics. In the attribution context, it converts a set function v(S) defined on feature subsets into Harsanyi dividends m(S)—where m(S) represents "the incremental contribution that only arises when all features in S are present," i.e., the pure synergistic value of coalition S. For a model with only two interacting features, the Harsanyi dividend concentrates on the interaction pair: m({a,b}) = 10, while m({a}) = m({b}) = 0. This single line intuitively demonstrates that neither feature can do anything alone. The Möbius inversion path completely bypasses factorial weights and permutation enumeration, providing a purely algebraic computational path for Shapley values. All three paths agree, constituting powerful cross-validation.

Checking All Four Axioms One by One

Shapley values aren't trusted because they happen to work. They are the only attribution scheme that simultaneously satisfies all four requirements:

• Efficiency—the sum of all feature contributions equals the prediction minus the baseline. This is exactly what leave-one-out fails to achieve.
• Dummy—a feature with no effect on the outcome scores exactly 0.
• Symmetry—features with interchangeable functions receive exactly the same value.
• Linearity—attribution for a sum of models equals the sum of their individual attributions.

The demo page verifies all four on every model, with symmetry re-derived by swapping columns of the data matrix rather than trusting the page's built-in checker.

Sampling Estimation: Efficiency Preserved for Free

Exact Shapley computation requires 2ⁿ model evaluations. Beyond roughly 20 features, it becomes completely infeasible. In practice, permutation sampling is used for approximation.

Here's a noteworthy property: no matter how many samples you draw, efficiency holds exactly—even with just one permutation. Because marginal contributions within each permutation telescope into v(full set) - v(empty set), regardless of which ordering you happen to draw. What sampling sacrifices is per-feature precision—and that precision is the number you actually need to report.

Why does exact computation require 2ⁿ evaluations? The factorial formula for Shapley values requires summing over all subsets S not containing feature i, and n features have 2ⁿ subsets in total. When n=20, that's 2²⁰ ≈ 1 million model evaluations; at n=30, over 1 billion—practically infeasible. Permutation sampling works by randomly drawing several orderings from the n! permutations, computing the feature's marginal contribution in each sequence, and averaging to approximate the Shapley value. Since marginal contributions within each permutation telescope to v(full set) − v(empty set), efficiency holds exactly regardless of which permutation is drawn—sampling error only affects per-feature estimation precision. This "exact total, approximate components" property makes sampling estimation highly practical in real applications.

Attribution Has Signs

A feature that drags down the prediction gets a negative Shapley score: a helpful feature gets +6, a harmful one gets −4. Methods that only report absolute magnitudes simply cannot express this—and "this feature is actively hurting" is precisely one of the most important conclusions attribution analysis needs to deliver.

Running the full validation suite: 470 verification assertions, 23 in-page checks, all passing.

One-line Summary

Leave-one-out distorts attribution under interaction and redundancy, while Shapley values provide reliable attribution through three independent paths and four axioms.

Paragraph Summary

Through a minimal interaction model, the article reveals two fatal flaws of leave-one-out attribution: double-counting contributions under feature interaction (two features each score 10, totaling 20, exceeding the true prediction of 10), and zeroing out real signal under feature redundancy (two features carrying identical information both get 0). Shapley values solve both problems by considering the expected marginal contribution across all feature arrival orderings, with factorial weights that have a precise probabilistic interpretation. The article further demonstrates cross-validation across three independent computational paths—factorial formula, permutation averaging, and Möbius inversion—along with the theoretical guarantee that Shapley values uniquely satisfy the four axioms of efficiency, dummy, symmetry, and linearity. Finally, it discusses the property that efficiency is preserved for free in permutation sampling estimation, and the important information carried by the sign of attribution.