Accuracy Misleads You: ROC AUC and Cross-Validation in Practice from ML Zoomcamp Module 4

In the early stages of learning machine learning, accuracy is often the first metric reported. ML Zoomcamp student Marcos Nieves trained a lead-scoring classifier in Module 3 and got an accuracy of 0.645, which looked "okay." But Module 4 shatters that illusion—when classes are imbalanced, accuracy hides everything that actually matters.

This article is based on Marcos's homework notes and breaks down the practical essentials of ROC AUC, the precision/recall trade-off, and cross-validation. All numbers come from his public Jupyter Notebook, and the code is reproducible.

1. Using AUC as a "feature importance flashlight"

Before training the model, Marcos did something smart: he treated each numeric column directly as a predictor and computed its ROC AUC against the target variable (inverting when AUC < 0.5, which indicates negative correlation). The results were:

• annual_income: 0.608
• number_of_courses_viewed: 0.723
• interaction_count: 0.765
• lead_score: 0.788 ← strongest single feature

This simple scan yielded two key insights. First, lead_score alone can separate positive and negative samples fairly well, which means that if the model performs poorly, you shouldn't blame the algorithm first. Second, there is a hidden trap: if you naively loop over all numeric columns, you'll also take the target column converted and compute its AUC against itself, yielding a perfect 1.000. Marcos fell into exactly this pitfall—you must explicitly exclude the target column.

This "raw-feature AUC scan" isn't rigorous, but as a quick diagnostic tool it tells you before modeling which features are worth keeping and which may drag you down.

ROC AUC stands for Area Under the Receiver Operating Characteristic Curve. The ROC curve plots the false positive rate (FPR) on the x-axis and the true positive rate (TPR, i.e., recall) on the y-axis, drawn by continuously moving the classification threshold. AUC ranges from 0 to 1, where 0.5 is equivalent to random guessing and 1.0 represents perfect ranking. It has an equivalent and intuitive probabilistic interpretation: the probability that, given a randomly drawn positive sample and a randomly drawn negative sample, the model assigns a higher predicted score to the positive one. For this reason, AUC measures the model's overall ranking ability rather than its classification performance at a fixed threshold—which is also why it is insensitive to class imbalance. When AUC is below 0.5, it indicates that the feature is negatively correlated with the target; inverting it (1 - AUC) makes it usable as a positive signal.

2. Model performance: AUC 0.732—accuracy isn't the whole story

The formal model uses DictVectorizer for one-hot encoding (fit on the training set only), logistic regression with solver="liblinear" and C=1.0, and obtains probabilities via predict_proba. The validation-set AUC is 0.732.

The interpretation of AUC is the most delightful part of Module 4: it represents the probability that, given a randomly drawn positive sample and a randomly drawn negative sample, the model gives the positive one a higher score. In other words, Marcos's model has a 73% chance of ranking a random converter ahead of a random non-converter. This number requires no threshold choice and is insensitive to class balance—accuracy cannot express any of this information.

It is worth emphasizing that DictVectorizer must be fit on the training set only; otherwise validation-set information leaks and leads to overly optimistic evaluation. This is an explicitly required step in the homework specification and a detail often overlooked in real projects.

DictVectorizer is a tool in scikit-learn for handling heterogeneous feature tables: it converts a list of dictionaries (each dictionary represents a row, with feature names as keys and feature values as values) into a numeric matrix. For string features, it automatically performs one-hot encoding; for numeric features, it keeps the original values. The resulting sparse matrix can be fed directly into linear models such as logistic regression. The key lies in the separation of fit and transform: the fit stage learns the mapping from feature names to column indices (i.e., the vocabulary), and must be called on the training set only, after which the same vectorizer is used to transform the validation or test set. If you fit on the full dataset first, category information that appears in the validation set enters the feature space in advance, causing data leakage and making the evaluation overly optimistic.

3. The threshold trade-off is real (you must plot it)

Sticking with the default 0.5 threshold is laziness. Marcos scanned thresholds from 0.00 to 1.00 in steps of 0.01 and observed that precision rises as recall falls, with the two crossing near 0.63 (P≈0.733, R≈0.725). The F1 peak occurs earlier, at 0.41 (F1≈0.758).

This means the default 0.5 is not optimal for any metric; it's just the library's default. Business decisions should be based on costs: if missing a converter costs more than sending an extra email, the threshold should move left (below 0.5, increasing recall); if harassment costs more, move it right (increasing precision). Draw the chart instead of letting the default decide for you.

Precision and recall differ in their denominators: precision = true positives / (true positives + false positives), answering "of the samples the model predicted as positive, how many are actually positive"; recall = true positives / (true positives + false negatives), answering "of all actual positive samples, how many were found." The two trade off against each other: lowering the threshold classifies more samples as positive, raising recall but usually lowering precision; raising the threshold does the opposite. The F1 score is their harmonic mean, used to balance the two in a single number, but it implicitly assumes precision and recall are equally important. In real business settings, weights should be adjusted according to the cost of misclassification rather than blindly maximizing F1.

4. Cross-validation: the standard deviation is the deliverable, and the solver is a trap

Marcos used 5-fold cross-validation (KFold(n_splits=5, shuffle=True, random_state=1)), refitting the vectorizer in each fold, and obtained per-fold AUCs of 0.731, 0.730, 0.736, 0.741, and 0.721. Mean ≈ 0.732, standard deviation 0.007.

The standard deviation itself is the deliverable—it tells you how trustworthy this number is on new data. A fluctuation of 0.007 indicates the model's performance is quite stable.

But here is a bug that could make you fail the homework: the specification requires solver="liblinear", and if you just write LogisticRegression(max_iter=1000), sklearn silently uses the lbfgs solver, which throws a ConvergenceWarning on this data, and the standard deviation becomes 0.013—exactly matching another multiple-choice option. Same data, same random seed, wrong solver, wrong answer. Read the specification carefully and copy the constructor verbatim.

K-fold cross-validation splits the training data into K parts, using one part as validation and the remaining K-1 parts for training each time, then averaging the metrics over K repetitions. Its value: a single train/validation split is heavily influenced by the random partition, whereas K-fold averaging reduces this variance, and the standard deviation of the per-fold scores directly quantifies the model's sensitivity to data perturbations. shuffle=True shuffles the sample order before splitting, preventing the data's inherent ordering (e.g., by time or category) from causing distribution shift in a fold; random_state fixes the random seed to ensure reproducibility. Note that if you perform feature engineering or encoding on the full dataset outside the cross-validation loop, information still leaks—hence the emphasis on refitting the vectorizer within each fold.

5. Tuning C: making regularization visible

Scanning C values within the same 5-fold loop:

• C=1e-06 → 0.617 ± 0.019 (regularization too strong, weights squashed, underfitting)
• C=0.001 → 0.749 ± 0.010 ← best
• C=1 → 0.732 ± 0.007

A very small C is not "safer"; it's just a model that barely learns anything. This scan turns the bias-variance trade-off from theory into a table visible to the naked eye. At the best C=0.001, AUC improves to 0.749 with a moderate standard deviation (0.010), indicating the model is neither overfitting nor underfitting.

Summary: from "it works" to "knowing what it's doing"

An accuracy of 0.65 once told Marcos that the Module 3 model "works." Module 4 provides the complete picture: an AUC of 0.732 shows decent ranking ability; the 0.5 threshold forces a precision/recall choice that no one made deliberately; a cross-validation standard deviation of ±0.007 shows the results are stable; and after tuning, C=0.001 reaches 0.749. The second description is the one worth trusting.

If you're also working on classification tasks, stop reporting only accuracy. Look at AUC, plot the threshold curve, and report the cross-validation standard deviation—these numbers are what tell you how your model actually behaves.

Full code: https://github.com/marcos-Nieves-24/machine-learning-zoomcamp/blob/main/homework-4/homework-4.ipynb