Machine Learning Fundamentals in Practice: A Complete Guide to Embeddings, TF-IDF, and Data Preprocessing

The success of machine learning depends not only on model selection but also on solid fundamentals — how to scientifically evaluate models, how to transform raw data into usable features, and how to handle missing values properly. This article provides an in-depth exploration of three core topics: Embeddings, Benchmarks, and Preprocessing.

How to Scientifically Evaluate Machine Learning Models

Model evaluation is the most underestimated step in the machine learning pipeline. Many beginners rely solely on Accuracy to judge model quality, but in cases of class imbalance or multi-class scenarios, this can be highly misleading. For example, in a fraud detection task with a 1:99 positive-to-negative sample ratio, a model that predicts all samples as "non-fraud" can still achieve 99% accuracy — yet it has zero practical value. This phenomenon is known as the "Accuracy Paradox," which reveals the fundamental limitations of relying on a single metric and underscores why a multi-dimensional evaluation framework is essential.

Understanding Top-k Error Metrics

Top-k error is an important tool for evaluating classification models, especially in tasks with a large number of categories (such as ImageNet image recognition with thousands of classes). The core idea is simple: as long as the true label appears among the model's top k predicted classes, the prediction is considered correct.

The widespread adoption of Top-k error metrics is closely tied to breakthroughs in deep learning for large-scale visual recognition. In 2012, AlexNet won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) by a significant margin, and the competition's official evaluation criterion was the Top-5 error rate. The ImageNet dataset contains over 14 million images spanning more than 20,000 categories (the commonly used competition subset includes 1,000 categories). In such a vast category space, requiring the model to hit the correct answer on its first try is extremely demanding, making Top-5 a more reasonable evaluation standard.

Taking Top-5 error as an example: if the true class of an image falls within the model's top 5 highest-probability predictions, it counts as a hit. This metric better reflects real-world application scenarios like recommendation systems or search ranking — users typically care about a "candidate list" rather than a single answer. This metric also profoundly influenced evaluation methods in recommendation systems, such as Hit Rate@K and NDCG@K (Normalized Discounted Cumulative Gain), which share the same core philosophy: focusing on whether the top positions in a ranked list contain content the user actually cares about. In contrast, Top-1 error requires the model's first prediction to be an exact match, setting a much stricter standard.

For ML model benchmarking, it's recommended to combine multiple dimensions for comprehensive evaluation:

• Accuracy: Overall correctness rate
• Precision: The proportion of true positives among all predicted positives
• Recall: The proportion of actual positives correctly identified
• F1 Score: The harmonic mean of Precision and Recall, particularly useful when there's a significant trade-off between the two. The harmonic mean is chosen over the arithmetic mean because it's more sensitive to lower values — F1 is high only when both Precision and Recall are high, and a severe deficiency in either will drag down the overall score
• Top-k Metrics: Whether the correct answer appears among the top k predictions

No single number can fully reflect a model's true capabilities. In real-world projects, tools like the ROC-AUC curve (which reflects the model's overall performance across different thresholds) and the Confusion Matrix should also be considered for a more comprehensive understanding of model behavior.

Feature Engineering: From One-Hot Encoding to Pre-trained Embeddings

Transforming raw categorical data into numerical representations that models can understand is a critical step in feature engineering. Choosing the right encoding method directly impacts model performance.

Why One-Hot Encoding Is Essential

Machine learning models fundamentally operate on numbers. When facing categorical features like "color" or "city," directly using integer encoding (e.g., red=1, blue=2, green=3) causes the model to incorrectly assume ordinal or magnitude relationships between categories.

One-Hot Encoding eliminates these spurious numerical relationships by creating an independent binary dimension for each category, ensuring all categories are orthogonal and treated equally in the vector space. Mathematically, it maps categorical variables onto standard orthonormal bases in Euclidean space, making the Euclidean distance between any two different categories equal (always √2), thus preventing the model from learning nonexistent ordinal relationships from the encoding.

However, the drawbacks of one-hot encoding are equally apparent: when the number of categories is massive (e.g., a vocabulary may contain tens of thousands of words), it produces extremely sparse, high-dimensional vectors that waste storage space and fail to capture semantic relationships. This problem is especially pronounced in natural language processing — a one-hot representation for a 50,000-word vocabulary means each word requires a 50,000-dimensional vector with only one position set to 1 and all others set to 0, resulting in extremely low information density.

Beyond one-hot encoding, the industry has developed several alternative encoding strategies: Label Encoding is suitable for ordinal variables with natural ordering (e.g., education levels "elementary < high school < college"); Target Encoding replaces category values with statistics of the target variable, performing well on high-cardinality features but requiring care to avoid overfitting; Feature Hashing uses hash functions to map high-dimensional categories into a fixed-dimensional space, dramatically reducing dimensionality at the cost of minor hash collisions. The choice of encoding method should consider category cardinality, data volume, and model type — tree-based models (such as XGBoost and Random Forest) are relatively insensitive to encoding methods, while linear models and neural networks heavily depend on proper encoding.

How Pre-trained Embedding Matrices Solve the High-Dimensional Sparsity Problem

This is exactly where pre-trained embeddings shine. Embeddings map high-dimensional sparse one-hot vectors into a low-dimensional dense space, where these low-dimensional vectors can encode semantic information — for example, the distance between "king" and "queen" in the embedding space reflects their semantic similarity.

The development of pre-trained embeddings has gone through several important stages. In 2013, Mikolov et al. at Google proposed Word2Vec, training word vectors on large-scale corpora through Skip-gram and CBOW architectures, validating the power of distributed semantic representations at scale for the first time — the famous analogy "king - man + woman ≈ queen" originated from this work. In 2014, Stanford University's GloVe (Global Vectors) combined the advantages of matrix factorization methods and local context window methods by factorizing global co-occurrence matrices. In 2018, ELMo introduced context-dependent embeddings — the same word receives different vector representations in different contexts, solving the polysemy problem (for example, "bank" should have different representations in "river bank" and "bank account"). Subsequently, large language models based on Transformer architectures like BERT and the GPT series further revolutionized embedding technology, producing contextual embeddings that significantly outperformed static word vectors on virtually all NLP downstream tasks.

Introducing pre-trained embedding matrices (such as Word2Vec, GloVe, or embeddings from large language models) means the model can reuse knowledge learned from massive corpora without training from scratch. This brings three major advantages:

1. Reduced training data requirements: Even small datasets can obtain good feature representations. This is especially critical in specialized domains like healthcare and law where labeled data is scarce
2. Lower computational costs: Skipping the lengthy training process for the embedding layer. GPT-3's training cost, for example, was estimated at over $4.6 million, while using its generated embeddings for downstream tasks costs virtually nothing
3. Improved generalization: More stable performance on downstream tasks, particularly effective in data-limited scenarios. This "Transfer Learning" paradigm has become the standard workflow in modern NLP and computer vision

Today, the application of embeddings extends far beyond text. Graph Embeddings are used for social network analysis and knowledge graphs, image embeddings for visual search and face recognition, and multimodal embeddings (such as CLIP) unify text and images into a shared vector space, driving unified representation learning across domains.

TF-IDF Deep Dive: A Classic Method for Text Feature Extraction

In the fields of text mining and natural language processing, TF-IDF (Term Frequency-Inverse Document Frequency) is a time-tested classic method. Its mathematical formulation is not arbitrary — it embodies profound information-theoretic intuition. Despite the emergence of numerous new text representation methods in the deep learning era, TF-IDF remains widely used in industry thanks to its strong interpretability, computational efficiency, and zero training requirements.

How TF-IDF Is Calculated

TF-IDF is the product of two components:

• Term Frequency (TF): Measures how frequently a word appears in a single document — higher frequency generally indicates greater importance. In practice, raw term frequency is often normalized (divided by the total number of words in the document) or sub-linearly scaled (e.g., using 1+log(tf)) to prevent high-frequency words in long documents from receiving disproportionately high weights
• Inverse Document Frequency (IDF): Typically written as log(N / df), where N is the total number of documents and df is the number of documents containing the word. In practice, 1 is usually added to the denominator (i.e., log(N / (df+1))) to avoid division-by-zero errors

Why IDF Uses a Logarithm

This is key to understanding TF-IDF. Without the logarithm, as the corpus size grows, IDF values would inflate linearly or even faster, causing the weights of rare words to be excessively amplified. The logarithm smooths this growth, compresses the dynamic range of values, and makes weight changes more reasonable.

The logarithmic design of IDF has a deep connection to the concept of "self-information" in information theory. In information theory, the information content of an event is defined as the negative logarithm of its probability: I(x) = -log(P(x)). The rarer an event, the lower its probability, and the greater its information content. Treating a word's appearance in a document as an event, its probability can be approximated as df/N (the number of documents containing the word divided by the total number of documents), giving the word's information content as -log(df/N) = log(N/df) — precisely the standard form of IDF. Therefore, IDF essentially measures the "information content" or "surprisal" of a word.

The deeper implications are:

• If a word appears in nearly every document (like stop words such as "the" or "is"), its discriminative power is extremely low, IDF approaches zero, and its weight is naturally suppressed in the final score
• Words that appear in only a few documents tend to be stronger topic indicators and receive higher weights

This design gives TF-IDF the natural ability to "highlight key information and suppress useless noise," and it's still widely used in search engine ranking and text classification tasks today.

It's worth noting that TF-IDF has also spawned important variant algorithms in practice. The most famous is BM25 (Best Matching 25), which builds on TF-IDF by introducing document length normalization and a term frequency saturation mechanism — the weight increase from higher term frequency has an upper bound, preventing infinite weight growth when a word appears extremely frequently in a document. BM25 has become one of the core ranking algorithms in modern search engines, serving as the default relevance scoring function in mainstream search systems like Elasticsearch and Apache Lucene, and is often used as a baseline method in the retrieval stage of RAG (Retrieval-Augmented Generation) systems.

Handling Missing Data: Understanding the Mechanism Matters More Than Blind Imputation

Real-world data is almost never perfect, and handling missing values is a daily challenge for data scientists. Statistics show that data scientists spend over 60% of their working time on average on data cleaning and preprocessing, with missing value handling being one of the most common tasks. The key is understanding the "mechanism" behind missingness rather than mechanically filling in values.

The Three Missing Data Mechanisms Explained

The three-mechanism classification of missing data was formally proposed by statistician Donald Rubin in 1976 and forms the theoretical cornerstone of modern missing data analysis. Statistically, missingness is typically categorized into three types:

• Missing Completely At Random (MCAR): Missingness is unrelated to any variable and is purely accidental. For example, data goes unrecorded due to sporadic equipment failures. In this case, simple deletion or mean imputation introduces relatively little bias. This can be statistically verified using Little's MCAR test
• Missing At Random (MAR): Missingness is related to other observed variables. For example, missing income data may correlate with respondents' age and education level — younger people or high-income groups may be more inclined to refuse answering income questions. Here, other observed features can be leveraged for more reasonable imputation
• Missing Not At Random (MNAR): Missingness is related to the unobserved values themselves, making this the most difficult to handle. A classic example is clinical trials where patients with more severe conditions are more likely to drop out, causing data loss — the missingness itself carries information about the missing values. Handling MNAR typically requires introducing Selection Models or Pattern Mixture Models, which require explicit parametric assumptions about the missing mechanism, making the robustness of conclusions highly dependent on the reasonableness of assumptions and usually requiring sensitivity analysis to assess reliability

When Missing Values Can Be Ignored

Not all missing data requires complex treatment. When the missing proportion is very low (typically below the empirical threshold of 5%) and the mechanism is MCAR, directly removing the affected samples has minimal impact on the overall analysis, making "ignoring missingness" a pragmatic and acceptable choice.

However, if the amount of missing data is large or systematic bias exists, hasty deletion or imputation can introduce serious bias and distort model conclusions. For example, in a model predicting user purchase behavior, if age information is systematically missing for high-value users (because they are more privacy-conscious), directly deleting these samples will severely compromise the model's ability to model the high-value user segment.

The Correct Workflow for Handling Missing Data

The first step in handling missing data is always "diagnosis":

1. Use visualization (such as missing value heatmaps and missing pattern matrix plots) and statistical tests to determine the missing pattern
2. Confirm whether the missingness is MCAR, MAR, or MNAR
3. Choose a strategy based on the diagnosis — deletion, imputation (mean/median/KNN/model prediction), or specialized modeling

Among imputation methods, KNN (K-Nearest Neighbors) imputation uses the K most similar complete samples in feature space for weighted averaging. Its advantage is that it requires no parametric assumptions about the data distribution and can adaptively capture local data structure (scikit-learn's KNNImputer is a commonly used implementation). Model-based prediction imputation includes using regression models, random forests (such as the MissForest algorithm), or deep learning models to predict missing values.

In recent years, the Multiple Imputation framework (such as MICE — Multiple Imputation by Chained Equations) has been regarded as the gold standard for handling missing data in statistical analysis. It works by iteratively building conditional models for each variable with missing values and sampling from them, generating multiple complete datasets, analyzing each separately, and then combining the results. This naturally incorporates the uncertainty introduced by imputation into the final estimates, yielding more honest confidence intervals and hypothesis test results. When choosing an imputation method, trade-offs must be made between computational cost, data scale, missing proportion, and downstream task requirements.

Conclusion

From Top-k evaluation metrics to the semantic encoding of pre-trained embeddings, from the logarithmic design of TF-IDF to the mechanism-based diagnosis of missing data, these seemingly basic concepts are precisely the building blocks of reliable machine learning systems. For developers looking to strengthen their ML fundamentals, deeply understanding these underlying principles offers far more long-term value than blindly chasing the latest models.

These foundational concepts are also deeply interconnected: embeddings are essentially a dimensionality-reducing feature engineering technique, TF-IDF can be viewed as a text feature representation method from the pre-embedding era, and the quality of data preprocessing (including missing value handling) directly determines the reliability of feature representations and model evaluation. Mastering the internal logic and interconnections of these components is what enables sound technical decisions when facing complex real-world problems. Solid fundamentals are the most powerful weapon for tackling complex practical challenges.