Data Cleaning in Practice: Essential Prep Work Before Embedding

The Underrated Data Cleaning Step

In a Reddit post shared by a developer, an engineer building a search feature articulated a pain point common to many AI application developers: the search API itself is the easy part — data cleaning is what actually eats up your time.

The developer mentioned having three "broken" datasets — price fields stored as strings, and category labels with no logical consistency. In the end, cleaning that dirty data took longer than writing the API itself. He raised a question that anyone working on RAG (Retrieval-Augmented Generation) or semantic search will inevitably face: how do you properly clean data before it goes into an embedding model?

This seemingly basic question actually touches on a long-ignored truth in AI engineering: a model's performance ceiling is usually determined by data quality, not the algorithm itself.

Why Data Cleaning Before Embedding Matters So Much

Embedding models convert text into high-dimensional vectors, bringing semantically similar content closer together in vector space. But this process is extremely sensitive to input quality.

Dirty Data Pollutes the Semantic Space

When a price like "$1,299.00" is embedded as plain text, the model may cluster it with other unrelated numeric text rather than understanding its business meaning as a "price." Similarly, inconsistent category labels cause retrieval results to drift from expectations — a user searching for "sneakers" might get "sports drinks" back due to noise in the category field.

Structured and Unstructured Fields Should Be Treated Differently

This is the most frequently overlooked key point in these discussions. Structured data like prices, dates, and category IDs generally should not be fed directly into an embedding model. They're better stored as metadata for filtering and ranking after vector retrieval. What actually needs to be embedded is semantically rich natural language text — product titles, descriptions, and so on.

Embedding models are neural networks that map text into a continuous vector space. Common examples include OpenAI's text-embedding-ada-002 and open-source models like BGE and E5. Their core assumption is that "semantically similar text should have similar vector representations." These models learn the semantic patterns of natural language during training, but they have no special understanding of structured symbols like currency formats, encoded strings, or ID sequences — they treat these as ordinary token sequences. This is precisely why embedding prices or category IDs directly pollutes the semantic space: the model can't distinguish whether "$1,299" is a price or just an arbitrary string of characters.

A Practical Data Cleaning Workflow

Here's a systematic approach to address the most common data quality issues.

Step 1: Field Type Normalization

For the "price stored as text" problem, the core solution is type conversion and normalization:

• Use regular expressions to strip currency symbols and thousands-separator commas, then convert to float or integer (storing values in cents avoids precision issues)
• Handle missing values: clearly distinguish "price is 0" from "price is unknown" — don't paper over both with the same default value
• Outlier detection: use simple quantile methods (e.g., IQR) to flag prices that are clearly unreasonable

Step 2: Semantic Alignment of Category Labels

"Meaningless categories" typically stem from inconsistent labeling across multiple data sources. Solutions include:

• Build a controlled vocabulary that maps all variants to a standard category set
• For labels that can't be automatically mapped, use an LLM for batch classification — let the model infer the appropriate category from the product description
• Keep the original label as a backup field for traceability

Step 3: Text Content Sanitization

For text fields that will actually go into the embedding model:

• Remove HTML tags, extra whitespace, and garbled characters
• Normalize casing and full-width/half-width characters (especially important for Chinese-language content)
• Handle texts that are too short or too long — empty descriptions should be filtered out, and overly long texts need to be chunked appropriately
• Deduplication: identical or near-identical records dilute retrieval quality

Chunking is one of the most critical decisions in RAG engineering. Embedding models typically have token length limits (e.g., 8,192 tokens), so long texts must be split and embedded separately. Common chunking strategies include: fixed character-count splitting (simple but may cut off semantic units), sentence or paragraph boundary splitting (better for semantic integrity), and sliding window splitting (adjacent chunks overlap to reduce loss of information at boundaries). Chunk size directly affects retrieval granularity — too large introduces noise, too small loses context. A range of 256–512 tokens with 10–15% overlap is generally recommended. For very short texts (e.g., incomplete descriptions of just a few words), the resulting embedding vectors tend to be directionally unstable, making it safer to filter them out.

Metadata Separation: Best Practice for Improving Retrieval Precision

One architectural principle that many developers only discover after making mistakes is this: embedding handles semantic matching; metadata handles structured filtering.

Modern vector databases (such as Pinecone, Weaviate, and Qdrant) all support attaching metadata fields to vectors and applying structured filters at query time. This means:

• Store cleaned prices, categories, and inventory status as metadata
• Only perform embedding on titles and descriptions
• At query time, first filter by metadata (e.g., "price < 500 AND category = shoes"), then rank results by semantic similarity

This layered design not only improves retrieval precision, but also significantly reduces your reliance on embedding quality — you don't need to hope that the model will "understand" pricing logic.

RAG (Retrieval-Augmented Generation) is now the dominant architecture pattern for AI applications: relevant document chunks are retrieved from an external knowledge base, then passed along with the user's question to a large language model to generate an answer. Vector retrieval is the most common retrieval method in RAG — documents are embedded in advance and stored in a vector database; at query time, the question is embedded the same way, and cosine similarity or approximate nearest neighbor (ANN) algorithms find the most relevant chunks. Metadata filtering acts as a "pre-filter" in this pipeline, dramatically narrowing the candidate set before semantic ranking — improving both precision and reducing latency. Vector databases including Pinecone, Qdrant, and Weaviate natively support combining ANN search with structured field filtering (i.e., pre-filtering or post-filtering strategies).

Building an Automated Cleaning Pipeline

Another hidden reason developers spend so much time on cleaning is that the process often lacks a reproducible pipeline. Here are recommended strategies:

• Write each cleaning step as an independent, testable function — not a one-off script
• Use Pandas, Polars, or dedicated data validation tools (such as Pandera or Great Expectations) to establish data quality assertions
• Log before-and-after statistics (record count, missing rate, number of anomalies) to verify the effect of each cleaning step

When data sources are updated, an automated pipeline turns "another late night of cleaning" into "just run the script."

Pandera and Great Expectations are two Python libraries focused on data quality validation. Pandera uses a declarative Schema approach to define rules for DataFrame fields — types, value ranges, non-null constraints, etc. — making it suitable for lightweight assertions within data processing functions. Great Expectations is more oriented toward team collaboration: it supports saving data expectations as configuration files and generating visual validation reports, making it well-suited as a quality gate in a data pipeline. In AI application contexts, it's recommended to add assertions at minimum for the final dataset before it enters embedding — for example: text fields have no null values, price fields are all positive, and category field values fall within the controlled vocabulary. These checks catch issues early when upstream data changes, preventing silent contamination of your vector index with dirty data.

Conclusion: Data Engineering Is the Foundation of AI Applications

This straightforward developer question actually reveals an industry-wide consensus: in AI application development, writing model-calling code is often just a small fraction of the work — data preparation is the real heavy lift.

For any project involving embedding and semantic search, rather than agonizing over which model to use, first get your data cleaning and metadata architecture solid. Because no matter how powerful an embedding model is, it cannot conjure clean semantics from a pile of price strings and chaotic categories.