Wrong Dedup Key Caused a Job Search API to Return the Same 50 Results for Every Query

A developer building a job search API shared what amounts to a textbook production incident: whether users searched for "python berlin" or "rust berlin," the returned listings looked almost identical — just in a different order. The first instinct was to blame a lazy upstream data vendor, but after digging in, the real bug turned out to be in their own result normalization logic.

Deduping on Title Alone Silently Throws Away Real Jobs

The root cause was in the dedup logic: the code deduplicated solely on the job title string, on the rationale that a title like "Senior Backend Engineer" should be unique enough within a single city.

That assumption turned out to be completely untenable. When three different companies in the same metro area all post "Senior Backend Engineer" in the same week, the dedup logic merges them into a single record and silently discards the other two.

So the API wasn't returning fewer jobs — it was returning the wrong jobs, and doing so confidently. This is the hardest class of failure to catch in monitoring: the endpoint doesn't error, latency is normal, the result count is stable — it's just wrong.

For any team building an aggregation product, this detail is worth pausing over: your dedup key doesn't just determine how clean your data is. It directly determines what users see and what they never see.

Silent failure is a class of system failure that stands in contrast to "loud failures": the service doesn't error, doesn't throw exceptions, and its latency and throughput metrics look normal — but the output itself is wrong. Because monitoring typically covers only availability and performance metrics (like HTTP 5xx, P99 latency, error logs), this kind of failure is hard for existing alerting systems to catch. For aggregation products, silent failures are especially damaging — users won't immediately notice missing data; instead, they gradually lose trust in the quality of the results. The industry calls this state of "all metrics green but users unhappy" a monitoring blind spot. The usual remedy is to introduce business-semantics-based validation, such as sampling for result diversity, cross-query result intersection analysis, and collecting explicit feedback from the user side.

The Fix: Use apply_url as the Stable Identifier

The eventual fix wasn't elegant, but it worked — change the dedup key to a composite of (title, company, location, apply_url).

Of those four fields, apply_url is the only identifier that stays stable across different aggregators. Titles get rewritten, companies get renamed, addresses get resolved into different coordinates by different geocoding services — but the application link is the one field nobody bothers to rewrite.

That conclusion applies far beyond job search. The original author specifically notes that if you're building an aggregation product for jobs, products, events, news, or the like, you should default to assuming "human-readable fields will lie to you" — and then find the field that won't.

Dedup key refers to the set of fields used to decide whether two records "are the same" at ingest or query time. It directly determines the database's UNIQUE constraints, the ETL merge logic, and the API's returned result set. An ideal dedup key should satisfy three conditions: stable across data sources, one-to-one with business semantics, and unaffected by presentation-layer rewrites. A common mistake in practice is choosing the fields "humans love to read most" (titles, names, descriptions), because those are the ones most likely to get SEO-style rewrites by each upstream during cross-source scraping. A more robust approach is to introduce an upstream entity ID, a normalized URL, or a content fingerprint (e.g., hashing key fields). When choosing a dedup key, you should also be clear about your preference for "rather duplicate than lose" versus "rather lose than duplicate": duplicates only degrade the experience, while losses directly alter the completeness of the result set.

All Tests Green — Because the Test Data Itself Was Wrong

The most painful part of the whole episode: the test suite stayed green the entire time. The reason is simple — in the synthetic data, every title was unique, so a real collision could never be triggered.

Real-world data is full of collisions. The probability of duplicate titles is far higher than intuition suggests, especially in verticals like recruiting, secondhand marketplaces, and local events.

The real lesson here isn't a technical detail like "use a composite key for dedup." It's this: the choice of dedup key is fundamentally a product decision, not a hygiene detail of data cleaning. It determines which data gets merged, which gets discarded, and therefore the entirety of what users can see.

Synthetic data refers to artificially constructed mock records used for testing, typically with deliberately highly regular fields to ensure readability and reproducibility. This practice leads to a severe disconnect between the test set and the real data distribution, especially in terms of collision rate — in the real world, the probability of overlap in high-entropy fields like names, titles, and place names is often far higher than intuition would suggest (the famous birthday paradox is the same phenomenon: among 23 people, the probability that two share a birthday exceeds 50%). To avoid this blind spot, common techniques include: sampling and sanitizing production data and feeding it back into tests, explicitly injecting same-name/same-title samples into the test set, running property-based testing specifically on the dedup logic, and tracking business-level metrics in CI such as "number of unique titles returned per query" so collisions get exposed at the testing stage.

From Finding the Bug to Packaging It as a Tool

The author eventually packaged the fix into a public job search tool hosted at x402.freeq.one/tools/jobs.html. According to their account, user complaints stopped the week the composite-key fix shipped.

The value of this case isn't in its technical complexity — the fix itself is just a few lines of code. It's in the class of easily overlooked bug it reveals: logically correct code, under wrong assumptions, silently produces wrong results. Monitoring struggles to catch it, tests can miss it because the data isn't realistic enough, and only the intuitive feedback of real users can expose it.

If you're building any kind of aggregation product, do one self-check first: when a collision happens in real data, which record does your current dedup key throw away?

Correct code, wrong assumptions is a highly representative class of defect in software engineering: the code itself runs according to its established rules — those rules are just built on a premise that doesn't match reality. It's harder to find than a syntax error or a boundary-condition bug, because the problem isn't inside the function — it's at the level of "the reason the function exists." Classic examples include: assuming usernames are unique, assuming time zones are consistent, assuming currency units are the same, assuming IDs are globally unique. Identifying this kind of problem requires stepping outside the code to examine the invariants — the premises that are assumed to hold throughout the system but were never explicitly verified or documented. Recording them and reviewing them separately during code review is an effective way to structurally reduce this risk.

One-line Summary

A job search API silently returned wrong results because it deduped on title alone; switching to a composite key including apply_url fixed it.

Paragraph Summary

A job search API developer discovered that whether users searched "python berlin" or "rust berlin," they got nearly identical job listings. Investigation revealed the problem was in the dedup logic: the code deduped only on job title, silently merging same-titled roles at different companies in the same city and discarding the others. The result was an API that didn't return fewer jobs but the wrong ones — no errors, normal latency, stable counts — a silent failure among the hardest to catch via monitoring. The fix was to change the dedup key to a composite of (title, company, location, apply_url), where apply_url is the only field stable across aggregators. The author emphasizes that tests stayed green because every title in the synthetic data was unique and could never trigger real-world collisions; choosing a dedup key is fundamentally a product decision, not a data-cleaning hygiene detail, and it directly determines what users can and cannot see.