Deduplication Key
A set of fields used when loading data into storage or querying it to determine whether two records represent the same entity. It directly affects database UNIQUE constraints, ETL merge logic, and the sets returned by APIs. An ideal deduplication key is stable across data sources and corresponds one-to-one with business semantics.
Timeline (last 90 days)
Traditional deduplication relying on exact match keys (such as title + company) will miss soft duplicates in the recruiting market; a more robust approach is fuzzy clustering, and after clustering salaries should be modeled as distributions rather than point values.
A deduplication key is the set of fields used at data ingestion or query time to determine whether two records 'belong to the same one', and it directly determines the database's UNIQUE constraints, the ETL merge logic, and the API's returned set.
An ideal deduplication key should satisfy three conditions: stable across data sources, one-to-one correspondence with business semantics, and unaffected by presentation-layer rewriting.
A common deduplication key mistake in practice is choosing the fields 'humans most like to look at' (titles, names, descriptions), because they are the most likely to be rewritten upstream in an SEO-style manner during cross-source crawling.
When choosing a deduplication key, one should clarify the preference of 'rather duplicate than lose' versus 'rather lose than duplicate': duplication only degrades the experience, while loss directly changes the completeness of the result set.
All Facts (5)
Traditional deduplication relying on exact match keys (such as title + company) will miss soft duplicates in the recruiting market; a more robust approach is fuzzy clustering, and after clustering salaries should be modeled as distributions rather than point values.
50%UnverifiedA deduplication key is the set of fields used at data ingestion or query time to determine whether two records 'belong to the same one', and it directly determines the database's UNIQUE constraints, the ETL merge logic, and the API's returned set.
50%UnverifiedAn ideal deduplication key should satisfy three conditions: stable across data sources, one-to-one correspondence with business semantics, and unaffected by presentation-layer rewriting.
50%UnverifiedA common deduplication key mistake in practice is choosing the fields 'humans most like to look at' (titles, names, descriptions), because they are the most likely to be rewritten upstream in an SEO-style manner during cross-source crawling.
50%UnverifiedWhen choosing a deduplication key, one should clarify the preference of 'rather duplicate than lose' versus 'rather lose than duplicate': duplication only degrades the experience, while loss directly changes the completeness of the result set.
50%