[PRD] Prevent Duplicate Contacts in CDP (contact-service) — Root-Cause Fix

HEADER BLOCK

Field	Value
PM	Zhelia Alifa
PRD Version	1.4
Status	DRAFT
PRD Type	TECH (Backend)
Epic	TBD
Squad	CDP Squad
RFC Link	TBD — unique index + atomic-upsert create
Repo	contact-service (Go + MongoDB)
Parent	PRD CDP Q3 2026
Sibling PRD	Duplicate Contact Cleansing (Pre-Migration)
Labels	epic:qontak-cdp | module:customers | feature:prevent-duplicate-contact
Last Updated	2026-07-24

---

0. Technical Context (TECH Conditional Block)

Technical problem: the contact-service create path is a non-atomic find-then-insert with no uniqueness constraint (grounded in §3), so concurrent / retried / cross-format-key creates for the same identity produce duplicate CDP contacts — recurring even after the pre-migration cleanse and corrupting identity resolution + associations for Qontak One clients.

Expected outcome: exactly one contact per identity per company on every create path — measured as 0 new duplicate groups / migrated company / week (§11), with contact-create p95 latency within budget (≤ baseline +20%).

User-facing UI changes: None. Backend-only — the create/upsert path, phone normalization, and new DB indexes. No new screen, component, page, or API response-shape change.

Scope	Owner	Items
PM-owned	Zhelia Alifa (CDP PM)	Identity semantics (what counts as "same person" — the DefineTarget signal priority); the dedup outcome (exactly one contact per identity per company); success metrics + rollout stage-gates; the policy call on whether phone-only merges are allowed (OQ-1/OQ-4).
Eng-owned	CDP BE	Exact upsert-filter construction; the Mongo isDuplicateKey helper; online/rolling partial-index build strategy; retry/backoff mechanics; where normalization is wired.

---

1. One-liner + Problem

One-liner: Stop contact-service from creating duplicate CDP contacts for the same identity (phone / email / chat account) — so that cleansing does not have to be re-run and Qontak One clients keep clean data after migration.

Problem: Even after the pre-migration cleanse, new duplicate contacts keep appearing in CDP (observed 2026-07-23: two contacts, same phone 6281210023793, same email achmadfery80@gmail.com, both channel_target: web_chat, sequential _ids, identical timestamp). The contact-create path is a non-atomic find-then-insert with no uniqueness constraint, so two concurrent creates for the same person both insert. Cleansing (the sibling PRD) removes existing duplicates; without this fix the create path re-creates them — so migration lands on data that re-dirties itself.

This is channel-agnostic, not a web_chat-only bug. web_chat is the most visible victim (its ephemeral primary key coincides with an absent stable secondary key — anonymous visitors have no email/phone), but the same defect is reached by every channel (wa, wa_cloud, instagram, telegram, line, facebook, tokopedia_chat, app_chat) and every create entrypoint (chat sync, CRM sync, direct API POST /contacts/create, bulk XLSX import, and the backfill cron) — because they all flow through the same non-atomic Perform → DefineTarget → Create path (merge_data.go:66-87). Three compounding gaps make duplicates land on channels that do have a stable identity: (1) phone is not normalized on the sync path, so +6281… / 081… / 62 81… for one person are three different keys; (2) the direct-API and bulk-import create paths use weaker dedup gates than chat sync (account matching disabled / a normalize-vs-store mismatch); and (3) cross-channel arrivals (same person via web_chat then whatsapp) unify only through the unnormalized phone field, which is exactly the field gap (1) breaks.

---

2. Target Users + Persona Context

Persona	Role	Goal	Pain	Workaround
Primary — CDP Backend Engineer	Owns contact-service create/sync	A create path that never produces a duplicate for the same identity	Non-atomic find-then-insert + no unique index → races create dupes	Repeatedly run the cleansing script
Secondary — CDP Data / PM	Owns data quality + migration	Migrate clean data that stays clean	Duplicates recur after cleanse; identity resolution + associations corrupt	Manual dedupe, endless cleanse cycles
Secondary — Client (Qontak One)	Uses the customer profile	One contact per real person	Sees two identical contacts (same phone/email)	—

---

3. Root Cause (grounded codebase assessment)

The create path is a TOCTOU (time-of-check to time-of-use) race with no database uniqueness:

1. Non-atomic find-then-insert. POST /contacts/create → ContactSyncHandler.Create (internal/app/handler/contact_sync_handler.go:76) calls DefineTarget (find-existing, :101); if target.ID == nil it calls MergeDataService.Create (:110) → MergeDataService.Create (internal/app/service/merge_data.go:632) → InsertContact (:648) → DBRepo.Create → plain InsertOne (internal/app/repository/db.go:113-114). No Upsert, no transaction, no per-identity lock.
2. No unique index on the Contact collection. Every identity index is non-unique — phone_index, email_index (db/migrations/001_create_contact.up.json), company_sso_with_email, company_sso_with_phone (db/migrations/011_contacts_with_sso.up.json), idx_contact_accounts_composite (008), idx_contact_company_sso_bsuid (021). Unique indexes exist only on field_properties / feature_flag / etc. — never contact. So MongoDB accepts both racing inserts.
3. Dedup match order (DefineTarget, merge_data.go:526-613): chat_data.id (SearchByAppContactID, :528) → email (:554) → accounts{channel,unique_id} (:617) → phone (:575) → bsuid (:594), run as concurrent goroutines, first non-nil wins.
4. Web_chat's primary key is ephemeral. chat_data.id = req.ContactID (contact_sync_request.go:109) — an anonymous web-chat visitor mints a new room contact id each session, so the first/most-specific lookup misses and it falls back to email/phone (ChannelMapping("web_chat") = "email", chat/contact_master/utils.go:16). Those fallbacks only catch a dup if the first insert already committed — the race defeats exactly that.
5. No request idempotency. RequestID is on the payload (contact_sync_request.go:27) but is never used to dedupe a retried/redelivered create (contact_sync_handler.go never queries by it).

Evidence signature: two sequential _ids allocated back-to-back + identical created_at = two creates that both passed the find-check before either insert committed.

3.1 Why this is channel-agnostic (not web_chat-only)

The race (#1) and the no-unique-index (#2) live in the shared Perform path (merge_data.go:66-87): target = DefineTarget(...); if target.ID == nil { Create(...) }. Every channel and every create entrypoint enters it, so the gap is structural, not channel-specific. web_chat is merely worst-hit because its ephemeral primary key (chat_data.id) coincides with an absent stable secondary key (anonymous → no email/phone).

6. Phone is NOT normalized on the sync/match path (cross-channel + same-channel dupes). The sync transform stores Phone: e.Phone raw (contact_sync_request.go:75); SearchByPhone matches the exact raw string (search.go:79) and DefineTarget passes form.Phone straight through (merge_data.go:623-624). So +6281…, 081…, 62 81… for one person are three distinct keys → duplicates, both same-channel (format variance) and cross-channel. A normalizer exists (internal/pkg/util/phone.go:21-34, strips +, 0→62) but is only wired into bulk import — and even there the dup check searches the raw phone while storing the normalized one (consumer/bulk_import_contact.go:534 search vs :548 store), so its own check and its stored value can disagree. (Email, by contrast, is normalized — lowercased on write + read, contact_sync_request.go:74, search.go:43 — so email dedup is sound.)

7. Multiple create entrypoints with weaker dedup gates than chat sync. Not every insert goes through the full DefineTarget priority:

   • Direct API POST /contacts/create → CreateContact → CheckDuplicate matches email + phone only — account matching is commented out (service/create_contact.go:74-86) → InsertOne (create_contact.go:133).
   • Bulk XLSX import → per-row SearchByEmail/SearchByPhone only, with the normalize-vs-store mismatch above (consumer/bulk_import_contact.go:505,534,548).
   • Backfill cron and async/bulk-merge consumers re-enter Perform/DefineTarget (cron/backfill_chat_contact.go:317, consumer/async_update_contact.go:48, consumer/bulk_merge.go:47,110). All are non-atomic; the front-door API and bulk paths are strictly weaker.

8. A race-safe template already exists in-repo. The secondary Postgres contact_master store relies on a DB unique constraint and handles the race correctly — Create catches Postgres 23505 and re-fetches (contact_master/contact_master.go:40-46, utils.go:53-60). The MongoDB side is missing exactly this. We can mirror the pattern rather than invent one.

---

4. Non-Goals

1. Not the cleansing of existing duplicates — that is the sibling PRD (Duplicate Contact Cleansing). This PRD stops new ones.
2. Not changing the dedup match keys' business meaning — chat_data.id / email / accounts / phone / bsuid stay the identity signals; we make the create atomic + constrained, not redefine identity.
3. Not a merge-UI — no user-facing merge screen.
4. No change to the read/serialization path — only the create/upsert path and indexes.

---

Scope Changes

Engineering surfaces this PRD touches (controlled vocab: Backend · Frontend · Mobile · Infra · Data · Design · Docs · None).

• Backend — contact-service: (1) make the contact-create path an atomic upsert keyed on the identity DefineTarget already computes (merge_data.go:632-648 + a new upsert method in repository/contact/create.go / db.go, mirroring the existing FindOneAndUpdate at db.go:150); handle duplicate-key as a merge/update (race loser merges into winner) at contact_sync_handler.go:110; add RequestID idempotency (contact_sync_request.go:27). (2) Normalize phone on the sync/match path — wire the existing util/phone.go normalizer into the transform (contact_sync_request.go:75) and SearchByPhone (search.go:79) so match + store use the same canonical form. (3) Create-path parity — the direct-API CheckDuplicate (create_contact.go:74-86) and bulk-import path (bulk_import_contact.go:505-548) must use the same identity resolution + atomic upsert as chat sync (no weaker/disabled gates).
• Infra (DB migration) — add partial unique indexes on the Contact collection ({company_sso_id, email} where email non-empty; {company_sso_id, phone} where phone non-empty; {company_sso_id, accounts.channel, accounts.unique_id}) — new db/migrations/NNN_*.up.json, mirroring db/migrations/010_field_properties_name_unique_index.up.json ("unique": true).

---

5. Constraints

Constraint	Value
Backward compatibility	Existing duplicates must be cleansed first (sibling PRD) — a unique index build fails if duplicates already exist. Sequencing: cleanse → build unique index → deploy atomic upsert.
Partial index	Unique indexes must be partial (only where the key is non-empty) so contacts legitimately missing a phone/email are not blocked.
Multi-key identity	A contact can match on any of email / phone / account / chat_data.id — the upsert filter must reflect the same priority as DefineTarget to avoid false merges.
No read-path change	Read/serialize behavior unchanged; is_deleted filtering unchanged.
Feature flag	cdp_atomic_upsert_create | default: OFF — enabled per company; flips the create path from legacy find-then-insert to atomic upsert.
Rollout	Index build online/background; the cdp_atomic_upsert_create flag gates the atomic-upsert switch for per-company rollback (see §9).

---

6. Proposed Solution

Plain-language overview of the fix (source: images/prevent-duplicate-solution.source.html). Problem → root cause → the 6 layered safeguards → rollout → target metric. The grounded technical detail for each layer is below.

Technical detail for each, grounded in code:

1. Atomic upsert on create (removes the race window). Replace the find-then-InsertOne with a single FindOneAndUpdate(filter = identity, update = SetOnInsert(...), Upsert: true) keyed on the same identity DefineTarget resolves. Two concurrent same-identity creates then converge on one document. Lands in MergeDataService.Create (merge_data.go:632-648) + new repo method (repository/contact/create.go / db.go, pattern at db.go:150).
2. Partial unique indexes (safety net). Add unique partial indexes on {company_sso_id,email}, {company_sso_id,phone}, {company_sso_id,accounts.channel,accounts.unique_id} so that even a future race / a code path that bypasses the upsert cannot insert a duplicate (MongoDB rejects with a duplicate-key error). New db/migrations/ file.
3. Duplicate-key → merge, not error. On a duplicate-key (E11000) from the upsert/insert, re-run DefineTarget and route to the Update/merge path (contact_sync_handler.go:110 / merge_data.go:648) so the race loser merges into the winner instead of returning 500. (Precedent: the Postgres isUniqueViolation helper at contact_master/utils.go:60 — add a Mongo equivalent.)
4. Normalize phone on the sync/match path (channel-agnostic dedup). Wire the existing normalizer (util/phone.go:21-34) into the sync transform (contact_sync_request.go:75) and SearchByPhone (search.go:79) so store + match use one canonical form (+62/0/spaces collapse to the same key). Fix the bulk-import search-vs-store mismatch (bulk_import_contact.go:534 vs :548) to use the normalized value on both sides. This is what unifies the same person across channels (e.g. web_chat → whatsapp) — the cross-channel path relies entirely on the phone field.
5. Create-path parity across all entrypoints. Route the direct API create (create_contact.go:74-86, currently email+phone with account matching commented out) and the bulk import path through the same DefineTarget identity resolution + atomic upsert as chat sync, so no entrypoint has a weaker dedup gate. The partial unique index (item 2) backstops any that slips.
6. Web_chat dedup key + request idempotency. Either (a) have the chat producer send a stable account_uniq_id for web_chat, and/or (b) add a RequestID idempotency guard on POST /contacts/create (contact_sync_handler.go:76, using contact_sync_request.go:27) so a retried/redelivered event collapses onto the existing contact.

Reference implementation already in-repo: the Postgres contact_master store already does duplicate-key → re-fetch (contact_master/contact_master.go:40-46) — mirror that pattern on the Mongo side rather than inventing one.

---

7. API & Webhook Behavior

#	Behavior	Entity	Triggered by	Expected behavior	Failure behavior
1	Create/sync a contact	contact	POST /contacts/create (chat / CRM sync)	Resolve identity, then atomic upsert — insert if new, else merge into the existing contact. Exactly one doc per identity per company.	On duplicate-key (concurrent race), re-resolve + merge into the winner; never a second doc, never a 500.
2	Enforce uniqueness at DB	contact (indexes)	Any insert	Partial unique indexes reject a second doc with the same {company_sso_id, email} / {…, phone} / {…, account}.	Duplicate-key surfaced to the app → routed to merge (behavior #1).
3	Idempotent retry	contact	Redelivered event with same RequestID	Second delivery is a no-op / update, not a new contact.	Missing RequestID → fall back to identity upsert (still safe via #1/#2).
4	Normalized phone match	contact	Any create/sync with a phone (any channel)	Phone is canonicalized (+62/0/spaces → one form) before match and store, so the same person via different formats/channels resolves to one contact.	Un-normalizable input stored as-is; still guarded by unique index #2.
5	Uniform dedup across entrypoints	contact	Direct API POST /contacts/create and bulk XLSX import	Same DefineTarget identity resolution + atomic upsert as chat sync — no entrypoint has a weaker/disabled gate.	Any bypass still hits the partial unique index #2 → routed to merge.

---

8. System Flow + User Stories + ACs

8.1 System Flow

1. A create/sync arrives from any entrypoint — chat sync (web_chat / WhatsApp / IG / …), CRM sync, direct API POST /contacts/create, or bulk import — all routed through the same identity resolution.
2. Normalize identity signals (phone → canonical +62 form; email already lowercased).
3. Resolve identity (DefineTarget: chat_data.id → email → account → phone → bsuid) on the normalized keys.
4. Atomic upsert on that identity → insert-if-absent, else merge.
5. If a concurrent insert already won → duplicate-key → re-resolve → merge into the winner.
6. Partial unique index guarantees at most one doc per identity per company.
7. (Ops sequencing) existing duplicates are cleansed first (sibling PRD) so the unique index can build.

8.2 User Stories

User Story	Importance	Mockup	Technical Notes	Acceptance Criteria
[DUP-S01] — Atomic upsert on contact create (no race window) As the system, I want contact creation to be an atomic upsert keyed on identity, so that two concurrent creates for the same person converge on one contact.	Must Have	N/A — backend	Grounded: today DefineTarget (read, contact_sync_handler.go:101) → InsertContact→InsertOne (db.go:113-114) is non-atomic. Change MergeDataService.Create (merge_data.go:632-648) to a FindOneAndUpdate(Upsert:true) keyed on the resolved identity (repo method in create.go/db.go; pattern at db.go:150). Before → After: Before — find-then-insert races. After — single atomic upsert; one doc per identity.	• AC-1: Given two concurrent POST /contacts/create for the same identity, when both run, then exactly one contact exists afterward (not two). • AC-2: Given an existing contact for the identity, when a create arrives, then it merges into the existing contact (no new doc). • AC-3: Given a genuinely new identity, then exactly one contact is inserted. — Error — • ERR-1: Given the upsert errors (non-duplicate), then the request fails safely and is retried; no partial/orphan doc.
[DUP-S02] — Partial unique indexes enforce identity uniqueness As the system, I want DB-level partial unique indexes on identity keys, so that a duplicate can never be inserted even if a code path bypasses the upsert.	Must Have	N/A — DB migration	Grounded: no unique contact index today (all db/migrations contact indexes non-unique). Add partial unique indexes {company_sso_id,email} (email non-empty), {company_sso_id,phone} (phone non-empty), {company_sso_id,accounts.channel,accounts.unique_id} — new migration mirroring 010_field_properties_name_unique_index.up.json. Before → After: Before — DB accepts duplicates. After — DB rejects them (E11000).	• AC-1: Given the migration is applied, when a second contact with the same {company_sso_id, email} (email non-empty) is inserted, then MongoDB rejects it with a duplicate-key error. • AC-2: Same for {company_sso_id, phone} (phone non-empty) and {company_sso_id, accounts.channel, accounts.unique_id}. • AC-3: Given a contact legitimately without a phone/email, then the partial index does not block it (multiple null-key contacts allowed). — Ops — • ERR-1: Given existing duplicates when the index builds, then the build fails — the cleanse (sibling PRD) must run first (sequencing constraint).
[DUP-S03] — Duplicate-key handled as merge (graceful race loser) As the system, I want a duplicate-key on create to merge into the winner, so that a race never returns an error or a second doc.	Must Have	N/A — backend	Grounded: on E11000, re-run DefineTarget and route to the Update/merge path (contact_sync_handler.go:110 / merge_data.go:648). Add a Mongo isDuplicateKey helper (precedent: Postgres isUniqueViolation, contact_master/utils.go:60).	• AC-1: Given a race where one insert wins and the other hits a duplicate-key, when the loser retries, then it merges into the winner and returns success. • AC-2: Given the merge, then no data from the losing request is lost (chat_data / accounts / fields merged via MergeDataService). — Error — • ERR-1: Given the re-resolve after duplicate-key finds no target (transient), then it retries with backoff; never inserts a second doc.
[DUP-S04] — Web_chat stable dedup key / request idempotency As the system, I want web_chat creates to dedupe reliably, so that anonymous-session churn does not spawn duplicates.	Should Have	N/A — backend + chat producer	Grounded: web_chat chat_data.id is ephemeral (contact_sync_request.go:109); ChannelMapping("web_chat")="email" (chat/contact_master/utils.go:16). Options: (a) chat producer sends a stable account_uniq_id for web_chat; (b) RequestID idempotency guard on POST /contacts/create (contact_sync_request.go:27).	• AC-1: Given a returning web_chat visitor with the same email/phone, when a new session's create arrives, then it resolves to the existing contact (no new doc). • AC-2: Given a redelivered event with the same RequestID, then it is a no-op / update (idempotent). • AC-3: Given web_chat with a stable account_uniq_id, then the account match dedups even before email/phone.
[DUP-S05] — Sequencing: cleanse before enabling prevention As a CDP engineer, I want the unique index to build only after cleansing, so that the rollout does not fail on existing duplicates.	Must Have	N/A — ops	Notes: order = run cleanse (sibling PRD) per company → build partial unique indexes → enable atomic-upsert (flag).	• AC-1: Given a company with existing duplicates, when the unique-index build is attempted before cleanse, then it is blocked until the cleanse completes. • AC-2: Given cleanse is complete, then the index builds and the atomic-upsert flag is enabled for that scope.
[DUP-S06] — Normalize phone on match + store (channel-agnostic dedup) As the system, I want phone numbers canonicalized before matching and storing, so that the same person via different formats or channels resolves to one contact.	Must Have	N/A — backend	Grounded: phone is stored raw (contact_sync_request.go:75) and matched exact-string (SearchByPhone, search.go:79; DefineTarget, merge_data.go:623-624); email already normalized (search.go:43). A normalizer exists but is unwired here (util/phone.go:21-34). Bulk import searches raw but stores normalized (bulk_import_contact.go:534 vs :548) — fix to normalize on both. Before → After: Before — +6281… / 081… / 62 81… are 3 keys. After — one canonical key.	• AC-1: Given two creates with phone +6281210023793 and 081210023793 for the same person, when both are processed, then they resolve to one contact. • AC-2: Given a web_chat (no phone) contact and a later whatsapp contact with the same normalized phone, then they resolve to the same contact (cross-channel). • AC-3: Given bulk import, then the duplicate search and the stored value use the same normalized form (no search-vs-store mismatch). — Error — • ERR-1: Given an un-normalizable phone string, then it is stored as-is and the unique index (DUP-S02) still prevents an exact-duplicate insert.
[DUP-S07] — Dedup parity across all create entrypoints As the system, I want every contact-create entrypoint to use the same identity resolution + atomic upsert, so that no path has a weaker gate that leaks duplicates.	Must Have	N/A — backend	Grounded: direct API CheckDuplicate matches email+phone only, account matching commented out (create_contact.go:74-86 → InsertOne :133); bulk import checks email/phone only (bulk_import_contact.go:505,534). Chat/CRM/cron use full DefineTarget (merge_data.go). Route the weaker paths through the same resolution + upsert. Before → After: Before — API/bulk gates weaker than chat sync. After — one uniform gate; index backstops any bypass.	• AC-1: Given a direct-API create for an identity that already exists via a chat account, when it runs, then it merges into the existing contact (account match no longer skipped). • AC-2: Given a bulk-import row matching an existing contact by normalized phone/email, then it merges (no new doc). • AC-3: Given any entrypoint, then a same-identity create hits the atomic upsert / unique index and never produces a second doc.

Dependencies: DUP-S02 & DUP-S05 depend on the sibling Duplicate Contact Cleansing PRD (existing dupes must be removed before a unique index can build). DUP-S01/S03/S06/S07 are independent BE changes; S06 (phone normalization) should land with or before the unique-index build so cross-format dupes are collapsed by the cleanse.

---

9. Rollout

Aspect	Detail
Feature flag (from §5)	cdp_atomic_upsert_create | default: OFF, enabled per company.
Rollout sequence	Per company: (1) cleanse existing duplicates (sibling PRD) → (2) build partial unique indexes online/background → (3) flip cdp_atomic_upsert_create ON. Widen company-by-company per the Launch Plan (§12).
Backward compatibility	Yes. Existing duplicates must be cleansed first — a unique-index build fails on existing dupes. Read/serialize path unchanged; is_deleted filtering unchanged.
Transition window (9.4)	Flag OFF → legacy find-then-insert runs; flag ON → atomic upsert runs. Both coexist behind the flag during rollout, so old and new create paths serve traffic per-company without a global cutover. Once built, the partial unique index is active for all writes and backstops both paths.
Rollback (9.5)	Flip cdp_atomic_upsert_create OFF to revert that company to the prior create path (upsert disabled; unique index remains as a passive guard). Trigger: the new-duplicate alert fires (§10) OR the upsert-error rate spikes. Per-company, no redeploy.

---

10. Observability

Events

Event	Trigger	Properties
contact_create_upsert	Any create/sync reaches the upsert	company_sso_id, channel, outcome (insert | merge), matched_key (chat_data.id / email / phone / account / bsuid)
duplicate_key_merge_triggered	Upsert/insert hits E11000 and routes to merge	company_sso_id, channel, matched_key
requestid_idempotent_hit	A redelivered event collapses via RequestID	company_sso_id, request_id
phone_normalization_applied	Phone canonicalized on match/store	company_sso_id, changed (bool)

Alerts

Condition	Threshold	Routing
New duplicate groups created for a migrated company (post-enable)	> 0 in a rolling 24h window	Page CDP BE — also triggers rollback consideration (§9)
Upsert error rate (non-duplicate errors)	> 1% of create calls over 15 min	Page CDP BE
duplicate_key_merge_triggered rate	> 5% of creates for a company (spike detection)	Slack CDP BE — expected small; a spike = producer sending racing dupes to investigate

Dashboard owner: CDP BE squad. Post-launch cadence (10.5): review daily for the first 2 weeks after each company is enabled, then weekly for the first month post-GA. Investigate immediately if the new-duplicate rate is > 0 for any migrated company.

---

11. Success Metrics

Category	Metric	Definition	Baseline	Target
Quality ⭐	⭐ New duplicate groups / migrated company / week	New duplicate identity groups created after enable, from a re-run of cdp_duplicate_mapping	Current recurrence trend in mekari_datamart.cdp_duplicate_mapping (e.g. the 2026-07-23 web_chat cases)	0 within 2 weeks of enabling per company
Quality	Concurrent-create convergence rate	% of same-identity concurrent creates that resolve to one contact (from a load test)	0% (today both insert)	100%
Quality	Cross-format phone dedup rate	% of same-person, different-format phone creates that resolve to one contact	Unmeasured (no normalization today)	100% after DUP-S06
Efficiency	Contact-create p95 latency	p95 of the create/upsert call	Current InsertOne-path p95	≤ baseline +20%
Efficiency	Partial unique index build time / company	Wall-clock of the online index build	N/A (new)	Within the maintenance window; no write downtime

⭐ Primary KPI = New duplicate groups / migrated company / week = 0. This is the initiative's whole point and the Gate-2 substitute for this UI-less TECH PRD.

---

12. Launch Plan & Stage Gates

Stage	Audience	Duration	Success gate
1 — Canary	1 internal / friendly migrated company (post-cleanse)	1 week	0 new duplicate groups (§11 ⭐); create p95 within budget; no upsert-error spike
2 — 10%	10% of migrated companies	1 week	Dup rate 0 across the cohort; duplicate_key_merge_triggered behaving (no runaway); no error spike
3 — GA	All migrated companies	Sustained 2 weeks	0 new duplicate groups sustained; §10 alerts quiet

Each gate references the §11 ⭐ metric and §10 alerts. Failing a gate → hold the rollout; if an already-enabled company regresses, roll back via the §9 flag.

---

13. Dependencies

Dependency	Owner	Deliverable	Blocking?
Duplicate Contact Cleansing (sibling PRD)	CDP Backend + Data	Remove existing duplicates so the unique index can build	YES (must precede the index build)
Mongo partial unique index support	CDP Infra	Index build on the production Contact collection (online)	YES
Chat producer stable web_chat key (if option a)	Chat / Omnichannel	Stable account_uniq_id for web_chat	NO (option b covers it)
MergeDataService reuse	CDP Backend	Merge-on-duplicate path	YES

---

14. Key Decisions + Alternatives Rejected

14a — Decisions Made

ID	Decision	Rationale
D-1	Atomic upsert + partial unique index together.	Upsert removes the race; the index is the DB-level safety net for any bypass/future race.
D-2	Cleanse first, then build the unique index.	A unique index build fails on existing duplicates.
D-3	Duplicate-key → merge, never 500.	Race loser must converge, not error, to keep the client experience seamless.
D-4	Fix is channel-agnostic + all-entrypoint, not web_chat-only.	The race/no-index gap lives in the shared Perform path; scoping to web_chat would leave phone-format and cross-channel dupes on every other channel.
D-5	Normalize phone; reuse the existing util/phone.go normalizer.	The normalizer already exists but is unwired on the sync/match path — wiring it (not writing a new one) closes the biggest cross-channel dup source.
D-6	Mirror the in-repo Postgres contact_master duplicate-key→re-fetch pattern.	A race-safe template already exists in this codebase; reuse beats inventing.

14b — Alternatives Rejected

Alternative	Why rejected
App-level lock per identity	Distributed lock adds latency + a new failure mode; DB upsert + unique index is simpler and stronger.
Keep find-then-insert, just add index	Insert would then throw duplicate-key on every race → errors unless merge handling (D-3) is added anyway; upsert avoids the error entirely.
Only cleanse (no prevention)	Duplicates recur — the whole point of this PRD.

---

15. Open Questions

#	Type	Question	Mitigation / Default	Owner	Deadline
OQ-1	Decision	Exact upsert filter when multiple identity signals disagree (email of A, phone of B).	Default: mirror DefineTarget priority (chat_data.id → email → account → phone → bsuid); if conflict, prefer the most-specific single key + flag for review.	CDP BE	2026-08-07 (RFC)
OQ-2	Risk	Building the partial unique index online on a large Contact collection.	Background/rolling build per company after cleanse; monitor.	CDP Infra	2026-08-07 (RFC)
OQ-3	Open	web_chat: chat producer stable key (option a) vs RequestID idempotency (option b) vs both.	Default: option b (self-contained in CDP); pursue option a with Chat for defense-in-depth.	CDP BE + Chat	2026-08-14
OQ-4	Assumption	Are there legitimate same-phone/same-email distinct contacts (shared phone) that a unique index would wrongly block? (cf. cleansing OQ-3).	Confirm with BI; if real, scope the unique key to include a distinguishing field or exclude those.	Data / BI	2026-08-07 (blocks DUP-S02)
OQ-5	Risk	Phone normalization: is the existing util/phone.go (strips +, 0→62; does not strip spaces/dashes/parens, phone.go:37-44) sufficient, or do we need E.164 parsing for non-ID numbers?	Default: extend the normalizer to strip separators + handle non-62 country codes; backfill-normalize existing phones before the unique-index build.	CDP BE	2026-08-14
OQ-6	Decision	Should the direct-API and bulk-import paths adopt full DefineTarget (incl. account matching, currently disabled) or a scoped subset?	Default: full parity; re-enable account matching in create_contact.go:74-86. Confirm no intentional reason it was disabled.	CDP BE	2026-08-14
OQ-7	Consistency (not a root cause)	The live Mongo create trusts accounts[].channel/unique_id verbatim from the payload (contact_sync_request.go:77), whereas delete + backfill derive channel via ChannelMapping (chat/contact_master/utils.go:12-39, called at merge_data.go:946 / backfill_chat_contact.go:260). This is fine as long as the producer always sends the same normalized channel ChannelMapping would compute. If they ever diverge (e.g. producer sends channel:"web_chat" where ChannelMapping would yield "email"), the account-match key is inconsistent across paths and account-level dedup can miss. Not a driver of the observed duplicates, but worth verifying.	Default: confirm producer payload matches ChannelMapping semantics; if not, apply ChannelMapping normalization on create too (defense-in-depth) — the phone normalization (DUP-S06) + unique index (DUP-S02) already backstop the phone/email cases.	CDP BE + Chat	2026-08-21

---

PRD CHANGELOG

Version	Date	By	Section	Type	Summary
1.0	2026-07-24	Zhelia Alifa (drafted w/ AI, grounded)	All	CREATED	First PRD for preventing new duplicate contacts in CDP — the root-cause fix that complements the pre-migration cleanse. Grounded root cause (§3): the create path is a non-atomic find-then-insert with no unique index (ContactSyncHandler.Create→DefineTarget→InsertOne, db.go:113-114; all contact indexes non-unique), and web_chat's primary dedup key (chat_data.id) is ephemeral — so concurrent same-identity creates both insert (observed: same phone+email, sequential _ids, same timestamp). Solution (§6): atomic upsert keyed on identity + partial unique indexes (company_sso_id+email / +phone / +account) + duplicate-key→merge + web_chat stable key / RequestID idempotency. Stories DUP-S01..S05 (5-col), sequenced after the cleanse (unique index can't build on existing dupes).
1.4	2026-07-24	Zhelia Alifa (drafted w/ AI, grounded)	§6	UPDATED	Readability: added a plain-language ("guest list at an event") visual overview to §6 Proposed Solution (problem → root cause → the 6 prevention layers → rollout → target metric), so non-technical stakeholders can explain the fix at a glance. The grounded technical detail is unchanged — it sits under a "Technical detail for each" sub-heading below the visual.
1.3	2026-07-24	Zhelia Alifa (drafted w/ AI, grounded)	§0 (new), §5, §9-§12 (new), §15	UPDATED	Added the mandatory TECH operational sections flagged by score-prd v3.3 (Layer 1). New: §0 Technical Context (PM-owned vs Eng-owned split + "UI changes: None"); §9 Rollout (named flag cdp_atomic_upsert_create | OFF, per-company sequence, transition window, rollback trigger); §10 Observability (4 events + 3 alerts + owner + daily/weekly cadence); §11 Success Metrics (⭐ 0 new duplicate groups/company/week + convergence/latency/index-build); §12 Launch Plan (Canary → 10% → GA stage-gates). Named the feature flag in §5; added a Deadline column to §15. No change to root cause (§3) or stories (§8).
1.2	2026-07-24	Zhelia Alifa (drafted w/ AI, grounded)	§3.1, §15	UPDATED	Demoted the ChannelMapping-not-on-create observation from root cause to a consistency note. ChannelMapping (chat/contact_master/utils.go:12-39) is a legitimate helper for the Postgres contact_accounts/contact_master subsystem (used on delete merge_data.go:946 + backfill backfill_chat_contact.go:260); the Mongo create path trusting the payload's accounts[].channel verbatim is not a driver of the observed duplicates — only a consistency risk if the producer's channel ever diverges from ChannelMapping semantics. Moved to OQ-7 (verify-only). Root cause list (§3.1) now = phone-not-normalized, weaker create-path gates, in-repo race-safe template.
1.1	2026-07-24	Zhelia Alifa (drafted w/ AI, grounded)	§1, §3, §6, §7, §8, Scope Changes, §14, §15	UPDATED	Generalized the root cause beyond web_chat to all channels/entrypoints after a grounded all-channel assessment. New findings: (a) the race + no-index gap lives in the shared Perform path every channel & entrypoint uses — web_chat is only worst-hit, not special (§3.1); (b) ChannelMapping (chat/contact_master/utils.go:12-39) isn't even applied on live create — accounts taken verbatim; (c) phone is not normalized on the sync/match path (contact_sync_request.go:75, search.go:79) though a normalizer exists unwired (util/phone.go) → same person in +62/0/spaced formats = duplicates, cross-channel & same-channel; (d) direct-API CheckDuplicate (account matching disabled, create_contact.go:74-86) and bulk import use weaker gates than chat sync; (e) an in-repo race-safe template exists (Postgres contact_master 23505 re-fetch). Added stories DUP-S06 (phone normalization) + DUP-S07 (create-path parity), decisions D-4..D-6, OQ-5/OQ-6, and §7 behaviors #4-5.