[PRD] Qontak CDP | Customers | Legacy Migration: CRM Contact Activity Logs → CDP (v2.0 — Grounded Rewrite)

Supersedes: [PRD] Legacy Migration: CRM Contact Activity Logs to CDP (page 51194134639). Why v2.0: v1.0 was validated against contact-service, qontak.com, and qontak-customer-fe. The write endpoint path, the CRM source schema (all named fields are computed attr_accessors, not DB columns), the changes[] source (audited_changes YAML, not why), the extraction API (does not exist), and the FE scope ("no new screens") were all incorrect. This version is grounded in code — see Appendix A.

Field	Value
PM	Zhelia Alifa
PRD Version	2.5
Status	DRAFT
PRD Type	NEW
Epic	TF-3182
Squad	CDP Squad
RFC Link	TBD — new migration endpoint + schema migration required
Figma Master	TBD — migration status indicator
Anchor	No — standalone single-squad feature
Labels	epic:qontak-cdp | module:customers | feature:crm-activity-log-migration
Last Updated	2026-07-01

---

Table of Contents

• 1. One-liner + Problem
• 2. What Happens If We Don't Build This
• 3. Target Users + Persona Context
• 4. Non-Goals
• 5. Constraints
• 6. New Features
• 7. API & Webhook Behavior
  • 7.1 CRM Source Schema (Grounded)
  • 7.2 CDP Target Schema (Grounded)
  • 7.3 Field Transformation Logic
• 8. System Flow + User Stories + ACs
• 9. Rollout
• 10. Observability
• 11. Success Metrics
• 12. Launch Plan & Stage Gates
• 13. Dependencies
• 14. Key Decisions + Alternatives Rejected
• 15. Open Questions
• Appendix A — Grounded Code References
• PRD CHANGELOG

---

1. One-liner + Problem

One-liner: Migrate CRM contact activity logs into CDP so clients retain full audit continuity when transitioning to Qontak One.

Problem: CDP currently has no mechanism to import historical activity log data from CRM Contact. When a client migrates from CRM to CDP, all contact activity history — field changes, association events — is left behind in the CRM audits table, creating a hard break in audit continuity. This forces Qontak's existing CRM clients to maintain parallel access to CRM indefinitely and directly blocks CDP General Release for these accounts.

Technical baseline (grounded): The CRM stores all activity in the audits table (model: Audit). The fields the v1.0 PRD listed (type, how, who, what, why, content, data, crm_person_id) are not DB columns — they are computed attr_accessors built at runtime by Audit#parse() from the real stored columns: auditable_id, auditable_type, user_id, username, action, audited_changes (YAML), and created_at. The CDP activity log write endpoint is POST /openapi/v1/customers/{customer_id}/activity_logs and currently lacks the external_id, source_tag, and multi-category support the migration requires — all of which must be added as net-new fields and constants.

---

2. What Happens If We Don't Build This

• CDP General Release is blocked for all existing Qontak CRM clients — the largest customer segment cannot fully transition without historical audit continuity.
• Agents who migrate to CDP see a completely blank activity timeline for every contact — losing years of field change history, association events, and interaction records.
• Clients in regulated industries face a compliance risk: audit trails exist in CRM but are inaccessible from CDP, requiring indefinite dual-system access.
• Without this, every CDP migration must be documented as a known data gap, undermining confidence in CDP as the authoritative system of record.

---

3. Target Users + Persona Context

Persona	Role	Goal	Pain	Workaround
Primary — CS / Sales Agent (post-migration)	Agent managing contacts in CDP	Access full contact interaction history — field changes, associations — without switching back to CRM	After migrating to CDP, the activity timeline shows only post-migration events. Pre-migration history is absent.	Keep CRM login active alongside CDP; manually copy key interactions to CDP notes
Secondary — Migration / Implementation Engineer	Internal engineer executing client migration	Complete full data migration (contacts + activity logs) with zero data gaps, in a single run	No tooling exists. Must either skip logs entirely (data gap) or write one-off ETL scripts per client	Skip activity log migration and document it as a known limitation, or write bespoke scripts

---

Scope Changes

Engineering surfaces this PRD touches (controlled vocab). Kept in sync with the scope_changes frontmatter above.

• Backend — contact-service: extend the activity-log write path with external_id/source_tag/multi-category support, a migration consumer with a durable job-state store, and a ContactResolver keyed on crm_data.id (CDP, indexed) / qontak_customer_id (CRM) — not source_id (D-9); plus the net-new CRM extraction contract.
• Data — one-time historical migration of the CRM audits table (qontak.com Postgres) → CDP activity_logs (MongoDB), parsing the dual audited_changes YAML structure (update vs destroy).

---

4. Non-Goals

1. This migration covers only CRM Contact (auditable_type = 'Crm::Person' and related) activity logs — Deal, Ticket, Task, and Company activity logs are out of scope for this phase.
2. This migration does not provide real-time or ongoing sync — it is a one-time historical batch per account.
3. This migration does not alter or clean up source CRM data — read-only extraction from the audits table.
4. This migration does not include activity log data from other Qontak products (Inbox, Campaign, Chatbot).
5. This migration does not add new UI screens — one targeted UI modification to an existing component: a migration status indicator on DetailPage.vue. This is a component change, not a new screen. Migrated logs themselves render in the existing CDP Activity Log view with no new UI.
6. The 1-year retention window is the target scope but the actual available range must be determined by querying SELECT MIN(created_at), MAX(created_at) FROM audits before migration runs — no code-enforced retention policy was found in qontak.com.
7. The CDP activity log UI caps visible entries at 5,000 per contact (ActivityLog.vue:50: "You have viewed 5,000 activity logs"). Contacts with large histories may silently truncate at this FE ceiling. Increasing or removing this cap is out of scope for this migration — agents on large accounts are already aware of this limit today.
8. No source badge / "Migrated from CRM" label on migrated activity logs — they render using the existing CDP activity-log structure, indistinguishable from native logs, within the existing 5,000-entry window.

---

5. Constraints

Constraint	Value
Platform	Backend migration service (Go, mirroring the existing ActivityLogMigrationConsumer/ActivityLogMigrationHandler pattern in contact-service). Migrated logs surface in existing CDP web + mobile via the CDP activity log API.
CDP write endpoint	A new dedicated S2S migration endpoint must be built — POST /private/activity-logs/migrate (or equivalent) — because the existing external endpoint (POST /openapi/v1/customers/{customer_id}/activity_logs) is single-record only, requires activity_log_code (mandatory, no CRM equivalent), and restricts category == 'transaction' only. A new endpoint bypasses these constraints.
Net-new schema fields required	ActivityLog struct + MongoDB document must add: external_id (string, CRM audit id as string), source_tag (new constant ActivityLogSourceCrmMigration = "crm_migration" alongside existing external/qontak), metadata (JSONB/embedded doc, initial value {legacy: true}). Unique index on (company_sso_id, external_id) required for idempotency.
CRM extraction	No REST extraction API exists with the described contract. Extraction is via direct query on the audits table: SELECT id, auditable_id, auditable_type, associated_id, associated_type, user_id, username, action, audited_changes, created_at FROM audits WHERE auditable_type IN ('Crm::Person', ...) AND created_at >= NOW() - INTERVAL '1 year'. The audited_changes YAML column is the source for changes[].
Performance	Migration job: P95 ≤ 24 hours per account. Batch size: up to 20 records per CDP write call (matching BulkCreate internal endpoint cap). CDP write throughput target: ≥ 500 records/second.
Idempotency	REQUIRED. external_id = CRM audit integer id stored as a string. Before inserting, check for an existing record with the same (company_sso_id, external_id). Skip if found; log duplicate_external_id.
Feature flag	cdp_crm_activity_log_migration_enabled | default: OFF — enabled per account by the migration/engineering team.
Rollback	A rollback pass must be possible: DELETE activity_logs WHERE source_tag = 'crm_migration' AND company_sso_id = {account}. The source_tag field is required before the first migration run for this to work.
CRM extraction query performance	The audits table has no composite index on (auditable_type, created_at) — only separate single-column indexes exist (db/schema.rb). The extraction query WHERE auditable_type IN (...) AND created_at >= :cutoff ORDER BY id will force a partial table scan at 15M+ records. Before Stage 0: run EXPLAIN ANALYZE on the extraction query; add a composite index if query time is unacceptable for the target volume.
Migration job state store	The existing ActivityLogMigrationService uses ephemeral Redis (7-day TTL, activity_log_migration_service.go:15-22). The CRM migration runs 24h+ per account — job state must be stored in a durable persistent store (MongoDB document or equivalent), not Redis. The gocraft/work + status endpoint pattern is reused; the storage backend is upgraded.
Contact mapping path	ContactResolver resolves uniquely via one of the two real CDP↔CRM linkage keys (not the unindexed source_id): CDP side — crm_data.id (contact/base.go:342-344, string, indexed crm_contact_index, queryable today via SearchByAppContactID(ctx, "crm", auditable_id.to_s)) which stores the CRM crm_person_id → match crm_data.id = auditable_id.to_s (no join); CRM side — qontak_customer_id (crm_people.qontak_customer_id, schema.rb:2098, indexed) which stores the CDP contact _id → reachable by joining audits.auditable_id → crm_people, then resolve the CDP contact by _id. These are two separate linkage pairs (crm_data.id ↔ crm_person_id and qontak_customer_id ↔ CDP _id) — not equal values. source_id is not used: it has no repository query method and no index. Confirm crm_data.id coverage per account before Stage 1 (OQ-7).
CRM user_id → SSO UUID resolution	audits.user_id is an integer FK to the CRM users table — not an email or SSO UUID. USMAN has no endpoint that accepts a CRM integer user_id. Resolution chain: (1) query CRM users by user_id integer → get email; (2) call USMAN by email → get SSO UUID. Deleted users: USMAN returns null → Actor = "[Deleted CRM User]". System actions (user_id = null): Actor = "Qontak System".
Backward compat	CRM audits table: read-only during extraction. No writes to CRM. Existing CDP native activity logs: unaffected.
Migration scope — contact population & log coverage	Two contact populations are in scope. (1) Contacts not yet synced to CDP — their CRM activity logs are backfilled in full once the contact is resolved/created in CDP (no CDP logs exist for them yet); depends on the contact being resolvable — see the Contact mapping path constraint + OQ-7. (2) Contacts already synced to CDP — two log-scope cases: (a) logs created before the contact was synced (historical backfill not yet present in CDP); and (b) logs associated after sync but still missing info/metadata (incomplete records — e.g. missing actor/SSO UUID, changes[], or metadata — re-processed to enrich). Note — idempotency tension: case (2b) is an enrich/update path, which differs from the strict Idempotency rule above (skip on duplicate (company_sso_id, external_id)). For incomplete post-sync logs the migration must update the existing CDP record (fill missing fields) rather than skip it — skip-vs-enrich policy to be confirmed (OQ).
Plan scope	All Qontak plans with existing CRM data. Migration is triggered by the internal migration team as part of account onboarding to CDP.

5.1 Data Lifecycle

Artifact	Retention	Visibility
Migration job log (per account)	1 year	Internal only
Failed-record queue (per account)	30 days	Internal migration team
Migration audit trail (external_id → CDP log id)	Permanent (idempotency + compliance)	Internal
Source CRM audits records	CRM retention policy (unchanged)	Read-only during migration

---

6. New Features

Backend migration service + two targeted FE component changes. No new screens.

Migration Pipeline component tree:

CRM Activity Log Migration Pipeline
├── MigrationJobRunner — entry point; validates preconditions; orchestrates
│   ├── IdempotencyChecker — checks existing CDP records by external_id before insert
│   ├── CRMExtractor — direct query on audits table (batched, date-filtered)
│   ├── SchemaTransformer — maps CRM audits → CDP activity log format
│   │   ├── ContactResolver — auditable_id → CDP customer_id via crm_data.id (CDP, indexed) or qontak_customer_id (CRM) — see D-9
│   │   ├── ActorResolver — resolves user_id integer → SSO UUID; handles deleted + system cases
│   │   ├── ChangesExtractor — parses audited_changes YAML → changes[] array
│   │   └── CategoryMapper — maps auditable_type + action → CDP category + action
│   ├── CDPActivityLogInserter — calls new S2S batch endpoint (up to 20/batch)
│   ├── ValidationRunner — count comparison (CRM source vs CDP inserted)
│   └── MigrationLogger — per-record success/failure; audit trail
└── MigrationMonitorAPI — GET /private/activity-logs/migration/status (reuse existing pattern)

FE component changes (net-new, not new screens):

Component	Change
features/customers/detail/views/DetailPage.vue	Add migration status indicator component. Polls GET /private/activity-logs/migration/status?account_id={id}. 4 UI states: Loading = progress % shown during in_progress; Empty = not shown when status = 'not_started'; Error = hidden (fail-silent, contact detail renders normally); Success = auto-dismissed when status = 'completed'.

---

7. API & Webhook Behavior

#	Behavior	Entity Affected	Triggered By	Expected Behavior	Failure Behavior
1	Trigger migration job	migration_job	Engineer enables flag + job enqueued (same gocraft/work consumer pattern as ActivityLogMigrationConsumer)	Validate: flag ON, account exists, CDP contacts migrated (prerequisite). Create job. Return job_id. Run async.	Flag OFF: 403. Contacts not migrated: abort, log contacts_not_migrated. Already completed: 409.
2	Extract CRM activity logs	audits table records	CRMExtractor batch query	SELECT id, auditable_id, auditable_type, associated_id, associated_type, user_id, username, action, audited_changes, created_at FROM audits WHERE auditable_type IN (scope list) AND created_at >= :cutoff ORDER BY created_at ASC — paginated by id cursor (not OFFSET)	DB unavailable: retry 3×; then halt + alert
3	Transform CRM record → CDP record	In-memory	SchemaTransformer per record	changes[] parsing — two branches by action type: • update action: audited_changes YAML = {field: [old_value, new_value]} → changes = [{field, from: val[0], to: val[1]}]. • destroy action: audited_changes stores the full attribute snapshot {field: value} (NOT [old,new] arrays) → changes = [{field, from: value, to: null}] per attribute. crm_person_id for association events: extract from comment JSON column first (JSON.parse(comment)['crm_person_id']); fall back to audited_changes YAML if absent (audit.rb:483-494). Contact mapping (D-9): resolve via either real linkage key — primary (CDP side): query CDP Contact where crm_data.id = auditable_id.to_s (contact/base.go:342-344, indexed crm_contact_index; SearchByAppContactID(ctx, "crm", …)) — no join; alternative (CRM side): join audits.auditable_id → crm_people.qontak_customer_id (schema.rb:2098, = the CDP contact _id) → resolve by CDP _id. Not source_id (no query method/index). Fall back to a separate mapping table only if crm_data.id coverage is incomplete. Actor resolution — 3 cases (use the Actor string field, activity_log/base.go:29, NOT a description field): • Active user: query CRM users by user_id integer → email → USMAN by email → SSO UUID. Actor = full_name. • Deleted user: USMAN returns null → user_id = null, Actor = "[Deleted CRM User]". • System action (user_id = null): user_id = null, Actor = "Qontak System". Set external_id = audit.id.to_s, source_tag = 'crm_migration', metadata = {legacy: true}.	Cannot map action type: skip, unmapped_action_type. Cannot resolve crm_person_id from either comment JSON or YAML: skip, contact_not_found. Invalid created_at: use ingestion time, log warning. audited_changes null on update: skip, invalid_change_format.
4	Write batch to CDP	ActivityLog documents in CDP	CDPActivityLogInserter	Call new S2S endpoint: POST /private/activity-logs/migrate with batch of ≤20 records; each record includes external_id, source_tag, metadata; endpoint checks (company_sso_id, external_id) uniqueness before insert; skip duplicates	5xx: retry once; then log batch_insert_failed. Failure rate > 10%: halt job + PagerDuty alert.
5	Get migration status	migration_job	GET /private/activity-logs/migration/status?account_id={id} (mirrors existing ActivityLogMigrationHandler.GetMigrationStatus)	Returns {status, progress_pct, records_migrated, records_skipped, records_failed, started_at, duration_seconds}	Job not found: {status: "not_started"}
6	Rollback migrated records	CDP ActivityLog where source_tag = 'crm_migration'	Migration team triggers rollback	Delete all records for the account where source_tag = 'crm_migration'. Fire crm_activity_log_migration_rolled_back event.	DB error: retry; log rollback_failed; alert on-call

7.1 CRM Source Schema (Grounded)

IMPORTANT: The fields listed in v1.0 (type, how, who, what, why, content, data, crm_person_id) are NOT database columns. They are in-memory computed attr_accessors set by Audit#parse() at runtime (audit.rb:14-17, audit.rb:1598-1981).

Real audits table columns (from db/schema.rb):

Column	Type	Notes
id	integer	CRM audit record ID — becomes external_id in CDP
auditable_id	integer	The record's PK (e.g. crm_person.id)
auditable_type	string	Model class name (e.g. 'Crm::Person', 'Crm::Deal')
associated_id	integer	Associated record PK (for association events)
associated_type	string	Associated model class (e.g. 'Crm::Deal')
user_id	integer	CRM integer user FK — resolve to SSO UUID via USMAN
username	string	Fallback display name if user_id lookup fails
action	string	'create', 'update', 'destroy', 'associate', 'disassociate'
audited_changes	TEXT (YAML)	Structured delta: {field_name: [old_value, new_value]} — source for changes[]
version	integer	Audit sequence number
comment	text	Optional comment
remote_address	string
request_uuid	string
created_at	datetime	Original event timestamp — normalize to UTC

7.2 CDP Target Schema (Grounded)

Existing ActivityLog fields in CDP (activity_log/base.go:17-34): id, contact_id (customer_id), company_sso_id, action, category, changes[], associated_id, associated_type, associated_name, user_id, description, source, timestamp, created_at, updated_at

Net-new fields to add:

Field	Type	Purpose
external_id	string	CRM audit id (as string). Unique index (company_sso_id, external_id) for idempotency.
source_tag	string	New constant ActivityLogSourceCrmMigration = "crm_migration" (join existing external/qontak in const.go). Used for rollback + validation (no FE display).
metadata	embedded doc / JSONB	{legacy: true} for all migrated records. Extensible for future migration metadata.

7.3 Field Transformation Logic

Category + Action mapping (from auditable_type + action):

CRM auditable_type	CRM action	CDP category	CDP action
Crm::Person (Contact)	update	customer_details	update
Crm::Phone	update	customer_details	update
Crm::Deal	associate / create	deals	linked
Crm::Deal	disassociate / destroy	deals	unlinked
Crm::Deal	update (status resolved)	deals	resolved
Ticket	associate / create	tickets	linked
Ticket	disassociate / destroy	tickets	unlinked
Ticket	update (status resolved)	tickets	resolved
Crm::Task	associate / create	tasks	linked
Crm::Task	disassociate / destroy	tasks	unlinked
Crm::Task	update (completed)	tasks	completed
Crm::Company	associate	company	linked
Crm::Company	disassociate	company	unlinked

changes[] extraction (from audited_changes YAML):

# CRM audited_changes YAML:
{ "phone_number": ["+628...", "628..."], "country_code": ["", "62"] }

# CDP changes[] output:
[
  { "field": "Phone Number", "from": "+628...", "to": "628..." },
  { "field": "Country Code", "from": "", "to": "62" }
]

Field name display mapping (CRM DB column name → human-readable) must be defined as a lookup table in the migration service.

Actor resolution (3 cases from Audit#mapping_who(), audit.rb:1815-1833):

Case	CRM stored value	CDP output
Active user	user_id = integer FK to users table	Query USMAN: user_id integer → SSO UUID → set user_id = UUID
Deleted user	user_id = integer (user may be deleted in USMAN)	Query CRM users by user_id → email → USMAN by email → returns null → user_id = null, Actor = "[Deleted CRM User]" (correct field: Actor string, activity_log/base.go:29)
System action	user_id = null, username = "Qontak system"	user_id = null, Actor = "Qontak System"

Contact resolution:

• For auditable_type = 'Crm::Person': extract auditable_id → resolve to the CDP contact via crm_data.id = auditable_id.to_s (CDP side, indexed) or qontak_customer_id (CRM side; = CDP _id) — see D-9. (Same linkage pattern as the notes migration.)
• For association events where auditable_type is not Crm::Person (e.g. a Deal audit that references a person): extract person reference from audited_changes YAML if present, otherwise skip.

Skip conditions:

• auditable_type not in the supported migration scope list
• audited_changes is null or empty AND action = 'update' (no structured change data)
• auditable_id cannot be resolved to a CDP contact via crm_data.id / qontak_customer_id (log contact_not_found)
• created_at is null or invalid
• Same (company_sso_id, external_id) already exists in CDP (log duplicate_external_id)

---

8. System Flow + User Stories + ACs

8.1 System Flow

1. Engineer enables cdp_crm_activity_log_migration_enabled = ON for account; job is enqueued via gocraft/work (mirrors ActivityLogMigrationConsumer.ProcessJob).
2. System validates prerequisites: flag ON, account exists, CDP contacts present (abort if not — logs must reference valid CDP customer_ids).
3. System fires crm_activity_log_migration_started event; migration status indicator appears in CDP Contact Detail UI.
4. CRMExtractor: cursor-paginated query on audits table: WHERE auditable_type IN (scope) AND created_at >= cutoff ORDER BY id ASC LIMIT 500. Collects id, auditable_id, auditable_type, associated_id, associated_type, user_id, username, action, audited_changes, created_at.
5. Per batch of records, SchemaTransformer processes each:
   • a. ContactResolver: resolve auditable_id → CDP contact via crm_data.id = auditable_id.to_s (CDP side, indexed) — or qontak_customer_id (CRM side, = CDP _id); see D-9. If neither resolves: skip, log contact_not_found.
   • b. ActorResolver: user_id integer → SSO UUID via USMAN (3-case handling: active/deleted/system).
   • c. ChangesExtractor: parse audited_changes YAML → changes[] array.
   • d. CategoryMapper: auditable_type + action → CDP category + action from lookup table.
   • e. Attach external_id = audit.id.to_s, source_tag = 'crm_migration', metadata = {legacy: true}.
6. IdempotencyChecker: for each transformed record, check if (company_sso_id, external_id) already exists in CDP. Skip if found.
7. CDPActivityLogInserter: batch of ≤20 transformed records → POST /private/activity-logs/migrate (new S2S endpoint). On 5xx: retry once. On second failure: log batch_insert_failed; continue with next batch.
8. Check failure rate after each batch: if failed / total > 10% → halt job, fire crm_activity_log_migration_failed, alert PagerDuty.
9. After all pages: retry pass for queued failed records.
10. ValidationRunner: compare CRM source count vs CDP inserted count (filtered by source_tag = 'crm_migration' AND company_sso_id = {account}). Compute accuracy_pct.
11. Fire crm_activity_log_migration_completed. If accuracy_pct < 90% → alert PagerDuty + PM.
12. Migration status indicator removed from CDP Contact Detail UI.

📊 System Flow — CRM Activity Log Migration

sequenceDiagram
    participant Eng as Migration Engineer
    participant Job as MigrationJobRunner
    participant CRM as CRM audits table
    participant Map as CDP Contact Mapping
    participant USM as USMAN (actor resolve)
    participant CDP as CDP S2S migrate endpoint
    participant FE as customer-fe (UI)
    Eng->>Job: enable flag + enqueue job {account_id}
    Job->>Job: validate prerequisites (flag, contacts migrated)
    Job-->>FE: migration status indicator shown
    loop cursor-paginated batches
        Job->>CRM: SELECT ... FROM audits WHERE account + date range
        CRM-->>Job: batch of audit records
        loop per record
            Job->>Map: auditable_id -> customer_id
            alt no mapping
                Job-->>Job: skip, log contact_not_found
            end
            Job->>USM: user_id -> SSO UUID
            USM-->>Job: uuid OR null (deleted/system)
            Job->>Job: parse audited_changes YAML -> changes[]
            Job->>Job: map auditable_type+action -> category+action
            Job->>Job: check (company_sso_id, external_id) duplicate
        end
        Job->>CDP: POST /private/activity-logs/migrate (batch ≤20)
        alt batch fails
            Job->>CDP: retry once
            alt still fails
                Job-->>Job: log batch_insert_failed
            end
        end
        Job->>Job: check failure_rate > 10%?
        alt halt
            Job-->>Eng: HALT + PagerDuty alert
        end
    end
    Job->>Job: validation (count compare) -> accuracy_pct
    Job-->>FE: migration status indicator removed
    Job-->>Eng: completed event + accuracy report

8.2 User Stories

User Story	Importance	Mockup	Technical Notes	Acceptance Criteria
[CALM-S01] — Run batch migration for an account As a Migration Engineer, I want to trigger a migration job for a specific account that extracts CRM contact activity logs from the audits table and inserts them into CDP, so that the client's full audit history is available in Qontak One on migration day.	Must Have	Figma: N/A — backend job, no UI	Data Fields: account_id (req), job_id (generated), status (enum), cdp_crm_activity_log_migration_enabled (flag), external_id (audit.id as string), source_tag = 'crm_migration' Before-After Behavior: Before: no tooling exists; history stays in CRM only. After: a gocraft/work consumer job (mirrors ActivityLogMigrationConsumer) extracts CRM audits records and inserts them into CDP via the new S2S endpoint.	— Happy Path — • AC-1: Given flag ON and CDP contacts present, when the engineer enqueues the migration job, then a job is created in_progress, job_id returned, async processing starts. • AC-2: Given a job in progress, when the engineer calls GET /private/activity-logs/migration/status?account_id={id}, then it returns progress_pct, records_migrated, records_skipped, records_failed. • AC-3: Given all batches complete with failure rate ≤ 10% and accuracy_pct ≥ 90%, then status = completed, counts + accuracy in response. — Error / Unhappy Path — • ERR-1: Given flag OFF, then job rejected 403 FLAG_DISABLED. • ERR-2: Given CDP contacts not yet migrated for the account, then job aborts with contacts_not_migrated. • ERR-3: Given failure rate > 10% during run, then halt + PagerDuty alert + status = halted. • ERR-4: Given a re-trigger of a completed migration, then 409 ALREADY_COMPLETED. — AC-4 (Contact mapping path) — • AC-4: Given a CRM auditable_id (the CRM crm_person_id integer), when ContactResolver runs, then it resolves the CDP contact via either real linkage key — CDP side: crm_data.id = auditable_id.to_s (contact/base.go:342-344, indexed crm_contact_index, SearchByAppContactID(ctx, "crm", …)); or CRM side: qontak_customer_id (crm_people.qontak_customer_id = the CDP contact _id); if found → use that customer_id; if neither resolves and a fallback mapping table exists → use that; else → skip record, log contact_not_found. (Not source_id — it has no query method/index.) — Permission Model — • CAN: internal migration/engineering team (S2S token only). • CANNOT: client admins, end users.
[CALM-S02] — Idempotent re-run; no duplicate records As a Migration Engineer, I want to safely re-run the migration for a halted or partially completed account without creating duplicate activity log entries in CDP, so that retries are safe.	Must Have	Figma: N/A	Data Fields: external_id (string, unique index (company_sso_id, external_id) on CDP activity_logs) Before-After Behavior: Before: no deduplication mechanism (external_id field absent from ActivityLog struct). After: every CRM audit id is stored as external_id; re-runs skip already-migrated records.	— Happy Path — • AC-1: Given a halted job re-triggered, then only records with external_id not yet in CDP are inserted; previously migrated records are skipped, logged as duplicate_external_id. • AC-2: Given a full re-run where all records are already present, then records_migrated = 0, records_skipped = N, status = completed. • AC-3: Given two concurrent migration jobs triggered for the same account, then only one runs (unique constraint on account_id + in_progress); second returns 409 JOB_ALREADY_RUNNING. — Error / Unhappy Path — • ERR-1: Given a record where (company_sso_id, external_id) already exists, when the inserter encounters it, then it is skipped silently; cdp_crm_activity_log_migration_record_skipped logged with reason duplicate_external_id.
[CALM-S03] — Transform CRM audit record to CDP activity log (grounded) As the Migration Pipeline, I want to correctly transform each CRM audits row into a CDP activity log using the real field mapping, so that migrated records are complete and accurate.	Must Have	Figma: N/A	Data Fields: auditable_id, auditable_type, user_id (int), username, action, audited_changes (YAML), created_at → CDP category, action, changes[], user_id (UUID), source_tag, external_id, timestamp. Before-After Behavior: Before: no transformation pipeline exists; CRM audit columns are not queryable as named fields. After: each CRM audits row is transformed by reading audited_changes YAML and applying the category/action mapping table.	— Happy Path — • AC-1: Given auditable_type = 'Crm::Person' and action = 'update', when transformed, then category = 'customer_details', action = 'update'. • AC-2: Given audited_changes = '{"phone_number": ["+628...", "628..."]}' and action = 'update', when ChangesExtractor runs, then changes = [{field: "Phone Number", from: "+628...", to: "628..."}]. (Source is audited_changes YAML [old,new] arrays — NOT the why field.) • AC-2b: Given action = 'destroy' and audited_changes = '{"first_name": "Budi", "phone_number": "+628..."}', when ChangesExtractor runs, then changes = [{field: "First Name", from: "Budi", to: null}, {field: "Phone Number", from: "+628...", to: null}]. (For destroy: audited_changes stores a full attribute snapshot, NOT [old,new] arrays — audit.rb:477-494.) • AC-3: Given user_id = 12 (active user), when ActorResolver queries CRM users by user_id integer → gets email → calls USMAN by email, then user_id in CDP = the SSO UUID; Actor = the user's full name. • AC-4: Given user_id resolves to a deleted user (USMAN returns null), then user_id = null, Actor = "[Deleted CRM User]" (using the Actor string field on ActivityLog, activity_log/base.go:29); record is NOT skipped. • AC-5: Given user_id = null and username = 'Qontak system', then user_id = null, Actor = "Qontak System"; record is NOT skipped. • AC-6: Given created_at with timezone offset, when stored in CDP, then timestamp = UTC-normalized ISO8601. • AC-7: Given any migrated record, then source_tag = 'crm_migration', external_id = audit.id.to_s, metadata = {legacy: true}. • AC-8: Given an association event where comment column contains JSON {"crm_person_id": 12345}, when ContactResolver runs, then it extracts crm_person_id = 12345 from the comment JSON first (before reading audited_changes YAML) per audit.rb:483-494. — Error / Unhappy Path — • ERR-1: Given auditable_type not in the supported scope list, when encountered, then skip + log unmapped_action_type. • ERR-2: Given auditable_id cannot be resolved via crm_data.id / qontak_customer_id (or fallback table), when ContactResolver fails, then skip + log contact_not_found. • ERR-3: Given audited_changes is null or unparseable YAML AND action = 'update', then skip + log invalid_change_format.
[CALM-S04] — Migrated logs visible in CDP Contact Activity view As a CS / Sales Agent, I want to see pre-migration CRM activity in the CDP contact timeline, so that I have full customer context without switching to CRM.	Must Have	Figma: N/A — uses the existing CDP Activity Log view (no new UI)	Data Fields (rendered): category, action, changes[], timestamp, actor — the standard CDP activity-log fields. Before-After Behavior: Before: CDP Activity Log shows only post-migration native events; no CRM history. After: migrated events appear inline in the existing Activity Log view, sorted chronologically alongside native CDP logs, rendered with the same structure as native entries (no source badge / migration label).	— Happy Path — • AC-1: Given migration status = 'completed' for an account, when an agent opens a contact detail, then all migrated CRM activity logs appear in the Activity Log panel sorted chronologically alongside native CDP logs. • AC-2: Given a migrated log entry, when rendered in the CDP Activity Log view, then it uses the existing CDP activity-log structure (category, action, changes[], timestamp, actor) — identical to a native entry, with no source badge or migration label. — Error / Unhappy Path — • ERR-1: Given an attachment or associated record referenced in a migrated log no longer exists in CDP, when the agent views it, then the log renders with "[Deleted reference]" placeholder; the log is NOT hidden.
[CALM-S05] — Migration status indicator on Contact Detail during in-progress job As a CS / Sales Agent, I want to see a clear indicator when activity log migration is in progress for a contact, so that I know incomplete history is expected and not a bug.	Should Have	Figma: TBD — banner/indicator component on DetailPage.vue	Data Fields: status (from GET /private/activity-logs/migration/status), progress_pct Before-After Behavior: Before: no migration awareness in the Contact Detail UI. After: a non-intrusive status indicator ("Importing CRM activity history…") appears during in_progress jobs and disappears on completed.	— Happy Path — • AC-1: Given a migration job with status = 'in_progress', when an agent opens any contact for that account, then a non-blocking status indicator shows: "Importing CRM activity history — some records may still be loading." • AC-2: Given the job reaches status = 'completed', then the indicator is removed and the full activity history is visible. • AC-3: Given status = 'not_started' (no migration triggered), then no indicator is shown. — Error / Unhappy Path — • ERR-1: Given the migration status API is unavailable, then the indicator is hidden (fail-silent); the contact detail renders normally.
[CALM-S06-NEG] — No duplicate records on re-run (Guard Rail) As the system, when a migration is re-triggered for an already-migrated account, then no duplicate activity log records are created.	Guard Rail	—	—	• NEG-1: Given CRM audit id 1341215908 was migrated in a prior run, when the migration encounters it again, then external_id = "1341215908" already exists → skip, log duplicate_external_id; no new record created. • NEG-2: Given the flag is OFF, when a migration attempt is made, then 403 FLAG_DISABLED; no job created, no records written.
[CALM-S07] — Migrate only pre-sync activity (no double-import) As the Migration Pipeline, I want to migrate only the CRM activity that predates the contact's CDP sync point, so that activity already captured live in CDP after the contact synced is not imported twice.	Must Have	Figma: N/A — backend job, no UI	Data Fields: SyncWatermarkResolver reads the CDP contact.created_at as the per-contact watermark; CRMExtractor/transform skips CRM audits rows where audits.created_at >= watermark. Before-After Behavior: Before: migrating every CRM audit row would double-import events CDP already recorded live after the contact synced. After: each CRM audit is compared to the contact's CDP sync watermark — only pre-watermark rows are migrated; post-watermark rows are skipped as post_sync_live_feed. (Engineering RFC: SyncWatermarkResolver · Decision 16. Implements the v2.4 log-scope split — case 2a pre-sync backfill vs post-sync live feed.)	— Happy Path — • AC-1: Given a CRM audit row whose created_at < the contact's CDP contact.created_at watermark, when the pipeline processes it, then the record is migrated. • AC-2: Given a CRM audit row whose created_at >= the watermark, when the pipeline processes it, then the record is skipped and logged as post_sync_live_feed (already captured natively in CDP — no double-import). • AC-3: Given the contact has no resolvable watermark (contact missing / not resolvable in CDP), when the pipeline processes its rows, then they are skipped and logged as contact_not_found. — Permission Model — • CAN: internal migration pipeline (S2S only). • CANNOT: client admins, end users.
[CALM-S08] — Reconcile association logs; fill missing metadata As the Migration Pipeline, I want to enrich association activity logs (link / unlink / resolved / completed) with the missing crm_person_id and linked-entity id/name, patching existing CDP logs in place, so that both migrated and already-synced association logs are complete.	Must Have	Figma: N/A — no FE change; patched rows render natively in the existing Activity Log view	Data Fields: AssociationLogEnricher — for association logs (link / unlink / resolved / completed) fill crm_person_id + the linked-entity id/name via CRM lookup; patch existing CDP logs in place (incl. live-feed logs), matched on the association match-key (see §7.3 mapping; exact match-key policy is an open question). Additive/enrich-only — an explicit exception to the strict skip-on-duplicate (company_sso_id, external_id) idempotency rule (this is the v2.4 case-2b enrich path, now resolved). Before-After Behavior: Before: association logs (migrated or captured live) may be missing crm_person_id or the linked-entity name. After: the enricher fills those fields via CRM lookup and patches the existing CDP log in place — no duplicate document created. (Engineering RFC: AssociationLogEnricher · Decision 17.)	— Happy Path — • AC-1: Given a migrated association log missing crm_person_id, when the enricher runs, then crm_person_id is filled via CRM lookup. • AC-2: Given an existing CDP live-feed association log missing the linked-entity name, when the enricher runs, then the entity name is patched in place (no new document created). • AC-3: Given a linked entity (referent) that was deleted, when the enricher cannot resolve it, then the field is left as "[Deleted reference]"; the log is NOT hidden. • AC-4: Given the enricher is re-run after a successful pass, when it processes already-complete logs, then it patches 0 records (idempotent re-run). — Error / Unhappy Path — • ERR-1: Given a log whose parent contact / match-key cannot be resolved, when the enricher runs, then the record is skipped (no patch) and logged contact_not_found.
[CALM-S09] — Gate on contact coverage (Phase A) As the Migration Pipeline, I want to gate the migration on a minimum contact-coverage threshold before it runs, so that activity logs are not migrated for an account whose contacts are not yet sufficiently resolvable in CDP.	Must Have	Figma: N/A — backend job, no UI	Data Fields: Phase-A coverage gate — if contact coverage < threshold → return 422 contacts_not_migrated (no job enqueued). Coverage is remediated by the upstream BackfillCustomerIdWorker (backfills the crm_data.id / customer-id mapping); this gate performs no contact/CRM write itself. Residual unresolved contacts after backfill are skipped as contact_not_found (not a CRM write). Before-After Behavior: Before: a migration could start against an account with incomplete contact coverage, producing a large contact_not_found skip volume. After: the migration is gated on coverage — below threshold it returns 422 and does not enqueue; the upstream backfill raises coverage, then the trigger succeeds. (Refines the CALM-S01 ERR-2 contacts_not_migrated abort into an explicit coverage threshold; relates to OQ-7.) (Engineering RFC: Phase-A trigger gate · Decision 18.)	— Happy Path — • AC-1: Given contact coverage below the Phase-A threshold, when the migration is triggered, then it returns 422 contacts_not_migrated and no job is enqueued. • AC-2: Given the upstream BackfillCustomerIdWorker has raised coverage to/above the threshold, when the migration is re-triggered, then the gate passes and the job succeeds. — Error / Unhappy Path — • ERR-1: Given coverage passes the gate but an individual contact remains unresolved during the run, when that record is processed, then it is skipped and logged contact_not_found — the gate does not write to CRM/contacts.

Dependencies: S01 → new S2S migration endpoint + external_id/source_tag/metadata fields; S02 → unique index (company_sso_id, external_id); S03 → audited_changes YAML parsing; S04 → renders in the existing CDP Activity Log view (no new FE field/badge); S05 → FE status component + status endpoint; S07 → SyncWatermarkResolver + CDP contact.created_at watermark read; S08 → AssociationLogEnricher + in-place patch path (association match-key) + CRM entity lookup; S09 → Phase-A coverage gate + upstream BackfillCustomerIdWorker.

🧪 Test Coverage Matrix — [CALM-S03]

Dimension	Coverage	Notes
Boundary values	⚠️ partial	AC-7 covers source_tag; ⚠️ QA: empty audited_changes YAML, >100 field changes, created_at at exact cutoff boundary, destroy record with empty snapshot
State transitions	✅ defined	ERR-1 (unmapped), ERR-2 (no contact), ERR-3 (invalid YAML); AC-2b covers destroy branch; AC-8 covers comment JSON path
Data validation	⚠️ partial	AC-2/AC-2b cover update/destroy branches; AC-8 covers comment JSON; ⚠️ QA: YAML with null values, non-UTF8 characters, malformed comment JSON
Concurrency	✅ defined	CALM-S02 AC-3 covers concurrent jobs
Network/timeout	⚠️ TBD	⚠️ QA: USMAN timeout per-record during actor resolution; CRM users table query timeout

---

9. Rollout

Field	Detail
Flag	cdp_crm_activity_log_migration_enabled | default: OFF — enabled per account by migration team.
Stage 0 (prerequisite)	Add external_id, source_tag, metadata fields to ActivityLog schema + unique index. Add crm_migration constant. Build new S2S migration endpoint. No behaviour change for existing records.
Stage 1 — Internal QA	1–2 synthetic test accounts. Verify idempotency, timestamp normalization, YAML parsing, deleted-user fallback, and the migration status indicator.
Stage 2 — Pilot	5–10 selected client accounts (migration-ready, with confirmed CDP contact migration completed). Monitor accuracy_pct, failure rate, job duration.
Stage 3 — Controlled Rollout	~20–50 accounts/week per migration schedule.
Backward compat	CRM read-only. Existing CDP native activity logs unaffected.

9.1 Migration Transition Window

• During in_progress: agents see existing CDP activity logs + the status indicator. Migrated records appear as they are inserted (progressive).
• After completed: full history visible inline, sorted chronologically; status indicator removed.
• On failure/halt: partial records present; status indicator shows last known state. Manual retry after root-cause resolution.

---

10. Observability

Event Name	Trigger	Properties
crm_activity_log_migration_started	Job triggered	job_id, account_id, triggered_by, date_range_start, estimated_record_count
crm_activity_log_migration_completed	Job finishes	job_id, account_id, records_migrated, records_skipped, records_failed, accuracy_pct, duration_seconds
crm_activity_log_migration_failed	Job halted (failure > 10%)	job_id, account_id, failure_reason, records_migrated_before_failure, failure_rate
crm_activity_log_migration_record_skipped	Record excluded (sampled 1:10)	job_id, account_id, external_id, skip_reason (unmapped_action_type | contact_not_found | invalid_change_format | duplicate_external_id)
crm_activity_log_migration_record_failed	Record failed to write after retry	job_id, account_id, external_id, failure_reason, retry_count
crm_activity_log_migration_rolled_back	Rollback triggered	job_id, account_id, triggered_by, records_removed, rollback_duration_seconds

Dashboard owner: CDP Engineering Squad. Alerts: failure rate > 10% in any account batch → PagerDuty CDP on-call; job duration > 48h → Slack #cdp-migration-ops; skip rate > 20% for any account → Slack #cdp-migration-ops (mapping gap); 3 consecutive account failures → PagerDuty + PM. Cadence: daily review first 2 weeks of pilot; weekly for first month of controlled rollout; monthly thereafter. Investigate immediately if accuracy_pct < 90% for any account.

---

11. Success Metrics

Metric	Definition	Baseline	Target
⭐ Migration accuracy (North Star)	accuracy_pct = records correctly migrated and field-mapped / total CRM source records (spot-checked via random sample verification per account)	0% (feature doesn't exist)	≥ 99% per account before client cutover
⭐ Accounts fully migrated	Accounts at completed status	0	100% of migrating CRM accounts by CDP GA
Idempotency correctness	Duplicate records created on re-run	N/A	0 (zero duplicates on any re-run)
Job reliability	failed / triggered	N/A	< 2% halt rate across all accounts in Stage 3
Skip rate — contact not found	contact_not_found skips / total records	N/A	< 1% (depends on contact migration completeness)

---

12. Launch Plan & Stage Gates

Stage	Audience	Duration	Success Gate	Owner
Stage 0 — Prerequisites	Engineering only	1–2 sprints	external_id + source_tag + metadata schema migration complete; new S2S endpoint stable in staging; unique index in place; migration status indicator in staging.	CDP Eng
Stage 1 — Internal QA	2 synthetic test accounts (small + large volume)	1 week	accuracy_pct = 100%; zero duplicates on idempotent re-run; all 3 actor cases (active/deleted/system) correct; status indicator appears + disappears correctly.	QA
Stage 2 — Pilot	5–10 CSM-approved client accounts	2 weeks	accuracy_pct ≥ 99% per account; zero halts from pipeline bugs; error log root-caused; client confirms history visible in CDP.	PM + CSM
Stage 3 — Controlled Rollout	~20–50 accounts/week	Ongoing until all accounts migrated	Halt rate < 2%; accuracy_pct ≥ 99% per account before cutover approval.	PM + Ops

---

13. Dependencies

Dependency	Owner	Deliverable	Blocking?
New S2S migration endpoint POST /private/activity-logs/migrate	CDP Backend	Batch insert (≤20/call), accepts external_id, source_tag, metadata; unique-check (company_sso_id, external_id) before insert; skip-on-conflict. Auth: internal S2S token.	YES
ActivityLog schema additions	CDP Backend	external_id (string), source_tag (string), metadata (embedded doc) on the ActivityLog struct + MongoDB document; unique index (company_sso_id, external_id); new constant ActivityLogSourceCrmMigration = "crm_migration"	YES
CRM audits table read access	CRM / Data Eng	Direct query access (or a dedicated extraction endpoint) to audits table with date-range + auditable_type filters. Must handle cursor-based pagination. Rate-limit implications to confirm before Stage 1.	YES
CDP Contact migration completed per account	CDP Squad	CDP contacts (with customer_id ↔ CRM crm_person_id mapping) must exist for an account before activity logs are migrated — logs reference CDP contact IDs.	YES (ordering prerequisite)
CRM user → SSO UUID resolution	CRM Squad + Platform / USMAN	USMAN has no endpoint that accepts a CRM integer user_id directly. Resolution chain: (1) query CRM users table by user_id integer → get email; (2) call USMAN by email → SSO UUID. CRM Squad must confirm that the CRM users read is available (endpoint or direct table access). USMAN must handle deactivated users gracefully (return null, not error).	YES
CDP Contact crm_data.id coverage confirmation per account	CDP / Data Eng	Confirm Contact.crm_data.id (contact/base.go:342-344, indexed crm_contact_index) stores the CRM crm_person_id as a string for all migrated contacts (and/or that CRM crm_people.qontak_customer_id holds the CDP _id). Run a coverage count per account before Stage 1 (OQ-7). If coverage < 100%, a fallback mapping table is needed for the gap.	YES
FE: migration status component	CDP FE	Add a migration status indicator to DetailPage.vue (polling GET /private/activity-logs/migration/status). Migrated logs render in the existing CDP Activity Log view — no new FE field or badge.	YES (Stage 1)
audited_changes YAML parsing library	CDP Eng	Confirmed YAML parsing library available in the Go migration service for audited_changes column. Field name display mapping table (DB column → human-readable).	YES

📊 Dependency Graph

graph LR
    M[Activity Log Migration] -->|BLOCKING| EP[New S2S migrate endpoint]
    M -->|BLOCKING| SCH[ActivityLog schema: external_id + source_tag + metadata]
    M -->|BLOCKING| CRM[CRM audits table read access]
    M -->|BLOCKING - ordering| CTM[CDP Contact migration completed first]
    M -->|YES| USM[USMAN user_id -> SSO UUID]
    M -->|Stage 1| FE[FE: status indicator]
    M -->|YES| YAML[audited_changes YAML parser + field name map]

---

14. Key Decisions + Alternatives Rejected

14a — Decisions Made

All decisions dated 2026-06-03 (grounded code review).

ID	Decision	Rationale (grounded)
D-1	Build a new dedicated S2S migration endpoint, not repurpose the existing external endpoint.	The existing POST /openapi/v1/customers/{customer_id}/activity_logs is single-record only, requires activity_log_code (mandatory, no CRM equivalent), and validates category == 'transaction' only (external_activity_log_request.go:122-127). A new endpoint bypasses all these constraints and can accept batch payloads.
D-2	Add external_id + unique index (company_sso_id, external_id) to ActivityLog as the idempotency key.	Neither external_id nor any deduplication mechanism exists in the current ActivityLog struct (base.go:17-34). This is mandatory to make re-runs safe.
D-3	Add source_tag as a new constant crm_migration alongside existing external/qontak in const.go.	The existing ActivityLog.Source is a plain string with only two defined values. Rather than repurposing Source, a dedicated source_tag field enables: (a) rollback by DELETE WHERE source_tag = 'crm_migration', (b) validation/immutability filtering, without affecting the existing Source semantics. (No FE display/badge — see Non-Goals.)
D-4	Parse changes[] from audited_changes YAML column, NOT from the why field.	The why/define_why field is a human-readable formatted string (e.g. "Added a contact Name Changed." — audit.rb:602) with no parseable structure. The structured delta is in audited_changes (YAML hash of {field: [old, new]}), which is the correct source for changes[].
D-5	Re-implement Audit#parse() mapping logic in the Go migration service, not invoke a live Rails process.	Calling a live Ruby process adds a runtime dependency, is not scalable for high-volume batch migration, and is fragile across CRM versions. The mapping logic in audit.rb:1598-1981 is deterministic and translatable to Go.
D-6	Handle three actor cases (active/deleted/system) explicitly; never skip records due to unresolvable user.	The mapping_who() method (audit.rb:1815-1833) produces three distinct formats. Deleted users and system actions are legitimate historical events — dropping them would silently reduce accuracy_pct.
D-7	Use cursor-based pagination (WHERE id > :cursor LIMIT 500) for CRM audits extraction, not OFFSET.	OFFSET-based pagination on large tables degrades with every page. Cursor-based pagination over the id primary key index is O(log n) regardless of depth.
D-8	Reuse the existing ActivityLogMigrationConsumer/ActivityLogMigrationHandler code pattern, but upgrade the state store from ephemeral Redis to a durable persistent store.	The existing ActivityLogMigrationService uses Redis with a 7-day TTL (activity_log_migration_service.go:15-22). The CRM migration runs 24h+ per account — Redis is insufficient. The gocraft/work + status endpoint pattern is reused; the storage backend is upgraded to a MongoDB document or equivalent persistent store.
D-9	ContactResolver resolves uniquely via crm_data.id (CDP side) or qontak_customer_id (CRM side) — not source_id.	The two real CDP↔CRM linkage fields are: CDP crm_data.id (string, indexed crm_contact_index, queryable today via SearchByAppContactID(ctx, "crm", …), contact/base.go:342-344) which holds the CRM crm_person_id → primary path crm_data.id = auditable_id.to_s (no join); and CRM qontak_customer_id (crm_people.qontak_customer_id, schema.rb:2098, indexed) which holds the CDP contact _id → resolve by CDP _id after joining audits.auditable_id → crm_people. These are two separate linkage pairs, not equal values. source_id is rejected — it has no repository query method and no index (a source_id lookup is net-new + unindexed; crm_data.id holds the same crm_person_id and is already indexed). A fallback mapping table is used only if crm_data.id coverage is incomplete per OQ-7.
D-10	Parse changes[] using two branches: update = [old, new] arrays; destroy = full attribute snapshot.	For destroy actions, the audited gem records the full record attributes in audited_changes (not [old,new] pairs) — audit.rb:477-494. A single parsing strategy produces wrong results for destroys. Both branches must be handled explicitly.

14b — Alternatives Rejected

All rejections dated 2026-06-03.

Alternative	Why Rejected
Repurpose existing POST /openapi/v1/customers/{customer_id}/activity_logs	Single-record, mandatory activity_log_code, category == 'transaction' only — all unworkable for migration.
Parse changes[] from the why field	why is a human-readable string, not structured data. audited_changes YAML is the correct source.
OFFSET-based pagination for CRM extraction	Performance degrades at large page depths; cursor-based on id PK is always O(log n).
Call live Ruby/Rails Audit#parse() process	Not scalable for high-volume batch; adds a cross-language runtime dependency; fragile.
Store CRM audit id as the CDP primary ID	CDP generates its own IDs. CRM id is stored as external_id for traceability only.
Skip records with unresolvable user_id	Deleted and system-action records are legitimate audit events. Dropping them reduces accuracy.
Reuse Redis for migration job state	Existing Redis state has a 7-day TTL (activity_log_migration_service.go:15-22). 24h+ migrations that pause and resume would lose state. Durable DB store required.
Single audited_changes parsing branch for all action types	For destroy actions, audited_changes stores a full attribute snapshot (not [old,new] arrays) — audit.rb:477-494. A uniform parser produces incorrect changes[] for destroys.
Resolve contacts via Contact.source_id	source_id (contact/base.go:68) has no repository query method and no index — a source_id lookup is net-new and unindexed. The indexed crm_data.id (crm_contact_index) holds the same CRM crm_person_id and is queryable today; use it (or the CRM-side qontak_customer_id) instead (D-9).
Separate external mapping table as the primary contact resolution path	CDP contacts already carry the indexed crm_data.id (= CRM crm_person_id) and CRM carries qontak_customer_id (= CDP _id). A separate table is only a fallback for coverage gaps.

---

15. Open Questions

#	Type	Question	Mitigation / Default	Owner	Deadline
OQ-1	Decision	Which interface for CRM data extraction: direct DB query on audits table, or a new internal CRM extraction API endpoint? Direct DB access is simpler but couples the migration to the CRM schema; an API is more decoupled but requires Platform work.	Default: direct DB query (read-only SELECT) with appropriate service-account credentials. Confirm access control with CRM/Platform team.	CDP Eng + CRM Squad	2026-06-17
OQ-2	Open Question	What is the actual date range of audits records (run SELECT MIN(created_at), MAX(created_at), COUNT(*) FROM audits WHERE auditable_type IN (scope) LIMIT 1)? The 1-year window is a target; no retention policy is enforced in code. Actual volume and oldest records must be confirmed before Stage 0.	Run query before Stage 0; adjust batch count and job-duration estimate accordingly.	CDP Eng + Data Eng	2026-06-10
OQ-3	Risk	Partial migration failure: if a job fails at 60%, the account has a mix of migrated and non-migrated logs — inconsistent state. Is the policy (a) full transactional rollback on failure or (b) resume from last successful record?	Mitigation: default to resume from last successfully processed external_id on re-run (idempotency prevents duplicates). Full rollback is available via DELETE WHERE source_tag = 'crm_migration' AND company_sso_id = {account} and is triggered if data corruption is detected or root cause cannot be resolved within 48h. Define the threshold that triggers rollback vs resume before Stage 1.	PM + CDP Eng	2026-06-17
OQ-4	Open Question	Some audits records have auditable_type values not in the initial scope list (e.g. custom field changes, admin actions). Should these be migrated (expand scope), skipped silently, or flagged for review?	Default: skip with unmapped_action_type log. Expand scope in a follow-up if the volume is significant.	PM	2026-06-17
OQ-5	Decision	Are migrated activity logs immutable post-migration (agents cannot delete them)? CRM audit logs are immutable. If CDP allows deletion by certain roles, migrated logs should be protected by source_tag = 'crm_migration'.	Proposed: migrated logs are append-only (CANNOT delete if source_tag = 'crm_migration'). Needs CDP platform confirmation.	CDP Squad	2026-06-17
OQ-6	Open Question	The audited_changes YAML field names are DB column names (e.g. phone_number, first_name). A human-readable field name mapping table (e.g. phone_number → "Phone Number") must be defined. Is this mapping centrally owned (shared with the CDP field definitions), or migration-service-local?	Default: migration-service-local lookup table (can be promoted to shared later).	CDP Eng	2026-06-17
OQ-7	Risk	The resolver depends on Contact.crm_data.id (contact/base.go:342-344, indexed crm_contact_index) holding the CRM crm_person_id for each migrated contact (and/or CRM crm_people.qontak_customer_id holding the CDP _id). If coverage is incomplete, some activity log records will fail with contact_not_found even though the contact is in CDP.	Mitigation: run a coverage count (db.contact.countDocuments({ "crm_data.id": null, company_sso_id: {account} })) per account before Stage 1. If coverage < 100%, backfill crm_data.id or build a fallback mapping table for the gap. Block migration for the account until coverage is ≥ 99%.	CDP / Data Eng	2026-06-25

---

Appendix A — Grounded Code References

contact-service (CDP, Go/MongoDB)

• Real external write endpoint: POST /openapi/v1/customers/{customer_id}/activity_logs → ExternalActivityLogHandler.CreateActivityLog — internal/server/rest_router.go:365.
• External request schema: source, category, activity, action, description, attributes, activity_log_code — internal/app/payload/external_activity_log_request.go:11-33. No external_id, source_tag, bulk params.
• activity_log_code is mandatory — external_activity_log_request.go:85-89.
• Category validation on external endpoint: category == 'transaction' only — external_activity_log_request.go:122-127.
• Bulk write internal endpoint (max 20): POST /api/v1/activity_logs → ActivityLogHandler.BulkCreate; service/activity_log.go:128-146.
• ActivityLog struct (no external_id, source_tag, metadata): internal/app/repository/activity_log/base.go:17-34.
• Source constants today: only ActivityLogSourceExternal = "external" and ActivityLogSourceQontak = "qontak" in const.go — no crm_migration.
• Existing migration framework to reuse: ActivityLogMigrationHandler + ActivityLogMigrationConsumer + GET /private/activity_logs/migration/status — rest_router.go:74; mirrors the pattern for the new activity log migration.
• Contact↔CRM linkage fields (ContactResolver, D-9): CDP Contact.crm_data.id (string, indexed crm_contact_index, queryable via SearchByAppContactID(ctx, "crm", …)) holds the CRM crm_person_id — internal/app/repository/contact/base.go:342-344 (CrmData{ID}, bson crm_data.id), index in db/migrations/001_create_contact.up.json. Contact.source_id (base.go:68) has no repository query method and no index. The live activity-log ingestion already resolves qontak_customer_id as the CDP contact _id via SearchByID — internal/app/service/activity_log.go:279-285.

qontak.com (Legacy CRM, Rails)

• Real table: audits — db/schema.rb. Real columns: id, auditable_id, auditable_type, associated_id, associated_type, user_id, username, action, audited_changes (TEXT/YAML), version, comment, remote_address, request_uuid, created_at.
• ALL named fields in v1.0 (type, how, who, what, why, content, data, crm_person_id) are attr_accessors — app/models/audit.rb:14-17. Computed at runtime by parse().
• Transformation logic: mapping_type(), mapping_who(), mapping_how_what_why() — audit.rb:1598-1981.
• who field formats: "Full Name (email)" / "Full Name <deleted user>" / "Qontak system" — audit.rb:1815-1833.
• audited_changes differs by action type: update = YAML {field: [old, new]}; destroy = full attribute snapshot {field: value} (NOT [old,new]) — audit.rb:477-494. Two parsing branches required.
• crm_person_id for association events: extracted from comment JSON column first; falls back to audited_changes YAML — audit.rb:483-494.
• why field = human-readable formatted string — unparseable into structured changes.
• No audits composite index on (auditable_type, created_at) — only separate single-column indexes exist (db/schema.rb). Extraction query needs query plan analysis.
• No confirmed REST extraction endpoint with the v1.0 described path/params. Contacts::Timeline calls an external CDP endpoint — app/services/contacts/timeline.rb:34-40.
• No code evidence of 1-year retention policy — audit_batch_sql_job.rb runs arbitrary SQL params only.
• CRM qontak_customer_id (ContactResolver, CRM side): crm_people.qontak_customer_id (string, indexed index_crm_people_on_qontak_customer_id) holds the CDP contact _id (set from @cdp_contact['id']) — db/schema.rb:2098,2117; app/services/contact360/cdp_incoming_contact_mapper.rb:97; app/services/contact360/params_mapper.rb:48. An audits row references the person by auditable_id (= crm_people.id); qontak_customer_id is not on the audit — reach it by joining audits.auditable_id → crm_people.id.

qontak-customer-fe (Nuxt 3)

• ActivityLog TypeScript interface: 10 fields, no source/source_tag/metadata — features/customers/store/CustomerStore.ts:145-156.
• ActivityLog.vue renders a pre-computed timeline[] object (name, title, caption, status) — NOT raw ActivityLog fields — features/customers/detail/components/ActivityLog.vue:23-50. Migrated logs reuse this existing rendering as-is; no source badge is added.
• ActivityLog.vue caps visible entries at 5,000 (page === 1000, line 50). Large migrated histories will silently truncate.
• ActivityLog.Actor (string) exists in the CDP ActivityLog struct (activity_log/base.go:29) — this is the correct field for deleted-user and system-action display names, not description.
• DetailPage.vue has no migration status component — features/customers/detail/views/DetailPage.vue:1-49.
• FE migration status indicator is net-new component work; the migrated logs themselves require no FE change (rendered by the existing Activity Log view).

---

PRD CHANGELOG

Version	Date	By	Section	Type	Summary
2.5	2026-07-01	New stories from engineering RFC	S8 (User Stories)	ADDED	Added three new user stories from the engineer's RFC: CALM-S07 — Migrate only pre-sync activity (no double-import) (SyncWatermarkResolver reads CDP contact.created_at; skips audits.created_at >= watermark as post_sync_live_feed; no-watermark → contact_not_found; RFC Decision 16 — implements the v2.4 case-2a/2b log-scope split); CALM-S08 — Reconcile association logs; fill missing metadata (AssociationLogEnricher fills crm_person_id + linked-entity id/name via CRM lookup and patches existing CDP logs in place, incl. live-feed; deleted referent → [Deleted reference]; idempotent re-run patches 0; additive/enrich-only exception to strict skip-on-duplicate — resolves the v2.4 case-2b enrich OQ; RFC Decision 17); CALM-S09 — Gate on contact coverage (Phase A) (coverage < threshold → 422 contacts_not_migrated, no job; remediated by upstream BackfillCustomerIdWorker; residual unresolved → contact_not_found; refines CALM-S01 ERR-2, relates to OQ-7; RFC Decision 18). Extended the story-dependency footnote for S07–S09.
2.4	2026-06-26	Migration scope — contact population	S5 (Constraints)	UPDATED	Added a Migration scope — contact population & log coverage constraint: two contact populations are in scope — (1) contacts not yet synced to CDP (full pre-sync backfill once resolvable) and (2) contacts already synced to CDP, covering (2a) logs created before sync and (2b) post-sync logs still missing info/metadata. Flagged the idempotency tension: case (2b) is an enrich/update path that conflicts with the strict skip-on-duplicate (company_sso_id, external_id) rule — skip-vs-enrich policy to be confirmed (OQ).
2.3	2026-06-25	Remove source-badge display	S4 (Non-Goals), S6 (New Features), S8 (CALM-S04), Schema, S13–S15, D-3, App.A	UPDATED	Removed all requirements to display a source badge / "Migrated from CRM" label on migrated activity logs. CALM-S04 reframed to "migrated logs visible in the existing CDP Activity Log view" — migrated entries render with the standard CDP structure (category / action / changes / timestamp / actor), indistinguishable from native logs (dropped badge ACs + the FE timeline[] threading note). Dropped the FE ActivityLog.vue badge work and the source_tag/metadata additions to the FE ActivityLog TS interface; FE scope is now just the migration status indicator (CALM-S05, unchanged). Backend source_tag field is retained — still used for rollback + validation + immutability; only its FE display is removed.
2.2	2026-06-22	ContactResolver correction (grounded)	S4, S6, S7, S8, S13, S14, S15, App.A	UPDATED	ContactResolver now resolves uniquely via the two real CDP↔CRM linkage keys instead of source_id: CDP-side crm_data.id (indexed crm_contact_index, queryable via SearchByAppContactID(ctx, "crm", …), holds the CRM crm_person_id) and CRM-side qontak_customer_id (crm_people.qontak_customer_id, indexed, holds the CDP contact _id). Grounded in contact-service (contact/base.go:342-344, activity_log.go:279-285) + qontak.com (schema.rb:2098, cdp_incoming_contact_mapper.rb:97). Rejected source_id (no query method, no index). Updated: Constraints contact-mapping path, component tree, API behavior #3, system-flow contact resolution + step 5a + skip condition, CALM-S01 AC-4, CALM-S03 ERR-2, dependency (crm_data.id coverage), D-9 rewritten, alternatives (source_id rejected), OQ-7 (crm_data.id coverage; deadline → 2026-06-25), Appendix A (linkage-field grounding both repos). Note the two linkage pairs are distinct (crm_data.id ↔ crm_person_id, qontak_customer_id ↔ CDP _id) — not equal values.
2.1	2026-06-03	Score fixes (9 grounded gaps)	S3, S4, S6, S7, S8, S13, S14, S15	UPDATED	Non-Goals: added FE 5000-entry cap (#7). Constraints: added CRM extraction query performance warning (no composite index on audits), durable state store requirement (Redis TTL insufficient), Contact.source_id mapping path, CRM user_id→SSO UUID resolution chain. API Behavior 3: two changes[] parsing branches (update=[old,new] vs destroy=snapshot); crm_person_id from comment JSON first; Actor field (not description) for deleted/system users. CALM-S01 AC-4: Contact.source_id resolution. CALM-S03: AC-2b (destroy branch), AC-4/AC-5 fix Actor field, AC-8 (comment JSON path). CALM-S04: FE transformation pipeline dependency note. Test matrix updated. S13: USMAN deliverable corrected (resolution chain: CRM users → email → USMAN); source_id coverage dep added. D-8 revised (durable state), D-9 (source_id mapping), D-10 (destroy parsing) added. OQ-3 strengthened mitigation. OQ-7 (source_id coverage, deadline 2026-06-10) added. Appendix A updated with destroy/comment/index findings.
2.0	2026-06-03	Grounded rewrite	All	REWRITE	Full rewrite grounded in contact-service, qontak.com, qontak-customer-fe. Critical corrections: (1) real write endpoint is POST /openapi/v1/customers/{id}/activity_logs, requires new S2S endpoint; (2) external_id/source_tag/metadata are all net-new fields; (3) CRM source is audits table — type, how, who etc. are computed attr_accessors, not DB columns; (4) changes[] must be parsed from audited_changes YAML, not why; (5) CRM extraction API doesn't exist — direct DB query required; (6) who has 3 formats (active/deleted/system); (7) FE badge and migration status indicator are net-new; (8) activity_log_code is mandatory on existing external endpoint; (9) bulk write capped at 20 on internal endpoint only; (10) category constants need expanding beyond transaction.
1.0	2026-05-28	(prior)	All	CREATED	Prior version (superseded — numerous endpoint, source schema, and FE scope errors).