Qontak | Chatbot & AI | AI Agent Impact Report — ANCHOR
Template: ANCHOR PRD v1.2 · Companion to PRD Section Reference v1.5 + Hierarchy v1.0
HEADER BLOCK
| Field | Value |
|---|---|
| PM | Dimas Fauzi Hidayat |
| PRD Version | 1.1 |
| Status | DRAFT |
| PRD Type | ANCHOR |
| Anchor | Yes — this IS the Anchor |
| Labels | epic:qontak-chatbot-ai | module:chatbot-ai | feature:ai-agent-impact-report |
| Last Updated | 2026-07-01 |
Status values:
DRAFT→ACTIVE→DEPRECATED
Table of Contents
- HEADER BLOCK
- 2. PHASE INDEX
- 3. One-liner + Problem
- 4. What Happens If We Don't Build This
- 5. Target Users + Persona Context
- 6. Success Metrics (Initiative-level)
- 7. Key Decisions + Alternatives Rejected
- 8. Open Questions
- PRD CHANGELOG
2. PHASE INDEX
Source of truth for all phases. Goal column must match each child Phase PRD's "Phase Goal" exactly.
| Phase | Goal | PRD Link | Epic Key | Status | Shipped |
|---|---|---|---|---|---|
| Phase 1: Live Impact Report (MVP) | Ship the customer-facing, per-org AI Agent Impact report off existing Postgres rooms data — honest-containment hero (net of 48h reopens), onboarding-baseline trend + next-period forecast, after-hours containment, work-absorbed (admin-set rupiah assumption), reopen rate, sentiment delta, turns-to-resolve, and the AI+human blended journey — in plain, non-technical language. No new backend instrumentation. | Phase 1 PRD (draft — link on publish) | QC-XXXXX | 📝 Draft | — |
| Phase 2: Diagnostic & Prescriptive Layer | Add the root-cause + action layer: answer coverage (in-scope KB-answered vs out-of-scope unanswered) with out-of-scope reason persisted from the AI-service response, escalation-reason taxonomy, containment-per-topic, and the knowledge-gap → projected-lift table with "Draft with AI". Requires new telemetry into the ai_activity_logs datamart. | — | — | ⏳ Not started | — |
| Phase 3: Audited Value & Quality | Reach competitive table-stakes: CSAT micro-survey, quality scoring across all conversations (Ada-grade Relevant+Accurate+Safe+Contained), audited rupiah (Billing rate + human handle-time benchmark), and action-execution success for the autonomous agent. | — | — | ⏳ Not started | — |
Status options: 📝 Draft · 🔄 In Progress · ✅ Shipped · ⏸ Paused · ❌ Cancelled
3. One-liner + Problem
One-liner: Give the org admin of a Qontak AI-agent account one plain-language report that shows what the AI actually did, how well it did it, and what that's worth.
Problem: Customers pay for the Qontak AI Agent but have no in-product way to see the value it delivers — no view of how much conversation load it absorbs on its own, how well it answers, how far it's improved, or what that is worth in rupiah or hours saved. The people who need this — non-technical org admins and CS supervisors at Indonesian SMEs, and the business owners who approve the spend — today judge the AI on gut feel from the inbox, and cannot answer their own boss's question, "is this AI thing worth the money?" The cost is direct: the paid AI add-on has no value narrative at renewal, expansion conversations lack evidence, and Qontak has no answer to Intercom Fin, Zendesk AI, and Ada — all of whom ship an AI resolution/ROI dashboard as standard.
4. What Happens If We Don't Build This
- The paid AI add-on renews on faith, not evidence. With no value narrative in-product, every renewal of the AI agent rests on the customer's gut feeling — the first budget cut when they can't see what they're paying for.
- We have no answer to competitors' ROI story. Intercom Fin (resolution rate + CX Score), Zendesk AI (automated-resolution dashboard + forecast), and Ada (Performance Center + ROI) all ship this as standard. Its absence is a visible gap in competitive deals and renewals in 2026.
- AI value stays anecdotal, so expansion stalls. Without data proving the AI absorbs real load, there's no basis to sell the customer into more channels, more autonomous actions, or a higher tier.
- Telemetry gaps never get prioritized. Confidence and answer-scope reasons remain unpersisted (AI-service-side only) as long as no consuming product forces the instrumentation contract — blocking not just this report but quality scoring and monitoring too.
5. Target Users + Persona Context
Primary Persona: Org Admin / CS Supervisor ("Bu Rina")
| Field | Detail |
|---|---|
| Role | Non-technical org admin or CS supervisor at an Indonesian SME running customer service on Qontak (WhatsApp-first), who approved the AI add-on and manages a small team of human agents |
| Goal | Know at a glance whether the AI is doing a good job and is worth the spend — and be able to defend that to her boss with a clear number |
| Pain | No in-product view of what the AI does; its value is invisible, so she judges it on inbox "feel" and can't answer "is it worth it?" |
| Workaround | Eyeballs the inbox, counts conversations by hand, guesses; no way to show trend, quality, or money saved |
Secondary Persona: Business Owner / Budget Decision-Maker
| Field | Detail |
|---|---|
| Role | The owner or manager who approves the AI add-on spend and the renewal at an SME |
| Goal | A simple, credible value story — hours saved / rupiah saved — to justify continuing (or expanding) the AI spend |
| Pain | Only sees the invoice, never the value; has to take the supervisor's word for whether the AI helps |
| Workaround | Asks the supervisor, who has no data to show; decision defaults to cost-cutting at renewal |
6. Success Metrics (Initiative-level)
North star for the full initiative. Per-phase metrics live in each Phase PRD. Targets marked (assumption) need PM/CS/BI confirmation — logged in Section 8.
Adoption & Usage:
| Metric | Definition | Baseline | Target |
|---|---|---|---|
| ⭐ Impact report adoption | % of AI-agent-enabled org accounts whose admin views the Impact report ≥1× per month | N/A — report does not exist today | ≥50% of AI-enabled orgs within 90 days of Phase 1 GA (assumption) |
| Repeat engagement | % of viewing admins who return in ≥2 consecutive months | N/A | ≥30% within 2 quarters of Phase 1 GA (assumption) |
Quality & Accuracy:
| Metric | Definition | Baseline | Target |
|---|---|---|---|
| Honest-containment coverage | % of reports that display the reopen-adjusted (net) containment rate and its false-resolution gap — never a gross-only vanity number | N/A | 100% of reports at Phase 1 GA |
| Quality-validated resolution coverage | % of AI conversations with persisted confidence + answer-scope reason enabling Ada-grade (Relevant+Accurate+Safe+Contained) scoring | 0% — confidence and reason not persisted (AI-service-side only) | ≥90% of AI conversations by Phase 3 GA |
Efficiency & Impact:
| Metric | Definition | Baseline | Target |
|---|---|---|---|
| AI add-on renewal / expansion (directional) | Renewal or expansion rate of the AI add-on for orgs that engaged the report vs. those that did not | Current AI add-on renewal rate — CS-owned, TBD | Positive lift for engaged orgs; tracked directionally, not committed (attribution is correlational) (assumption) |
| Value made visible | % of AI-enabled orgs shown a non-zero "work absorbed / cost saved" figure (admin-set assumption in P1; audited in P3) | 0% — no value figure exists | 100% of AI-enabled orgs at Phase 1 GA |
7. Key Decisions + Alternatives Rejected
7a — Decisions Made
| Date | Decision | Rationale |
|---|---|---|
| 2026-07-01 | Hero metric = honest containment (RESOLVE_AI, no human attached, net of same-contact reopens within 48h) — show net, not gross | A reopen-adjusted number is the credible signal; a believable ~62% beats a vanity ~68% and inoculates the report against "the customer just gave up." Mirrors Intercom's confirmed-vs-assumed resolution. |
| 2026-07-01 | Containment is defined from rooms.closed_reason='RESOLVE_AI' + assign_channel_agent_id IS NULL + closed_at — not is_failed, forward_request, or exit[:id] | Code dig: is_failed is fire-and-forget to Billing (not queryable) and conflates AI-doesn't-know with technical error; forward_request/forwarded_at are dead columns (zero write sites); exit[:id] is intent-routing only. Bare RESOLVE conflates human-resolve with idle-timeout and is excluded from the win. |
| 2026-07-01 | Layered monetization — hard units (conversations, hours) by default; rupiah "cost saved" as an optional, admin-set assumption | No rupiah rate or human handle-time benchmark exists in the chatbot BE; a customer-set assumption is both feasible now and, per Ada's model, more trusted than a vendor-imposed figure. |
| 2026-07-01 | Position on the AI+human blended journey + honesty, not on prescriptive gap insight | Competitive scan: Intercom/Ada already detect gaps and auto-draft articles (we are parity-to-behind there). The blended journey is a structural moat pure-AI vendors (e.g. Decagon) explicitly cannot report, because Qontak's AI lives inside the omnichannel inbox with human agents. |
| 2026-07-01 | Two artifacts — an annotated spec wireframe (for build) and a plain-language customer view (what ships) | Readability audit against the non-technical admin ("Bu Rina") showed jargon (containment, in/out-of-scope, KB) and internal markers (phase tags, competitor names) make a single combined artifact unreadable to the actual user. |
| 2026-07-01 | Phased delivery: P1 live-data MVP → P2 instrumentation-dependent diagnostic/prescriptive → P3 table-stakes | The hero metric + trend ship dependency-free off Postgres rooms; insight value and dependency surface rise together, so ship credible value first and pull instrumentation forward per phase. |
| 2026-07-01 | The report's metric spec drives the ai_activity_logs datamart / telemetry contract (flip the dependency) | Don't let existing data cap the report; the report defines what to instrument next (persisted confidence, answer-scope reason, escalation reason, action result). |
7b — Alternatives Rejected
| Alternative | Why Rejected | Date |
|---|---|---|
| Existing-data-only descriptive report (volume/percentage counts) | Vanity metrics; gets opened once and fails the renewal goal. The insight (quality, coverage, prescriptive) is what makes it worth building. | 2026-07-01 |
| Gross containment as the hero number | Over-counts silent abandonment and idle-timeout closes (bare RESOLVE is conflated); fails the credibility bar the report is meant to set. | 2026-07-01 |
| Qontak-computed rupiah ROI figure | No token→rupiah rate or human benchmark in-product; a vendor-imposed money number is contestable. Admin-set assumption chosen instead. | 2026-07-01 |
| Lead the product on prescriptive gap→lift as the differentiator | Incumbents are ahead here (Intercom/Ada auto-draft gap articles ranked by impact). Repositioned to parity-to-reach, with quantified lift as a minor twist. | 2026-07-01 |
Pure Metabase-iframe report (reuse /reports shell only) | Can't host the interactive cost-assumption toggle, forecast, or config; a native surface is required for those. Static charts may still ride Metabase. | 2026-07-01 |
| Single artifact serving both PM/build and the customer | Internal markers and jargon make it unusable for the non-technical admin; readability audit forced the split into two views. | 2026-07-01 |
8. Open Questions
| # | Type | Question | Owner | Deadline |
|---|---|---|---|---|
| 1 | Risk | The AI-service answer-scope / reason (why a question was out-of-scope) is not persisted today — it lives only in the AI-service response body. It is the highest-leverage Phase-2 dependency (feeds out-of-scope reason, escalation reason, and gap-lift). Mitigation: Phase 1 ships without it; scope a telemetry change into the ai_activity_logs datamart + AI-service before Phase 2 commits. | Dimas + AI-Service / BI | Before Phase 2 kickoff |
| 2 | Open Question | Is a per-org token→rupiah rate queryable from the Qontak Billing service at report time, or must the report apply an admin-set rate to token volume? | Dimas + Billing | Before Phase 3 scope |
| 3 | Assumption | 48h is the right reopen window, and "same contact + similar topic" is the right false-resolution match rule. | Dimas + BI | Phase 1 grooming |
| 4 | Open Question | Cross-store feasibility/performance: containment is in Postgres rooms, while AI durations, turns, sentiment, and scorecards live in MongoDB (ask_airene_history). Involvement/speed metrics need a cross-store join. | Dimas + BE | Phase 1 grooming |
| 5 | Assumption | A per-org business-hours configuration exists (or can be derived) to compute the "after-hours containment" metric. | Dimas + BE | Phase 1 grooming |
| 6 | Resolved | Squad ownership + build surface — RESOLVED (2026-07-01): owning squad = BOT — Bot, AI & Automation; Phase 1 is a native page sourced from the ai_activity_logs datamart (not the Metabase /reports iframe). See Phase 1 PRD §15. | Dimas | Resolved 2026-07-01 |
| 7 | Risk | Renewal/expansion attribution to report engagement is correlational, not causal. Mitigation: track as a directional impact metric only (Section 6), never a committed target. | Dimas + CS | Ongoing |
Types:
Assumption·Open Question·RiskRisk-type rows without a mitigation path block ACTIVE status. Rows 1 and 7 each carry an explicit mitigation.
PRD CHANGELOG
Append-only. Claude generates entries — PM never writes to this table manually.
| Version | Date | By | Section | Type | Summary |
|---|---|---|---|---|---|
| 1.0 | 2026-07-01 | Claude | All | CREATED | Initial ANCHOR PRD created from product-ideation → feature-comparison → code-grounding → mekari-taste wireframe session. Locks the honest-containment hero, the 3-phase architecture, and the initiative-level decisions/alternatives. |
| 1.1 | 2026-07-01 | Claude | Open Questions | MODIFIED | Resolved OQ#6 after Phase 1 grooming: owning squad = BOT — Bot, AI & Automation; Phase 1 is a native page on the ai_activity_logs datamart. |