Qontak | Chatbot & AI | Bot-vs-Human Traffic Split — Phase 1: Per-Conversation Split

Phase 1 (NEW) PRD under the Bot-vs-Human Traffic Split initiative.

HEADER BLOCK

Field	Value
PM	Dimas Fauzi Hidayat
PRD Version	1.2
Status	DRAFT
PRD Type	NEW
Epic	TBD — add once Epic is created
Squad	Chatbot
RFC Link	N/A — to be created via rfc-starter after READY
Figma Master	Pending — draft wireframes in §6.1 await designer validation → Figma
Anchor	No — standalone, single-squad (Phase 2 sticky-per-customer tracked in the initiative README)
Labels	epic:qontak-chat | module:regular-chatbot | feature:bot-traffic-split
Last Updated	2026-06-16

Status values: DRAFT → READY → BUILD → SHIPPED

Table of Contents

• HEADER BLOCK
• 2. One-liner + Problem
• 3. Target Users + Persona Context
• 4. Non-Goals
• 5. Constraints
• 6. New Features
  • 6.1 Design Draft — for Designer review & validation
• 7. API & Webhook Behavior
• 8. System Flow + User Stories + ACs
  • 8.1 System Flow
  • 8.2 User Stories
• 9. Rollout
• 10. Observability
• 11. Success Metrics
• 12. Launch Plan & Stage Gates
• 13. Dependencies
• 14. Key Decisions + Alternatives Rejected
• 15. Open Questions
• PRD CHANGELOG

2. One-liner + Problem

One-liner: Let a Chatbot Admin route a configurable percentage of chats to the bot and the rest to human agents, validating bot quality before full automation.

Problem: Today a Qontak chatbot is all-or-nothing — once enabled on a channel it handles 100% of incoming chats. A Chatbot Admin who isn't yet confident in the bot has no safe way to start small: they can't expose it to a controlled slice of real customer traffic while humans cover the rest, and they get no apples-to-apples comparison of bot vs human outcomes on the same audience. So Admins either over-commit to a bot that may underperform — risking CSAT and lost conversations — or never launch automation at all, leaving deflection value on the table. The only workarounds today (Bot Preview, or a low-stakes test channel like Telegram) are synthetic — they prove nothing about how the bot behaves with the Admin's real customers on their real channel.

3. Target Users + Persona Context

Persona	Role	Goal	Pain	Workaround
Primary — Chatbot Admin	Admin / SPV who configures the chatbot for a Qontak account	Gain confidence the bot performs well on real customers before committing to full (100%) automation	The bot is all-or-nothing per channel — there is no way to limit real-traffic exposure while ramping up trust	Tests via Bot Preview (tree-diagram simulator) or spins up a low-stakes channel like Telegram — but neither reflects real customers on the real production channel, so the resulting confidence is not trustworthy
Secondary — Human Agent	CS agent who receives conversations in the Inbox	Receive the human-assigned share of chats and handle them as normal	Sudden, unplanned bot handoffs create unpredictable load	Handles whatever the bot escalates today, with no notion of a deliberate traffic share

Plan availability and the feature flag live in Section 5 (Constraints).

4. Non-Goals

1. No sticky per-customer bucketing — Phase 1 splits per conversation (each new incoming chat is bucketed independently). A returning customer may land in the bot arm on Monday and the human arm on Tuesday. Deterministic per-unique-customer assignment is Phase 2.
2. No mid-conversation re-bucketing — the arm (bot vs human) is decided once at conversation start and does not change for the life of that conversation. Normal bot→human escalation still works, but that is a handover, not a re-roll of the experiment.
3. No bot-A-vs-bot-B testing — Phase 1 compares the chatbot against human agents only. Routing a percentage to a second, different chatbot is out of scope.
4. No automatic winner selection or auto-ramp — the system does not read the results and change the percentage by itself. The Admin reads the metrics and adjusts the percentage manually.
5. No change to the no-agent-available behavior — when a chat is assigned to the human arm and no agent is available, it follows the channel's existing queue/offline behavior. This feature does not add a new "bot rescues the queue" fallback (see Section 14, decision rejected).
6. No statistical significance engine — Phase 1 surfaces the raw comparison metrics (resolution, CSAT, handover) per arm; it does not compute confidence intervals or declare a "statistically significant" winner.
7. No per-segment / per-topic targeting — the split is a flat random percentage across all incoming chats on the channel, not conditioned on customer attributes, intent, or message content.

5. Constraints

Platform:       Routing engine — backend (chatbot). Config UI — web (chatbot-fe) only. No mobile config in Phase 1.
Scope unit:     Per channel integration (the "Path"). Each bot-enabled channel has its own
                independent split percentage. Not a single global account-level percentage.
Percentage:     Integer 0–100 (whole numbers). 0 = all human, 100 = all bot (current default behavior).
Performance:    The split decision adds no perceptible latency to message handling — it is an
                in-process check during find_default_path, with a target of ≤ 5 ms added per
                incoming message and no extra network/DB round-trip on the hot path beyond
                reading the already-loaded Path config.
Determinism:    Phase 1 bucketing is random per conversation (rand(100) < bot_percent). No
                identity hashing in Phase 1 — that is Phase 2.
No-agent path:  When the human arm is selected but no agent is available, the chat enters the
                existing human queue / channel offline behavior (SendMessageAutoAssignAgentWorker).
                The bot does NOT pick it up. Trade-off accepted to keep the comparison clean
                (see Section 14).
Plan scope:     [OPEN — see Section 15] Proposed: Professional + Enterprise plans that already
                have the chatbot module. Not available on plans without chatbot access.
Feature flag:   bot_traffic_split_enabled | default: OFF. Enabled per organization during rollout.
Read/write:     Configure split % — Chatbot Admin / Admin role. View results — Admin + SPV.
                Human Agents and end-customers cannot see or change the percentage.

5.1 Data Lifecycle

The experiment introduces one new persisted attribute (the per-conversation variant tag) plus event logs. No file artifacts or background batch jobs.

Artifact Type	Retention Period	Cleanup Trigger	User-Visible Effect
Per-conversation variant tag (bot / human) on the conversation/room record	Same lifetime as the conversation record itself	Inherits existing conversation retention (acts_as_paranoid soft-delete) — no separate cleanup	None — internal attribute used for reporting/segmentation
Split-decision analytics events (e.g. bot_traffic_split_assigned)	Per existing product analytics retention	Existing analytics pipeline TTL	None — feeds the comparison dashboard

6. New Features

Feature: Bot Traffic Split configuration (within the channel's chatbot settings)

URL:     Within the existing bot/channel configuration screen for a channel integration
         (e.g. /settings/chatbot/{channel_integration_id}) — a new "Traffic Split" control.
Access:  Chatbot Admin / Admin (configure). Admin + SPV (view).

Component Tree:

ChatbotChannelSettings (existing screen)
└── TrafficSplitSection                  — new section; only visible when bot_traffic_split_enabled flag is ON
    ├── EnableSplitToggle                — turns the A/B split on/off for this channel
    ├── BotPercentageInput               — integer 0–100; the % of new chats routed to the bot
    ├── SplitPreviewLabel                — live helper text, e.g. "~30% to bot, ~70% to human agents"
    └── SaveButton                       — persists the percentage to the channel's Path config

UI States:

Empty:    Split disabled (default) — toggle OFF, percentage field hidden/greyed. Helper text:
          "All incoming chats are handled by the bot (100%)." (i.e. current behavior.)
Loading:  Save in flight — Save button disabled + spinner. Existing value remains until confirmed.
Error:    Invalid input (not 0–100 / not an integer) → inline validation error, Save blocked.
          Save fails (network/5xx) → "Couldn't save traffic split. Try again." + Retry. Log event.
Success:  Toggle ON, percentage saved → confirmation toast "Traffic split updated: 30% bot / 70% human."

Figma: Pending — see Stitch UI Prompts (generated at READY). Frame-level link to be added.

📊 UI State Diagram — Traffic Split configuration

stateDiagram-v2
    [*] --> Disabled: Section loads (default)
    Disabled --> Editing: Admin toggles split ON
    Editing --> Loading: Admin clicks Save
    Editing --> Editing: Invalid value (inline error, Save blocked)
    Loading --> Success: Saved (toast shown)
    Loading --> Error: Save fails (5xx/network)
    Error --> Loading: Admin clicks Retry
    Success --> Editing: Admin edits percentage again
    Success --> [*]
    Disabled --> [*]

6.1 Design Draft — for Designer review & validation

🎨 Status: DRAFT — pending designer validation. These low-fidelity wireframes were produced via the designer-repo mekari-taste imagination phase (Pixel 2.4; components validated against the Pixel MCP). They are structural proposals, not final design — included here so PM, Eng, and Design collaborate on one document. The designer owns turning these into Figma frames using production Pixel 3 components; please leave comments inline or replace with Figma links once frames exist. Open design questions are listed at the end of this block.

Screen A — Traffic Split configuration (pattern: layout-shell + form-view + inline banner)

░░ = surface #F1F5F9 (frame)   ▓▓ = brand #4B61DC (reserve for primary action + active nav)   ── = white content
┌──────────────────────────────────────────────────────────────────────────┐
│ ░░ qontak                                              🔔   (DF) Dimas ░░  │  top bar (frame)
├────────────┬───────────────────────────────────────────────────────────────┤
│ ░ Inbox    │  Chatbot Settings ›  WhatsApp – Sales            (breadcrumb)   │  page header (frame)
│ ░ Broadcast│  ───────────────────────────────────────────────────────────   │
│ ░▓Chatbot▓ │   Traffic Split (A/B)                          (H3 sub-section) │  active nav = brand
│ ░ Contacts │                                                                 │
│ ░ Reports  │   ◉───  Enable traffic split            ← MpToggle             │
│ ░ Settings │         Send only part of new chats to the bot; the rest go    │
│            │         to human agents.                  (#description, 12px)  │
│            │                                                                 │
│            │   Bot handles                                                   │
│            │   ┌───────────────────┬─────┐                                   │
│            │   │  30               │  %  │  ← MpInputGroup + MpInputRightAddon│
│            │   └───────────────────┴─────┘                                   │
│            │   ~30% to bot, ~70% to human agents      (live preview, 12px)   │
│            │                                                                 │
│            │   ┌ ⓘ When no agent is available, human-arm chats wait in the ┐ │
│            │   │   normal queue — the bot will not take them over.          │ │  ← MpBanner info (inline)
│            │   └────────────────────────────────────────────────────────────┘ │
│            │                                        Cancel        ▓ Save ▓   │  footer (Save=primary)
└────────────┴───────────────────────────────────────────────────────────────┘
States ▸ Disabled (default): toggle off, % field + preview + banner hidden · Loading: Save disabled + spinner
        Error: inline "Enter a whole number 0–100" / save-failed + Retry · Success: toast top-center

Screen B — Bot vs Human comparison (pattern: layout-shell + index-view summary strip + table + empty-state)

┌──────────────────────────────────────────────────────────────────────────┐
│ ░ Chatbot ›  Traffic Split  ›  Comparison                                  │  page header (frame)
│   Bot vs Human Comparison                              (H1)                 │
│   ┌ Channel: WhatsApp – Sales ▾ ┐  ┌ Last 14 days ▾ ┐        (MpSelect ×2)  │  toolbar (frame)
│ ──────────────────────────────────────────────────────────────────────────│
│  ░░ Summary strip (96px): Conversations │ ▓Resolution parity 92%▓ │ CSAT −0.3 │ Handover 24% ░░
│ ──────────────────────────────────────────────────────────────────────────│
│   Metric              │  ● Bot arm        │  ● Human arm    │  Δ / Parity    │  MpTable header (52px)
│   Conversations       │  372              │  868            │   —            │
│   Resolution rate     │  78%              │  85%            │  92% parity ▓  │  ⭐ only KPI w/ brand
│   CSAT (avg /5)        │  4.4              │  4.7            │  −0.3          │
│   Handover to human   │  24%              │  —             │   ≤30% ✓        │
│   [● Bot] [● Human] (MpBadge legend)           Updated 16 Jun 2026, 14:20   │  accountability: timestamp
└──────────────────────────────────────────────────────────────────────────┘
States ▸ Loading: skeleton rows · Bot-arm empty (%=0): "No bot data yet" (not 0%) · Filtered-empty:
        "No conversations in this range yet" · Error: Retry, render no partial/misleading numbers

Validated Pixel components (2.4): MpToggle (#description slot) · MpInputGroup+MpInput+MpInputRightAddon (%, size="md", has-background) · MpBanner (variant="info", is-inline) · MpButton (Save primary / Cancel ghost) · MpSelect (size="md", placeholder for "all") · MpTable (plain MpTableHead) · MpBadge (information = bot, completed = human) · toast.notify (top-center, 3000ms).

Mekari taste conventions applied: form left-aligned ≤6 columns, no card wrapper (divider + H3); rows not cards in the comparison; brand color reserved for the primary action, active nav, and the single ⭐ resolution-parity KPI; accountability timestamp on the comparison.

Open design questions for the designer:

• D1 — Does Traffic Split sit inside the existing channel chatbot-settings screen, or warrant its own tab? (proposed: inline section)
• D2 — Recommended starting-% nudge / guard-rail copy (ties to §15 Open Q5).
• D3 — Comparison-view CSAT cell when the channel doesn't collect CSAT → "CSAT not available" treatment (ties to §15 Risk #3).
• D4 — Exact parity-formula display (ratio vs delta vs both) in the summary strip and Δ column.

7. API & Webhook Behavior

Plain-language behavior. HTTP methods, paths, and JSON schemas are resolved in the RFC.

#	Behavior	Entity Affected	Triggered By	Expected Behavior	Failure Behavior
1	Save traffic-split config	Channel's bot config (Path: traffic_split_enabled, bot_percent)	Admin clicks Save in the Traffic Split section	Validate bot_percent is an integer 0–100 and the flag is ON for the org; persist on the Path; return the saved value to the UI	Validation fails → 422 with field error, nothing saved. Flag OFF for org → 403, section should not have been shown. Persist fails → 5xx, UI keeps prior value and shows Retry
2	Decide arm for an incoming chat	New conversation/room + its variant tag	A new incoming message creates/opens a conversation on the channel (find_default_path in process_incoming_message_with_resolve)	If split is enabled: roll rand(100). If >= bot_percent → human arm (set is_auto_assign_agent, clear bot intent_id); else → bot arm (existing bot flow). Stamp the conversation with variant = bot | human. If split disabled → 100% bot (unchanged)	Config read fails / ambiguous → fail safe to the bot arm (current default behavior) and log, so no chat is dropped. Decision must be made exactly once per conversation (idempotent on the conversation)
3	Hand human-arm chat to an agent	Conversation assignment / human queue	Arm decision = human (behavior #2)	Reuse the existing SendMessageAutoAssignAgentWorker path: assign to an available agent, or enter the existing human queue / channel offline behavior if none available	No change to existing assignment failure handling — chat waits in queue; it is NOT re-routed to the bot
4	Emit split-assignment analytics	Analytics event stream	Arm decision made (behavior #2)	Fire bot_traffic_split_assigned with { channel_integration_id, variant, bot_percent, conversation_id, timestamp } for the comparison dashboard	Analytics emit is best-effort and must never block or fail the chat routing

8. System Flow + User Stories + ACs

8.1 System Flow

Flow: Per-Conversation Bot-vs-Human Split
Type: User Journey + System Decision

1.  Chatbot Admin opens the channel's chatbot settings, enables Traffic Split, sets bot % (e.g. 30), saves.
2.  System validates 0–100 + flag ON, persists bot_percent on the channel Path.
3.  A real customer sends a first message → a new conversation opens on that channel.
4.  System reaches the routing decision (find_default_path) and reads the channel's split config.
5.  Decision point — split enabled?
      • No  → 100% bot (current behavior). Stop.
      • Yes → roll rand(100).
6.  Decision point — rand(100) >= bot_percent?
      • No  → BOT ARM: existing bot flow runs (intent_id). Stamp conversation variant = bot.
      • Yes → HUMAN ARM: set is_auto_assign_agent, clear bot intent. Stamp conversation variant = human.
7.  Human arm → SendMessageAutoAssignAgentWorker assigns an available agent.
8.  Failure branch — human arm but no agent available → chat enters existing human queue / channel
      offline behavior. It is NOT handed back to the bot.
9.  System emits bot_traffic_split_assigned analytics event with the variant.
10. Admin later opens the comparison view and reads resolution / CSAT / handover per arm to decide
      whether to raise the bot %.

📊 System Flow — Bot-vs-Human Traffic Split

graph TD
    A[New incoming message → conversation opens] --> B[find_default_path reads channel split config]
    B --> C{Split enabled?}
    C -- No --> D[Bot arm: existing intent flow<br/>variant = bot]
    C -- Yes --> E[Roll rand 0-99]
    E --> F{roll >= bot_percent?}
    F -- No --> D
    F -- Yes --> G[Human arm: is_auto_assign_agent<br/>clear bot intent, variant = human]
    G --> H{Agent available?}
    H -- Yes --> I[SendMessageAutoAssignAgentWorker assigns agent]
    H -- No --> J[Existing human queue / offline behavior<br/>bot does NOT take over]
    B -. config read fails .-> K[Fail safe to bot arm<br/>log decision_fallback]
    D --> L[Emit bot_traffic_split_assigned]
    G --> L
    K --> L

8.2 User Stories

User Story	Importance	Mockup / Technical Notes	Acceptance Criteria
[BTS-S01] — Configure the bot traffic split for a channel As a Chatbot Admin, I want to set what percentage of incoming chats my bot handles on a channel, so that I can expose the bot to a controlled slice of real traffic.	Must Have	Figma: Pending — see Stitch prompt Data Fields: • channel_integration_id (int, required) — URL/route param • traffic_split_enabled (boolean, required) — User input (toggle) • bot_percent (int 0–100, required when enabled) — User input Before-After Behavior: Before, a channel's bot is implicitly 100% with no control; after, the Admin sees a Traffic Split section and can persist a 0–100 bot percentage per channel.	— Happy Path — • AC-1: Given the bot_traffic_split_enabled flag is ON for the org and I am a Chatbot Admin, when I enable Traffic Split and set bot % to 30 and Save, then the value persists on that channel's Path and a confirmation toast "Traffic split updated: 30% bot / 70% human" is shown. • AC-2: Given I set the bot % to 100, when I Save, then the channel behaves exactly as today (100% bot) and no human-arm assignment occurs. • AC-3: Given I set the bot % to 0, when I Save, then every new chat on the channel is assigned to the human arm. — Error / Unhappy Path — • ERR-1: Given the Traffic Split section, when I enter a value that is not an integer 0–100 (e.g. 150, -5, "30%"), then Save is blocked with inline error "Enter a whole number between 0 and 100" and nothing is persisted. • ERR-2: Given a valid value, when Save fails (network/5xx), then the prior value is kept, an error "Couldn't save traffic split. Try again." + Retry is shown, and event bot_traffic_split_save_failed is logged. — Permission Model — • CAN: Chatbot Admin / Admin (configure). • CANNOT: Human Agent, SPV (view-only), end-customer. • Unauthorized: Traffic Split section not rendered; direct save attempt returns 403. — UI States — • Loading: Save disabled + spinner. • Empty: Split OFF by default — "All incoming chats are handled by the bot (100%)." • Error: Inline validation / save-failed message. • Success: Toggle ON, value saved, confirmation toast.
[BTS-S02] — Route an incoming chat by the configured split and tag its arm As a Chatbot Admin, I want each new incoming chat to be sent to either the bot or a human according to my percentage, and recorded as such, so that the bot is tested on real traffic and I can compare the two arms later.	Must Have	Figma: N/A — backend routing decision Data Fields: • bot_percent (int, required) — Path config (from BTS-S01) • traffic_split_enabled (boolean, required) — Path config • variant (enum: bot | human, required) — System-set on the conversation/room Before-After Behavior: Before, find_default_path always routes to the bot; after, when split is enabled it rolls a per-conversation dice and either keeps the bot arm or switches to is_auto_assign_agent, stamping the conversation with its variant. (Business rule per S5 §Determinism: rand(100) >= bot_percent → human arm.)	— Happy Path — • AC-1: Given split is enabled with bot % = 30, when a new conversation starts and the roll is < 30, then the bot handles the conversation (existing intent_id flow) and the conversation is tagged variant = bot. • AC-2: Given split is enabled with bot % = 30, when a new conversation starts and the roll is ≥ 30, then the conversation is assigned to the human arm (is_auto_assign_agent, bot intent not run) and tagged variant = human. • AC-3: Given a conversation has already been assigned an arm, when subsequent messages arrive in that same conversation, then the arm is NOT re-rolled — the original variant persists for the life of the conversation. • AC-4: Given split is disabled on the channel, when a new conversation starts, then the bot handles it (100%) exactly as today and no variant human assignment occurs. — Error / Unhappy Path — • ERR-1: Given the split config cannot be read or is ambiguous, when the routing decision runs, then the system fails safe to the bot arm (current default), logs bot_traffic_split_decision_fallback, and no chat is dropped. — Permission Model — • CAN: System (automatic). No human triggers this directly. • CANNOT: The assigned arm is decided once and CANNOT be changed for the life of the conversation — it is not user-reversible by design (a normal bot→human escalation is a handover, not a re-roll). No role (Admin, Agent, customer) can flip a conversation's variant after assignment. • Unauthorized: N/A — not user-invokable. — UI States — • Loading / Empty / Error / Success: N/A — backend decision, no UI surface for the customer beyond receiving the bot or agent reply.
[BTS-S03] — Human-arm chat with no agent available waits in queue As a Chatbot Admin, I want a human-arm chat to follow the normal human queue when no agent is available, instead of bouncing back to the bot, so that my bot-vs-human comparison stays clean.	Must Have	Figma: N/A — reuses existing queue/offline behavior Data Fields: • variant (enum, required) — from BTS-S02 • agent_availability (system, required) — existing assignment logic Before-After Behavior: Before, only bot-escalated chats hit the human queue; after, deliberately human-arm chats use the same SendMessageAutoAssignAgentWorker queue/offline path — with no new "bot rescues the queue" branch (per S4 Non-Goal 5).	— Happy Path — • AC-1: Given a conversation is tagged variant = human and at least one agent is available, when assignment runs, then the chat is assigned to an available agent via the existing SendMessageAutoAssignAgentWorker. • AC-2: Given a conversation is tagged variant = human and no agent is available (off-hours / all at capacity), when assignment runs, then the chat enters the existing human queue / channel offline behavior and remains tagged variant = human. — Error / Unhappy Path — • ERR-1: Given a human-arm chat is waiting in queue, when no agent picks it up, then the bot does NOT take it over and the conversation is never re-tagged to variant = bot. — Permission Model — • CAN: System (automatic). • CANNOT: N/A. • Unauthorized: N/A. — UI States — • Loading / Empty / Error / Success: N/A — backend; customer sees the channel's existing offline/queue experience.
[BTS-S04] — Compare bot vs human outcomes per arm As a Chatbot Admin, I want to see resolution, CSAT, and handover rate split by arm, so that I can decide whether the bot is good enough to raise its percentage.	Should Have	Figma: Pending — see Stitch prompt Data Fields: • variant (enum, required) — from BTS-S02 • resolution_status (existing conversation metric) • csat_score (existing, where collected) • handover_flag (derived — bot-arm chats later escalated to human) Before-After Behavior: Before, conversation metrics are not segmentable by experiment arm; after, the Admin can view resolution / CSAT / handover filtered by variant = bot vs human for the channel and date range.	— Happy Path — • AC-1: Given conversations tagged with a variant, when I open the comparison view for the channel and a date range, then I see resolution rate, CSAT, and handover rate for the bot arm and the human arm side by side. • AC-2: Given the bot arm has zero conversations in the range (e.g. bot % was 0), when I open the view, then the bot column shows an explicit "No data yet" state rather than 0%/blank that reads as a real result. — Error / Unhappy Path — • ERR-1: Given the metrics query fails, when I open the view, then "Couldn't load comparison. Try again." + Retry is shown and no partial/misleading numbers are rendered. — Permission Model — • CAN: Chatbot Admin / Admin / SPV (view). • CANNOT: Human Agent, end-customer. • Unauthorized: Comparison view not rendered. — UI States — • Loading: Skeleton rows while fetching. • Empty: "No conversations in this range yet." • Error: Retry message. • Success: Two-arm comparison table/chart.
[BTS-S01-NEG] — Split control hidden when feature flag is OFF (Guard Rail — from Non-Goal: per-segment/global) As a Chatbot Admin in an org without the flag, when I open chatbot settings, then I do not see or have access to the Traffic Split control.	Guard Rail	—	• NEG-1: Given bot_traffic_split_enabled is OFF for my org, when I open the channel's chatbot settings, then the Traffic Split section is not rendered and any direct save call returns 403.
[BTS-S02-NEG] — No mid-conversation re-bucketing (Guard Rail — from Non-Goal 2) As a returning customer in an active conversation, when I send more messages, then my conversation is not re-rolled into a different arm.	Guard Rail	—	• NEG-1: Given a conversation already tagged variant = bot, when more messages arrive within that same conversation, then no new dice roll occurs and the variant remains bot (and symmetrically for human).
[BTS-S01-NEG2] — Ineligible plan cannot use Traffic Split (Guard Rail — enforces S5 §Plan scope) As an Admin on a plan without Traffic Split eligibility, when I try to access or save the split, then the system refuses.	Guard Rail	—	• NEG-1: Given my account is not on an eligible plan (per S5 §Plan scope), when I open the channel's chatbot settings, then the Traffic Split section is not rendered. • NEG-2: Given an ineligible account, when a save request for bot_percent is sent directly (e.g. crafted API call), then it is rejected with 403 and no value is persisted.

Dependencies: BTS-S02 depends on BTS-S01 (config must exist). BTS-S03 depends on BTS-S02 (arm must be assigned). BTS-S04 depends on BTS-S02 (variant tag must exist).

9. Rollout

Feature flag:    bot_traffic_split_enabled (see Section 5 — OFF by default)
Rollout:         Stage 1 → Internal QA org(s) on a Telegram test channel (low-stakes) to verify
                          the split mechanic and variant tagging end-to-end.
                 Stage 2 → 2–3 friendly design-partner accounts, enabled manually, running a real
                          experiment on one production channel each (e.g. WhatsApp) at a low bot %.
                 Stage 3 → Professional + Enterprise accounts with the chatbot module, on request.
                 GA       → Self-serve for all eligible plans (toggle in chatbot settings).
Backward compat: Yes — split defaults to disabled; existing channels keep 100% bot behavior untouched.
Migration:       None — additive config field + new variant tag. No existing data is migrated.

10. Observability

Key Events:

Event Name	Trigger	Properties
bot_traffic_split_config_saved	Admin saves the split config	channel_integration_id, bot_percent, enabled, actor_id, timestamp
bot_traffic_split_assigned	Arm decided for a new conversation	channel_integration_id, conversation_id, variant, bot_percent, timestamp
bot_traffic_split_decision_fallback	Config unreadable → failed safe to bot arm	channel_integration_id, conversation_id, reason, timestamp
bot_traffic_split_save_failed	Config save errored	channel_integration_id, actor_id, error_code, timestamp
bot_arm_handover_to_human	A bot-arm conversation later escalates to a human	channel_integration_id, conversation_id, timestamp

Dashboard owner: Chatbot Squad (squad: Chatbot)

Alerts:
- bot_traffic_split_decision_fallback rate > 1% of assignments in 1h → Slack: #chatbot-alerts
  (signals the config read is failing and traffic is silently defaulting to 100% bot)
- Human-arm queue wait p90 > 15 min during configured experiment hours → Slack: #chatbot-alerts
  (signals the human arm is under-staffed and CX is degrading)

10.1 Post-Launch Monitoring Cadence

Review cadence:    Weekly for the first 4 weeks post-GA, then monthly.
Owner:             Chatbot Squad PM (Dimas Fauzi Hidayat).
Review scope:      bot_traffic_split_assigned (volume + actual vs configured ratio),
                   bot_traffic_split_decision_fallback rate, bot_arm_handover_to_human,
                   plus the Section 11 comparison metrics per arm.
Trigger thresholds:
  - Observed bot share deviates > 10 percentage points from configured bot_percent over a week
    → investigate the bucketing logic.
  - bot_traffic_split_decision_fallback > 1% sustained for 2 consecutive days → escalate to Eng.
Rollback consideration:
  If fallback rate cannot be brought under 1% within 24h, PM disables bot_traffic_split_enabled
  for affected orgs (reverting to standard 100% bot) pending root cause.

11. Success Metrics

The KPIs are deliberately comparative — bot arm vs human arm on the same channel and audience — because the entire purpose is to prove the bot is good enough to expand. Targets are framed as parity (bot within a tolerance of human), not absolute numbers, since the human arm is the baseline the experiment generates.

Quality & Accuracy (the proof):

⭐ Primary KPI: Resolution / containment parity
   Definition:  Bot-arm resolution rate ÷ human-arm resolution rate, on the same channel and
                date range. "Resolved" uses the existing conversation resolution definition.
   Baseline:    N/A — no controlled bot-vs-human comparison exists today (this feature creates it).
   Target:      Bot-arm resolution ≥ 90% of human-arm resolution sustained over a 2-week run
                → signal the bot is "good enough" to raise the percentage.

- CSAT parity
   Definition:  Average CSAT of bot-arm conversations vs human-arm conversations (where CSAT is collected).
   Baseline:    N/A — not previously segmentable by arm.
   Target:      Bot-arm CSAT within 0.5 points (5-pt scale) of human-arm CSAT over the run.

- Handover rate (bot arm)
   Definition:  % of bot-arm conversations that get escalated to a human anyway.
   Baseline:    N/A — captured for the first time via bot_arm_handover_to_human.
   Target:      ≤ 30% handover on the bot arm (i.e. the bot genuinely deflects the majority it is given).

Adoption & Usage (did the tool work):

- Experiment adoption
   Definition:  # of accounts that enable Traffic Split on ≥ 1 production channel and run it ≥ 2 weeks.
   Baseline:    0 (new feature).
   Target:      ≥ 10 accounts running a real experiment within 60 days of GA.

- Routing fidelity
   Definition:  |observed bot share − configured bot_percent| averaged across active experiments.
   Baseline:    N/A.
   Target:      Within 5 percentage points (the random split lands where the Admin set it).

12. Launch Plan & Stage Gates

Stage	Audience	Duration	Success Gate to Advance	Owner
Internal Alpha	Internal QA org, Telegram test channel	1–2 weeks	Variant tagging correct on 100% of test conversations; routing fidelity within 5pp; 0 P0/P1 bugs; decision_fallback ≈ 0	PM + QA
Closed Beta	2–3 design-partner accounts, 1 production channel each	3–4 weeks	≥ 2 partners complete a 2-week run; routing fidelity within 5pp in production; no unhandled human-arm dead-air incidents attributable to the feature	PM + CSM
Open Beta	Professional + Enterprise w/ chatbot, on request	4 weeks	decision_fallback < 1% sustained; comparison view used by ≥ 5 accounts; no P0/P1 for 2 weeks	Eng Lead
GA	All eligible plans (self-serve toggle)	Ongoing	All Open Beta gates sustained 2 weeks; PMM launch approved	PM + PMM

13. Dependencies

Dependency	Owning Team	Deliverable Needed	Blocking?
Incoming-message routing (process_incoming_message_with_resolve / find_default_path)	Chatbot	Hook point for the split decision (already exists)	NO — already present
Human auto-assignment (SendMessageAutoAssignAgentWorker)	Chatbot	Reused as-is for the human arm + existing queue/offline behavior	NO — already present
Chatbot settings UI (chatbot-fe)	Chatbot	New Traffic Split section + save endpoint	YES — config UI is in Phase 1 scope
Conversation resolution metric	Chatbot / Data	A consistent "resolved" definition segmentable by variant for the ⭐ KPI	YES — primary KPI cannot be computed without it
CSAT collection	Chatbot / Data	CSAT data available and joinable to variant (note: not all channels collect CSAT)	NO — CSAT parity is a secondary metric; degrade gracefully if absent
Product analytics pipeline	Data	Ingest the new bot_traffic_split_* events for the comparison view	YES — comparison view (BTS-S04) depends on it

14. Key Decisions + Alternatives Rejected

14a — Decisions Made

Date	Decision	Rationale
2026-06-16	Phase 1 splits per conversation (random per new chat), not per customer	Sticky per-customer needs deterministic identity hashing across channels; deferring it keeps Phase 1 small while still delivering real-traffic validation. Sticky is Phase 2.
2026-06-16	When the human arm has no agent available, the chat queues / follows existing offline behavior — the bot does not rescue it	Keeps the bot-vs-human comparison clean; a bot fallback would contaminate the "human" arm. Off-hours wait is mitigated by running experiments during staffed hours.
2026-06-16	Percentage is configured per channel (Path), not per account	Routing already happens at the Path; per-channel lets the Admin isolate the experiment to one channel (e.g. one WhatsApp number) without affecting others.
2026-06-16	Self-serve UI in chatbot settings is in Phase 1 (not backend-only)	The primary persona is a Chatbot Admin who must configure the split themselves to build their own confidence.
2026-06-16	On any config-read error, the router fails safe to the bot arm (current default)	Never drop or delay a chat because of the experiment; a failed read should behave like today.

14b — Alternatives Rejected

Alternative	Why Rejected	Date
Bot takes over human-arm chats when no agent is available	Contaminates the experiment — some "human" outcomes would actually be bot-handled, breaking the parity comparison	2026-06-16
Sticky per-unique-customer bucketing in Phase 1	Requires reliable cross-channel identity resolution + deterministic hashing; too much scope for the first validation. Deferred to Phase 2	2026-06-16
Single global account-level percentage	Too blunt — can't isolate the experiment to one channel or compare channels; Admin couldn't protect high-stakes channels	2026-06-16
Backend/internal-only config (no UI)	The persona is a self-serve Chatbot Admin building their own confidence; an internal-only toggle wouldn't serve them	2026-06-16
Keep using Bot Preview / a Telegram test channel as the "test"	Synthetic — the explicit user need is confidence from real customers on the real channel, which these cannot provide	2026-06-16

15. Open Questions

#	Type	Question	Owner	Deadline
1	Open Question	Which Qontak plans/tiers get Traffic Split (proposed: Professional + Enterprise with the chatbot module)?	PM + Commercial	2026-07-15
2	Open Question	What is the exact "resolved" definition reused for the ⭐ resolution-parity KPI, and is it already segmentable by conversation?	PM + Data	2026-07-15
3	Risk	CSAT is not collected on every channel, so CSAT-parity may be unmeasurable for some experiments. Mitigation: treat CSAT as a secondary, best-effort metric; the comparison view shows "CSAT not available on this channel" and the go/expand decision can rest on resolution + handover alone.	PM + Data	2026-07-22
4	Risk	Concurrency — if two messages for a brand-new conversation arrive almost simultaneously, the arm could be decided twice. Mitigation: make the arm decision idempotent per conversation (decide-once, persist variant atomically on first assignment; later reads use the stored value, per BTS-S02 AC-3).	Eng	2026-07-22
5	Assumption	A reasonable default/guardrail for the starting bot % (e.g. recommend ≤ 30% for a first run) helps Admins not over-expose. Assumed advisory-only helper text, not an enforced cap. Confirm.	PM	2026-07-15

PRD CHANGELOG

Version	Date	By	Section	Type	Summary
1.0	2026-06-16	Claude	All	CREATED	Initial NEW PRD generated from coaching session — per-conversation bot-vs-human split, queue-&-wait no-agent behavior, per-Path self-serve config, resolution/CSAT/handover parity KPIs
1.1	2026-06-16	Claude	S8	MODIFIED	Post-score fixes: added BTS-S01-NEG2 (plan-tier enforcement AC for S5 §Plan scope) and made BTS-S02 arm immutability explicit in its Permission Model (closes Layer 2.5 reversibility gap)
1.2	2026-06-16	Claude	S6	ADDED	Added §6.1 Design Draft (2 wireframes + plan) from designer-repo mekari-taste imagination phase, marked as collaboration artifact pending designer review/validation; updated Figma header field