Qontak | AI Agent | Testing — Phase 1: Historical Validation

Historical Validation — Phase 1 PRD under the AI Agent: Testing ANCHOR. The first test-case source ("Generate from inbox"): sample resolved human-handled conversations, generate AI shadow answers, compare side-by-side vs the human "golden" answer, rate per question, and roll up a confidence score. Imported from Confluence and reconciled against code (chatbot, chatbot-fe, qontak-designer).

HEADER BLOCK

Field	Value
PM	Dimas Fauzi Hidayat
PRD Version	1.2
Status	IN-REVIEW
PRD Type	PHASE
Epic	BOT-3351
Squad	BOT
RFC Link	AI Agent Testing: Generate from Inbox (Conversation History) (Confluence — BE technical design, Eko Aprianto) · repo RFC: `../rfcs/historical-validation.md` (review: PROCEED 8.5)
Figma Master	N/A — design source is the `qontak-designer` prototype (`feat/ai-agents-testing`): `app/pages/bot-automation/testing/{index,[id]}.vue` + `app/components/bot-automation/testing/*` (see §8).
Anchor	AI Agent: Testing — ANCHOR (Confluence)
Labels	`epic:qontak-chatbot` \| `module:ai-agent` \| `feature:ai-agent-testing`
Last Updated	2026-07-20

Scope Changes

Backend · Frontend · Data — new Testing page + test-case endpoints (chatbot, chatbot-fe), and the sampling/shadow-generation pipeline (Data).

2. Phase Context

Anchor PRD: AI Agent: Testing — ANCHOR
Phase Number: Phase 1 of 3 (phasing by test-case source: Historical → Knowledge → Imported)
Phase Goal: Validate AI Agent quality against a sample of historical, resolved human conversations so SPV/Admin can confidently go live. (Matches the ANCHOR Phase Index Goal for Phase 1.)
Prior phases: N/A — this is Phase 1, no prior phases.
This phase: The Testing page + the "Generate from inbox" test-case source — sample resolved human-handled rooms, generate AI shadow answers, compare side-by-side vs the human "golden" answer, rate per question, roll up a confidence score.
Deferred to next: Phase 2 "Generate from knowledge" and Phase 3 "Imported question list" test-case sources (scaffolded in design/code but out of scope here).
Cross-phase deps: Test cases bind to a specific AI Agent version (ai_agent_history_id). The ai_agent_test_cases / ai_agent_test_case_questions schema established here must remain stable for Phase 2/3, which reuse the same tables and comparison UI.

3. One-liner + Problem

One-liner: Let Qontak SPVs and Admins validate an AI Agent against a sample of their own resolved conversations — comparing AI answers to their human agents' — before going live.

Problem: Before activating an AI Agent, SPV/Admins have no way to preview its quality at scale, so they run ~6-hour/day manual "War Rooms" during onboarding to catch errors. This phase gives them a self-serve, evidence-based comparison against historical "golden" human answers, so they can reach a confidence bar and activate without babysitting. For full initiative context, see the ANCHOR PRD.

4. Target Users + Persona Context

Persona	Role	Goal	Pain	Workaround
Primary — SPV / Chatbot Admin	SPV / Chatbot Admin responsible for customer-interaction quality	Reach a confidence bar that proves the AI is safe to launch, then activate it	No scalable way to preview AI answers on real customer questions before go-live	~6 hours/day of manual War Room monitoring during onboarding
Secondary — Super Admin	Decision-maker who purchased the AI module	Activate the subscription fully once the SPV signs off	No objective proof to justify activation	Waits on the SPV "green light"; keeps AI inactive

5. Non-Goals

Live Shadow Mode — not a real-time system running in parallel during actual customer chats. This phase is strictly historical.
Model fine-tuning UI — answers are fixed by updating the Knowledge Base/Context, not by "teaching" the AI in this workspace.
Multi-modal validation — MVP is text-only; tickets with images/voice/attachments are excluded.
Knowledge-based and imported test sets — deferred to Phase 2 and Phase 3.
Editing the AI's answer in the workspace — the comparison view is read-only.
Mobile app — Testing is web-only in this phase.

6. Constraints

Platform: Web only (Qontak web app — Bot Automation → Testing).
Performance: Batch generation is asynchronous (~2–5 min for ~50 items). Must respect the LLM provider's TPM/RPM so it never blocks live production traffic. Historical room fetch should read from a replica DB where available, to avoid slowing the live inbox.
Data limits: Lookback = last 90 days. Sample = 10% of eligible rooms, capped at 50–70 rooms per batch (≤50 shown when a batch exceeds 100). Text-only tickets.
Plan scope: All plans with the AI Agent enabled.
Feature flag: ai_agent_testing | default: OFF. Enabled per organization during beta.
Read/write: Roles owner / supervisor / admin can read + write (generate, rate). Standard agents have no access (menu hidden). Enforced server-side via set_role on every test-case endpoint.
Data lifecycle: ai_agent_test_cases and ai_agent_test_case_questions are soft-deleted (deleted_at, acts_as_paranoid) when a user deletes a test case; the transient LLM inference payload is not persisted beyond the response. Hard-purge window TBD (Open Question #8).

7. Feature Changes

CHG-001 — Confidence score surfaced in the Tree Diagram

Change Type: Modified component (Bot Flow tree diagram).
Page: /bot-automation/bot-flow/{id} (Tree Diagram).
Before: Adding/selecting an AI Agent node shows the agent with no quality signal.
After: When a tested AI Agent is selected, the node shows its average Confidence Score from testing (average across that agent's completed test cases).

Element	Before	After
AI Agent node (Tree Diagram)	No confidence indicator	Shows average Confidence Score from testing per AI Agent

Backend touchpoint: get_tree_diagram_v3 (chatbot). JIRA: BOT-3976. Design source: qontak-designer bot-flow prototype (⚠ target — the confidence badge is not yet rendered there; see AITEST-S10).

8. New Features

Feature: AI Agent Testing page

URL: /bot-automation/testing — top-level item under Bot Automation (AI agents · Resources · Actions · Testing · Analytics · Bot flow). Test-case detail opens from a row.
Access: owner / supervisor / admin only. Standard agents: menu hidden.

Component tree: (reconciled against the qontak-designer prototype 2026-07-10 — deltas from the original draft flagged [drift])

index.vue — Testing index (/bot-automation/testing)
- Page header ("AI testing" h1) → "Generate test case" button → GenerateTestCaseModal
- Filter toolbar — type popover ("All types") + search input
- Test-cases MpTable — columns: Test case name · AI agent name · Testing type · Score · Status · Last updated · Actions. [drift] the prototype carries an AI agent name column (not in the original draft), Score has a tooltip + sort, and Status is a "Passed"/"Need review" MpBadge.
- GenerateTestCaseModal — full-screen source picker (three option cards):
  - "Generate from Inbox" → GenerateFromInboxDrawer (this phase; disabled "not enough conversation" state when <50 past conversations)
  - "Generate from Resources" → GenerateFromKnowledgeDrawer (Phase 2)
  - "Import questions" → UploadManuallyDrawer (Phase 3)
  - [drift] labels are "Generate from Resources" / "Import questions" (not "Generate from knowledge" / "Imported question list"), and the prototype currently renders all three cards enabled — Phase-1 build must hide/disable the Resources + Import cards to honour Non-Goal #4 / NEG-4.
[id].vue — Test-case detail (per test case) [drift] async progress is inline on the detail page as phases, not a separate TestCaseGeneratingModal:
- Test-case header — title + source meta + result badge (score % + Passed/Need review)
- Phase generating — progress bar ("n of 50") + warning banner + revealing question cards + skeletons
- Phase organizing — centered MpLoader "Organizing questions into topics"
- Phase complete — "Show unreviewed questions only" toggle + collapsible topic groups (x/y badge) → expandable rows → QuestionComparisonCard (Human vs AI side-by-side + metrics + thumbs)
- Dislike → "Add sources to improve response" improve card → AddKnowledgeDrawer

UI States (index page):

Empty: /testing-empty.png illustration + "No test cases yet" + helper; primary action = Generate test case. No table.
Loading: skeleton rows while the list is fetched.
Error: blank slate "Couldn't load test cases" + Retry. Log: ai_workspace_load_failed.
Search not found / Filter not found: /not-found.png blank slate + recheck-keywords/recheck-filters copy. [drift] two distinct not-found states exist in the prototype beyond the original Empty/Loading/Error/Success set.
Success: table of test cases with agent, type, score, status, last updated, row actions.

📊 UI state diagram — Testing index (index.vue)

stateDiagram-v2
    [*] --> Loading
    Loading --> Empty: no test cases
    Loading --> Success: test cases returned
    Loading --> Error: fetch failed (log ai_workspace_load_failed)
    Error --> Loading: Retry
    Success --> SearchNotFound: search yields nothing
    Success --> FilterNotFound: filter yields nothing
    SearchNotFound --> Success: clear search
    FilterNotFound --> Success: clear filter
    Empty --> Loading: Generate test case

📊 UI state diagram — Test-case detail ([id].vue) phase machine

stateDiagram-v2
    [*] --> Generating: Generate clicked (toast: test case saved)
    Generating --> Organizing: all 50 questions generated
    Organizing --> Complete: questions grouped into topics
    Generating --> Complete: reopen an already-completed test case
    Complete --> [*]: close

Design source: qontak-designer app/pages/bot-automation/testing/{index,[id]}.vue + app/components/bot-automation/testing/* (GenerateTestCaseModal, GenerateFromInboxDrawer, GenerateFromKnowledgeDrawer, UploadManuallyDrawer, QuestionComparisonCard).

9. API & Webhook Behavior

Base path: /api/frontend_service/v1/ai_agent. All endpoints gated server-side to owner / supervisor / admin via set_role. Technical fields (JSON schemas, error codes) resolved during RFC.

#	Behavior	Entity Affected	Triggered By	Expected Behavior	Failure Behavior
1	Create test case (triggers batch)	New `ai_agent_test_cases` row (bound to `ai_agent_id` + `ai_agent_history_id`)	SPV/Admin clicks Generate in the Generate-from-Inbox drawer (body: `type`, `version_id`)	Test case created (status `processing`); Sidekiq `FetchRoomConversationsWorker` (queue `:ai_agent`) enqueued to fetch assigned rooms, extract Q/A pairs, and (target) generate AI shadow answers; UI shows progress	Version not found → 404, no test case. Create fails → 422. Worker error on a room → skip + Rollbar; batch continues
2	List / poll test cases	Read `ai_agent_test_cases` (paginated)	Opening Testing page; polling during a batch	Returns test cases with `status` + `confidence_score`; UI polls `status` until `completed`	Unauthorized role → forbidden (menu should not have been visible)
3	Get test-case detail	Read one `ai_agent_test_cases` + its `ai_agent_test_case_questions`	SPV/Admin opens a test case	Returns each question: `topic`, `question`, `answer` (AI), `parameters.human_answer` (golden), `response_time`, `confidence`, `sources`, `score`, `status`	Test case not found → 404
4	Rate a question	Update one `ai_agent_test_case_questions` row	SPV/Admin clicks thumbs up/down (`score` 0/1 + scored_by metadata)	Saves `score` + `is_score` + `scored_by`/`scored_at`; (target) recomputes the test case aggregate `confidence_score`	Question not found → 404. Invalid score (not 0/1) → 422

Implementation note: today RateTestCaseQuestion only persists the per-question score; the aggregate confidence_score recompute is not yet wired (Open Question #2).

10. System Flow + User Stories + ACs

10.1 System Flow

SPV/Admin opens Bot Automation → Testing.
Clicks "Generate test case" → chooses "Generate from inbox".
Names the test case, selects the AI Agent version, clicks Generate.
System creates the test case (status processing) and enqueues FetchRoomConversationsWorker.
Worker fetches assigned rooms (last 90 days), extracts customer→agent question/answer pairs, (target) samples 10% capped at 50–70, and generates an AI shadow answer per question.
Decision point: if total eligible rooms < 10 → use all available rooms.
Failure branch: if a room fetch / LLM call fails → skip that room, log to Rollbar, continue.
On completion, status → completed; UI surfaces the test case with questions grouped by topic.
SPV/Admin opens a question → sees Human answer (left) vs AI answer (right) + confidence / response-time / sources.
SPV/Admin rates each answer thumbs up/down → (target) confidence meter updates.
When the meter reaches ≥80%, the agent is "Ready to Launch"; SPV/Admin proceeds to activate.

📊 System flow diagram (User Journey — includes the <10-room decision point and the fetch/LLM failure branch)

flowchart TD
    A[SPV/Admin opens Bot Automation → Testing] --> B[Generate test case → Generate from inbox]
    B --> C[Name test case, select AI Agent version, Generate]
    C --> D[Create test case status=processing + enqueue FetchRoomConversationsWorker]
    D --> E[Worker fetches assigned rooms last 90d, extracts Q/A pairs]
    E --> F{Eligible rooms < 10?}
    F -- Yes --> G[Use 100% of available rooms]
    F -- No --> H[Sample 10%, cap 50–70 target]
    G --> I[Generate AI shadow answer per question]
    H --> I
    I --> J{Room fetch / LLM call fails?}
    J -- Yes --> K[Skip room, log Rollbar, continue]
    J -- No --> L[Store answer + confidence + sources]
    K --> M[status=completed, group questions by topic]
    L --> M
    M --> N[SPV opens question: Human vs AI side-by-side]
    N --> O[Rate thumbs up/down → confidence meter updates]
    O --> P{Meter ≥ 80%?}
    P -- No --> N
    P -- Yes --> Q[Ready to Launch → activate]

10.2 User Stories

All 10 stories carry their original JIRA tickets. MoSCoW preserved from the source "Importance". Implementation-status notes flag where ACs describe target behavior not yet in code.

Each story closes with a two-column Technical Notes · Mockup table. Technical Notes = backend touchpoints, code refs, implementation status, and story dependencies (engineering-facing). Mockup is grounded against the qontak-designer prototype and details three things — Design spec (grounded UI/interaction), Component(s) (Pixel 3 / prototype components), and Page (route + prototype file). Where a story is server-side only, Mockup is marked N/A; where a screen is not yet built in the prototype, that is flagged explicitly.

AITEST-S01 — Workspace access control | Must Have

Story: As an SPV/Admin, I want to access the Testing page, so that I can validate AI performance before activation.

Before: No Testing page exists; there is no place to validate an AI Agent against history. After: A Bot Automation → Testing page is visible to owner/supervisor/admin only.

Data Fields: ai_agent_id (uuid, required — route), role (enum, required — auth session)

Happy Path:

AC-1: Given I am a Super Admin or Supervisor, when I open Bot Automation, then I see the "Testing" menu item and can open the Testing page.
AC-2: Given I am on the Testing page, when it loads, then I can create a new test case and open existing ones.

Error Path:

ERR-1: Given the test-case list fails to load, when the page renders, then I see a "Couldn't load" blank slate with Retry, and event ai_workspace_load_failed is logged.

Permission Model: CAN: owner/supervisor/admin. CANNOT: standard agents. Unauthorized: Testing menu not rendered; direct route forbidden.

UI States: Loading (skeleton rows), Empty ("No test cases yet" + Generate CTA), Error (blank slate + Retry), Success (test-case table).

Technical Notes: Route-level role guard (owner/supervisor/admin); Testing menu item not rendered for standard agents. List-fetch failure logs ai_workspace_load_failed. Dependencies: None.

Mockup (grounded in qontak-designer)

Design spec	Component(s) used	Page
Testing index — page header "AI testing" (h1) + primary "Generate test case" button (top-right). Below: filter toolbar (type popover "All types" 240px + search input) then a stage-bg card wrapping `MpTable`. Columns as built: Test case name · AI agent name · Testing type · Score · Status · Last updated · Actions (Score has an info tooltip + sort; Status is an `MpBadge` "Passed"/"Need review"; row-hover → `text.link`, row click → detail). States: Loading = 5 skeleton rows; Empty = `/testing-empty.png` + "No test cases yet" (h3) + helper; Search/Filter not-found = `/not-found.png` blank slate.	`MpFlex`, `MpText`, `MpButton`, `MpInputGroup`/`MpInput`, `MpPopover`, `MpTableContainer`/`MpTable`/`MpTableRow`/`MpTableCell`, `MpBadge`, `MpSkeleton`, `MpTooltip`, `DevToolsWidget`	`/bot-automation/testing` — `app/pages/bot-automation/testing/index.vue`

AITEST-S02 — Historical data sampling (10% rule) | Must Have

Story: As an SPV/Admin, I want the system to fetch a 10% sample of past resolved chats, so that I have a "golden standard" to test against.

Before: No mechanism to pull historical conversations into a test set. After: Generating a test case samples eligible resolved, human-handled rooms from the last 90 days.

Data Fields: lookback_days (int, fixed 90), sample_pct (int, 10), cap (int, 50–70)

Happy Path:

AC-1: Given 200 eligible human-handled rooms in the last 90 days, when I generate a validation set, then the system selects ~10% (≈20) at random.
AC-2: Given fewer than 10 eligible rooms, when I generate, then the system uses 100% of available rooms.
AC-3: Given 5,000 eligible rooms, when I generate, then the system caps the sample at 50–70 rooms (≤50 shown when the batch exceeds 100).

Error Path:

ERR-1: Given the room-list API is unavailable, when generation runs, then the test case surfaces an error state and can be retried.

Permission Model: CAN: owner/supervisor/admin. CANNOT: standard agents.

UI States: Loading (generating modal), Empty ("no eligible rooms found"), Error (retry), Success (sample loaded).

Technical Notes: Sampling is server-side. Implementation status: the worker currently fetches assigned rooms (90 days, paginated, LIMIT=100) and extracts Q/A pairs, but the 10% sampling + 50–70 cap are not yet implemented. Dependencies: AITEST-S03; Chat Service room APIs (§15).

Mockup (grounded in qontak-designer)

Design spec	Component(s) used	Page
No sampling params are exposed to the user (matches §16b rejected alt) — the only UI is the Generate from Inbox drawer: "Test case name" (required, 24-char counter, inline error) + "AI agent" required `MpSelect` in a popover, with a Cancel/Generate footer and a "Leave this process?" discard-confirm modal when dirty. Entry point is the full-screen Generate test case modal (3 source cards); the Inbox card exposes a disabled "not enough conversation" state (requires ≥50 past conversations). Sample size surfaces downstream as "n of 50" on the detail progress bar.	`MpDrawer*`, `MpFormControl`/`MpFormLabel`/`MpFormErrorMessage`, `MpInput`, `MpSelect`+`MpPopover`, `MpButtonGroup`, discard `MpModal`; entry `MpModal` option-card grid	drawer `app/components/bot-automation/testing/GenerateFromInboxDrawer.vue`, opened from `GenerateTestCaseModal.vue` on `/bot-automation/testing`

AITEST-S03 — Data integrity & filtering | Must Have

Story: As the System, I must filter out invalid or non-compatible chats, so that the test set is representative and safe.

Before: No filtering rules for what counts as a valid test conversation. After: Bot-only and non-text conversations are excluded from the sample.

Data Fields: participant_type (enum: customer/agent/system), message_type (enum, must be text)

Happy Path:

AC-1: Given a room resolved entirely by a bot with no human reply, when sampling runs, then that room is excluded.
AC-2: Given a room with at least one human-agent text reply to a customer question, when sampling runs, then it is eligible.
AC-3: Given a ticket containing an image, voice note, or attachment, when the sample is selected, then that ticket is skipped in favor of text-only inquiries.

Error Path:

ERR-1: Given a room returns zero eligible question/answer pairs, when extraction runs, then the room contributes no items and is not counted toward the sample.

Permission Model: System rule (runs under an owner/supervisor/admin-triggered batch).

UI States: N/A — server-side filtering; result reflected in the generated sample.

Technical Notes: ExtractConversationPairs pairs a customer question with the next agent text reply; status filter = assigned (human-handled) rooms; bot-only and non-text (image/voice/attachment) rooms excluded. Dependencies: AITEST-S02.

Mockup: N/A — server-side filtering, no UI surface. The result is reflected implicitly in the generated sample rendered on the detail page (app/pages/bot-automation/testing/[id].vue); there is no dedicated screen for this rule.

AITEST-S04 — Shadow mode execution (zero leakage) | Must Have

Story: As a PM, I want the AI to generate responses in shadow mode without messaging actual customers, so that testing is safe.

Before: The AI Agent only answers live customers. After: The AI generates answers to historical inquiries in shadow mode, never sent to customers.

Data Fields: answer (text), confidence (int), sources (jsonb), status (enum)

Happy Path:

AC-1: Given the AI generates a shadow response for a 3-month-old inquiry, when generation completes, then the send_message service is NOT triggered and no notification/email reaches the customer.
AC-2: Given a shadow response is generated, when it is stored, then the AI answer is saved on ai_agent_test_case_questions.answer and the human answer on parameters.human_answer — not in the live conversation log.

Error Path:

ERR-1: Given the LLM call fails for an inquiry, when generation runs, then that question is marked failed with a status_description and the batch continues.

Permission Model: CAN: system (triggered by owner/supervisor/admin). Unauthorized: not executed if flag OFF.

UI States: Loading (per-question generating), Empty (N/A), Error (per-question failed badge), Success (answer + confidence + sources shown).

Technical Notes: AI answer saved on ai_agent_test_case_questions.answer, human golden on parameters.human_answer; send_message is never triggered (zero leakage). Per-question LLM failure → status_description, batch continues. Dependencies: AITEST-S03; LLM/AI service (§15).

Mockup (grounded in qontak-designer)

Design spec	Component(s) used	Page
Generation is server-side; the visible surface is the detail page "generating" phase — collapsed question cards reveal one-by-one as shadow answers arrive (accordion, ratable as they appear) with `MpSkeleton` cards filling the remaining slots. A question that fails generation shows a "could not generate" state on the AI side of the comparison card rather than a blank panel.	`QuestionComparisonCard`, `MpCollapse`, `MpProgress`, `MpSkeleton`	`/bot-automation/testing/[id]` — `app/pages/bot-automation/testing/[id].vue`

AITEST-S05 — Side-by-side validation UI | Must Have

Story: As an SPV/Admin, I want to compare human vs AI responses side-by-side, so that I can judge accuracy effectively.

Before: No way to compare AI vs human answers. After: A side-by-side comparison per question, grouped by topic.

Data Fields: question (text), answer (text), parameters.human_answer (text), confidence (int), response_time (int), sources (jsonb), topic (string)

Happy Path:

AC-1: Given a completed batch, when I open a validation item, then I see the inquiry at the top, the human response (golden standard) on the left, and the AI response on the right.
AC-2: Given the AI response card, when it renders, then it shows Confidence, response time, and cited Sources.
AC-3: Given questions in the test case, when the list renders, then questions are grouped by topic.

Error Path:

ERR-1: Given a question failed shadow generation, when I open it, then the AI side shows a "could not generate" state instead of a blank panel.

Permission Model: CAN: owner/supervisor/admin. CANNOT: standard agents.

UI States: Loading (skeleton), Empty ("no questions in this test case"), Error (retry), Success (comparison rendered).

Technical Notes: Detail returns question, answer (AI), parameters.human_answer (golden), confidence, response_time, sources, topic. Schema stores confidence + sources only — no separate "relevance" field; if required it must live in parameters (Open Question #6). Dependencies: AITEST-S04.

Mockup (grounded in qontak-designer)

Design spec	Component(s) used	Page
The `QuestionComparisonCard` — two columns: Human agent (left, neutral card: employee avatar + "Human agent" + answer capped at 320px + "Response time" footer) vs AI agent (right, purple `#8270db` outer wrapping a highlight inner card with gradient border: "airene-brand" avatar + agent name + answer + "Sources:" pills, max 3 then "View more" popover + a purple footer bar showing Response time and Confidence % with an info tooltip). Questions sit in collapsible topic groups (each with an x/y reviewed badge) under a "Show unreviewed questions only" toggle; expanding a row opens the side-by-side card. When the source type is AI-only (Resources/Import), the human column is hidden.	`QuestionComparisonCard`, `MpCollapse`, `MpToggle`, `MpTooltip`, `MpPopover`, `MpIcon` (`airene-brand`), source pills	`/bot-automation/testing/[id]` — `[id].vue` + `QuestionComparisonCard.vue`

AITEST-S06 — Confidence meter & feedback | Must Have

Story: As an SPV/Admin, I want to rate AI responses, so that the system can calculate a confidence meter for launch readiness.

Before: No rating or roll-up of AI answer quality. After: Each answer can be rated thumbs up/down; ratings roll up into a confidence meter.

Data Fields: score (int 0/1, required), is_score (boolean), confidence_score (int — test-case aggregate)

Happy Path:

AC-1: Given I am viewing a comparison, when I click thumbs up, then the item is marked "Pass" (score = 1) and the meter increments.
AC-2: Given I previously marked an item thumbs down (score = 0), when I change it to thumbs up, then the aggregate confidence meter recalculates immediately.
AC-3: Given a test case with N rated items, when the meter renders, then it equals (thumbs-up ÷ total sample) × 100, with <80% = Low Confidence and ≥80% = Ready to Launch.

Error Path:

ERR-1: Given the rating save fails, when I click thumbs up/down, then the previous state is restored and an inline error is shown.

Permission Model: CAN: owner/supervisor/admin. CANNOT: standard agents.

UI States: Loading (saving rating), Empty (meter at 0% before any rating), Error (save failed inline), Success (meter updated).

Technical Notes: Rating persists the per-question score (0/1) + is_score/scored_by; the confidence_score aggregate recompute is not yet wired (Open Question #2). Threshold 80. Dependencies: AITEST-S05.

Mockup (grounded in qontak-designer)

Design spec	Component(s) used	Page
Under each AI answer, a feedback row "Was this response good and accurate?" with thumbs up (like) / down (dislike). Like → toast "Response rated", auto-collapse and scroll to the next unscored question. Dislike → an anchored "Add sources to improve response" improve card (Not now / Add source → opens `AddKnowledgeDrawer`). Roll-up is rendered as a result badge in the detail header (score % + a Passed ≥80 green / Need review <80 amber pill), and as the Score column + `MpBadge` on the index table — note the prototype shows a score/result badge, not a progress-"meter" bar.	feedback buttons + improve card in `QuestionComparisonCard`, `AddKnowledgeDrawer`, `toast`, header result badge, `MpBadge`	`[id].vue` (feedback + result badge) and `index.vue` (Score column)

AITEST-S07 — Activation gatekeeping | Should Have

Story: As a PM, I want to prevent "Go Live" until the AI reaches a safe quality threshold, so that low-quality agents don't go live.

Before: An AI Agent can be activated regardless of any test result. After: Activation is gated until the confidence meter reaches the ≥80% threshold.

Data Fields: confidence_score (int), threshold (int, default 80)

Happy Path:

AC-1: Given the confidence meter is <80% (Low Confidence), when I open the agent's main settings, then the "Activate Agent" button is disabled/greyed out.
AC-2: Given the confidence meter is ≥80%, when I open main settings, then "Activate Agent" is enabled.

Error Path:

ERR-1: Given I attempt to activate via API while <80%, when the request is made, then it is rejected with a clear reason.

Permission Model: CAN: owner/supervisor/admin. CANNOT: standard agents.

UI States: Disabled (locked) below threshold; Enabled at/above threshold.

Technical Notes: publish_ai_agent.rb does not check any confidence threshold today — no activation gate exists yet (Open Question #2). Should-Have for this phase. Dependencies: AITEST-S06.

Mockup: ⚠ Not yet in the qontak-designer prototype — no component or disabled-state exists in the testing prototype today, and the repo's agent-settings page does not render a confidence gate.

Design spec (target)	Component(s) used	Page (target)
Below-threshold → "Activate Agent" disabled/greyed with a reason; ≥80% → enabled.	— (to be built)	AI-agent main settings (to be built)

AITEST-S08 — Background processing (async) | Must Have

Story: As a user, I want large batches processed in the background, so that the UI doesn't freeze.

Before: No batch generation pipeline. After: Batches run in a background queue so the UI never blocks.

Data Fields: status (enum: processing/completed/failed), test_case_id (uuid)

Happy Path:

AC-1: Given I trigger a batch of ~50 items, when the request is sent, then the UI shows a progress/processing state and remains responsive.
AC-2: Given the batch is running, when I poll the test case, then its status reflects progress until completed.

Error Path:

ERR-1: Given a worker job errors, when it fails, then the failure is logged (Rollbar) and the test case surfaces an error/partial state rather than hanging.

Permission Model: CAN: owner/supervisor/admin.

UI States: Loading (TestCaseGeneratingModal / progress), Empty (N/A), Error (failed/partial badge), Success (completed).

Technical Notes: Implemented with Sidekiq (FetchRoomConversationsWorker, queue :ai_agent) — not Kafka. status = processing/completed/failed; worker error → Rollbar + partial/error state (never hangs). Dependencies: AITEST-S02, AITEST-S04.

Mockup (grounded in qontak-designer)

Design spec	Component(s) used	Page
On Generate, a success toast "Test case saved. Questions are being generated" fires and the user is routed to the detail page (a new index row appears with status "Need review"). Detail "generating" phase = linear `MpProgress` "Generating questions… n of 50" + a warning banner "Generating questions may take up to 30 minutes. You can leave this page…". When the count completes, the "organizing" phase shows a centered `MpLoader` + "Organizing questions into topics" before the complete view. UI stays responsive throughout (polling).	`MpProgress`, warning banner, `MpLoader`, `toast`, `MpBadge`	`[id].vue` (generating / loading / organizing phases)

AITEST-S09 — Manual override & audit | Could Have

Story: As an Admin, I want to force-activate the AI with a business justification, so that I'm not blocked when I have a valid reason.

Before: No override path if the gate blocks a justified activation. After: Admins can force-activate below threshold with a logged reason.

Data Fields: override_reason (text, required), actor_id (uuid), score_at_override (int)

Happy Path:

AC-1: Given the meter is below 80%, when I choose "Force Activate", then I must provide a reason before the action proceeds.
AC-2: Given I provide a reason and confirm, when force-activation completes, then the action is recorded in an audit trail (who, when, reason, score at time).

Error Path:

ERR-1: Given I submit force-activate without a reason, when I confirm, then the action is blocked with a validation message.

Permission Model: CAN: owner/admin. CANNOT: supervisor (override is admin-level). Unauthorized: option not shown.

UI States: Loading (submitting), Empty (N/A), Error (validation), Success (activated + audit entry).

Technical Notes: Depends on the (not-yet-built) activation gate. Could-Have for this phase. Dependencies: AITEST-S07.

Mockup: ⚠ Not yet in the qontak-designer prototype — the force-activate reason modal + audit trail has no component in the prototype today.

Design spec (target)	Component(s) used	Page (target)
Below-threshold "Force Activate" → required-reason modal (blocks on empty) → confirm records an audit entry (actor, timestamp, reason, score-at-override); owner/admin only (option hidden for supervisor).	— (to be built)	AI-agent main settings (to be built)

AITEST-S10 — Confidence score in Tree Diagram | Should Have

Story: As an SPV/Admin or Bot Specialist, I want to see the confidence score per AI Agent in the Tree Diagram, so that I can judge quality while composing a flow.

Before: Tree Diagram AI Agent nodes show no quality signal. After: A selected, tested AI Agent shows its average confidence score from testing.

Data Fields: ai_agent_id (uuid), avg_confidence_score (int, computed)

Happy Path:

AC-1: Given I am on the Tree Diagram page, when I click +, choose "AI Agent", and select a configured agent, then the node shows that agent's Confidence Score from testing.
AC-2: Given an agent has multiple completed test cases, when the score renders, then it is the average across those test cases.
AC-3: Given an agent has no completed test cases, when selected, then the node shows "no score yet" rather than 0%.

Error Path:

ERR-1: Given the confidence lookup fails, when the node renders, then it falls back to "no score yet" rather than erroring the diagram.

Permission Model: CAN: owner/supervisor/admin/bot-specialist.

UI States: Loading (node fetching), Empty ("no score yet"), Error (falls back to "no score yet"), Success (average score shown).

Technical Notes: Backend get_tree_diagram_v3; score = average across the agent's completed test cases; untested agent → "no score yet" (not 0%); lookup failure falls back to "no score yet". Dependencies: AITEST-S06.

Mockup: ⚠ Partially prototyped — the Bot Flow builder exists with an "AI agent" step/node, but the per-node confidence score badge is not yet rendered.

Design spec	Component(s) used	Page
Builder canvas (`bot-flow/[id].vue`) offers an "AI agent" node from the "Add step" menu; the confidence badge is not built. Target: a selected, tested AI-agent node shows its average Confidence Score from testing; untested → "no score yet".	builder canvas node (existing); confidence badge (to be added)	`/bot-automation/bot-flow/[id]` — `app/pages/bot-automation/bot-flow/[id].vue`

Negative Scenarios

NEG-1: Given I am a standard agent, when I look for AI Agent testing, then the Testing menu is not rendered and the route is forbidden.
NEG-2: Given a conversation contains only images/voice/attachments, when sampling runs, then it is excluded from the test set.
NEG-3: Given I am in the comparison view, when I try to edit the AI's answer, then no edit affordance exists — I must update the Knowledge Base instead.
NEG-4: Given the current phase, when I open "Generate test case", then "Generate from knowledge" and "Imported question list" are not selectable (deferred to Phase 2/3).

11. Rollout

Feature flag: ai_agent_testing | default: OFF. Enabled per organization during beta.
Stage 1: Internal — telesales POC org(s) (the original POC team).
Stage 2: Closed beta — 3–5 Enterprise/Pro orgs (manually enabled).
Stage 3: All orgs with AI Agent enabled, on request.
GA: All orgs with AI Agent enabled (self-serve toggle).
Backward compat: Yes — additive. AI Agents with no test cases show "no score yet"; existing publish/activate behavior is unchanged until the activation gate (AITEST-S07) ships.
Migration: Additive tables (ai_agent_test_cases, ai_agent_test_case_questions). No backfill.

Semantic regression rollback: ai_agent_testing is the per-org kill switch. If beta orgs report the score is misleading, or the gate suppresses legitimate activations, PM toggles ai_agent_testing OFF per org (no deploy); the gate reverts to advisory-only.

12. Observability

Event Name	Trigger	Properties
`ai_workspace_opened`	User opens the Testing page	user_id, org_id, bot_id, timestamp
`ai_validation_generated`	User clicks Generate (from inbox)	sample_size, date_range, test_case_id, timestamp
`ai_response_graded`	User clicks thumbs up/down	grade (pass/fail), confidence_score, inquiry_id, timestamp
`ai_validation_completed`	Confidence meter reaches the ≥80% threshold	total_time_spent, total_items_reviewed, timestamp
`ai_agent_activated`	User clicks Go Live	previous_validation_score, timestamp

Dashboard owner: BOT squad (Mixpanel + Tableau).

Alerts:

ai_validation_generated batch failure rate > 10% in 1h → Slack: #bot-ai-alerts (on-call).
LLM error rate during shadow generation > 5% in 15m → Slack: #bot-ai-alerts + PagerDuty on-call.

Post-Launch Monitoring Cadence: Weekly for the first 4 weeks post-GA, then monthly. Owner: Dimas Fauzi Hidayat (BOT squad PM). Triggers: activation conversion drops > 10% WoW → investigate within 48h; batch failure rate > 10% for 2 consecutive weeks → PM review + eng escalation. Rollback: if batch failure rate > 20% unresolved within 24h, PM disables ai_agent_testing for affected orgs.

13. Success Metrics

⭐ Primary KPI: Conversion Rate (Configured → Live)

Definition: % of AI Agents activated within 7 days of finishing configuration
Baseline: N/A — new capability (today activation takes a minimum of ~3 days, often weeks)
Target: ≥ 60% within 7 days, within 90 days of GA

Adoption: "Confidence Bar" completion rate

Definition: % of Admins who review enough items to reach ≥80% on the bar
Baseline: N/A — new capability
Target: ≥ 70% of started test cases reach ≥80%

Quality: Shadow-generation success rate

Definition: % of sampled questions that produce a valid AI shadow answer (no LLM failure)
Baseline: N/A — new capability
Target: ≥ 95% within 60 days of GA

Efficiency: Time-to-Confidence

Definition: Time from first setup to a go-live-ready validation result per agent
Baseline: ~6 hours/day of manual War Room monitoring during onboarding
Target: < 1 hour of reviewing the comparison report per agent, within 90 days of GA

Targets assume a re-baselined launch timeline (Open Question #4) — the original "May 2026 / Q1 2026" dates are past.

14. Launch Plan & Stage Gates

Stage	Audience	Duration	Success Gate	Owner
Internal Alpha	Telesales POC org(s)	2 weeks	0 P0/P1 bugs; shadow-generation success ≥ 90%; zero customer-message leakage	PM + QA
Closed Beta	3–5 Enterprise/Pro orgs	3–4 weeks	≥ 70% of started test cases reach ≥80% bar; batch failure rate ≤ 10%	PM + CSM
Open Beta	All orgs with AI Agent, on request	3 weeks	Shadow-generation success ≥ 95% sustained 1 week; no P0/P1 open	Eng Lead
GA	All orgs with AI Agent enabled	Ongoing	All Open Beta gates sustained 2 weeks; PMM launch approved	PM + PMM

15. Dependencies

Dependency	Owning Team	Deliverable Needed	Blocking?
Chat Service (Hub)	Inbox / Platform	Assigned-room list + room messages APIs (`FetchAssignedRoomIds`, `FetchRoomMessages`) over a 90-day window	YES
LLM / AI service	AI squad	Batch shadow inference within TPM/RPM limits (`sync_to_ai_service`, `qontak_nlp/predict`)	YES
Data team	Data	10% sampling + 50–70 cap algorithm (currently unbuilt)	YES
Channel Integration	Platform	Access tokens for room fetch (`ChannelIntegrations::GetTokens`)	YES
AI Agent versioning	BOT	`ai_agent_histories` stable — test cases bind to a version	YES

16. Key Decisions + Alternatives Rejected

Initiative-level decisions live in the ANCHOR PRD §5. Below are phase-specific decisions.

16a — Decisions Made

Date	Decision	Rationale
2025-12-19	Sample resolved, human-handled rooms over the last 90 days; cap 50–70 per batch	Recent data reflects current policies; cap controls token cost/latency on large accounts
2026-06-18	Store AI answer in `answer` and the human golden answer in `parameters.human_answer` on the question row	Matches the implemented schema; keeps comparison data on one record
2026-06-18	Use Sidekiq (`FetchRoomConversationsWorker`) for batch generation	Already the chatbot async stack; avoids introducing Kafka for this workload

16b — Alternatives Rejected

Alternative	Why Rejected	Date
Expose sampling parameters (date range, %, cap) in the Generate drawer	Adds user effort; system defaults (90d/10%/cap) are sufficient for the trust signal	2026-06-18
Separate `ai_validation_sessions` / `ai_validation_items` tables (original draft)	Superseded by the implemented `ai_agent_test_cases` / `ai_agent_test_case_questions` schema	2026-06-18

17. Open Questions

#	Type	Question	Owner	Deadline
1	Risk	Historical tickets contain PII sent to a 3rd-party LLM. Mitigation: covered by existing DPA; transient inference only; not used to train the public model.	Dimas (PM)	2026-07-24
2	Risk	Confidence-meter recalc (S06) + activation gate (S07) are not yet built. Mitigation: ship advisory-only for beta; enforce the gate before GA.	Eng (BOT)	2026-07-15
3	Risk	Original launch dates ("May 2026" GA, "late Q1 2026" beta) are now in the past. Mitigation: re-baseline the timeline with stakeholders before READY.	Dimas (PM)	2026-07-24
4	Open Question	Is the 80% confidence threshold fixed or org-configurable?	Dimas (PM)	2026-07-15
5	Open Question	Per-batch token budget across plan tiers — does the 50–70 cap hold for all?	Data team (Reza)	2026-07-15
6	Open Question	Is a separate "relevance" metric required (schema only has `confidence`)? If yes, store in `parameters`.	Dimas (PM)	2026-07-15
7	Assumption	A single human reply is a sufficient "golden answer" when a room has multiple agent messages.	Data team (Reza)	2026-07-15
8	Open Question	Hard-purge window for soft-deleted test cases/questions.	Eng (BOT)	2026-07-31
9	Risk	The `qontak-designer` prototype currently renders all three "Generate test case" source cards (Inbox / Resources / Import) as enabled, but Phase-1 scope (Non-Goal #4, NEG-4) requires "Generate from Resources" (Phase 2) and "Import questions" (Phase 3) to be hidden/disabled — shipping the prototype as-is would expose out-of-scope flows. Mitigation: FE gates the Resources + Import cards behind the phase flag before Phase-1 release; re-enable per phase. See §8 `GenerateTestCaseModal` [drift].	Dimas (PM) + FE	2026-07-24

PRD CHANGELOG

Version	Date	By	Section	Type	Summary
1.0	2026-06-18	Claude	All	CREATED	Phase 1 PRD authored in the documents-repo template under the new AI Agent: Testing ANCHOR. Reconciled with the Confluence draft and current code (`chatbot`, `chatbot-fe`, `qontak-designer`): endpoints, schema (`ai_agent_test_cases` / `ai_agent_test_case_questions`), async queue (Sidekiq), and sampling status filter (`assigned`) aligned to implementation; not-yet-built items (10%/cap sampling, confidence-meter recalc, activation gate) flagged as Open Questions. 10 stories with composite AC ids (AITEST-S01…S10).
1.1	2026-07-10	Claude	Header, §7, §8, §10, §17	UPDATED	Split every user story's footer into separate Technical Notes and a designer-grounded Mockup (3-column Design spec · Component(s) · Page) sub-table, grounded against the `qontak-designer` prototype (`index.vue`, `[id].vue`, `QuestionComparisonCard`, `GenerateTestCaseModal`, `GenerateFromInboxDrawer`). S07/S09 (activation gate, force-activate) and S10's per-node confidence badge flagged not yet built in the prototype. Reconciled §8 component tree + UI states with the prototype (added "AI agent name" column, inline generating/organizing/complete phases, corrected source-picker labels, search/filter not-found states) with [drift] markers. Post-score fixes: bumped Header to 1.1 + Last Updated 2026-07-10; re-baselined overdue OQ #1 & #3 deadlines to 2026-07-24; added CHG-001 Figma frame link and three Tier-1 diagrams (§10.1 system flow, §8 index + detail UI-state machines). Wired the real Confluence RFC (QON/50904990129, BE technical design) + repo RFC into the Header (replacing the stale to-be-created placeholder); flipped Status `draft → in-review`; added OQ #9 (source-picker phase-gating drift).
1.2	2026-07-20	Claude	Header, §7, §8, §10.2	MODIFIED	Removed all Figma links/node citations (header Figma Master field, CHG-001, §8 Mockup-column description, every §10.2 story's Technical Notes/Mockup cell) — design source is now stated as the `qontak-designer` prototype only. The two "not yet built" flags (S07 activation gate, S09 force-activate audit) are kept but no longer cite the absent Figma frame they were sourced from.

HEADER BLOCK​

Scope Changes​

2. Phase Context​

3. One-liner + Problem​

4. Target Users + Persona Context​

5. Non-Goals​

6. Constraints​

7. Feature Changes​

8. New Features​

9. API & Webhook Behavior​

10. System Flow + User Stories + ACs​

10.1 System Flow​

10.2 User Stories​

AITEST-S01 — Workspace access control | Must Have​

AITEST-S02 — Historical data sampling (10% rule) | Must Have​

AITEST-S03 — Data integrity & filtering | Must Have​

AITEST-S04 — Shadow mode execution (zero leakage) | Must Have​

AITEST-S05 — Side-by-side validation UI | Must Have​

AITEST-S06 — Confidence meter & feedback | Must Have​

AITEST-S07 — Activation gatekeeping | Should Have​

AITEST-S08 — Background processing (async) | Must Have​

AITEST-S09 — Manual override & audit | Could Have​

AITEST-S10 — Confidence score in Tree Diagram | Should Have​

Negative Scenarios​

11. Rollout​

12. Observability​

13. Success Metrics​

14. Launch Plan & Stage Gates​

15. Dependencies​

16. Key Decisions + Alternatives Rejected​

17. Open Questions​

PRD CHANGELOG​

HEADER BLOCK

Scope Changes

2. Phase Context

3. One-liner + Problem

4. Target Users + Persona Context

5. Non-Goals

6. Constraints

7. Feature Changes

8. New Features

9. API & Webhook Behavior

10. System Flow + User Stories + ACs

10.1 System Flow

10.2 User Stories

AITEST-S01 — Workspace access control | Must Have

AITEST-S02 — Historical data sampling (10% rule) | Must Have

AITEST-S03 — Data integrity & filtering | Must Have

AITEST-S04 — Shadow mode execution (zero leakage) | Must Have

AITEST-S05 — Side-by-side validation UI | Must Have

AITEST-S06 — Confidence meter & feedback | Must Have

AITEST-S07 — Activation gatekeeping | Should Have

AITEST-S08 — Background processing (async) | Must Have

AITEST-S09 — Manual override & audit | Could Have

AITEST-S10 — Confidence score in Tree Diagram | Should Have

Negative Scenarios

11. Rollout

12. Observability

13. Success Metrics

14. Launch Plan & Stage Gates

15. Dependencies

16. Key Decisions + Alternatives Rejected

17. Open Questions

PRD CHANGELOG