AI Agent Testing
Give Qontak SPVs and Admins a dedicated Testing workspace to validate an AI Agent's quality before go-live, so they can replace the manual ~6-hour/day "War Room" monitoring with self-serve, evidence-based validation. Testing is a standalone top-level page under Bot Automation (Bot Automation → AI agents · Resources · Actions · Testing · Analytics · Bot flow), with multiple test cases per AI Agent — each bound to a specific agent version. The initiative phases by test-case source, with Historical Validation (sample real resolved conversations) first.
QA Lane
Lane B — keeps a human QA gate. No Lane-B trigger is present, but no E2E test specs exist for this initiative yet, so the Lane-A entry bar (100% E2E, spec-mapped coverage) is unmet — Lane B by default. Classified 2026-06-29.
Master index (ANCHOR)
ai-agent-testing-anchor.md— the ANCHOR PRD: the initiative master index (identity, Phase Index, north-star metrics, decisions). It carries no acceptance criteria of its own (those live in the phase PRDs), so it sits at the initiative root rather than underprds/. Synced with the Confluence AI Agent: Testing ANCHOR and reconciled against code (chatbot,chatbot-fe,qontak-designer).
Phases
| Phase | Source | Status | Epic |
|---|---|---|---|
| Phase 1 — Historical Validation | Generate from inbox (resolved conversations) | PRD draft; partial backend (test-case CRUD + worker shipped; sampling/recalc/gate pending) | BOT-3351 |
| Phase 2 — Generate from Knowledge | Questions synthesised from the knowledge source | Planned | TBD |
| Phase 3 — Imported Question List | PM/SPV-curated question set uploaded into a test case | Planned | TBD |
Phase 2 and Phase 3 are seeded from the three test-case sources already scaffolded in the design/code (
GenerateTestCaseModal: "Generated from inbox" / "Generated from knowledge" / "Imported question list"). They are placeholders until their phase PRDs are written.
Contents
prds/— phase PRDs (each with its own ACs → Jira Epic) land here.rfcs/— Request for Comments (technical design proposals).tests/— E2E / acceptance test specs.