RFC: Unified Agent Quality Scorecard — Phase 2: AI Auto-Scoring & In-Room Scorecard

Document Conventions (do not remove)

This RFC follows the Qontak RFC Template format for governance — the metadata table, sections 1–6, and Comment log are mandatory.

It is also agent-execution-ready: §1 PRD-to-Schema Derivation (BE half) + §2.A UI Contract (FE half), §2.0 Repo Reading Guide for both layers, mermaid diagrams, §2.G Cross-Layer Contract Verification, and §4 Agent Execution Plan + Verification & Rollback Recipe are complete.

Agent-execution-ready RFC derived 1:1 from ../prds/phase-2-auto-scoring-and-in-room-scorecard.md. Phase 2 builds the scoring engine + in-room panel — consumes Phase 1 config; produces the per-conversation scores Phase 3 (report) and Phase 5 (gate) read. Backend = Qontak Chatbot (chatbot, Rails 7.1 / Grape / Clean Architecture / Sidekiq). Frontend = Omnichannel inbox (hub-chat, Nuxt 4 / Vue 3 / Pinia / Pixel3) — not chatbot-fe (which owns the Phase 1 settings surfaces).

Metadata

Field	Value	Notes
Status	RFC	Working vocabulary IDEA/RFC/AGREED/ABANDON. YAML status: uses linter enum (RFC→in-review); kept draft until reviewed.
Owner (DRI)	Dimas Fauzi Hidayat	Mirrors frontmatter dri. Single accountable owner; staffing lives in delivery/.
Source PRD	../prds/phase-2-auto-scoring-and-in-room-scorecard.md	PRD v1.4.
Anchor	../unified-agent-scorecard-anchor.md	Initiative master index.
Prior RFC	rfc-phase-1-settings-and-rubric-config.md	Config layer this phase consumes (is_ai_auto_score, ai_passing_grade, custom-param prompt, default-rubric constant).
Delivery	not yet handed to delivery	Timeline/effort/rollout scheduling lives in delivery/.
Type	full-stack	Backend (Grape API + Sidekiq + chatbot_gpt DB) + Frontend (hub-chat in-room panel).
Squad	BOT — Bot, AI & Automation
Infosec approver	required at review — see §7	Stores scored conversation transcripts (PRD Open Q#5 / Strategy G7).
Last Updated	2026-07-17

Sections at a Glance

§	Section	Type hint
§1	Overview, traceability, decisions index	PRD coverage, AC map, schema derivation
§2	Technical design	Repo reading guide, infra topology, ADRs, ER + sequence + state diagrams, API + UI contracts
§3	HA & Security	Perf, auth matrix, failure catalog, error catalog, a11y
§4	Backwards compat & rollout	Flag contract, test plan, agent execution plan, rollback recipe
§5	Concerns / open questions	Carried from PRD + grounding gaps
§6	Comment log
§7	Ready for agent execution	The readiness gate

---

1. Overview

Phase 2 makes the SkillPack engine's 9-metric evaluator output visible and durable, behind the same ai_qa_unified_scorecard flag Phase 1 shipped behind. It does four things:

1. Ingests the engine's per-conversation 9-metric evaluator result at terminal exit/handoff through a new internal endpoint (POST /v1/ai_agent/evaluations) — verified: no ingestion path exists today (§2.0 Source Verification).
2. Scores the conversation asynchronously (Sidekiq): tier-1 weighted score (uniform 0.11 weights), tier-2 custom-param judge scoring via one Repositories::Gpt::Completion call for all rubric'd params, veto check (Groundedness/Policy floors is_pass), compared against Phase 1's ai_passing_grade — persisted as an AI-actor agent_scorecards record keyed (organization_id, room_id, agent_id) alongside (never replacing) any human record for the same room.
3. Surfaces the score in the hub-chat in-room Scorecard panel: a new "Auto-scored by AI" mode (graded metrics + judge reasons + cited KB sources + veto flags + total & pass/fail badge) plus an actor selector for rooms served by both the AI agent and human agent(s). Human manual mode is byte-for-byte unchanged.
4. Allows override: Supervisor/Admin can correct any AI metric score, consuming the scorecard's existing one-edit correction slot (edit_count/correction_at), with paper_trail audit and total/is_pass recompute.

No report, no gate, no validation harness — those are Phases 3/5/4.

Success Criteria

• ≥95% of AI-agent conversations auto-scored within 60s of terminal exit/handoff (PRD §7/§13).
• Tier-2 params with a non-empty Phase 1 rubric are judge-scored and merged; empty-rubric params are excluded (manual-only) without affecting tier-1 (P2-S01/AC-2, AC-4).
• A veto-metric failure floors is_pass = false regardless of the weighted total (AC-5).
• In-room panel renders the AI mode ≤ 2s P95; actor selector lists every scored actor (P2-S02).
• Override recomputes total/is_pass, marks "edited by [SPV]", audits via paper_trail, and consumes the single correction slot (P2-S03).
• Human manual scoring and the live human GPT auto-scorer (auto_agent_scoring.rb) are behaviorally unchanged — except the room-resolve skip guard is made actor-aware (§2.2 ADR-8) so an AI record never suppresses human scoring or vice versa.
• Scoring never blocks live conversation handling (async; ERR-1).

Out of Scope

Analytics report + export (P3), validation/testing harness (P4), go-live gate (P5), weight tuning (DSAI; uniform 0.11 here), mobile, real-time per-message scoring, any change to human manual scoring UX, selectable scorecard templates / multi-human scoring UX (P8), per-team RBAC scoping (descoped — §2.2 ADR-10, PRD Open Q#7).

Related Documents

Document	Path	What was taken from it
Phase 2 PRD v1.4	../prds/phase-2-auto-scoring-and-in-room-scorecard.md	All requirements, ACs, rubric, data lifecycle, events.
Phase 1 RFC	rfc-phase-1-settings-and-rubric-config.md	Config contract consumed here: is_ai_auto_score, ai_passing_grade (new columns — not the PRD's "extended is_auto_score" wording), custom-param prompt (text, ≤4000), Constants::ScorecardAiDefaultRubric (metric codes), flag mechanism (ADR-5), forward note on the room-resolve skip guard.
Initiative anchor	../unified-agent-scorecard-anchor.md	Phase map; "consume the engine's evaluator, don't build a second judge" decision.
Phase 1 PRD	../prds/phase-1-settings-and-rubric-config.md	Rubric content + judging prompts (Appendix A).
chatbot AGENTS.md API rules	chatbot/AGENTS.md §"API Specification Rules"	Mandatory OpenAPI bundle/split/validate workflow (§4.B).
hub-chat AGENTS.md	hub-chat/AGENTS.md	Feature-folder structure, Pinia setup-syntax stores, Pixel3-only styling, flag sources.

Assumptions

• A1 — The DSAI evaluator pushes its per-conversation 9-metric result to us at terminal exit/handoff, using the PROPOSED contract in §2.4 Inbound row 1. The contract is not yet confirmed by DSAI (PRD Open Q#2, still blocking as of 2026-07-03). The build proceeds against the PROPOSED contract behind the flag; enabling ingestion for real orgs gates on DSAI confirmation (§5 #1, §7).
• A2 — The evaluator payload carries the segment boundary (segment.from/to message ids + turns) for the AI-handled turns. Grounded fallback: chatbot BE cannot derive it from handover events (verified absent BE-side — §2.0), so a payload without segment is scored whole-conversation and flagged (segment_source: "missing"), validated in Internal Alpha against the 15-CID set (PRD Open Q#3).
• A3 — The AI agent is not a hub room participant (verified: participants are fetched type: ["agent"] and typed Models::User/customer only). The AI actor therefore exists only as a scorecard record (actor_type: "ai_agent", agent_id = chatbot AI-agent id); the FE actor selector is built from the scorecard actor list, not roomParticipant (ADR-2; resolves PRD Open Q#6).
• A4 — Phase 1 is deployed before Phase 2 scoring is enabled: the is_ai_auto_score / ai_passing_grade columns, widened prompt, and Constants::ScorecardAiDefaultRubric exist. Phase 2 code reads them and fails closed (no scoring) when absent/off.

Dependencies

Dependency	Owner	Needed	Blocking?
Evaluator output contract (§2.4 Inbound row 1)	DSAI / engine team	Confirm PROPOSED payload + push trigger	YES — for enabling ingestion (build proceeds behind flag; §5 #1).
Phase 1 config live	BOT (Phase 1 RFC)	is_ai_auto_score, ai_passing_grade, custom-param prompt, default-rubric constant	YES — deploy order (§4 Rollout).
Net-new scorecard schema fields	BOT (this RFC)	§2.3 DDL	NO — in scope.
PII hashing + credential vault (G7)	Security / Infosec	PII hashed before judge/log; credential leak closed before transcripts stored	YES before GA (§5 #4; snapshots are the new blast radius).
Design frames (panel AI mode + actor selector)	Design squad	Figma for CHG-003	YES for FE pixel-fidelity — interim spec = PRD Appendix B Stitch prompt (§5 #3).
9-metric weight tuning	DSAI	Tuned weights	NO for P2 (uniform 0.11, measurement-only) · gates Phase 5.

Detail 1.A — Coverage Matrices

1.A.1 — PRD Section Coverage

PRD §	Title	Covered in
2	Phase Context	§1 Overview, Assumptions A4
3	One-liner + Problem	§1 Overview
4	What happens if we don't ship	§1 (motivation)
5	Target users + persona	§1.A.3 Roles, §3 Role × Endpoint matrix
6	Non-Goals	§1 Out of Scope
7	Constraints (+7.1 Data Lifecycle)	§3 Performance, §2.3 retention DDL + workers, §4.A flag contract
8	Feature Changes (CHG-003)	§2.A UI Contract, §2.4 APIs
9	API & Webhook Behavior (B1 ingest, B2 override)	§2.4 (Inbound row 1; Outbound rows 1–2)
10	System Flow + Stories + ACs	§2.1a Sequences, §1.A.4 AC map, §1.C
11	Rollout (+11.1 semantic regression rollback)	§4 Rollout, §4.A weights flag, §4.D
12	Observability (+12.1 cadence)	§3 Monitoring & Logging, §4.D signals
13	Success Metrics	§1 Success Criteria, §4.D signals
14	Launch Plan & Stage Gates	§4 Rollout (technical view; scheduling → delivery/)
15	Dependencies	§1 Dependencies
16	Key Decisions + Alternatives	§1.B + §2 ADRs
17	Open Questions	§5
App. A	AI Scoring Rubric	§2.4 metric codes (reuse Phase 1 Constants::ScorecardAiDefaultRubric)
App. B	Stitch UI Prompt	§2.A interim design spec

1.A.2 — UI / Consumer Surface Coverage

Surface	PRD ref	Backing read endpoint	RFC anchor
In-room Scorecard panel — AI mode (/inbox right panel, agent_scorecard tab)	CHG-003, S02	GET /api/v1/gpt/omnichannel/agent_scorecards/:room_id (extended: agent_id param + actors[])	§2.A, §2.4 row 1
Actor selector (same panel)	CHG-003, S02/AC-4	same GET (actors[] summary)	§2.A, §2.4 row 1
Override control (same panel)	S03	PATCH .../agent_scorecards/:room_id (extended) — n/a — covered by writes; state re-read via GET	§2.A, §2.4 row 2
Scoring pipeline (no UI)	S01	n/a — backend; surfaces via the panel GET	§2.4 Inbound row 1

1.A.3 — Role Coverage

PRD persona	Grounded role/permission	Access
QA Lead / Supervisor	role supervisor (hub-core enum, per Phase 1 RFC ADR-7) + Usman inbox_scorecard_view/inbox_scorecard_manage	View AI scores; override (manage)
Bot / AI Builder (Agent Owner)	owner / admin	View; override
End CS agent	agent / member	GET stays role-permitted as today (agent/member allowed on the room GET — grounded, see §3), gated by Usman scorecard perms; no override (PATCH excludes them)

Grounding corrections (PRD vs code): (1) "team scope" does not exist — Usman scorecard permissions are org-wide (inbox_scorecard_view/inbox_scorecard_manage, hub-chat UsmanStore.ts:14-16); Phase 2 descopes to org-wide (ADR-10, resolves PRD Open Q#7). (2) The existing room GET already permits agent/member (agent_scorecards.rb:127-143 — roles owner/supervisor/admin/member/agent), so "end CS agents: no access to others' scores" is enforced by Usman gating in the panel (view-only visibility rules in AgentScorecard.vue:216-226), not by the Grape role list. Unchanged here.

1.A.4 — Acceptance-Criteria → Design Element Map

PRD Story	Composite AC ids	Design element	Test spec ref
P2-S01 — Auto-score AI conversation (two-tier, per actor/segment, veto)	P2-S01/AC-1..AC-6, /ERR-1, /ERR-2, /NEG-1, /NEG-2	§2.4 Inbound row 1 (ingest) · §2.3 DDL · §2.2 ADR-1..6,8 · §4.C chunks 1–7, 10	tests/phase-2-auto-scoring-and-in-room-scorecard.md (to author — §4.C chunk 16)
P2-S02 — View AI auto-score in panel (+ actor selector)	P2-S02/AC-1..AC-5, /ERR-1, /NEG-1	§2.4 row 1 (GET ext) · §2.A ActorSelector + AiScorecardView · §4.C chunks 8, 12–14	″
P2-S03 — Override an AI auto-score	P2-S03/AC-1..AC-4, /ERR-1, /NEG-1	§2.4 row 2 (PATCH ext) · §2.2 ADR-7 · §2.A override control · §4.C chunks 9, 13	″

1.A.5 — PRD-to-Schema Derivation (BE)

PRD entity/attribute/rule	table.column	Exposed by	Enforced at	PRD ref
AI-actor score record (separate from human)	agent_scorecards row with new actor_type ('ai_agent'); existing unique key (organization_id, room_id, agent_id)	GET/PATCH :room_id (+agent_id)	unique partial index (existing, chatbot_gpt_schema.rb:124)	S01/AC-6
Score source (auto vs manual)	existing agent_scorecards.is_use_ai (bool) + new actor_type	GET response is_use_ai, actor_type	write path sets both	CHG-003 "score source label"
Scoring lifecycle ("scoring…", partial, unavailable)	new agent_scorecards.scoring_status (string enum, nullable — null for manual)	GET response	state machine §2.3	S01/ERR-1,2; S02 UI states
Veto outcome	new agent_scorecards.is_veto_failed (bool) + veto_reason (string); new agent_scorecard_details.is_veto (bool)	GET response	scoring use case (veto before pass calc)	S01/AC-5, App. A
Segment boundary	new agent_scorecards.segment (jsonb: {from_message_id,to_message_id,from_turn,to_turn,source})	GET response	ingest contract; source: evaluator|missing	S01/AC-6, Open Q#3
Per-metric judge reason (13-mo retention)	new agent_scorecard_details.judge_reason (text)	GET response	ingest/judge write	S02/AC-2, §7.1
Cited KB sources + transcript snapshot (90-day TTL)	new table agent_scorecard_snapshots (transcript jsonb, cited_sources jsonb, failed_payload jsonb, expires_at)	GET response (sources, transcript_available)	nightly cleanup worker	§7.1, S02/AC-2,5
Tier-1 metric identity	agent_scorecard_details.scorecard_parameter_code reused for the 9 metric codes from Phase 1 Constants::ScorecardAiDefaultRubric (no new parameter rows)	GET response	scoring use case maps code→detail	App. A
Tier-2 custom-param scores	agent_scorecard_details rows with existing scorecard_custom_parameter_id/is_custom_parameter	GET response	judge merge; empty rubric excluded	S01/AC-2,4
Pass verdict vs threshold	existing is_pass, passing_grade (populated from Phase 1 ai_passing_grade)	GET response	scoring use case	S01/AC-3
Idempotency recency + weights audit	new agent_scorecards.evaluated_at (datetime) + weights_version (string) — on the 13-month record, not the 90-day snapshot	GET response	ingest upsert guard; scoring use case stamps version	§9 B1 (retry), §11.1 (weights rollback audit)
Override audit ("edited by [SPV]")	existing edit_count, correction_at, correction_by* + paper_trail	GET response	PATCH use case (one-slot guard)	S03/AC-1,2,4
13-month score retention	TTL cleanup on agent_scorecards (AI rows) created_at	—	nightly worker (§2.3)	§7.1

Detail 1.B — Decisions Closed (index → §2 ADRs)

#	Decision	ADR
1	Ingest via a new internal endpoint POST /v1/ai_agent/evaluations (push from AI Service), not by extending the per-message async-reply	ADR-1
2	AI actor = a scorecard record (actor_type) keyed by the existing (org, room, agent_id) index; not a room participant	ADR-2
3	Extend agent_scorecards/agent_scorecard_details + one new agent_scorecard_snapshots table; no parallel AI-score tables	ADR-3
4	Segment boundary comes from the evaluator payload; no BE derivation from handover events (they don't exist BE-side)	ADR-4
5	Tier-2 judge = one Repositories::Gpt::Completion call for all rubric'd custom params; partial-failure downgrades to scoring_status: "partial"	ADR-5
6	Tier-1 weights uniform 0.11 constant; tier-2 params join at the same uniform weight; total = weighted mean; versioned by ai_qa_unified_scorecard_v2_weights	ADR-6
7	Override reuses the existing one-edit correction slot (edit_count/correction_at) via the extended PATCH	ADR-7
8	Make the room-resolve skip guard actor-aware (Phase 1 forward note) so AI and human records never suppress each other	ADR-8
9	Retention via sidekiq-cron nightly workers in config/schedule.yml (90-day snapshots, 13-month AI scores, 7-day failed payloads)	ADR-9
10	RBAC stays org-wide Usman scorecard permissions; per-team scoping descoped (PRD Open Q#7 resolved for P2)	ADR-10
11	Feature gate reuses ai_qa_unified_scorecard org-feature (BE OrganizationFeatures::FindFeature; hub-chat checkSubscription / organizationSettings precedent)	ADR-11

Detail 1.C — Per-Story Change Map

Story	Layer scope	Changes (concrete artifacts)	Acceptance criteria	RFC anchors
P2-S01	BE-only	BE: 3 migrations (§2.3); Grape app/api/internal_service/v1/ai_agent/evaluations.rb (NEW) + contract; AiAgentScoringWorker (NEW, queue :gpt_agent_scoring); use case UseCases::Gpt::Omnichannel::AiAgentScoring (NEW): tier-1 compute (uniform weights vs Constants::ScorecardAiDefaultRubric codes) → tier-2 via Repositories::Gpt::Completion (single call) → veto → persist via new Repositories::Gpt::AgentScorecards::CalculateAiAgentScore (+ snapshot row); actor-aware fix to room_resolve_interactions.rb:48-63 guard; retention workers + config/schedule.yml; events via SendMixpanelEventWorker + ChatGptLogWorker related_key: 'ai_agent_scoring'; OpenAPI.	P2-S01/AC-1 record ≤60s w/ tier-1 score; AC-2 rubric'd tier-2 merged; AC-3 is_pass vs ai_passing_grade; AC-4 empty rubric excluded; AC-5 veto floors; AC-6 separate per-actor records on (org,room,agent); ERR-1 malformed → unavailable + retry + scorecard_autoscore_failed; ERR-2 judge fail → partial, tier-1 kept; NEG-1 human conv not AI-scored; NEG-2 no live per-message scoring.	§2.3 · §2.4 Inbound r1 · §2.2 ADR-1..6,8,9 · §4.C c1–7,10,11
P2-S02	FE + BE	BE: extend AgentScorecard::Get + GetScorecard repo (optional agent_id; actors[]; AI fields; default-to-human when param absent); entity additions; OpenAPI. FE (hub-chat): ActorSelector.vue (NEW), AiScorecardView.vue (NEW: graded metric rows + expandable judge_reason + source links + veto banner + total/pass badge + "Auto-scored by AI" badge); extend AgentScorecard.vue mode switch on actor_type; extend useAgentScorecard.ts (agent_id param); types in AgentScorecardTypes.ts; flag gate via checkSubscription("ai_qa_unified_scorecard") ∥ organizationSettings (SwitchBar precedent); $mixpanel scorecard_panel_load_failed.	P2-S02/AC-1 AI actor → graded groups + total + badge; AC-2 reason expand + Groundedness source link; AC-3 veto red flag + is_pass=false; AC-4 selector lists AI + human, human view unchanged; AC-5 expired snapshot → scores render, transcript "expired"; ERR-1 unavailable state + retry, no crash, event logged; NEG-1 human-only room → manual mode only.	§2.4 r1 · §2.A · §2.B · §4.C c8,12–14
P2-S03	FE + BE	BE: extend AgentScorecard::Update + repo: AI-override path (agent_id, overrides[]{metric_code,score_value,reason}), 0–100 validation, one-slot guard (edit_count), recompute total/is_pass (veto still floors), set correction_*, paper_trail; events scorecard_override_saved/_failed; OpenAPI. FE: per-metric override control in AiScorecardView.vue (pencil → inline input → save), "edited by [SPV]" chip, gated by Usman inbox_scorecard_manage + roles; error+retry toast.	P2-S03/AC-1 in-range override recomputes + chip; AC-2 paper_trail version recorded; AC-3 out-of-range → 422, nothing persisted; AC-4 second override blocked "already corrected"; ERR-1 save fail → error+Retry, no partial state, scorecard_override_failed; NEG-1 non-supervisor sees no control.	§2.4 r2 · §2.2 ADR-7 · §2.A · §4.C c9,13

---

2. Technical Design

Detail 2.0 — Repo Reading Guide

Repo Map (slice this RFC touches)

flowchart LR
  subgraph AIS["AI Service (DSAI)"]
    eval["SkillPack 9-metric evaluator"]
  end
  subgraph BE["chatbot (Rails 7.1 / Grape / Sidekiq)"]
    inb["app/api/internal_service/v1/ai_agent/evaluations.rb (NEW)"]
    wrk["app/workers/ai_agent_scoring_worker.rb (NEW)"]
    uc["app/core/use_cases/gpt/omnichannel/ai_agent_scoring.rb (NEW)"]
    judge["app/core/repositories/gpt/completion.rb (tier-2 judge)"]
    calc["app/core/repositories/gpt/agent_scorecards/calculate_ai_agent_score.rb (NEW)"]
    apiget["app/api/frontend_service/v1/gpt/omnichannel/agent_scorecards.rb (GET/PATCH ext)"]
    guard["webhook/room_resolve_interactions.rb (guard fix)"]
    cron["config/schedule.yml + retention workers (NEW)"]
  end
  db[("chatbot_gpt DB - Postgres<br/>agent_scorecards + details<br/>agent_scorecard_snapshots NEW")]
  subgraph FE["hub-chat (Nuxt 4 / Vue 3 / Pinia / Pixel3)"]
    panel["room-details/features/agent-scorecard/views/AgentScorecard.vue (ext)"]
    actor["components/ActorSelector.vue (NEW)"]
    aiview["components/AiScorecardView.vue (NEW)"]
    comp["composables/useAgentScorecard.ts (ext)"]
    gate["SwitchBar.vue flag pattern + UsmanStore"]
  end
  oai["OpenAI (tier-2 judge LLM)"]

  eval -->|"POST evaluations (PROPOSED)"| inb --> wrk --> uc
  uc --> judge --> oai
  uc --> calc --> db
  cron --> db
  panel --> actor & aiview
  panel --> comp -->|"GET / PATCH room scorecard"| apiget --> db
  gate -.gates.-> panel
  guard -.-> wrk

Existing Code Anchors (read before writing)

#	Path	What to learn
1	chatbot/app/core/use_cases/gpt/omnichannel/auto_agent_scoring.rb	The live human auto-scorer this pipeline sits beside: participant pick (L63-70), preference gate (L76-78), OpenAI call + 2 attempts (L160-196), result validation (L232-236), persistence handoff (L274-283).
2	chatbot/app/core/repositories/gpt/agent_scorecards/calculate_agent_score.rb	Transactional scorecard+details write, weighted-score math, passing_grade default fill (L21-22) — the shape CalculateAiAgentScore mirrors.
3	chatbot/app/api/frontend_service/v1/gpt/omnichannel/agent_scorecards.rb	Per-room Grape contract: POST L45-62, PATCH L99-116, GET L127-143 (+roles), batch L154-169 — the endpoints extended here.
4	chatbot/app/core/repositories/gpt/agent_scorecards/get_scorecard.rb	find_by(organization_id, room_id) single-record read (L16) — must become actor-aware.
5	chatbot/app/api/internal_service/v1/ai_agent/use_cases/process_async_reply.rb	AI-Service→chatbot inbound pattern: contract (L38-51), context resolution (L25-35), qontak_chat_room_resolve detection (L180-183) — auth + param conventions for the new evaluations endpoint.
6	chatbot/app/core/use_cases/api/internal_service/v1/webhook/room_resolve_interactions.rb	The AutoAgentScoringWorker trigger + skip guard (L48-63) this RFC makes actor-aware.
7	chatbot/app/core/repositories/gpt/completion.rb	Tier-2 judge call: .new(messages:, gpt_timeout: 240, config:), {success:, message:, token usages} return, rescue-all + ChatGptLogWorker logging (L61-107).
8	chatbot/app/workers/auto_agent_scoring_worker.rb + config/schedule.yml	Worker shape (retry: 1, queue: :gpt_agent_scoring) and sidekiq-cron entry shape (delete_old_logs_job, cron 0 1 * * * Asia/Jakarta).
9	hub-chat/features/inbox/features/main-section/features/room-details/features/agent-scorecard/ (views/AgentScorecard.vue, components/ScorecardForm.vue, composables/useAgentScorecard.ts, @types/AgentScorecardTypes.ts)	The panel to extend: one-edit guard (AgentScorecard.vue:124-131), roomParticipant[0] assumption (L187), binary rating (ScorecardForm.vue:443-451), API URLs (useAgentScorecard.ts:7-48), unused is_use_ai/is_ai_failed types (AgentScorecardTypes.ts:39,45).
10	hub-chat/features/inbox/features/main-section/features/room-details/components/SwitchBar.vue + common/store/UsmanStore.ts	Tab gating precedent (isScorecardActive, L156-164: org setting ∥ checkSubscription + Usman) and permission store (inbox_scorecard_view/manage, L14-16; isHasPermission L97-102).

Patterns to Follow

Concern	Reference file (opened)	Pattern
Inbound AI-Service endpoint	chatbot/app/api/internal_service/v1/ai_agent/use_cases/process_async_reply.rb	APIAbstractUseCase + contract do params …; context resolution; mounted under internal_service (internal_service/api.rb:29 precedent).
Sidekiq worker	chatbot/app/workers/auto_agent_scoring_worker.rb	include Sidekiq::Worker; sidekiq_options retry: 1, queue: :gpt_agent_scoring; perform(params) delegating to a use case with a plain hash.
LLM judge call	chatbot/app/core/repositories/gpt/completion.rb	OpenAI::Client.new(request_timeout: gpt_timeout).chat; returns {success:, message:, …}; never raises to caller.
Scorecard persistence	chatbot/app/core/repositories/gpt/agent_scorecards/calculate_agent_score.rb	Build scorecard + details hashes → single transactional create; Rollbar.error(e) on failure.
Scheduled cleanup	chatbot/config/schedule.yml (delete_old_logs_job → DeleteOldLogsWorker)	sidekiq-cron entry, nightly 0 1 * * * Asia/Jakarta, maintenance queue.
Feature flag (BE)	chatbot/app/core/repositories/organization_features/find_feature.rb (per Phase 1 RFC ADR-5)	OrganizationFeatures::FindFeature.new(feature_code:, organization_id:).call_by_organization.
GPT-call logging	chatbot/app/core/use_cases/gpt/omnichannel/auto_agent_scoring.rb:198-226	ChatGptLogWorker.perform_async with related_key ('auto_agent_scoring' → ours: 'ai_agent_scoring').
Async analytics	chatbot/app/workers/send_mixpanel_event_worker.rb	SendMixpanelEventWorker.perform_async(org, event, props.as_json) (per Phase 1 RFC ADR-6).
Panel component	hub-chat/.../agent-scorecard/components/ScorecardForm.vue:322-342	<script setup lang="ts">; Pixel3 Mp* imports; css() helper; token-string props; toast from @mekari/pixel3.
Panel API client	hub-chat/.../agent-scorecard/composables/useAgentScorecard.ts	$customFetch + chatbotServiceUrl runtime config; unwrap response.data; rethrow error._data?.message.
Panel mount/tab	hub-chat/.../room-details/views/RoomDetailsView.vue:134-144	<Transition> + v-if="activeMenu === 'agent_scorecard'"; props room-status/room-agent/room-participant.
FE flag	hub-chat/.../components/SwitchBar.vue:156-164	organizationSettings.value?.X ∥ checkSubscription("CODE") && Usman gate.
FE state	hub-chat/features/inbox/store/mainSectionStore.ts:51-77 + common/store/UsmanStore.ts:67	Pinia; new feature-local store uses setup syntax per hub-chat AGENTS.md; roomParticipant stays untouched (actors come from the API).

Reading Order for the Agent

1. chatbot/AGENTS.md (§Workflow Commands + §API Specification Rules)
2. Anchor #1 (auto_agent_scoring.rb) → #2 (calculate_agent_score.rb)
3. Anchor #3 (agent_scorecards.rb Grape) → #4 (get_scorecard.rb)
4. Anchor #5 (process_async_reply.rb) → #6 (room_resolve_interactions.rb)
5. Anchor #7 (completion.rb) → #8 (worker + schedule.yml)
6. chatbot/db/chatbot_gpt_schema.rb:52-125 (both scorecard tables) + Phase 1 RFC §2.3 (columns it adds)
7. hub-chat/AGENTS.md → Anchor #9 (the whole agent-scorecard/ feature folder)
8. Anchor #10 (SwitchBar.vue + UsmanStore.ts)
9. hub-chat/.../room-details/views/RoomDetailsView.vue:134-144, 1169-1183 (mount + tab switch)
10. chatbot/docs/openapi/ + scripts/openapi/split.rb (spec workflow)

Existing-Endpoint Check (reuse / extend / new)

Endpoint	Surface(s)	Tag	Evidence
POST /v1/ai_agent/evaluations	internal_service (AI Service → chatbot)	new-with-justification	Verified absent: no ingestion path for evaluator/judge/metric output exists anywhere in app/ (grep evaluator|judge|metric|rubric — only NLP intent-confidence hits); process_async_reply.rb ingests only output_text/references/exit/token_usage (contract L38-51) and is per-message, while the evaluation is per-conversation at terminal exit — overloading the reply contract would couple scoring to message delivery and violate "scoring never blocks live handling" (ERR-1).
GET /api/v1/gpt/omnichannel/agent_scorecards/:room_id	frontend_service	extended	agent_scorecards.rb:127-143; adds optional agent_id param, actors[] summary, AI fields. Default behavior (no param) returns the human record → today's contract preserved.
PATCH /api/v1/gpt/omnichannel/agent_scorecards/:room_id	frontend_service	extended	agent_scorecards.rb:99-116; adds the AI-override body variant (agent_id + overrides[]). Existing human category/parameter shape untouched.
POST /api/v1/gpt/omnichannel/agent_scorecards/:room_id (manual create)	frontend_service	reused	agent_scorecards.rb:45-62 — unchanged; manual human scoring stays as-is.
GET /api/v1/gpt/omnichannel/agent_scorecards (batch)	frontend_service	reused	agent_scorecards.rb:154-169 — unchanged in P2 (P3 report revisits).
gpt_service mirror GET /v1/omnichannel/agent_scorecards/:room_id	gpt_service	extended	app/api/gpt_service/v1/omnichannel/agent_scorecard.rb (GET only, L18) — same Get use case, gains the same fields for parity.

Source Verification

Claim	Evidence (file:line / identifier)
Live human auto-scorer exists; picks the earliest active Models::AgentParticipant ("first agent")	chatbot/app/core/use_cases/gpt/omnichannel/auto_agent_scoring.rb:63-70 (_fetch_participants); entry #result L12-59
It is triggered from the room-resolve webhook and skipped when custom params or an existing room scorecard exist	chatbot/app/core/use_cases/api/internal_service/v1/webhook/room_resolve_interactions.rb:48-63 (unless is_custom_parameter || scorecard_exists, features include 'auto_agent_scoring')
agent_scorecards columns incl. is_use_ai (default false), is_ai_failed, edit_count, correction_at, correction_by*, remarks	chatbot/db/chatbot_gpt_schema.rb:93-125 (edit_count L111, correction_at L116)
Per-actor unique key already exists: (organization_id, room_id, agent_id) where deleted_at IS NULL	chatbot/db/chatbot_gpt_schema.rb:124
agent_scorecard_details columns incl. scorecard_parameter_code, scorecard_custom_parameter_id, is_custom_parameter, score_value, weighted_score_value	chatbot/db/chatbot_gpt_schema.rb:52-91
No judge_reason / veto / segment / snapshot / actor_type / scoring_status fields exist today	same schema ranges — columns absent (net-new per PRD §7 "Net-new build scope")
Both models are acts_as_paranoid + has_paper_trail	chatbot/app/models/chatbot_gpt/agent_scorecard.rb:4-5; agent_scorecard_detail.rb:4-5
Room GET is single-record find_by(org, room_id) (actor-blind)	chatbot/app/core/repositories/gpt/agent_scorecards/get_scorecard.rb:16
Grape roles: POST/PATCH owner/supervisor/admin; GET also member/agent	chatbot/app/api/frontend_service/v1/gpt/omnichannel/agent_scorecards.rb:45-62, 99-116, 127-143
Transactional scorecard+details write + passing_grade default fill	chatbot/app/core/repositories/gpt/agent_scorecards/calculate_agent_score.rb:21-34
LLM call pattern with timeout/retry + rescue-all + logging	chatbot/app/core/repositories/gpt/completion.rb:17,43-52,61-77,81-107; direct-client variant auto_agent_scoring.rb:160-196 (request_timeout: 240, max_attempts = 2)
Sidekiq worker + queue convention	chatbot/app/workers/auto_agent_scoring_worker.rb (retry: 1, queue: :gpt_agent_scoring)
sidekiq-cron + nightly cleanup precedent	chatbot/Gemfile:170 (gem 'sidekiq-cron'); config/schedule.yml (delete_old_logs_job, cron 0 1 * * * Asia/Jakarta, DeleteOldLogsWorker)
AI-Service inbound contract fields available	chatbot/app/api/internal_service/v1/ai_agent/use_cases/process_async_reply.rb:38-51 (vendor_ai_agent_id, room_id, status, output_text, references, exit, token_usage); resolve detection L180-183 (qontak_chat_room_resolve)
No evaluator/9-metric ingestion exists anywhere	grep evaluator|metric|rubric|judge|confidence over app/api/frontend_service/v2/ai_agent/, app/api/internal_service/v1/ai_agent/, app/ — only NLP intent-confidence hits (wit_ai/get_intent_v2.rb, qontak_nlp/predict.rb)
No handover/takeover model or events exist in chatbot BE	grep agent_take_room|remove_agent|handover|takeover over app/**/*.rb — only AI-assist prompt text hits
Migration dialect: Postgres, chatbot_gpt DB, ActiveRecord::Migration[7.1]	db/chatbot_gpt_schema.rb:15-16 (pgcrypto); db/chatbot_gpt_migrate/20241205085125_add_sequence_to_agent_scorecard_detail_reasons.rb
BE test/lint commands	.rspec; bitbucket-pipelines.yml:78 (RAILS_ENV=test bundle exec rspec …), :379 (bundle exec rubocop); Gemfile L77-91 (brakeman/reek/fasterer)
Feature-flag mechanism (BE) chosen in Phase 1	Phase 1 RFC ADR-5; chatbot/app/core/repositories/organization_features/find_feature.rb:4-22
hub-chat panel: binary rating, single-agent assumption, one-edit guard	ScorecardForm.vue:443-451 (score_value = isPass ? 100 : 0); AgentScorecard.vue:187 (roomParticipant[0]), :124-131 (edit_count === null guard), :104-114 ("edit 1 time" warning)
hub-chat scorecard API client URLs (GET/POST/PATCH same path)	useAgentScorecard.ts:7-48 (${chatbotServiceUrl}api/v1/gpt/omnichannel/agent_scorecards/${roomId})
is_use_ai / is_ai_failed typed but unconsumed in-room	@types/AgentScorecardTypes.ts:39,45; grep — type declarations only
Participants are human-only (type: ["agent"]; Models::User/customer types; no bot type)	ChatRoomView.vue:468-487 (filter L472); RoomDetailTypes.ts:142-153; wa-call/@types/index.ts:147-153; grep Models::Bot|"ai"|"bot" participant type — absent
Handover/actor events exist FE-side (payloads incl. actor,agent)	common/composables/push-notification/useEventHandler.ts:118-156; common/types/eventNotificationTypes.ts:250-308 (AgentHandoverRoomExtraNotification L267-283)
Usman scorecard permissions are org-wide; panel gating logic	common/store/UsmanStore.ts:14-16, 67-102; AgentScorecard.vue:208-226; SwitchBar.vue:156-164
Panel mount + tab switching	RoomDetailsView.vue:134-144, 342-346, 903, 1169-1183; SwitchBar.vue:40-48, 95-100
FE flag precedent (org setting ∥ subscription code)	SwitchBar.vue:156-164; ChatBotStore.ts:167-172 (checkSubscription)
ai_qa_unified_scorecard absent in hub-chat today	grep ai_qa_unified_scorecard|Auto-scored|actorSelector over *.ts,*.vue — 0 hits
FE test/lint commands	hub-chat/package.json scripts: vitest --dom --pool=forks, eslint --ext ".ts,.vue" … + prettier --check ., vue-tsc --noEmit; pnpm

Detail 2.1 — Infrastructure Topology

flowchart TB
  user(["QA Lead / Supervisor (web inbox)"])
  lb["LB / Ingress"]
  fe["hub-chat pods (Nuxt 4 inbox)"]
  api["chatbot pods (Puma · Grape frontend_service + internal_service)"]
  hub["hub-service /users/me + rooms API (auth + room detail)"]
  ais["AI Service (DSAI) — SkillPack engine + evaluator"]
  pg[("Postgres — chatbot_gpt DB<br/>agent_scorecards / details / snapshots")]
  redis[("Redis (Sidekiq queues)")]
  skq["Sidekiq workers<br/>gpt_agent_scoring + maintenance"]
  cron["sidekiq-cron (schedule.yml)"]
  oai["OpenAI (external) — tier-2 judge"]
  mp["Mixpanel (external)"]

  user --> lb --> fe -->|"GET/PATCH agent_scorecards (Bearer + X-Auth)"| lb
  lb --> api
  api -->|"validate token / room detail"| hub
  ais -->|"POST /v1/ai_agent/evaluations (async push)"| lb
  api -->|"enqueue"| redis --> skq
  skq -->|"tier-2 judge (timeout 240s, 2 attempts)"| oai
  skq -->|"write scores + snapshot"| pg
  api -->|"read scores"| pg
  cron --> skq
  skq -.->|"events"| mp
  fe -.->|"track panel events"| mp

Per-service responsibilities

Service	Use cases (this RFC)	Internal calls (owner)	External APIs
AI Service (DSAI)	Emit per-conversation 9-metric evaluator result at terminal exit/handoff (PROPOSED contract)	chatbot POST /v1/ai_agent/evaluations (BOT)	—
chatbot (Grape + Sidekiq)	Ingest evaluation; tier-1/tier-2/veto scoring; persist per-actor records + snapshots; serve extended GET; AI-override PATCH; retention cleanup; feature-gate	hub-service /users/me + rooms (Core); Mixpanel worker	OpenAI (tier-2 judge)
hub-chat	Render AI mode + actor selector + override; Usman/flag gating; fire panel events	chatbot frontend_service (BOT)	Mixpanel (browser)
chatbot_gpt DB	Store extended agent_scorecards/agent_scorecard_details + new agent_scorecard_snapshots (+ paper_trail versions)	—	—

Detail 2.1a — Sequence Diagrams (happy + failure paths)

S01 — Ingest + score (happy path, tier-2 judge, veto)

sequenceDiagram
  participant AIS as AI Service (evaluator)
  participant LB as LB
  participant API as chatbot internal_service
  participant Q as Redis (gpt_agent_scoring)
  participant W as Sidekiq AiAgentScoringWorker
  participant DB as chatbot_gpt (Postgres)
  participant OAI as OpenAI (tier-2 judge)
  participant MP as Mixpanel (async)
  AIS->>LB: POST /v1/ai_agent/evaluations (9 metrics + segment + transcript)
  LB->>API: forward
  API->>API: contract validate + flag & is_ai_auto_score gate
  API->>Q: enqueue payload
  API-->>AIS: 202 accepted
  W->>Q: dequeue
  W->>DB: read Phase 1 config (ai_passing_grade, custom params with prompt)
  W->>W: tier-1 weighted score (uniform 0.11, metric codes from default rubric)
  alt custom params with non-empty rubric exist
    W->>OAI: one Completion call scoring ALL rubric'd params (timeout 240s, 2 attempts)
    OAI-->>W: per-param scores + reasons
  else none
    Note over W: tier-2 skipped — tier-1 only
  end
  W->>W: veto check (groundedness / policy) — failure floors is_pass
  W->>DB: transactional insert - scorecard (actor_type ai_agent) + details + snapshot
  W-)MP: scorecard_autoscore_completed

S01-F — Ingest failure paths (malformed payload / judge failure)

sequenceDiagram
  participant AIS as AI Service
  participant API as chatbot internal_service
  participant W as AiAgentScoringWorker
  participant DB as chatbot_gpt
  participant OAI as OpenAI
  participant MP as Mixpanel
  AIS->>API: POST evaluations (malformed - missing metrics)
  API-->>AIS: 422 contract failure (not enqueued)
  Note over API: room untouched — live handling never blocked
  AIS->>API: POST evaluations (valid envelope, bad metric codes)
  API->>W: enqueue
  W->>W: metric codes unknown vs default rubric
  W->>DB: upsert record scoring_status "unavailable" + failed_payload snapshot (7d TTL)
  W-)MP: scorecard_autoscore_failed (reason, retry_count)
  Note over W: Sidekiq retry (retry 1) then dead — record stays "unavailable"
  AIS->>API: POST evaluations (valid, tier-2 org)
  API->>W: enqueue
  W->>OAI: tier-2 judge call
  OAI--xW: timeout / error after 2 attempts
  W->>DB: persist tier-1 details, tier-2 params "unscored", scoring_status "partial"
  W-)MP: scorecard_tier2_judge_failed (custom_param_id, reason)

S02 — Panel load + actor select (happy + unavailable)

sequenceDiagram
  participant U as QA Lead
  participant FE as hub-chat panel
  participant LB as LB
  participant API as chatbot frontend_service
  participant HUB as hub-service
  participant DB as chatbot_gpt
  U->>FE: open Scorecard tab
  FE->>LB: GET agent_scorecards/:room_id (no agent_id)
  LB->>API: forward
  API->>HUB: validate token + room exists
  API->>DB: read records for room (all actors)
  DB-->>API: human record + actors[] summary
  API-->>FE: 200 human record (today's shape) + actors[]
  FE-->>U: manual mode + actor selector (AI + human)
  U->>FE: select AI actor
  FE->>API: GET agent_scorecards/:room_id?agent_id=AI_AGENT_ID
  API->>DB: read AI record + details + snapshot
  alt scoring_status completed or partial
    API-->>FE: 200 graded metrics + reasons + sources + veto + total
    FE-->>U: "Auto-scored by AI" mode
  else scoring_status unavailable or pending
    API-->>FE: 200 record with scoring_status
    FE-->>U: "scoring unavailable" + retry indicator (no crash)
  end

S03 — Override (happy + failure)

sequenceDiagram
  participant SPV as Supervisor
  participant FE as hub-chat panel
  participant API as chatbot frontend_service
  participant DB as chatbot_gpt
  participant MP as Mixpanel
  SPV->>FE: edit metric score (in range) + reason
  FE->>API: PATCH agent_scorecards/:room_id {agent_id, overrides[]}
  API->>API: role owner/admin/supervisor + 0-100 validation
  alt edit_count already consumed
    API-->>FE: 422 "already corrected"
  else slot free
    API->>DB: tx - update detail score_value, recompute total + is_pass (veto still floors), set edit_count=1 + correction_* (paper_trail)
    alt save fails
      API-->>FE: 500, no partial state (tx rollback)
      FE-)MP: scorecard_override_failed
      FE-->>SPV: error toast + Retry
    else ok
      API-->>FE: 200 recomputed record
      FE-)MP: scorecard_override_saved
      FE-->>SPV: "edited by [SPV]" chip + new total
    end
  end

Detail 2.1b — Branch Flowcharts

Tier-2 rubric gate + veto (P2-S01/AC-2,4,5):

flowchart TD
  A["Evaluation ingested (9 tier-1 metrics)"] --> B{"org custom params<br/>with non-empty prompt?"}
  B -->|Yes| C["One judge call scores all rubric'd params"]
  B -->|No| D["Tier-2 skipped - params marked manual-only"]
  C --> E{"judge call succeeded?"}
  E -->|No| F["params unscored - scoring_status = partial"]
  E -->|Yes| G[Merge tier-2 into total]
  D --> H{"veto metric failed?<br/>(groundedness / policy)"}
  F --> H
  G --> H
  H -->|Yes| I["is_pass = false + is_veto_failed + veto_reason"]
  H -->|No| J["is_pass = total >= ai_passing_grade"]
  I --> K[Persist per actor]
  J --> K

Detail 2.2 — Technical Decisions (ADR-format)

ADR-1 — Ingest via a new internal push endpoint, not the per-message async reply

• Context. The evaluator result is per conversation, emitted at terminal exit/handoff. Verified: no ingestion path exists; the only AI-Service→chatbot inbound is process_async_reply.rb, which is per message (output_text/references/exit, contract L38-51).
• Options.
  • A. New POST /v1/ai_agent/evaluations (internal_service), enqueue → Sidekiq. Clean conversation-level contract; 202-and-forget keeps scoring off the reply path (ERR-1); idempotent on the record key. Con: DSAI must add one push call.
  • B. Extend the async-reply payload with an optional evaluation object — Con: couples scoring to message delivery; the final reply may precede evaluator completion; bloats a hot path; violates "never blocks live handling".
  • C. Chatbot polls the AI Service for results — Con: Strategy doc lists the read APIs (GET /models, /thread/message) as OPEN/unconfirmed; polling adds latency vs the 60s budget and a scheduler burden.
• Decision. Option A.
• Rationale. Matches the existing inbound-webhook shape (room_resolve_interactions precedent), keeps the 60s budget (enqueue ≈ instant), and gives DSAI a single, explicit contract to confirm (Open Q#2).
• Consequences. New Grape resource + contract + worker; the payload is the PROPOSED contract in §2.4 Inbound row 1; DSAI confirmation gates org enablement, not the build.
• Reversibility. High — endpoint is additive; switch to B/C later without schema change.

ADR-2 — The AI actor is a scorecard record, not a room participant

• Context. PRD Open Q#6: verified the AI agent is not a participant — hub-chat fetches type: ["agent"] humans only; there is no bot participant type; the panel assumes roomParticipant[0].
• Options.
  • A. Attach the AI score to (org, room_id, agent_id = chatbot AI-agent id) with a new actor_type column; FE builds the actor selector from the GET's actors[]. Con: the selector's actor list comes from scorecard data, so the AI actor appears only once scored (acceptable: "Not scored yet" empty state covers the gap).
  • B. Introduce a bot participant type in hub/hub-core — Con: cross-squad platform change to participants, presence, and events; far outside this initiative; PRD's own mitigation says attach to room + agent.
• Decision. Option A (PRD Q6 mitigation adopted).
• Rationale. The unique index (org, room_id, agent_id) already supports multiple records per room (schema L124) — zero platform coupling; S01/AC-6's "separate record" falls out of the existing key.
• Consequences. actor_type column ('human' default, 'ai_agent'); GET grows actors[]; human flows keep resolving via roomParticipant.
• Reversibility. Medium — if a bot participant type lands platform-wide later, the FE selector source can switch; records don't move.

ADR-3 — Extend the existing scorecard tables + one snapshot table; no parallel AI tables

• Context. PRD §7 lists net-new fields; the panel, batch API, paper_trail, and Phase 3 report all read agent_scorecards/agent_scorecard_details.
• Options.
  • A. Add columns to both tables + new agent_scorecard_snapshots for the 90-day payload. Con: nullable columns on manual rows (documented).
  • B. New ai_agent_scorecards tables — Con: duplicates the panel/report read path, the override/audit machinery, and the unique-key semantics; two sources of truth for "the room's quality".
• Decision. Option A. Snapshot data (transcript, cited sources, failed payloads) goes to a separate table because its retention (90d/7d) differs from the score record's (13 months) — TTL cleanup must not touch scores (§7.1).
• Rationale. One quality record per actor per room, one read path, existing audit.
• Consequences. 3 migrations (§2.3); details rows for AI metrics use scorecard_parameter_code = metric code with no scorecard_parameter_id (no seeded parameter rows — the 9 metrics stay owned by Phase 1's constant; scorecard_parameter_id is already optional on the detail model).
• Reversibility. Medium — columns are additive and nullable; snapshot table droppable.

ADR-4 — Segment boundary comes from the evaluator payload

• Context. PRD Open Q#3 proposed delimiting segments via hub-chat handover events — but those events exist only in the FE/hub layer (useEventHandler.ts); verified: no handover model/table/event exists in chatbot BE. The engine, however, knows exactly which turns it handled.
• Options.
  • A. Require segment in the evaluation payload (evaluator-owned truth); tolerate absence by scoring whole-conversation with segment.source = "missing".
  • B. Derive the segment in chatbot BE from its own AI reply history + resolve detection — Con: chatbot sees its outbound AI messages but not the human agent's takeover moment (no handover signal BE-side); boundary would be a guess exactly where precision matters.
  • C. Subscribe chatbot BE to hub handover events — Con: new cross-squad event integration (Omnichannel/Core) for a value the evaluator already has.
• Decision. Option A — segment is part of the PROPOSED contract; Internal Alpha validates it against the 15-CID transcripts (PRD launch gate).
• Rationale. Single source of truth co-located with the scoring input; keeps the cross-squad surface at one contract.
• Consequences. segment jsonb persisted verbatim + source tag; panel shows the scored span; missing-segment records are measurable (segment.source) for the alpha gate.
• Reversibility. High — fallback derivation can be added later without contract change.

ADR-5 — Tier-2 judge: one Completion call for all rubric'd params

• Context. PRD decision (S16): "one judge call per parameter multiplies token cost; score all params in one call." The repo has a ready judge-call repository.
• Options. A. One Repositories::Gpt::Completion call (gpt_timeout: 240, worker-level retry) with a system prompt assembling every non-empty prompt rubric and a JSON-scores response format. B. Per-param calls — rejected by the PRD (cost). C. Send tier-2 rubrics to the DSAI evaluator — Con: makes org config part of the DSAI contract, which is exactly the drift the anchor decision avoids.
• Decision. Option A.
• Rationale. Matches the live scorer's proven single-call pattern (auto_agent_scoring.rb scores all manual params in one call today); Completion returns {success: false} instead of raising, mapping cleanly to partial.
• Consequences. Judge failure after attempts → affected params unscored, scoring_status: "partial", tier-1 still stored (ERR-2). Missing judge input: if the payload carries no transcript while the org has rubric'd params, tier-2 is skipped the same way — params unscored, record partial with reason no_transcript, tier-1 unaffected (no fetch-from-elsewhere fallback in P2). Stored rubric text is treated as untrusted input in prompt assembly (Phase 1 §3 forward note honored: delimit rubric content, never interpolate into instructions).
• Reversibility. High.

ADR-6 — Uniform weights as a versioned constant; total = weighted mean

• Context. Weights are untuned (uniform 0.11, DSAI tunes later — PRD Non-Goal #5, Open Q#4). The PRD does not fix the tier-2 blend.
• Options. A. AI_TIER1_WEIGHT = 0.11 constant; tier-2 params join at the same uniform weight; total = Σ(wᵢ·sᵢ)/Σwᵢ (a weighted mean, robust to param count). B. Store per-metric weights in DB now — Con: premature config surface for values DSAI will replace; Phase 5 owns tuned weights. C. Tier-2 outside the total (display-only) — Con: contradicts P2-S01/AC-2 "merges into the total".
• Decision. Option A, versioned behind ai_qa_unified_scorecard_v2_weights (PRD §11.1): the flag selects the weights version at scoring time; the record stores the version used (agent_scorecards.weights_version — on the 13-month record, not the 90-day snapshot) so agreement metrics compare like-for-like across the full retention window.
• Rationale. Simplest faithful reading of the PRD; normalization keeps totals in 0–100 regardless of how many tier-2 params an org adds.
• Consequences. Weight change = constant + flag flip, no migration; rollback = flag off (PRD §11.1 path, no deploy).
• Reversibility. High.

ADR-7 — Override consumes the existing one-edit correction slot

• Context. PRD decision + S03/AC-4. Grounded: the slot is edit_count/correction_at on agent_scorecards; hub-chat renders the edit button only when edit_count === null (AgentScorecard.vue:124-131) and warns "You can only edit scorecard 1 time" (L104-114).
• Options. A. The AI-override PATCH sets edit_count = 1 + correction_at/by* on the AI record; one override submission may change multiple metrics (it is one correction event); further overrides 422 "already corrected". B. A new per-metric edit ledger — Con: diverges from the established audit model the PRD explicitly reuses.
• Decision. Option A (per-record slot, matching existing semantics; paper_trail keeps the field-level diff).
• Rationale. Keeps one audit model across human and AI records; FE guard logic is reused as-is.
• Consequences. The correction event stores per-metric override_reason in the detail row (new nullable column not needed — reason goes into the existing detail bad_reason_description? No — that field has bad-rating semantics; add override_reason string to details, see §2.3); "edited by [SPV]" renders from correction_by_name.
• Reversibility. High.

ADR-8 — Make the room-resolve skip guard actor-aware

• Context. Phase 1 RFC forward note: room_resolve_interactions.rb:48-63 skips AutoAgentScoringWorker unless is_custom_parameter || scorecard_exists. With Phase 2, an AI-actor record created at terminal exit would make scorecard_exists true and silently suppress human auto-scoring on resolve (and a pre-existing human record is irrelevant to AI scoring).
• Options. A. Scope both existence checks by actor: human auto-scoring skips only if a actor_type: 'human' record exists for the room; AI ingestion upserts only its own (org, room, ai_agent_id) record. B. Leave the guard — Con: regression on the live human path, violating "No change to human manual scoring" (PRD Non-Goal #6).
• Decision. Option A (no alternative seriously considered beyond the fix — the regression is objective).
• Consequences. One-line-ish change in the webhook use case + repo query; regression spec required (§4.C chunk 7).
• Reversibility. High.

ADR-9 — Retention via sidekiq-cron nightly workers

• Context. PRD §7.1: snapshots 90d, AI score records 13 months, failed payloads 7d. Grounded precedent: sidekiq-cron (Gemfile L170) + config/schedule.yml (delete_old_logs_job, nightly 01:00 Asia/Jakarta).
• Options. A. Two workers: DeleteExpiredScorecardSnapshotsWorker (nightly: expires_at < now — covers both 90d snapshots and 7d failed payloads via per-row expires_at) and DeleteOldAiScorecardsWorker (nightly: AI-actor records created_at < 13.months.ago, paranoid-destroy). B. Postgres pg_cron/partitioning — Con: no precedent in the repo; ops surface.
• Decision. Option A, queue application_maintenance (existing precedent).
• Consequences. Two schedule.yml entries; deletes are batched (in_batches) to bound lock time; S02/AC-5 "transcript expired" falls out of the snapshot row's absence.
• Reversibility. High.

ADR-10 — RBAC stays org-wide Usman permissions (team scoping descoped)

• Context. PRD Open Q#7: "team scope" doesn't exist — Usman inbox_scorecard_view / inbox_scorecard_manage are org-wide (verified UsmanStore.ts:14-16).
• Options. A. Descope to org-wide for Phase 2: view = existing panel gating; override = inbox_scorecard_manage (FE) + Grape roles owner/admin/supervisor (BE — the PATCH's existing role set). B. Build per-team scoping — Con: net-new platform RBAC feature, cross-squad (Usman/Platform), not a P2 deliverable.
• Decision. Option A; per-team scoping recorded as a follow-up (§5 #6).
• Reversibility. High — additive later.

ADR-11 — Feature gate reuses ai_qa_unified_scorecard (both layers)

• Context. Phase 1 ADR-5 established the org-feature (BE OrganizationFeatures::FindFeature). hub-chat's scorecard tab is already gated by organizationSettings?.chatbot_score_card ∥ checkSubscription(code) + Usman (SwitchBar.vue:156-164); checkSubscription reads chatbot-served subscription features (ChatBotStore.ts:167-172).
• Decision. BE: scoring pipeline + extended GET/PATCH AI fields gate on OrganizationFeatures::FindFeature(feature_code: 'ai_qa_unified_scorecard') and scorecard_preferences.is_ai_auto_score. FE: AI mode + actor selector render only when checkSubscription("ai_qa_unified_scorecard") (provisioned with the org feature) — falling back dark: flag off ⇒ panel behaves exactly as today. (no alternative considered — mechanism decided in Phase 1 ADR-5; this ADR only extends it to hub-chat's existing gating idiom).
• Consequences. One provisioning row serves both layers; Stage gating = provisioning.
• Reversibility. High — toggle off restores today's panel.

Minimum-coverage checklist

• Storage — chatbot_gpt Postgres; extend 2 tables + 1 new snapshot table (ADR-3).
• Sync vs async — ingestion/scoring fully async via Sidekiq (ADR-1); GET/PATCH sync; analytics async.
• Caching — n/a — panel GET is a single indexed read (unique key); no cache layer added; 2s P95 met by query shape (§3).
• Third-party — OpenAI via existing Repositories::Gpt::Completion (ADR-5); Mixpanel via existing worker.
• Consistency — strong within a record (single transaction, scorecard+details+snapshot); AI and human records independent by key (ADR-2, ADR-8).
• Multi-tenancy — organization_id from the validated token (FE surface) / from the evaluation payload validated against the room's org (internal surface); never trusted from the FE body (§3 Security).
• Reuse vs new — 1 new inbound endpoint (justified), 2 extended, 2 reused (Existing-Endpoint Check).

Detail 2.3 — Database Model

erDiagram
  AGENT_SCORECARDS ||--o{ AGENT_SCORECARD_DETAILS : has
  AGENT_SCORECARDS ||--o| AGENT_SCORECARD_SNAPSHOTS : has
  AGENT_SCORECARDS {
    bigint id PK
    string organization_id "UK (org+room+agent) where deleted_at IS NULL"
    string room_id
    string agent_id "human agent id OR chatbot AI-agent id"
    string actor_type "NEW - human (default) or ai_agent"
    string scoring_status "NEW nullable - pending/completed/partial/unavailable"
    boolean is_veto_failed "NEW default false"
    string veto_reason "NEW nullable"
    jsonb segment "NEW - from/to message ids + turns + source"
    datetime evaluated_at "NEW - evaluator emission time, idempotency guard"
    string weights_version "NEW - weights flag version used (13-mo audit)"
    boolean is_use_ai "existing - auto-scored marker"
    boolean is_ai_failed "existing"
    float passing_grade "existing - AI rows filled from ai_passing_grade"
    boolean is_pass "existing"
    float score_percentage "existing"
    integer edit_count "existing - one-edit slot"
    datetime correction_at "existing"
  }
  AGENT_SCORECARD_DETAILS {
    bigint id PK
    bigint agent_scorecard_id FK
    string scorecard_parameter_code "tier-1 - the 9 metric codes"
    uuid scorecard_custom_parameter_id "tier-2 - existing"
    boolean is_custom_parameter "existing"
    float score_value "existing - graded 0-100 for AI rows"
    text judge_reason "NEW nullable"
    boolean is_veto "NEW default false"
    string override_reason "NEW nullable"
    string scoring_state "NEW nullable - scored/unscored/manual_only/overridden"
  }
  AGENT_SCORECARD_SNAPSHOTS {
    uuid id PK
    bigint agent_scorecard_id FK
    string organization_id
    jsonb transcript "judge input - 90d TTL"
    jsonb cited_sources "KB chunk refs - 90d TTL"
    jsonb failed_payload "raw payload on failure - 7d TTL"
    datetime expires_at "cleanup key"
  }

DDL (Rails DSL, chatbot_gpt connection — pattern: db/chatbot_gpt_migrate/20241205085125_*):

# db/chatbot_gpt_migrate/<ts>_add_ai_actor_fields_to_agent_scorecards.rb
class AddAiActorFieldsToAgentScorecards < ActiveRecord::Migration[7.1]
  def change
    add_column :agent_scorecards, :actor_type, :string, null: false, default: 'human'
    add_column :agent_scorecards, :scoring_status, :string
    add_column :agent_scorecards, :is_veto_failed, :boolean, null: false, default: false
    add_column :agent_scorecards, :veto_reason, :string
    add_column :agent_scorecards, :segment, :jsonb
    add_column :agent_scorecards, :evaluated_at, :datetime
    add_column :agent_scorecards, :weights_version, :string
    add_index  :agent_scorecards, [:organization_id, :actor_type]
  end
end

# db/chatbot_gpt_migrate/<ts+1>_add_ai_fields_to_agent_scorecard_details.rb
class AddAiFieldsToAgentScorecardDetails < ActiveRecord::Migration[7.1]
  def change
    add_column :agent_scorecard_details, :judge_reason, :text
    add_column :agent_scorecard_details, :is_veto, :boolean, null: false, default: false
    add_column :agent_scorecard_details, :override_reason, :string
    add_column :agent_scorecard_details, :scoring_state, :string
  end
end

# db/chatbot_gpt_migrate/<ts+2>_create_agent_scorecard_snapshots.rb
class CreateAgentScorecardSnapshots < ActiveRecord::Migration[7.1]
  def change
    create_table :agent_scorecard_snapshots, id: :uuid do |t|
      t.bigint   :agent_scorecard_id, null: false
      t.string   :organization_id,    null: false
      t.jsonb    :transcript
      t.jsonb    :cited_sources
      t.jsonb    :failed_payload
      t.datetime :expires_at,         null: false
      t.datetime :deleted_at
      t.timestamps
    end
    add_index :agent_scorecard_snapshots, :agent_scorecard_id
    add_index :agent_scorecard_snapshots, :expires_at
  end
end

No backfill: existing rows get actor_type: 'human' via the column default — which is the truthful value for every record that exists today (verified: only human records are ever created). Regenerate db/chatbot_gpt_schema.rb after migrating.

Per-status lifecycle (scoring_status — the one new status enum):

stateDiagram-v2
  [*] --> pending : evaluation enqueued
  pending --> completed : tier-1 (+ tier-2) scored
  pending --> partial : tier-2 judge failed after attempts
  pending --> unavailable : payload invalid / scoring error
  unavailable --> pending : Sidekiq retry / re-push
  completed --> completed : override (status unchanged)
  partial --> partial : override (status unchanged)

Status	Visible as	Retention	Restore	Transitions out
null (manual/human rows)	today's manual scorecard	existing behavior (no TTL)	n/a	none — human rows never enter the AI lifecycle
pending	"scoring…" skeleton	until scored or retries exhausted	n/a	→ completed / partial / unavailable
completed	graded metrics + total + badge	13 months (AI rows)	n/a	none (override edits values, not status)
partial	graded tier-1 + "unscored" tier-2 chips	13 months	re-push may complete it	→ completed (idempotent re-ingest)
unavailable	"scoring unavailable" + retry indicator	13 months (record) / 7 days (failed_payload)	retry → pending	→ pending

State Surface Contract:

Entity	Surfaced to	Field(s)	Visibility	Audit
agent_scorecards (AI rows)	in-room panel (GET)	actor_type, scoring_status, is_veto_failed, veto_reason, segment, is_pass, score_percentage, edit_count, correction_by_name	Usman scorecard perms + flag on	paper_trail (existing)
agent_scorecard_details (AI rows)	panel metric rows	score_value, judge_reason, is_veto, override_reason, scoring_state	same	paper_trail (existing)
agent_scorecard_snapshots	panel (transcript/source links)	transcript, cited_sources, presence → transcript_available	same	n/a — TTL'd evidence, not config
actor list	panel actor selector	actors[] {agent_id, actor_type, display_name, is_pass, scoring_status}	same	n/a — derived

Detail 2.4 — APIs

Auth (frontend_service rows): Bearer + X-Auth-Token via middlewares/auth.rb → hub-service /users/me (unchanged). Auth (internal row): the internal_service surface's existing service-to-service auth, matching process_async_reply / room_resolve_interactions (same mount, internal_service/api.rb).

Inbound Row 1 — new-with-justification — Evaluation ingest (AI Service → chatbot)

POST /v1/ai_agent/evaluations          # internal_service; flag + is_ai_auto_score gated

Request (PROPOSED — pending DSAI confirmation, Open Q#2):

{
  "organization_id": "org-uuid",
  "room_id": "channel-room-id",
  "vendor_ai_agent_id": "ai-agent-uuid",        // becomes agent_scorecards.agent_id
  "conversation_id": "conv-uuid",
  "exit_reason": "skill_completed",             // free-form engine exit id (PRD B1)
  "evaluated_at": "2026-07-17T08:00:00Z",
  "segment": {                                   // ADR-4; may be absent → source:"missing"
    "from_message_id": "msg-1", "to_message_id": "msg-14",
    "from_turn": 1, "to_turn": 14
  },
  "metrics": [                                   // exactly the 9 default-rubric codes
    { "code": "groundedness", "score": 82.0, "veto_failed": false,
      "reason": "All claims cite KB chunks …",
      "sources": [ { "kb_chunk_id": "kb-123", "title": "Refund policy" } ] },
    { "code": "policy", "score": 95.0, "veto_failed": false, "reason": "…" }
    // … 7 more
  ],
  "transcript": [ { "turn": 1, "role": "customer", "text": "…" } ]  // judge input, 90d TTL
}

Responses: 202 { "message": "accepted" } (enqueued); 422 contract failure (unknown org/room, missing metrics array, non-numeric score); 200 { "message": "skipped" } when flag or is_ai_auto_score is off for the org (not an error — engine needn't care). Idempotency: re-push upserts the same (org, room_id, agent_id) record (a completed record is only re-scored if the new payload's evaluated_at is newer than the stored agent_scorecards.evaluated_at — this re-score wins over a prior supervisor override, whose values remain recoverable via paper_trail); duplicates never create a second row (unique index). Tenancy/room validation runs synchronously pre-enqueue against chatbot's own DB (resolve chatbot_room by room_id and compare its organization — the process_async_reply context pattern, its L25-35; no hub call on the internal surface): unknown room or org mismatch → 422. transcript absent while the org has rubric'd tier-2 params → tier-2 skipped as partial/no_transcript (ADR-5). Timeout/retry (caller side): AI Service should treat non-2xx as retryable; chatbot-side worker retry retry: 1 + unavailable terminal state (ERR-1).

Outbound Row 1 — extended — Per-room scorecard read

GET /api/v1/gpt/omnichannel/agent_scorecards/:room_id?agent_id=<id>   # roles: owner|admin|supervisor|member|agent (existing)

• No agent_id (today's callers): returns the human record exactly as today (actor-aware default — ADR-8), plus the new actors[] summary. If the room has no human record (e.g. AI-only conversation), the no-param GET returns today's default template response (existing _build_default_agent_scorecard fallback, get_scorecard.rb) plus actors[] — the panel shows the manual empty state and the selector reveals the AI actor.
• agent_id present: returns that actor's record (AI or human).

Response 200 (AI actor selected — additions shown):

{
  "data": {
    "id": 123, "room_id": "…", "agent_id": "ai-agent-uuid",
    "actor_type": "ai_agent", "is_use_ai": true, "scoring_status": "completed",
    "is_veto_failed": false, "veto_reason": null,
    "segment": { "from_turn": 1, "to_turn": 14, "source": "evaluator" },
    "passing_grade": 80.0, "score_percentage": 86.4, "is_pass": true,
    "evaluated_at": "2026-07-17T08:00:00Z", "weights_version": "v1",
    "edit_count": null, "correction_by_name": null,
    "transcript_available": true,
    "actors": [
      { "agent_id": "ai-agent-uuid", "actor_type": "ai_agent", "display_name": "Billing Bot",
        "scoring_status": "completed", "is_pass": true },
      { "agent_id": "user-42", "actor_type": "human", "display_name": "Rina",
        "scoring_status": null, "is_pass": false }
    ],
    "groups": [
      { "code": "qontak_ai_quality_default", "name": "Qontak AI Quality (default)",
        "parameters": [
          { "code": "groundedness", "name": "Groundedness / factual accuracy",
            "score_value": 82.0, "is_veto": true, "scoring_state": "scored",
            "judge_reason": "All claims cite KB chunks …",
            "sources": [ { "kb_chunk_id": "kb-123", "title": "Refund policy" } ],
            "override_reason": null } /* … 8 more */ ] },
      { "code": "custom_org", "name": "Custom (org)",
        "parameters": [
          { "custom_parameter_id": "uuid", "name": "BANT capture",
            "score_value": 70.0, "scoring_state": "scored", "judge_reason": "…" },
          { "custom_parameter_id": "uuid2", "name": "Greeting",
            "scoring_state": "manual_only" } ] }
    ]
  },
  "message": "OK"
}

Errors: 404 room/record not found (existing message); 401/403 as today. Human-record response is byte-compatible with today's shape plus the additive actors[] / actor_type keys.

Outbound Row 2 — extended — AI-score override (write)

PATCH /api/v1/gpt/omnichannel/agent_scorecards/:room_id      # roles: owner|admin|supervisor (existing)

AI-override body variant (existing human categories[] body unchanged):

{
  "agent_id": "ai-agent-uuid",
  "overrides": [
    { "metric_code": "groundedness", "score_value": 90.0, "reason": "judge misread the KB citation" }
  ]
}

Response 200: the recomputed record (Row 1 shape). One PATCH = one correction event = the record's single edit slot (ADR-7). Each touched metric's detail row is marked scoring_state: "overridden" (+ optional override_reason) — the per-metric anchor for the FE "edited by [SPV]" chip (name from the record's correction_by_name). Errors: 422 score_value outside 0–100 → "Override score only between 0 - 100 are allowed"; 422 slot consumed → "Scorecard already corrected"; 422 unknown metric_code; 403 role; 500 save fail (tx rollback, no partial state) → FE logs scorecard_override_failed.

APIs (Inbound — other services → us): Row 1 above is the inbound table for this RFC (one row). The existing room-resolve webhook (/v1/room_resolve_interactions) is modified only in its skip-guard predicate (ADR-8) — no contract change.

Detail 2.A — UI Contract (FE — hub-chat)

Design status: Figma Pending (PRD CHG-003). Interim spec = PRD Appendix B Stitch prompt. Components use Pixel3 (Mp*) only; feature-folder co-location per hub-chat AGENTS.md. All files live under features/inbox/features/main-section/features/room-details/features/agent-scorecard/.

Component	File	Purpose	Key Pixel3 elements	Backing endpoint
AgentScorecard (extended)	views/AgentScorecard.vue	Mode switch: selected actor actor_type === 'ai_agent' → AI view; else today's manual flow untouched; hosts selector	existing	GET row 1
ActorSelector (new)	components/ActorSelector.vue	Lists actors[] (AI + human) with name + pass/fail dot; emits selection; hidden when only one actor	MpFlex/MpText/MpBadge (or MpSelect if >3 actors)	GET row 1 (actors[])
AiScorecardView (new)	components/AiScorecardView.vue	"Auto-scored by AI" badge; per-group metric rows: graded score chip + expandable judge_reason; Groundedness source links; red veto banner; total % + pass/fail badge vs passing_grade; "edited by [SPV]" chips; Remarks	MpTable*/MpBadge/MpIcon/MpTooltip/MpText/MpButton; toast for errors	GET row 1
Override control (in AiScorecardView)	(same file)	Per-metric pencil → inline numeric input (0–100) + reason; save → PATCH; gated inbox_scorecard_manage + slot free (edit_count === null, reusing L124-131 guard semantics)	MpInput/MpButton(is-loading)/MpPopover	PATCH row 2
useAgentScorecard (extended)	composables/useAgentScorecard.ts	getScorecardData(roomId, agentId?); new overrideAiScore({roomId, agentId, overrides})	—	GET/PATCH
Types (extended)	@types/AgentScorecardTypes.ts	ActorSummary, AiMetric, AiScorecardData, scoring_status union; wire the already-typed is_use_ai/is_ai_failed	—	—

Design ↔ Code Mapping: n/a — Figma pending; layout follows the existing panel shell (AgentScorecard.vue) + PRD Appendix B. Re-verify on frames (§5 #3).

Detail 2.B — Data-Fetching Strategy (FE)

• No new Pinia store: panel state stays component-local (refs), matching today's AgentScorecard.vue (scorecardData, isEdit, … L178-181). The selected-actor id is a local ref; switching actors re-fetches via the composable. roomParticipant (mainSectionStore) is not the actor source (ADR-2).
• useAgentScorecard.ts keeps the $customFetch + chatbotServiceUrl pattern; agent_id passed as a query param; response consumed snake_case end-to-end (no camelCase mapping — same rule as Phase 1 REV-6).

// @types/AgentScorecardTypes.ts additions (snake_case, mirrors BE entities)
export type ScoringStatus = "pending" | "completed" | "partial" | "unavailable";
export interface ActorSummary {
  agent_id: string; actor_type: "human" | "ai_agent"; display_name: string;
  scoring_status: ScoringStatus | null; is_pass: boolean;
}
export interface AiMetricSource { kb_chunk_id: string; title: string; }
export interface AiMetric {
  code: string; name: string; score_value: number | null;
  is_veto: boolean; scoring_state: "scored" | "unscored" | "manual_only" | "overridden";
  judge_reason: string | null; sources: AiMetricSource[]; override_reason: string | null;
}
export interface AiScorecardData /* extends existing ScorecardData fields */ {
  actor_type: "human" | "ai_agent"; scoring_status: ScoringStatus | null;
  is_veto_failed: boolean; veto_reason: string | null;
  segment: { from_turn: number; to_turn: number; source: "evaluator" | "missing" } | null;
  transcript_available: boolean; actors: ActorSummary[];
  groups: { code: string; name: string; parameters: AiMetric[] }[];
}

Detail 2.C — UI State Matrix

State	ActorSelector	AiScorecardView	Override control
Loading	skeleton row	"scoring…" skeleton metrics (also for scoring_status: pending)	hidden
Empty	hidden (single actor) / "Not scored yet" for unscored AI actor	"Not scored yet"	hidden
Error / unavailable	actors still listed	"scoring unavailable" + retry indicator; $mixpanel scorecard_panel_load_failed on hard load error	hidden
Partial	AI actor dot amber	tier-1 graded; tier-2 chips "unscored"	shown (tier-1 rows)
Success	AI + human listed, selected highlighted	graded groups + reasons + sources + veto + total/badge; transcript "expired" note when transcript_available: false	pencil per metric (manage perm + slot free)
Veto-failed	AI actor dot red	red "Failed" banner + veto metric flagged; total shows is_pass: false	shown

Detail 2.D — Scope Boundaries

In scope	Out of scope
Evaluation ingest endpoint + worker + scoring use case; 3 migrations; actor-aware guard fix; extended GET/PATCH (+gpt_service GET mirror); retention workers; OpenAPI; hub-chat ActorSelector + AI mode + override + types + flag gating; events; contract tests	Report/export (P3), harness (P4), gate (P5), weight tuning, bot-as-participant platform change, per-team RBAC, batch-GET changes, human manual UX changes, chatbot-fe changes (Phase 1 owns settings), mobile

Detail 2.E — Branch & Skip Catalog

Branch / skip	Condition	Behavior	Owner
Flag/pref off skip	org feature off or is_ai_auto_score false	ingest returns 200 "skipped", nothing scored; FE renders today's panel	BE + FE (ADR-11); S01 CAN/CANNOT
Human-conversation skip	no evaluation pushed (engine only evaluates AI conversations)	no AI record; NEG-1	DSAI (contract) + BE
Live-conversation skip	no terminal exit yet	no scoring (per-conversation only); NEG-2	DSAI + BE
Empty-rubric exclusion	tier-2 param prompt blank	param manual_only, excluded from total	BE (ADR-5); AC-4
Veto floor	groundedness/policy veto_failed	is_pass=false regardless of total	BE; AC-5
Partial downgrade	tier-2 judge fails after attempts	params unscored, record partial, tier-1 kept	BE (ADR-5); ERR-2
Unavailable + retry	payload invalid / scoring raises	record unavailable, Sidekiq retry, failed_payload kept 7d	BE (ADR-1/9); ERR-1
Missing-segment fallback	payload lacks segment	score whole conversation, segment.source: "missing" (alpha gate metric)	BE (ADR-4); Open Q#3
No-transcript tier-2 skip	payload lacks transcript + org has rubric'd params	tier-2 params unscored, record partial (no_transcript); tier-1 unaffected	BE (ADR-5); REV-2
Actor-aware guard	AI record exists at room resolve	human auto-scoring still runs (and vice versa)	BE (ADR-8)
Already-corrected block	edit_count non-null	override 422 "already corrected"	BE+FE (ADR-7); AC-4
Non-manage viewer	Usman view-only / agent role	override control not rendered; PATCH 403 by role	FE+BE (ADR-10); NEG-1 (S03)
Snapshot expired	snapshot row TTL'd	scores render, transcript/sources "expired"	BE (ADR-9); S02/AC-5

Detail 2.G — Cross-Layer Contract Verification

Endpoint	PRD-to-Schema row (§1.A.5)	Interim design (App. B Stitch)	Match?
POST evaluations (inbound)	rows 1–8 (record, status, veto, segment, reasons, snapshot)	n/a — no UI (pipeline)	yes
GET room scorecard (ext)	rows 1–3, 5–10 (record + groups + actors + verdict)	Stitch #1 (panel AI mode + actor selector + states)	yes
PATCH override (ext)	row 12 (override audit) + 10 (verdict recompute)	Stitch #1 (override → "edited by [SPV]")	yes

---

3. High-Availability & Security

Performance Requirement

• Score ≤ 60s after terminal exit (PRD §7): enqueue is O(1); worker cost = one DB read batch + (optional) one OpenAI call (240s timeout is the ceiling, not the norm; the human scorer runs the same shape in production today). The 60s budget holds for tier-1-only orgs structurally; tier-2 orgs depend on judge latency — surfaced as duration_ms on scorecard_autoscore_completed and alerted (below).
• Panel ≤ 2s P95 (PRD §7): GET is one query on the unique (org, room_id, agent_id) index with includes(details, snapshot) — same shape as today's GET plus one join; no N+1.

Monitoring & Alerting

Events (PRD §12) via SendMixpanelEventWorker (BE) / $mixpanel (FE), LLM-call logs via ChatGptLogWorker with related_key: 'ai_agent_scoring' (naming precedent: 'auto_agent_scoring', auto_agent_scoring.rb:223):

Event	Trigger	Props
scorecard_autoscore_completed	AI record persisted	org_id, agent_id, total_score, is_pass, tier2_count, veto_failed, duration_ms, weights_version, segment_source
scorecard_autoscore_failed	ingest/scoring failed	org_id, room_id, reason, retry_count
scorecard_tier2_judge_failed	tier-2 call failed	org_id, custom_param_id, reason
scorecard_panel_load_failed	panel hard load error (FE)	org_id, room_id, reason
scorecard_override_saved	override persisted	org_id, score_record_id, metric_id
scorecard_override_failed	override save failed	org_id, reason

Alerts (PRD §12): scorecard_autoscore_failed > 5% of AI conversations in 1h → Slack #bot-ai-oncall; judge-vs-human agreement < 85% during calibration → #bot-ai-quality (agreement computed from scorecard_override_saved deltas — PM-owned program metric; this RFC ensures the events carry the fields).

Logging

Rollbar.error(e) in use cases/repos (existing pattern). Never log transcript content, judge reasons, or rubric text at error level — log org_id, room_id, reason, retry_count only. Add transcript, metrics, overrides to the Rollbar scrub list.

Tracing

Inherits existing Grape request spans (ddtrace) + Sidekiq instrumentation. The async chain is correlated by room_id + ChatGptLogWorker rows (related_key: 'ai_agent_scoring') — same observability story as the human scorer. Cross-service trace stitching with the AI Service is out of scope (no shared trace header in the PROPOSED contract; revisit with DSAI).

Security Implications

• AuthN (FE surface): unchanged — middlewares/auth.rb (Bearer + X-Auth-Token).
• AuthN (internal surface): the evaluations endpoint mounts under internal_service with the same service auth as room_resolve_interactions / process_async_reply; it is not reachable from the FE gateway.
• AuthZ: GET keeps existing roles + Usman panel gating (grounding note §1.A.3); PATCH override keeps owner/admin/supervisor + FE inbox_scorecard_manage. End CS agents never see the override control (NEG-1).
• Tenancy (critical): FE surface takes organization_id only from the validated token (existing behavior). The internal surface receives organization_id in the payload — the use case must resolve the room and verify it belongs to that org before enqueueing — synchronously, against chatbot's own DB (pattern: process_async_reply context resolution L25-35; no hub call is possible on the internal surface) — rejecting unknown rooms and org mismatches with 422. Request specs must assert org-A token cannot read org-B's room scorecard.
• PII / transcripts (G7 — the big one): the snapshot table stores conversation transcripts for 90 days. Per PRD Open Q#5 this widens the plaintext-credential blast radius and gates GA: (1) DSAI hashes/redacts PII before the payload leaves the engine (contract field expectation, to confirm with the contract itself); (2) chatbot stores the payload as received but never logs it; (3) the G7 credential-leak closure is a hard GA gate (§5 #4); (4) 90-day TTL enforced by ADR-9 workers; (5) snapshots are excluded from API responses except transcript_available + sources (the panel gets source refs, not the raw transcript, until a Phase 3 drill-down defines that surface).
• Prompt injection (tier-2): org rubric prompt text and transcript content are untrusted input to the judge call — assemble with explicit delimiters; instructions come only from the system template (Phase 1 §3 forward note honored). Judge output is parsed as JSON scores; non-conforming output → partial, never executed.
• Input validation: metric scores coerced float + range 0–100 at the ingest contract; override values validated 0–100 (Dry rule, mirroring Phase 1 ADR-8 bounds); unknown metric codes rejected against Constants::ScorecardAiDefaultRubric.
• Audit: paper_trail already on both models; override sets correction_by* from current_user (existing PATCH pattern); the ingest write is system-attributed.
• DoS / size: transcript payload capped (contract: ≤ 500 turns / 1MB body — internal surface, single caller); ingest is 202-enqueue so floods degrade to queue depth, not API latency; per-org dedup by unique key bounds row growth to actors × rooms.

Role × Endpoint Authorization

Endpoint	owner	admin	supervisor	agent	member	AI Service (svc auth)
POST /v1/ai_agent/evaluations	—	—	—	—	—	✅
GET agent_scorecards/:room_id (± agent_id)	✅	✅	✅	✅*	✅*	—
PATCH agent_scorecards/:room_id (override variant)	✅	✅	✅	❌ 403	❌ 403	—
POST agent_scorecards/:room_id (manual, unchanged)	✅	✅	✅	❌ 403	❌ 403	—

* existing GET role list (grounded); visibility for agents is governed by Usman panel gating (AgentScorecard.vue:216-226) — unchanged.

Detail 3.A — Failure Mode Catalog

Failure	Detection	Behavior	Recovery
Malformed evaluation payload	Grape/Dry contract	422 to AI Service; nothing enqueued	AI Service retries (their side); alert on rate
Unknown metric codes / bad shape post-enqueue	scoring use case validation	record unavailable + failed_payload (7d) + scorecard_autoscore_failed	Sidekiq retry (retry: 1); re-push upserts
Tier-2 judge timeout/error	Completion returns success: false after 2 attempts	params unscored, record partial, tier-1 stored	re-push re-scores; event + alert
DB write fails mid-score	tx raises → rescued + Rollbar	full rollback — no partial record (single transaction)	Sidekiq retry
Duplicate/replayed push	unique (org,room,agent) + evaluated_at guard	idempotent upsert; older payloads ignored	n/a
Room/org mismatch (tenancy)	context resolution fails	422, nothing written	DSAI fixes payload
Override save fails	save! raises in tx	500, rollback, no partial state; scorecard_override_failed	FE Retry
Concurrent overrides (two SPVs)	one-slot guard checked in tx (edit_count re-read under lock)	first wins; second 422 "already corrected"	by design (ADR-7)
Snapshot cleanup deletes early/late	worker query on expires_at	scores unaffected (separate table)	rerun worker; expires_at is data
Flag off mid-stream	gate checked at ingest	200 "skipped"; existing records still readable	re-enable
hub-service down (FE GET)	auth middleware nil	401 (existing)	FE re-auth

Detail 3.B — Error Message Catalog

Code	Message	Surface
422	"Override score only between 0 - 100 are allowed"	override control
422	"Scorecard already corrected"	override control
422	"Unknown metric code"	(API-only; FE prevents)
404	"Agent Scorecard not found" (existing) / "Room not found" (existing)	panel
500	"Couldn't save. Try again."	override control
—	"scoring…" / "scoring unavailable" / "Not scored yet" / "Transcript expired"	panel states (from scoring_status / transcript_available)

Detail 3.C — Accessibility

Pixel3 components as-is. Veto state conveyed by text + badge, never color alone; graded scores are text chips; judge-reason expanders are keyboard-reachable buttons with aria-expanded; override input has a label + error message component; actor selector is a list of buttons (or MpSelect) with visible focus. Score totals announced as text.

Detail 3.D — Browser Support & FE Performance Budget

Inherits hub-chat's existing targets — no new matrix, no new dependency (Pixel3 + existing $mixpanel). The AI view renders ≤ 9 + N(custom) rows; reasons lazy-expand; no bundle-side data processing beyond the GET payload. Components live inside the already-lazy room-details feature — no initial-bundle weight added.

---

4. Backwards Compatibility and Rollout Plan

Compatibility

Additive only. Existing rows become actor_type: 'human' (truthful default). The no-param GET returns the human record (today's shape) + additive keys — existing hub-chat callers unaffected. Manual POST/PATCH human bodies unchanged. The human GPT auto-scorer's behavior is preserved by the actor-aware guard (ADR-8) — with a regression spec. is_use_ai / is_ai_failed keep their existing semantics on human rows.

Rollout Strategy

Flag ai_qa_unified_scorecard (same org feature as Phase 1) default OFF. Stage 1 internal QA (3–5 accounts + 15-CID set: validate segment attribution + ingestion contract); Stage 2 closed beta (TransGo, Talenta LMS + 3 partners: calibration/agreement ≥85%); Stage 3 Pro+Ent on request; GA per PRD §14 gates. Weight changes ride ai_qa_unified_scorecard_v2_weights (per-org, no deploy — PRD §11.1). Detailed scheduling lives in delivery/ (not here).

Cross-Layer Rollout Compatibility

Order	Step	Safe if FE not yet shipped?	Safe if BE not yet shipped?
1	Phase 1 BE live (config columns + rubric constant)	yes (prereq A4)	—
2	BE migrations (this RFC)	yes (inert columns/table)	—
3	BE ingest + scoring + extended GET/PATCH + OpenAPI	yes (flag-gated; GET additive)	—
4	DSAI starts pushing evaluations (per confirmed contract)	yes (records accumulate dark)	no — 404s; do not enable before step 3
5	hub-chat AI mode + selector behind checkSubscription	—	FE no-ops (flag off / fields absent → manual mode)
6	Provision feature for alpha orgs	—	—

Deploy BE before FE before provisioning. Rollback in reverse order.

Detail 4.A — Configuration Contract

Key	Type	Default	Where
ai_qa_unified_scorecard	org feature	OFF	BE OrganizationFeatures::FindFeature; FE checkSubscription (ADR-11)
ai_qa_unified_scorecard_v2_weights	org feature (weights version selector)	OFF (= v1 uniform weights)	scoring use case (ADR-6)
is_ai_auto_score, ai_passing_grade	scorecard_preferences columns (Phase 1)	false / 75	read by scoring use case (A4)
AI_TIER1_WEIGHT	constant	0.11 (decided — ADR-6)	scoring use case
Tier-2 judge model/temperature	system_preferences row code: 'ai_agent_scoring', group_code: 'omnichannel' (pattern: 'auto_agent_scoring' row, auto_agent_scoring.rb:80-109)	I18n defaults fallback	Repositories::Gpt::SystemPreference::FindBy
Judge timeout / attempts	args to Completion	240s / 2 (decided — matches live scorer)	worker
Snapshot TTLs	expires_at at write	90d transcript/sources; 7d failed_payload (decided — PRD §7.1)	scoring use case
AI-record TTL	worker constant	13 months (decided — PRD §7.1)	DeleteOldAiScorecardsWorker
Override range	Dry rule	0–100 inclusive (decided — mirrors ADR-8 Phase 1)	update use case

Locked for build; DSAI weight tuning and rubric confirmation change constants/flags only.

Detail 4.B — Test Plan (commands from the repos)

Backend (chatbot — .rspec, bitbucket-pipelines.yml:78,379, AGENTS.md §API rules):

RAILS_ENV=test bundle exec rails db:migrate
bundle exec rspec spec/api/internal_service spec/api/frontend_service/v1/gpt_spec.rb \
  spec/core/use_cases/gpt/omnichannel spec/core/repositories/gpt
bundle exec rubocop
bundle exec brakeman
# OpenAPI (MANDATORY when endpoints change):
ruby scripts/openapi/split.rb
npx --yes @apidevtools/swagger-cli validate docs/openapi/openapi.yaml
npx --yes @stoplight/spectral-cli lint docs/openapi/openapi.yaml --fail-severity=error

Frontend (hub-chat/package.json scripts; pnpm):

pnpm lint            # eslint + prettier --check
pnpm type-check      # vue-tsc --noEmit
pnpm test            # vitest --dom --pool=forks

Cross-boundary contract test. BE request spec asserts the exact JSON of (a) the extended GET (AI actor: actor_type, scoring_status, actors[], groups[] with judge_reason/sources/is_veto) and (b) the override PATCH response. hub-chat vitest parses the same fixture into the §2.B interfaces (AiScorecardData, ActorSummary, AiMetric) and asserts the mode switch + selector render from it. Shared fixture = the contract anchor (same mechanism as Phase 1 REV-2).

Detail 4.C — Agent Execution Plan

Order respects dependencies (Phase 1 prereq → migrations → pipeline → API → FE). Use the chatbot repo's openapi-spec-sync skill for endpoint chunks. Chunk 2 builds against the PROPOSED contract; enabling real ingestion gates on DSAI confirmation (§5 #1).

#	Chunk	Files	Commands	Acceptance
1	Migrations (3) + schema regen	chatbot/db/chatbot_gpt_migrate/<ts>_add_ai_actor_fields_to_agent_scorecards.rb, <ts+1>_add_ai_fields_to_agent_scorecard_details.rb, <ts+2>_create_agent_scorecard_snapshots.rb; db/chatbot_gpt_schema.rb	RAILS_ENV=test bundle exec rails db:migrate	schema shows the §2.3 columns/table; existing rows readable with actor_type='human'
2	Ingest endpoint + worker	app/api/internal_service/v1/ai_agent/evaluations.rb (NEW; mount in internal_service/api.rb); app/api/internal_service/v1/ai_agent/use_cases/ingest_evaluation.rb (NEW — contract per §2.4 Inbound r1); app/workers/ai_agent_scoring_worker.rb (NEW, queue: :gpt_agent_scoring, retry: 1)	bundle exec rspec spec/api/internal_service	request spec: valid payload → 202 + job enqueued; malformed → 422; flag off → 200 "skipped"
3	Scoring use case (tier-1 + veto + persist)	app/core/use_cases/gpt/omnichannel/ai_agent_scoring.rb (NEW); app/core/repositories/gpt/agent_scorecards/calculate_ai_agent_score.rb (NEW — mirrors calculate_agent_score.rb tx shape; writes scorecard actor_type:'ai_agent' + evaluated_at + weights_version + 9 detail rows by metric code + snapshot with expires_at)	bundle exec rspec spec/core/use_cases/gpt/omnichannel	unit specs: uniform-weight total; veto floors is_pass; ai_passing_grade comparison; unknown code → unavailable; idempotent upsert on re-push
4	Tier-2 judge merge	extend ai_agent_scoring.rb: load org custom params where prompt non-empty (Phase 1 field); one Repositories::Gpt::Completion call (system template + delimited rubrics + transcript); parse JSON scores; merge; partial on failure	bundle exec rspec	specs: rubric'd params merged into total; empty-rubric → manual_only; success:false → partial, tier-1 intact; scorecard_tier2_judge_failed fired
5	Events + logging	wire SendMixpanelEventWorker (scorecard_autoscore_completed/_failed, scorecard_tier2_judge_failed) + ChatGptLogWorker related_key: 'ai_agent_scoring'; Rollbar scrub additions (transcript, metrics, overrides)	bundle exec rspec	specs assert events enqueued with §3 props; no transcript in Rollbar payloads
6	Retention workers	app/workers/delete_expired_scorecard_snapshots_worker.rb, app/workers/delete_old_ai_scorecards_worker.rb (NEW); config/schedule.yml entries (nightly 01:00 Asia/Jakarta, queue: application_maintenance)	bundle exec rspec spec/workers	specs: expired snapshot rows destroyed, fresh kept; AI rows >13mo paranoid-destroyed, human rows untouched
7	Actor-aware guard fix	app/core/use_cases/api/internal_service/v1/webhook/room_resolve_interactions.rb (scope scorecard_exists to actor_type: 'human')	bundle exec rspec	regression spec: AI record present → human auto-scoring still enqueued; human record present → still skipped (today's behavior)
8	Extended GET (+ gpt_service mirror)	agent_scorecards.rb GET (optional agent_id); use_cases/.../agent_scorecard/get.rb; repositories/gpt/agent_scorecards/get_scorecard.rb (actor-aware default = human; actors[]; includes snapshot presence); response entities (+groups[] AI shape)	bundle exec rspec spec/api/frontend_service/v1/gpt_spec.rb	request specs: no-param → human record byte-compatible + actors[]; AI-only room no-param → default template + actors[]; agent_id → AI record with §2.4 r1 shape; unavailable record serializes scoring_status
9	Override PATCH	agent_scorecards.rb PATCH (override body variant); use_cases/.../agent_scorecard/update.rb (0–100 rule, slot guard in tx, recompute total/is_pass with veto floor, correction_*, per-detail override_reason)	bundle exec rspec	specs: in-range override recomputes + sets edit_count=1; second → 422 "already corrected"; 150 → 422, nothing persisted; paper_trail version exists
10	Tenancy + security specs	request specs across chunks 2/8/9	bundle exec rspec; bundle exec brakeman	org-A token cannot read org-B room; ingest org/room mismatch → 422; brakeman clean
11	OpenAPI sync	docs/openapi/openapi.yaml + dist; SESSION-LOG	ruby scripts/openapi/split.rb; swagger-cli + spectral	both validators pass; 3 changed ops documented
12	FE types + composable	hub-chat/.../agent-scorecard/@types/AgentScorecardTypes.ts (§2.B additions); composables/useAgentScorecard.ts (agentId? param; overrideAiScore)	pnpm type-check; pnpm test	vitest: composable builds correct URLs (?agent_id=), parses the shared fixture
13	FE ActorSelector + AI mode + override	components/ActorSelector.vue, components/AiScorecardView.vue (NEW); views/AgentScorecard.vue mode switch + flag gate (checkSubscription("ai_qa_unified_scorecard")) + Usman manage gating for override	pnpm test; pnpm lint	component tests: actor_type:'ai_agent' → AI view w/ 9 graded rows + veto banner + badge; human-only fixture → manual mode untouched; view-only perm → no pencil; flag off → no selector/AI view
14	FE states + events	loading/pending skeleton, unavailable retry state, partial chips, transcript-expired note; $mixpanel scorecard_panel_load_failed, override events	pnpm test	state-matrix tests per §2.C rows
15	Cross-boundary contract test	BE: exact-JSON assertions (chunk 8/9 specs); FE: __tests__ parsing the same fixture	bundle exec rspec; pnpm test	shared fixture round-trips; key rename breaks both suites
16	Test specs doc	documents/chatbot/unified-agent-scorecard/tests/phase-2-auto-scoring-and-in-room-scorecard.md	n/a	covers_acceptance_criteria lists every P2-S0x/{AC,ERR,NEG}-n

Detail 4.D — Verification & Rollback Recipe

Pre-merge (in order): BE rails db:migrate → rspec → rubocop/brakeman → OpenAPI split + swagger-cli + spectral; FE pnpm lint → pnpm type-check → pnpm test.

Post-deploy signals:

• scorecard_autoscore_completed events flow for alpha orgs within 60s of AI conversation closes (duration_ms P95 < 60,000); scorecard_autoscore_failed < 5%/h.
• segment_source: "missing" rate ≈ 0 on the 15-CID set (Internal Alpha gate, PRD §14).
• Manual: enable flag for one internal org → run an AI conversation to terminal exit → panel shows the AI actor with graded metrics/reasons/sources; override a metric → chip + recompute; second override blocked.
• Regression: a human-only resolved room still gets its human GPT auto-score (AutoAgentScoringWorker enqueued — chunk 7 spec + a staging smoke).
• Coverage metric: % AI conversations scored (target ≥90% Open Beta, ≥95% GA — PRD §13/14).

Rollback (numbered):

1. Disable ai_qa_unified_scorecard for affected orgs → ingest returns "skipped", FE reverts to today's panel (instant, no deploy).
2. Weight/rubric regression (PRD §11.1): toggle ai_qa_unified_scorecard_v2_weights back per org (no deploy); confirm judge_human_agreement recovers.
3. If BE bug: revert the BE PR — GET additions are additive; hub-chat FE without them falls back to manual mode (fields absent ⇒ no AI view).
4. If FE bug: revert the FE PR — BE records keep accumulating dark.
5. Migrations are forward-only in prod (columns default-safe, inert when flag off; snapshot table isolated). Do not drop columns as a rollback step.
6. Confirm scorecard_autoscore_failed back to baseline and human auto-scoring intact.

---

5. Concerns, Questions, or Known Limitations

#	Item	Type	Owner	Status
1	DSAI evaluator output contract is unconfirmed (PRD Open Q#2, still blocking 2026-07-03). §2.4 Inbound row 1 is PROPOSED; chunks build against it behind the flag, but no org is enabled until DSAI signs the contract (payload fields, push trigger, PII handling, segment). (= rfc-reviewer REV-8.)	Blocker (enablement, not build)	DSAI + BOT	Open — due 2026-07-17 (PRD)
2	Metric definitions/order are PROPOSED (PRD Open Q#1); metric codes are pinned to Phase 1's Constants::ScorecardAiDefaultRubric — DSAI confirmation changes constants only.	Open (accuracy)	DSAI	PRD Open Q#1, due 2026-07-15
3	Figma frames for the panel AI mode + actor selector pending; FE builds against PRD Appendix B Stitch prompt; pixel deviations re-checked on frames.	Blocker (FE fidelity)	Design	Open (PRD Dep "Design — YES")
4	Storing transcripts widens the G7 plaintext-credential/PII blast radius; GA gates on G7 closure + PII hashing upstream (§3 Security). Infosec approver required.	Blocker (GA)	Security/Infosec	PRD Open Q#5, due 2026-09-30
5	Segment attribution accuracy: contract-supplied segment validated on 15-CID in Internal Alpha; segment.source:"missing" rate is the gate metric (ADR-4). The PRD's proposed handover-event mechanism was not grounded — those events exist only in the hub FE layer.	Risk (mitigated)	Bot/AI + DSAI	PRD Open Q#3 recast
6	Per-team RBAC scoping descoped (org-wide Usman perms — ADR-10). Follow-up if team scoping lands platform-wide.	Known limitation	PM + Platform	PRD Open Q#7 resolved for P2
7	Judge-vs-human agreement instrumentation is derived from override events (PM-owned program metric); a dedicated calibration surface is Phase 5's concern.	Known limitation	PM + DSAI	New
8	60s scoring budget for tier-2 orgs depends on OpenAI latency (240s ceiling); monitored via duration_ms, alert if P95 breaches.	Risk (monitored)	BOT	New
9	The actors[] display name for the AI actor requires resolving the AI agent's name from its id (chatbot owns AI-agent config); human names come from room detail. Implementation detail flagged so the entity builder resolves both. (= rfc-reviewer REV-7, accepted-risk.)	Note	BOT BE	New
10	FE per-component defineProps/defineEmits typings for ActorSelector / AiScorecardView are deferred to implementation (interfaces exist in §2.B). Close in §2.A when Figma frames land. (= rfc-reviewer REV-6 — same posture Phase 1 shipped with, its §5 #7.)	Open (low risk)	BOT FE	New (rfc-reviewer R1)

---

6. Comment Log

Date	Author	Note
2026-07-17	rfc-starter (Claude)	Initial draft from PRD v1.4. Grounded against chatbot (BE) + hub-chat (FE). Grounding corrections vs PRD: (1) Phase 1 shipped new is_ai_auto_score/ai_passing_grade columns — this RFC reads those, not an "extended is_auto_score"; (2) is_use_ai/is_ai_failed already exist on agent_scorecards (reused, not net-new); (3) no handover events exist in chatbot BE — segment comes from the evaluator payload (ADR-4); (4) the per-actor unique key already exists (schema L124); (5) Usman scorecard RBAC is org-wide → team scoping descoped (ADR-10); (6) the room-resolve skip guard must be made actor-aware to protect the live human scorer (ADR-8). All 9 mermaid blocks validated with @mermaid-js/mermaid-cli (mmdc) — no parse errors.
2026-07-17	rfc-reviewer (Claude)	R1 review → 8.0 (Strong / PROCEED with notes). Applied R2 fixes: evaluated_at + weights_version columns on agent_scorecards (idempotency guard + 13-month weights audit — REV-1); tier-2 no_transcript branch in ADR-5/§2.4/Branch Catalog (REV-2); scoring_state: "overridden" per-metric override marker (REV-3); tenancy check pinned to chatbot-DB sync pre-enqueue, no hub call (REV-4); AI-only-room no-param GET behavior specified (REV-5). Re-score → 8.5 (Agentic-Ready / PROCEED with notes). Open: REV-6 (§5 #10), REV-7 (§5 #9), REV-8 (§5 #1 — enablement gate). erDiagram re-validated with mmdc after edits. See rfc-phase-2-auto-scoring-and-in-room-scorecard-review.md.

---

7. Ready for Agent Execution

• Every PRD story + composite AC id traced in §1.A.4 / §1.C
• Topology, repo map, ER, 4 sequence (happy + failure), state, and branch diagrams — all mermaid, all parser-validated
• Every endpoint tagged reused/extended/new with evidence (Existing-Endpoint Check)
• Source Verification table backs every "existing" claim with file:line (incl. three verified-absents)
• ADRs cover storage, sync/async, caching, third-party, consistency, multi-tenancy, reuse/new (11 ADRs)
• Test/build commands sourced from chatbot CI config + hub-chat/package.json
• Agent Execution Plan: 16 ordered chunks with files + commands + assertable acceptance
• Rollback recipe concrete (flag → weights flag → revert order → forward-only migrations)
• DSAI evaluator contract confirmation (§5 #1) — gates enabling ingestion for real orgs; chunks 1–16 are executable against the PROPOSED contract behind the flag
• Figma frames (§5 #3) — FE chunks proceed on the interim Stitch spec; re-verify on frames
• Infosec sign-off (§5 #4) — gates GA (transcript storage / G7)

Ready for agent execution: yes (build). All chunks are executable now behind ai_qa_unified_scorecard; the three unchecked gates block org enablement / GA, not the build. Recommended next: hand to rfc-reviewer for a second-pass score, then rfc-task-breakdown for the sprint slicing.