Gap Analysis — RFC "Qontak Chatbot AI - Autonomous Agent" vs shipped code

• RFC: QON/PMDAT 51153994292, DSAI Team (Grasia Meliolla), last modified 2026-06-22
• Code baseline: chatbot (Rails BE) and chatbot-fe local checkouts, analyzed 2026-07-04
• Analyst: Dimas Fauzi Hidayat (PM) via grounding sweep

Headline

The RFC is not a future proposal — it is partially shipped, and the document was never updated to match. The Rails V2 AI Agent engine is the de-facto implementation: SyncToAiService translates Rails capabilities into the RFC's skill_pack and POSTs mode: 'autonomous' to qontak-ai-noncore-mrag. But the implementation diverged from the RFC on its central promise (closed-enum exits), on the model-config source of truth, on the async callback contract, and it silently drops several skill_pack blocks (fallback, turn_budget, idle_action). The document itself also contains internal contradictions and broken examples.

---

Part 1 — Errors inside the RFC document

A. Internal contradictions

#	Where	Problem
D1	§6 vs §8.1/§8.2/§13	§6 says up to 6 LLM calls/turn including an "intent detector" and "milestone detector". §8.1 says LLM is used ONLY for planner, executor, setup-time drafting. §5.3 says milestone firing is deterministic (action map) or tag-parsing (string strip — not LLM). §13 Cons says "One additional LLM call per turn". Four sections, three different call counts; the two "detector" LLM calls are never explained anywhere.
D2	§5.1 vs §10.2	§5.1 specifies two-level KB binding (agent-level + capability-level with entity_type='ai_agent_capability', per-skill vector store overrides). §10.2 "Where do action / KB references come from?" says the exact opposite: "No per-skill KB binding, no RAG schema changes", KB agent-level only. Code implements §5.1's model (capability-level), so §10.2 is the wrong one.
D3	§11.1 vs §10.4	§11.1: "Test in simulator button — opens a chat preview that calls /run-autonomous-agent directly". §10.4: that endpoint is "Internal AI team, mekari-agent only, called by qontak-chatbot-ai", bearer-auth shared secret. FE cannot call it. Both can't be true.
D4	§10 intro vs body	Intro says "Two new endpoints. One existing endpoint extended." The body defines six new endpoints (draft-skill-pack, refine-skill-pack, GET /models, run-autonomous-agent, POST /predictions/async, GET /predictions/async/{id}). The §10.1 summary table omits /models and both async endpoints.
D5	§5.1/§9 vs §12.2	"Migration path is a one-field update, reversible" vs CHECK constraint mode='tree_node' AND skill_pack IS NULL. Reverting to tree_node forces nulling skill_pack — config is destroyed, so it's neither one-field nor reversible.
D6	§5.1 fallback vs §6.1 enum	fallback.no_skill_match / turn_budget_exceeded take the value exit_to_human — which is not in the closed exit-reason enum (§6.1). How exit_to_human maps to the enum the BE receives is never defined.
D7	§5.1 routing_rules vs §8.2 thesis	{"when": {"user_intent": "talk_to_human"}} rules require free-text intent classification at runtime — the exact LLM-classifier pattern the RFC exists to eliminate. The mechanism for evaluating user_intent is never specified (it is presumably the undocumented "intent detector" from D1). The doc's determinism claim (§8.2) doesn't hold for these rules.

B. Broken examples and spec bugs

#	Where	Problem
D8	§12.2 DDL	Invalid SQL: ADD COLUMN skill_pack JSONB; — the semicolon terminates the ALTER mid-statement, orphaning the next four ADD COLUMN clauses; stray trailing ;; mixes ADD COLUMN and ADD COLUMN IF NOT EXISTS styles.
D9	§10.2 / §12.2 / §10.4 / §10.5	Default-model chaos: schema default openai/gpt-5.1 (§10.2) vs DDL default openai/gpt-4o-mini (§12.2) vs example gpt-4.1-mini (§10.4) vs token_usage gpt-5-mini-2025-08-07 (§10.5). Operationally live: Rails omits model_id when no category is set, so the AI-side default decides — and the RFC gives two different answers.
D10	§10.2 request example	Invalid JSON — missing comma after "language": ["ID"]. Also language is typed as an array but described as a single ISO code with scalar default "EN".
D11	§10.3 example response	Circular milestone gate: resolve_agent gets requires_milestone='conversation_closed', but milestone_action_map fires conversation_closed only when resolve_agent succeeds → the tool is permanently blocked. This is exactly the anti-pattern §5.3 warns against, produced by the drafter's own defaulting rule ("defaulted requires_milestone=last milestone") and presented as a good example. The drafter default is itself the bug.
D12	§5.1 example	routing_rules switch to skill "registration" — which doesn't exist in the example's skill list (only fleet_search_and_order). Dangling reference in the doc that advertises reference-filtering as an anti-hallucination guarantee.
D13	§5.2 vs §5.1	Workflow-state example slots (pickup_date, return_date, location) don't match required_slots (date, duration, location_id, fleet_id); fleet_id holds a string where the action arg type is int. With these names, the "all required slots filled" tool-call-enforcement validator would never fire — the doc's own Pattern-B fix wouldn't trigger on its own example data.
D14	§10.4 response	workflow_state_snapshot.last_response_snapshot embeds the full workflow_state_snapshot again — recursive self-nesting. Persisted as-is, the snapshot grows every turn.
D15	§12.1	Redundant index: idx_workflow_state_lookup duplicates the index already created by UNIQUE (room_id, autonomous_agent_id). Also notes / last_planner_decision have no columns — acknowledged as piggybacking on last_response_snapshot, which couples state persistence to the idempotency cache (fragile).
D16	§10.3 response	suggested_actions / suggested_guidance appear both inside skill_pack and at the response top level — duplicated, schema ambiguous.
D17	§10.3 response	milestone_action_map: {"answered": "file_search"} maps a milestone to an action type, not a registered action name — contradicts §5.3's definition, and file_search has no 2xx-tool-call semantics to fire on.
D18	§10.2 response	Skill assign_support_agent has required_slots: [assign_to, assign_target_id] but its only action declares no args and slot_action_args_map is empty — the slots can never be filled by the documented mechanism.
D19	§11.1/§5.1	Description limit spec left as a range "2000–4000 chars" — FE needs one number. (Shipped FE uses 10,000.)

---

Part 2 — RFC vs actual code (chatbot + chatbot-fe)

C. Contract-level divergences (highest risk)

C1. The closed-enum exit model — the RFC's core thesis — did not ship; free text came back.

• FE routing-rule "exit" is a free-text textarea (placeholder "Example: Customer is angry") — chatbot-fe/modules/bot-automation/components/AiAgentEditor.vue:1637, saved verbatim via useSaveAgent.ts:323.
• Rails passes it through untouched: build_routing_then → exit: {reason: <free text>} (sync_to_ai_service.rb).
• Runtime still sends tenant tree-edge exit_conditions [{id, description}] in the prediction payload and routes on exit.id → Response → next_intent_id (send_message_with_resolve.rb:1707,1751; process_async_reply.rb process_success). There is no BE handling for any of the six enum reasons (skill_completed, user_request_human_handoff, …). §9's claim "BE transfer-to-human driven by exit reasons (closed enum)" is false in code.

C2. Advance-settings source of truth is inverted.

• RFC §10.5: model_id/temperature/reasoning are NOT in the /predictions request; chatbot-ai reads them from the AI-side ai_agents row.
• Code: Rails builds llm_config: {model_id, model_settings: {temperature, reasoning}} into the request (send_message_with_resolve.rb ~1854–1895), resolving the model from the Rails ai_agent_models catalog by category.
• Two competing model catalogs exist: Rails ai_agent_models (Gemini 2.5 Flash-Lite, Gemini 3 Flash Preview, Claude Haiku 4.5, GPT-5.1 + GET /v1/ai_agents/models) vs RFC's valid_model_configurations + GET /models. The RFC's example models (gpt-4o-mini, gpt-4.1-mini, gpt-5-mini) are in neither.

C3. Async prediction contract mismatch.

• RFC 10.5b: caller supplies required callback_url; callback body includes assign_agent, assign_reason, resolve; 3 retries w/ backoff; plus a GET /predictions/async/{message_id} pull endpoint.
• Code: no callback_url field — the callback is the convention-fixed internal route /api/internal/v1/ai_agents/{vendor_ai_agent_id}/predictions/async/reply; the handler validates only {status, output_text, references, exit, token_usage} and would drop assign_agent/assign_reason/resolve if the AI side sent them (process_async_reply.rb:43,95-115). Resolve is signalled via references[].action_type == 'qontak_chat_room_resolve' instead. Failure path = fallback intent, not assign_agent: true.
• Also: /predictions/async is presented as new, but Rails already calls it behind the ai_agent_prediction_async flag. No consumer of the pull endpoint exists.

C4. Rails never sends several skill_pack blocks the RFC treats as core. build_skill_pack omits:

• fallback (no_skill_match / turn_budget_exceeded / default_turn_budget) — turn-budget behavior is whatever the AI side defaults to; invisible and unconfigurable to tenants. It also overlaps uncoordinated with Rails' own message_limit Redis counter — two caps, two owners.
• per-skill turn_budget — not mapped from any FE field.
• idle_action — kept Rails-side in parameters.profile.idle_rule (enforced by BE). §9's "reused as idle_action field in skill_pack" is false — and correctly so, since the AI service has no idle timer; the RFC should remove it from skill_pack.
• agent-level knowledge_base is hardcoded {files: [], status: 'idle'} — agent-level KB fallback (per §5.1) effectively doesn't exist; only per-capability vector stores are real (confirming §10.2 is wrong, see D2).

C5. Action taxonomy collapsed; the Rails execution path is missing from the RFC.

• RFC action types: internal_tool_calls, file_search, qontak_function_call, external_api_calls. Rails maps every action — including action_type: 'api' — to qontak_function_call (build_skill_actions); internal_tool_calls (get_time) is not configurable from the product.
• The RFC says action definitions/credentials live in the AI-side functions table (POST /knowledge-base/function) and mekari-agent dispatches them. In reality, execution flows back into Rails via POST /api/internal/v1/ai_agents/:vendor_id/actions/execute → ActionExecutorFactory (V2 reads actions out of parameters['capabilities'][].actions[]). Credentials (mekari-action HMAC, qontak company token, Midtrans OAuth) are Rails-owned. This entire callback execution path — the most security-sensitive part of the system — is absent from the RFC.

C6. Versioning/publish is ignored. RFC: "Edits go live immediately; no deploy required." Shipped: draft→publish workflow with ai_agent_histories, active_version_id, per-version vendor_ai_agent_id (runtime resolves the active version's vendor id), and test cases (ai_agent_test_cases) bound to versions. The RFC's direct-PUT edit model would bypass publish gates and version-pinned testing.

D. Product/UX divergences

#	RFC says	Shipped reality
P1	Mode toggle inside the existing AI Agent form; "tenants do not learn a new motion; only a review table added"	A separate bot-automation module with a brand-new editor (Identity / Goals / Knowledge sources / Actions / Validation / Publish). V1 agents are read-only ("View details" disabled). No mode field — gating is `engine_version 1
P2	Description limit raised 256 → 2000–4000	FE maxlength=10000 (CreateAiAgent.vue:139).
P3	Skill catalog review table, then Save	Full editor; draft generation exists (POST /v2/ai_agents/generate → AI draft-skill-pack → SkillPackMapper back into Rails capabilities), so the concept shipped, translated.
P4	POST /ai-agent/refine-skill-pack + "Refine with AI" side-panel chat	No consumer anywhere — zero references in chatbot or chatbot-fe.
P5	Test in simulator → direct /run-autonomous-agent	Testing is via bot-preview channel + ai_agent_test_cases through Rails.
P6	Tenants pick from a rich actions pool	FE ACTION_LIST hardcodes 4 action types (qontak_crm_deal_create, qontak_crm_ticket_create, qontak_crm_ticket_update, api); the actions page still ships ACTION_FALLBACK stub data pending backend. The RFC's setup vision is bottlenecked by FE exposure.
P7	Vocabulary: skills / intent_hints / instructions / user_intent / mode	Rails-FE vocabulary: capabilities / triggers / instruction / user_says / engine_version. The mapping lives only inside sync_to_ai_service.rb + skill_pack_mapper.rb and is documented nowhere — this is how drift keeps happening.

---

Recommended fix queue

1. [RFC] Update the doc to as-built (or mark superseded sections): async contract (C3), llm_config in request (C2), execution-callback path (C5), skills↔capabilities vocabulary map (P7). The RFC is being read as a contract by three teams; today it describes a system nobody runs.
2. [Product decision] Exit model: either implement the closed-enum + deterministic routing that justified the RFC (FE dropdown for exit reasons, BE mapping enum → assignment/resolve), or formally accept free-text exit reasons and delete §6.1/§9's claims. Right now Pattern A's fix — the RFC's raison d'être — is not actually shipped.
3. [BE] Send fallback + turn_budget in skill_pack (or document AI-side defaults) and reconcile with the Rails message_limit counter — decide one owner for conversation caps.
4. [AI/DSAI] Fix the drafter's gate-defaulting rule (D11) — it manufactures circular deadlock gates.
5. [BE/AI] Align async callback schema — either AI stops sending assign_agent/assign_reason/resolve or Rails starts consuming them.
6. [FE] Unblock the actions pool (P6) — the 4-item ACTION_LIST caps everything the RFC promises about self-service transactional bots.
7. [RFC] Fix the mechanical errors for the next revision: D8 SQL, D9 default model, D10 JSON, D12/D13 example coherence, D5 reversibility constraint.