Fast Track chatbot-ai — recurring issues, grounded fixes and action items

A formatted Word version of this report is committed alongside it: 2026-07-27-fast-track-chatbot-ai-recurring-issues.docx. This markdown file is the canonical, diffable source; the .docx is the shareable rendering.

Context

Dimas Fauzi Hidayat asked for every Fast Track Jira issue tagged chatbot ai, the recurring patterns across them, and fix suggestions grounded in the actual code rather than guesses.

• Source: Jira jurnal.atlassian.net, filter issuetype = "Fast Track" AND labels = "chatbot-ai" → 36 tickets, project BOT (Qontak ChatBot). ("Fast Track" is an issue type, id 10845; "chatbot ai" resolves to the label chatbot-ai.)
• Grounded against: chatbot (BE, Rails) and chatbot-fe (FE, Nuxt 3 / Vue 3). Every root cause below cites file:line.
• Read-only research — no Jira fields and no application code were changed.

Executive summary

The 36 tickets cluster into six recurring themes. The single largest theme — wrong / KB-mismatched answers (~11 tickets) — is chronically closed as Won't Fix, because retrieval and embedding run in the external AI Service (DSAI / Mekari RAG), not in our application. That is a cross-team dependency, not an app bug.

The genuinely app-fixable recurring pain, in priority order:

Priority	Cluster	Why it matters	Effort
P0	F — Availability / latency	Contains the only open P1 (BOT-4739); customers get no reply during AI-service degradation	M
P0	D — Config not persisting	4 tickets; two confirmed bugs (BE + FE); erodes trust in the product	S
P1	C — KB ingestion lifecycle	9 tickets; "stuck Processing" forever, source_type error, missing delete/resync	M
P1	B — Language ID↔EN	3 tickets; cheap, high-visibility win	S
P2	A — Answer quality	~11 tickets but DSAI-owned; only app lever is hardening the re-embed on KB edit	S (app part)

All 36 tickets (newest first)

Key	Status	Summary	Cluster
BOT-4746	In Progress	AI Resource Upload Stuck in Processing	C
BOT-4739	In Progress · P1	AI Service Downtime → AI Agent Stop Responding	F
BOT-4726	Won't Fix	Bot AI Generates Mixed Indonesian/English	B
BOT-4693	Won't Fix	AI Answer Repeats Greeting Message	A
BOT-4519	Won't Fix	Unable to Open AI Agent Details by Clicking Name	D
BOT-4492	Done	Unable to Rename AI Agent (name reverts)	D
BOT-4383	Done	AI Agentic Prompt Changes Not Persisting After Save	D
BOT-4376	Won't Fix	Kendala AI Chatbot	A
BOT-4336	Won't Fix	AI Agentic Response Delayed ~1 Hour	F
BOT-4311	Won't Fix	Outdated KB Response Reappeared After Changing KB	A/C
BOT-4276	Done	NLP/Intent Classification Not Synced	G
BOT-4264	Done	Inconsistent Language in AI Summary	B
BOT-4240	Done	Missing delete icon/button for AI resource	C
BOT-4142	Won't Fix	Error When Removing Knowledge Source & Saving Bot Response	C
BOT-4128	Done	Error Publishing Bot	E
BOT-4094	Done	AI Prompt Changes Not Persisted After Save	D
BOT-4060	Done	AI Response Exceeds Character Limit → Message Send Failure	H
BOT-3889	Won't Fix	AI Assist Not Triggered — "Package is Expired" despite active	H
BOT-3720	Won't Fix	AI Response Not Matching KB & Inconsistent Language	A+B
BOT-3718	Won't Fix	AI Not Responding According to KB in Preview	A
BOT-3705	Won't Fix	Intent Misclassification	G
BOT-3635	Won't Fix	AI Source PDF Stuck in Processing, cannot Resync/Delete	C
BOT-3592	Done	Request to Disable AI Page	H (not a bug)
BOT-3547	Won't Fix	AI Generated Response Is Incorrect	A
BOT-3540	Done	AI Resources Status Changed to Failed After Active	C
BOT-3511	Done	AI Generated Response Is Incorrect	A
BOT-3455	Done	AI Generated Response Is Incorrect	A
BOT-3246	Won't Fix	AI Chatbot Providing Incorrect Responses	A
BOT-3018	Done	Error when publishing chatbot flow	E
BOT-2990	Done	Chatbot preview not showing but working on WhatsApp	H
BOT-2964	Done	Failed to Add Training Source URL	C
BOT-2903	Done	"source_type does not have a valid value" on AI Resources	C
BOT-2743	Won't Fix	AI Provides Incorrect Schedule Information	A
BOT-2690	Won't Fix	AI Unable to Answer Certain Qs from Same KB	A
BOT-2175	Done	User Input Disappears When Editing AI Source	C
BOT-1686	Done	Incorrect Routing "Table Booking" → Handover not UI	G

Cluster legend: A = Answer quality / KB · B = Language · C = Ingestion lifecycle · D = Config persistence · E = Publish · F = Availability / latency · G = Intent / routing · H = Other one-offs.

Recurring clusters

• A. Answer quality / KB-not-matched — ~11 (BOT-4693, 4376, 3720, 3718, 3547, 3511, 3455, 3246, 2743, 2690, 4311). Mostly Won't Fix → retrieval is external.
• C. KB / resource ingestion lifecycle — ~9 (BOT-4746, 3635, 3540, 2964, 2903, 4240, 4142, 2175; plus freshness 4311).
• D. AI Agent config not persisting — 4 (BOT-4383, 4094, 4492, 4519).
• B. Language inconsistency — 3 (BOT-4726, 4264, 3720).
• F. Availability / latency — 2, including the only open P1 (BOT-4739, 4336).
• G. Intent / routing / sync — 3 (BOT-3705, 4276, 1686) · E. Publish — 2 (BOT-4128, 3018).

Grounded fixes

D. Config not persisting (BOT-4492, 4383, 4094, 4519) — highest ROI, app-side, confirmed

Root causes:

1. BE — existing-agent name/prompt never written back to ai_agents. create_history.rb:114-136 only does FindBy for an existing agent; name/description are written to the DB only on first create (l.121-122); the edit lands only in the history snapshot (l.62). Any list or detail view reading ai_agents.name shows the old value → "reverts".
2. FE — name-revert display bug. handleNameUpdated (AiAgentModal.vue:865-867) only flips a boolean; the displayedName watch (AiAgentIdentityForm.vue:346-368) re-syncs from the stale props.agentData and overwrites the new name. The prompt has a formInitialized guard; the name does not.
3. BE — save is a Redis draft (1h TTL) plus an async worker. update_session.rb:255 writes $redis_cache.setex(key, 3600, …); DB propagation is deferred to CreateAiAgentHistoryPropertyWorker (create_history.rb:78). If that worker fails or the draft expires before publish, the "AI Agent Saved" toast lied. The present? guard at update_session.rb:180-190 also silently drops blank values.

Fix: (a) update ai_agents.name/description on the existing-agent edit path in create_history.rb; (b) FE: propagate name-updated into agentDataForSetup/props.agentData (or add a name guard) so the watch stops reverting; (c) confirm persistence before the success toast (AiAgentModal.vue:1042-1046). The "press-Enter-first" symptom (BOT-4094) is pixel3 MpTextarea v-model commit timing (in node_modules); the app-side flush at AiAgentModal.vue:873-896 is correct — (a) is the load-bearing fix.

C. KB ingestion lifecycle (BOT-4746, 3635, 3540, 2964, 2903, 4240, 4142, 2175)

1. Stuck in "Processing" forever. V1 AiKnowledgeSource has no worker, no timeout, no terminal-failure; status refreshes lazily only when the list is opened (get.rb:33 → update_status.rb). determine_status (update_status.rb:51-53) blindly upcases the AI-service value; anything outside the enum raises and is swallowed by the rescue at l.44, so the row silently stays IN_PROGRESS. Fix: add a scheduled poller with a poll budget that forces IN_PROGRESS → FAILED after N attempts — the V2 path already implements exactly this: mirror refresh_capability_vector_status.rb:35,198-220 plus refresh_ai_agent_vector_status_worker.rb. Map unknown statuses safely instead of upcase-and-swallow.
2. source_type does not have a valid value (BOT-2903). Enum divergence: the model allows FILE/URL/TEXT/API/OCR (en.yml:273-278), but endpoints disagree — ai_knowledge_sources.rb:20 allows FILE/URL/CONVERSATION and v2/ai_assist.rb:100 allows FILE/URL/TEXT/OCR. The FE sends URL/FILE/TEXT plus OCR (AddKnowledgeDrawer.vue:477-481) and passes the list filter value through raw (knowledge-list.vue:436-442), so an unaccepted value hits Grape's values: validator and the raw error is surfaced verbatim (knowledge-list.vue:449-460). Fix: reconcile the lists to one canonical enum; have the FE normalize/validate before sending.
3. Missing delete/resync (BOT-4240). The FE renders row actions only for row.type ∈ {URL, FILE, TEXT, OCR} (knowledge-list.vue:167-264); an unknown or empty type renders no popover at all. Fix: add a default fallback action group so Delete always renders. (BE delete.rb has no status guard, so stuck rows are deletable once the button exists.)
4. No FE polling. The status badge is static (knowledge-list.vue:122-144); "Processing" never updates without a manual refresh and never times out. Fix: poll with a client-side give-up.

F. Availability / latency (BOT-4739 P1, BOT-4336) — app-side

• The async path has no watchdog. If the AI service accepts the async request (2xx) but never calls back /predictions/async/reply, the BE sets text = '{{EMPTY}}' and returns with no scheduled fallback (send_message_with_resolve.rb:1931-1938; the idempotency key TTL is only 60s in process_async_reply.rb:18). This is the most likely code cause of "no reply / ~1h delay" when the AI service is degraded but not returning a fast error.
• Fix: at async dispatch, schedule a delayed fallback job keyed by (room_id, message_id) that fires human handover or the default answer if no callback arrives within the timeout; the callback cancels it. The primitives already exist: _execute_ai_assist_fallback (send_message_with_resolve.rb:1824-1831), _assign_agent (get_answer.rb:198-244), and process_fallback (process_async_reply.rb:185-196).
• There is no retry (max_retry: 0) and no circuit breaker on the answer path (lib/http.rb, prediction.rb:43, ai_agent.rb). Consider a lightweight breaker plus surfacing 504/408.

B. Language ID↔EN (BOT-4726, 4264, 3720) — app-side, cheap

• The AI Assist payload carries no language field (payload_v2.rb, custom_prompt.rb). The AI Agent has a language config (profile.rb:67, default 'EN') that is sent only at training time (sync_to_ai_service.rb:83), not per message (send_message_with_resolve.rb:1902-1920). The 'EN' default explains English replies to Indonesian users.
• Fix: (a) add an explicit "respond in <language>" instruction to the prompt/payload; (b) include the agent's language in the per-message prediction payload; (c) default to the detected user/org locale instead of 'EN'.

A. Answer quality (~11 tickets) — mostly a cross-team DSAI dependency

• Retrieval, embedding, and ranking run in the external AI Service, not in this repo (lib/ai_service/knowledge_base.rb, docs/architecture/flows/knowledge-training/README.md). This is why the cluster is chronically Won't Fix.
• The app-side lever is hardening the re-embed on KB edit. post_knowledge_sources.rb:57-78 re-embeds with timeout: 5 and swallows failures into a non-2xx code (http.rb:152-157) with no retry or verification, so a failed rebuild leaves the old vectors in place — exactly the "outdated KB answer reappeared" report (BOT-4311). Fix: verify create_vector_db success, retry, and surface rebuild failures.
• Recall problems such as "same KB, one question answered, another not" (BOT-2690) are genuinely inside DSAI. Per the dependency-relevancy rule, flag these with an avoidability check before they enter any PRD or RFC.

Action items

#	Action	Evidence	Cluster · Tickets	Priority	Effort
A1	Add an async-reply watchdog: a delayed fallback job keyed by (room_id, message_id), cancelled by the callback	send_message_with_resolve.rb:1931-1938; reuse _execute_ai_assist_fallback l.1824-1831	F · BOT-4739 (P1), 4336	P0	M
A2	BE: write back ai_agents.name/description on the existing-agent edit path	create_history.rb:114-136	D · BOT-4492, 4383	P0	S
A3	FE: propagate name-updated into the canonical agentData (or add a name guard); gate the success toast on real persistence	AiAgentModal.vue:865-867,1042-1046; AiAgentIdentityForm.vue:346-368	D · BOT-4492, 4094	P0	S
A4	Add a scheduled poll-budget worker forcing IN_PROGRESS → FAILED; stop swallowing invalid statuses	mirror refresh_capability_vector_status.rb:35,198-220; fix update_status.rb:44,51-53	C · BOT-4746, 3635, 3540	P1	M
A5	Reconcile the source_type enum across endpoints to one canonical set; FE pre-validates before sending	ai_knowledge_sources.rb:20, v2/ai_assist.rb:100, en.yml:273-278; knowledge-list.vue:436-442	C · BOT-2903	P1	S
A6	FE: default row-action group so Delete/Resync always render for an unknown row.type; add status polling with a give-up	knowledge-list.vue:122-144,167-264	C · BOT-4240, 3635	P1	S
A7	Add language to the per-message payload plus an explicit "respond in <lang>" instruction; default to user/org locale, not EN	send_message_with_resolve.rb:1902-1920; payload_v2.rb; profile.rb:67	B · BOT-4726, 4264, 3720	P1	S
A8	Harden the re-embed on KB edit: verify create_vector_db, retry, surface rebuild failure	post_knowledge_sources.rb:57-78; http.rb:152-157	A/C · BOT-4311	P2	S
A9	Flag the DSAI dependency: raise answer recall/quality (cluster A core) with the AI-Service team; run an avoidability check before any PRD/RFC	—	A · BOT-2690, 3246, 2743, 3718	P2	—

Suggested sequencing: A1 plus A2/A3 first (closes the open P1 and the most trust-damaging config bug), then the C batch (A4–A6) and B (A7) as a fast follow, A8 alongside C, and A9 as a parallel cross-team conversation.

Caveats

• Every root cause is grounded in chatbot / chatbot-fe at the cited file:line.
• One item is unverifiable from our repos: the pixel3 MpTextarea v-model commit timing lives in node_modules, so the "press-Enter-first" symptom cannot be confirmed here — but the BE write-back (A2) is the load-bearing fix regardless.
• Cluster A's core is not an app fix; treat it as a DSAI dependency rather than a Fast Track engineering task.