Skip to main content

AI service async deployment downtime and late-reply handling

Context​

The AI Service team reported downtime for the end-to-end asynchronous process during a deployment. The affected process is used by Qontak Support. Dimas Fauzi Hidayat raised the issue as urgent and requested a list of Autonomous AI Agent implementations already using the asynchronous flow so the team could prepare for Support reports.

The downtime was reported as roughly 1.5 hours, from 16:50 to 18:30 on 2026-07-23. The team found no Chatbot alert for the failure.

Notes​

  • Eko Aprianto identified five implementations using the async flow:
CIDCustomer / workspace
382618Telesales Qontak
635808Qontak Support
473584PERSEROAN TERBATAS - BADAN ASIA OUTSOURCING SERVICES
796793Transgo.id / Travel Pro
544392Sandbox Chatbot 7 - Ikhsan Trial
  • Sample affected room:
    • https://chat.qontak.com/inbox-v2?status=unassigned&room=f786c4d8-921e-41da-9e73-e9970207d2f7
  • Puji Triwibowo explained the expected async flow:
    1. Chatbot asks AI Service to process an async request.
    2. AI Service sends a webhook after processing finishes.
  • During the failure, Chatbot successfully triggered AI Service, but AI Service errored and did not send the completion webhook.
  • Chatbot only processes an async result after receiving that webhook. Because no webhook arrived, the existing flow did not trigger an alert.
  • Dimas proposed detecting rooms for which AI Service had not sent a webhook within five minutes, potentially using a 10% sample of rooms using AI Agent.
  • Puji noted that sampling could produce false alarms because the traffic volume is low.
  • Puji suggested a sweeping process as another option, but noted that it could be resource-intensive.
  • Eko Aprianto proposed adding Chatbot-side tracing for async requests.
  • The proposed handling uses a delayed background job to check whether the AI has responded.
  • Puji Triwibowo noted that the Data team may provide a polling endpoint for fetching the result; this is unconfirmed.
  • The team discussed preventing a very late async response from being forwarded to the user after the timeout path has already run.
  • Dimas raised a concern that retrying an async request could disrupt the conversation context.
  • Eko highlighted the risk of an async reply arriving tens of minutes later.
  • Dimas noted that the team should respond more proactively to incidents like this rather than only reacting after users make requests.
  • Puji confirmed the RCA template:
    • https://jurnal.atlassian.net/wiki/spaces/DEVOPS/pages/49426071557/Template+-+RCA+V2

Decisions / changes​

  • The five implementations listed above form the current Support-readiness inventory for this incident.
  • Use a five-minute response budget for the async AI response.
  • The detailed behavior after five minutes was discussed but is not yet a confirmed implementation design:
    • Add a log or trace.
    • Apply an AI fallback if no response has arrived.
    • Do not forward a response that arrives after the allowed window.
  • A 10% room-sampling alert and a full sweeping process were discussed but not adopted because of false-alarm and resource-cost concerns.
  • Retry behavior and its effect on conversation context remain unresolved.
  • The root cause of the AI Service downtime itself remains with the AI Service team and was not established in this conversation.

Follow-ups​

  • Validate and maintain the five-entry Autonomous async implementation inventory for Support readiness.
  • Ask the AI Service team to provide the deployment downtime root cause and prevention measures.
  • Confirm whether the Data team provides a polling endpoint and document its contract.
  • Evaluate delayed-job tracing, polling, sampling, and sweeping approaches against traffic volume, detection coverage, false-alarm risk, and resource cost.
  • Define the timeout, fallback, late-response suppression, retry, and idempotency behavior in the relevant RFC or decision log before implementation.
  • Add observability that distinguishes delayed, timed-out, retried, suppressed-late, and successfully processed async replies.
  • Prepare an RCA using the confirmed RCA V2 template and document proactive detection and communication improvements.
  • Record confirmed customer or delivery impact in a dated status update.