Skip to main content

Bot space alerts and Datadog latency no-data false alarm

Context​

Dimas Fauzi Hidayat reported that alerts in the Bot space had been firing since the previous night. Although the service recovered, he asked the team to investigate because the preceding weeks had been quiet.

Notes​

  • Farras Herdicho identified one alert as a false alarm.
  • Datadog could not retrieve latency data and reported a no-data condition.
  • Farras planned to raise the Datadog issue with DevOps.
  • Dimas noted that there appeared to be other alerts in addition to the latency no-data alert.
  • The identities and causes of the other alerts are unconfirmed.

Decisions / changes​

  • Treat the known Datadog latency no-data alert as a false alarm.
  • Continue investigating the additional alerts rather than assuming they share the same cause.

Follow-ups​

  • Farras to follow up with DevOps about Datadog being unable to retrieve latency data.
  • Review the other Bot-space alerts, identify their causes, and confirm whether any indicate a real service issue.
  • Record any confirmed delivery blocker or incident impact in a dated delivery status update.