Skip to main content

Auto-resolve stale monitoring alerts when the anomaly clears

Jira: TBD — to be minted as a Story under Epic BIF-8695 · Epic: BIF-8695

Moderator panel → Support › Monitorings.

Overview

  • Before: The daily Hologres (datamart) ingest only ever creates or updates monitoring alerts for CIDs still flagged anomaly_status = 'Anomaly'. It iterates only over CIDs that are still anomalies, so an existing open alert whose CID has dropped out of the next day's anomaly set is never touched — it lingers open forever and clutters the support monitoring queue with a self-cleared false positive.
  • After: On the daily ingest, an open alert that (a) no one has edited and (b) whose CID is no longer in the fresh anomaly set is auto-resolved — its status is moved off open and a system log is written — so the queue reflects only live, actionable anomalies.

This originated as an "add deletion logic" request; per review it is implemented as an auditable auto-resolve (status flip + log) rather than a hard row delete, so there is a trace of what the system cleaned up.

Scope Changes

  • Backend (moderator-be) — extend the daily Hologres ingest to auto-resolve stale, untouched open alerts whose CID is absent from the fresh anomaly set.

Technical notes (grounding)

  • Ingest: Core::UseCases::MonitoringAlerts::IngestFromHologres (app/domains/core/use_cases/monitoring_alerts/ingest_from_hologres.rb), scheduled daily 03:00 Asia/Jakarta (config/schedule.yml). The fresh anomaly CID set is the company_id (→ external_company_id) column of the Hologres rows where anomaly_status = 'Anomaly' — already in memory during the run.
  • Statuses (MonitoringAlert::STATUSES): open, in_progress, fixed, on_hold. Only open is eligible — in_progress/on_hold are being handled manually and are already skipped by today's ingest; fixed is inactive.
  • "Edited by anyone" maps to real fields: last_updated_by blank and internal_comment blank and no monitoring_alert_logs. The ingest's own create/update never sets those, so an untouched alert has them all empty.
  • Auto-resolve = set current_status off open + write a MonitoringAlertLog (log_type: 'status') attributed to the automated actor.

Out of scope

  • No change to alert creation/update behaviour for CIDs still in the anomaly set.
  • No UI change beyond the alert leaving the open queue (existing status/list views already render by current_status).

8. User Stories + Acceptance Criteria

User StoryImportanceMockup / Technical NotesAcceptance Criteria
[MONALERT-S01] — Auto-resolve stale, untouched open alerts

As the support/moderator team, I want an open monitoring alert to auto-resolve once its anomaly has cleared in the next datamart refresh and no one has touched it, so that the monitoring queue only shows live, actionable anomalies instead of self-cleared false positives.
Should HaveRuns inside the daily Hologres ingest (the fresh anomaly CID set is already loaded). Eligible = current_status: 'open' and untouched (last_updated_by, internal_comment blank, no logs).— Happy Path —
• AC-1: Given an open alert that no one has edited (no last_updated_by, no internal_comment, no monitoring_alert_logs), when the daily ingest runs and that alert's external_company_id is not in the fresh anomaly set, then the alert is auto-resolved (moved off open) and a system status log is recorded.
• AC-2: Given an open alert whose CID is still in the fresh anomaly set, when the ingest runs, then it is updated in place as today and not auto-resolved.
— Guard-rails —
• NEG-1: Given an alert that has been edited (any of last_updated_by set, internal_comment present, or ≥1 log), when its CID drops out of the anomaly set, then it is not auto-resolved — a human is/was handling it, so it is left as-is.
• NEG-2: Given an alert in in_progress / on_hold / fixed, when the ingest runs, then this logic does not touch it (only open alerts are eligible) — unchanged from today's behaviour.

Open questions

  1. Resolved status — reuse the existing fixed status, or add a dedicated auto_resolved status so system cleanup is distinguishable from a human fix? (Affects STATUSES enum + any status filter in the UI.)
  2. "Untouched" definition — is last_updated_by blank a sufficient signal, or require the full belt-and-suspenders (also zero logs and blank internal_comment)? AC-1 currently assumes the full check.
  3. System attribution — what updated_by_email / updated_by_id value identifies the automated actor on the log (e.g. system / datamart-ingest)?
  4. "Next day" gating — is "not in the current anomaly set" sufficient, or must the alert have survived ≥1 prior ingest cycle? (An alert created in the same run is always in the current set, so it is inherently safe from same-run resolution — confirm this is enough.)