Skip to main content

Runbook — Stale-hold sweeper, 30-day expiry (BIF-8745 / T6)

Billings::SweepStaleWaHoldsWorker (hub_core) releases reservations for messages that were sent but never delivered. A held row whose webhook never arrived (see webhook-hold-transitions.md — Meta may send no webhook at all on TTL expiry) would reserve balance forever; past the ~30-day Meta window it can no longer deliver, so the sweeper flips it to expired.

What it does — and deliberately does NOT do​

  • Scope: only state='held' rows with held_at < 30.days.ago (STALE_HOLD_HORIZON_DAYS). delivered holds belong to settlement (eod-settlement-settle-daily.md); terminal states (settled/refunded/expired) are never revisited. held rows with a NULL held_at are skipped (nothing to age against).
  • No money movement — pools/credit are untouched; only settlement moves money. Expiring releases the reserve: the hold leaves the held|delivered set that reserved_wa_balance sums, and the short-TTL Redis cache is cleared per touched package so the freed reserve is visible to the very next send-gate check.
  • Race-safe: rows are processed in_batches(of: 500) with a per-row with_lock re-check — a hold the delivery webhook flips to delivered between the batch query and the write is left alone (it settles via T5 instead).
  • retry: 0, queue billing_settle_wa_holds; a crashed run self-heals on the next night (stale rows only get staler). Unexpected errors go to Rollbar; the run logs expired N stale holds.

Scheduling​

Register in the host app's Sidekiq Cron after the settlement dispatcher (T7), so a hold that could still settle today is consumed by settlement before the sweeper sees it:

sweep_stale_wa_holds:
  cron: '0 23 * * *'   # ~06:00 Asia/Jakarta — after the settlement fan-out has drained
  class: 'Billings::SweepStaleWaHoldsWorker'
  queue: billing_settle_wa_holds

The 30-day horizons are deliberately aligned: settlement force-closes unfilled batches at 30d (shortfall), the sweeper expires undelivered holds at 30d — nothing reserves or stays pending forever.

Verify a run​

-- chat_billing shard: what just expired
SELECT organization_id, count(*), sum(estimated_amount)
FROM wa_balance_holds
WHERE state = 'expired' AND expired_at >= now() - interval '1 day'
GROUP BY 1 ORDER BY 2 DESC;

-- leak check: nothing held should be older than the horizon after a green run
SELECT count(*) FROM wa_balance_holds
WHERE state = 'held' AND held_at < now() - interval '30 days';

The second query should return 0 after every successful nightly run.

Triage​

  • Large expiry counts for one org — that org's messages are being sent but their status webhooks never arrive (webhook wiring broken, or wamids never reach the send subscriber). Correlate with the T4 wa_hold_missing_on_webhook metric and the settlement shortfall records; fix the webhook path — the sweeper is masking a delivery-signal gap, not causing one.
  • A hold expired that later did deliver — the webhook transition treats terminal states as no-ops, so the delivery is intercepted but nothing is charged (bounded under-charge, SC-4). If Meta billed for it, it surfaces as a settlement shortfall on that bucket. No manual reversal — fix the latency source.
  • Manual run (safe, idempotent — expired rows don't match the query again): Billings::SweepStaleWaHoldsWorker.new.perform