Runbook — Webhook hold transitions & dual-emit guard (BIF-8743 / T4)

For orgs on the hold/settlement path, delivery/failure webhooks now only move a hold's state — no live pool deduction, no WaConversationLog row, no live broadcast report. Money moves once, at EOD settlement (T5).

State transitions (NewPricingWaDeduction#call)

Webhook status	Hold state	Result
delivered / read	held	→ delivered (+delivered_at)
delivered / read	delivered/terminal	no-op (idempotent)
failed (/expired*)	held or delivered	→ refunded (reserve released, nothing charged)
failed (/expired*)	terminal	no-op

* 'expired' is a defensive entry only — Meta's Cloud API documents just sent|delivered|read|failed. TTL expiry arrives as failed with an expiry/undeliverable error (131026/131049 family), or as no webhook at all — the silent case is exactly what the T6 sweeper's held → expired (30-day) transition covers. The internal hold state expired is set only by the sweeper, never by a webhook.

Mechanics worth knowing:

• The hold row itself is the routing signal. The transition hook looks up wa_balance_holds.external_id = wamid (indexed, idx_wbh_external_id) at the very top of #call — before the pricing guards, because failed/expired webhooks may carry no pricing payload. Hold found ⇒ transition and return; no hold ⇒ fall through.
• Missing-hold guard — if the org IS toggled but no hold matched (send-side miss), the legacy deduction is never run as a fallback (mutual exclusion, SC-3). The webhook returns Success, logs a warning, and emits the wa_hold_missing_on_webhook Datadog metric (enable via preference wa_hold_missing_on_webhook_metric + DATADOG_ENABLED). The miss is a bounded under-charge (SC-4) that EOD settlement reconciles.
• Dual-emit guard is structural — a toggled org returns from the transition hook or the missing-hold guard before the live deduction/WaConversationLog/broadcast-report code, so the live broadcast report can't double-count with T5's settlement report.
• Concurrency — StaleObjectError/QueryCanceled re-enqueues to Billings::WaDeductionWorker (same recovery as the live path); a concurrent duplicate transition hits the partial unique (external_id, state) and is treated as already-done (RecordNotUnique → Success).
• The transition clears the org's wa_reserved Redis cache so Available balance updates on the next gate check.
• Toggle-off cost: one indexed point-SELECT on an (initially empty) partial-indexed table per delivered/read/failed/expired webhook; the legacy path is otherwise byte-identical.

Known consequences — pending stakeholder sign-off (BIF-8824 / RFC OQ-7)

For toggled orgs, wa_conversation_logs behavior changes — implied by the RFC ("report row written only by settlement, only Meta's number", SC-1) but needing explicit product/analytics sign-off before the pilot:

1. Free/service conversations get NO log rows. Legacy wrote a billed_to='free' row for every conversation; on the hold path free categories never get holds, the webhook returns at the missing-hold guard, and settlement only writes rows for settled holds. If any consumer counts service-conversation volume from this table, the cheap fix is to keep writing the free-only row from the webhook path (no money movement).
2. Billable rows arrive next-day (settlement ~01:00 ICT), not live. Intraday spend visibility is the Available balance, not the log.

Hold outcome	WaConversationLog row?
delivered → settled at EOD	✅ by T5, Meta's actual amount (reconciliation_settlement)
refunded / swept expired	❌ never — nothing was charged
shortfall at 30-day horizon	✅ by T5 (reconciliation_shortfall)
free/service (no hold)	❌ never (⚠️ pending BIF-8824 decision)

Monitoring

-- Transition health for a pilot org (chat_billing shard):
SELECT state, count(*) FROM wa_balance_holds
WHERE organization_id = '<organization_id>' AND created_at >= now() - interval '1 day'
GROUP BY 1;
-- Expect held → delivered flow during the day; refunded for failed sends;
-- a growing 'held' backlog means delivery webhooks aren't matching (check wamids).

Watch wa_hold_missing_on_webhook — near-zero is healthy (SC-4); a spike means CreateHold (T2) is missing sends (wiring gap or hold-insert failures).

Rollback

Toggle the org off — webhooks for new messages route to the legacy live deduction again (their wamids have no holds). Existing holds keep transitioning (hold presence routes them), which is exactly what you want: they'll be settled or refunded, never double-charged.

Sources (grounded against hub_core)

• app/core/domains/repositories/v2/billings/new_pricing_wa_deduction.rb — transition_wa_balance_hold (top-of-#call hook), report_missing_wa_hold (guard before the deduct transaction).
• Spec: co-located new_pricing_wa_deduction_spec.rb — "Deduction V2 hold transitions (BIF-8743)": delivered/read idempotency, refund from held AND delivered, terminal no-op, missing-hold metric + no fallback deduction, toggle-off legacy path unchanged.