Skip to main content

Runbook — EOD settlement against Meta (BIF-8744 / T5)

At end of day the hold ledger is reconciled against Meta's actual cost to the cent: Billings::SettleWaHoldsDispatcherWorker → Billings::SettleWaHoldsWorker → Billings::Repositories::V1::WaHoldSettlement::SettleDaily (all in hub_core). Holds created at send (create-hold-at-send.md) and marked delivered by webhooks (webhook-hold-transitions.md) are settled FIFO here — this is the only place money moves for orgs on the hold/settlement path.

What it does​

For every (waba_id, phone_recipient, category, meta_date) bucket:

  1. Meta cost/volume are fetched per Meta day (MetaPricingAnalytics, window (today−7d)..(today−1d) in the WABA timezone) and frozen into a wa_reconciliation_batches row on first sight (find_or_create_by on the unique bucket key — idempotent across re-runs and concurrent workers). The window is lazy: a day that already has batch rows is skipped (totals are frozen anyway), so steady state is one Meta call per WABA per night (~6k calls fleet-wide at GA, not ~42k); the trailing week is only fetched for days a previous run missed. A zero-usage day yields no rows and is re-probed until it ages out of the window — a cheap, empty response.

  2. Unassigned wa_balance_holds are consumed FIFO (delivered_at ASC, aged held last) up to meta_volume. Consumed states: delivered, plus held holds older than AGED_HELD_THRESHOLD (1h) — a still-held hold Meta already charged for is a lost delivered webhook, so it settles with its real customer_ref/country/broadcast_id (BIF-9080 Enhancement A); recent held rows are genuinely in-flight and skipped. The batch↔hold match is normalization-tolerant (BIF-9080 Bug #1/#2): phone_recipient matches both the normalized .to_phone key and the legacy +-prefixed twin, and the category match accepts hold-vocabulary aliases — a MARKETING_LITE batch also consumes legacy marketing-tagged MM Lite holds.

  3. Each hold settles at the Meta unit price (meta_cost_total / meta_volume); the hold that completes the batch takes the rounding residual so Σ settled_amount == meta_cost_total exactly.

  4. Pools move through the live-deduction ladder — balance_initial → balance → postpaid (v3 postpaid only) → negative balance. Each hold is charged its Meta base share + the org's configured margin (WaPricing total_price − base_price: conversation fee, discount, tax — the same components live deduction charges). One optimistic-locked whatsapp_packages write per bucket.

  5. One actual-amount wa_conversation_logs row per hold with origin_type='reconciliation_settlement', is_auto_deduct=false. base_price mirrors Meta exactly (reporting/reconciliation); total_price/credit carry the amount actually deducted (base + margin). Monthly-reset gap queries (which filter is_auto_deduct: true, single_reset_package.rb) ignore these rows.

  6. Holds with a message_broadcast_id enqueue Billings::BroadcastDeductionReportWorker with the actual settled amount.

  7. Phantom settlement (BIF-9080 Enhancement B): at the 30-day horizon a bucket whose Meta volume was never fully covered by holds no longer under-charges — the residual (meta_cost_total − Σ settled base) is deducted as one synthetic wa_conversation_logs row (origin_type='reconciliation_gap_deduct', conversation_id='gap-deduct-<batch.id>') so Σ settled_amount == meta_cost_total exactly. Base is Meta's residual; margin is WaPricing's markup × phantom count, priced with a blended country (most common country among the batch's already-settled sibling holds; WaPricing OTHER/business-number fallback if the bucket settled no hold). Always on (new flow, not flag-gated); idempotent (the batch closes settled, so a late hold for the slot can never re-settle).

Key code (hub_core): app/apps/billings/repositories/v1/wa_hold_settlement/settle_daily.rb (find_or_create_batch, fifo_holds / phone_recipient_candidates / hold_categories_for, hold_amount, apply_ladder, close_at_horizon! / gap_deduct_phantom! / phantom_margin / blended_country, report_broadcast_deductions). Phone normalization on the hold write path: normalize_phone_recipient in app/core/domains/repositories/billings/helpers.rb (applied in create_hold.rb); MM Lite hold tagging in app/apps/wa_cloud/repositories/broadcast/send.rb.

Scheduling​

Add the dispatcher to the host app's Sidekiq Cron schedule (hub_core has no production cron registry — it is not self-scheduling):

settle_wa_holds_dispatcher:
  cron: '0 21 * * *'   # 04:00 Asia/Jakarta — after Meta closes the WABA day
  class: 'Billings::SettleWaHoldsDispatcherWorker'
  queue: billing_settle_wa_holds

Both workers run with retry: 0 on queue billing_settle_wa_holds; the per-org worker is throttled to 1 000 jobs/hour. Because the fetch window covers 7 days and re-runs are idempotent, a missed night self-heals on the next run — no manual catch-up needed inside 7 days.

Feature gating / kill switch​

Enablement gate — company-scoped Services::Billing::FeatureFlag (as-built; see wa_hold_settlement-flag-registration.md):

TierHowEffect on dispatcher
Global GAwa_hold_settlement preference is_global = true (billing-preferences service)every package is dispatched — all statuses, trial or not (the dispatcher enumerates organization_packages and filters via wa_hold_settlement_enabled?)
Pilotadd the company_id to the wa_hold_settlement preference_unique_ids allow-listonly opted-in companies (any status/trial)

organization_packages.extras['wa_hold_settlement'] is not the mechanism — that two-tier Services::Preference ∨ extras design (RFC Decision 6) was not shipped.

Turning the flag off does NOT stop settlement for organizations that still have open (held/delivered) holds — the dispatcher keeps enqueuing them until the ledger drains. This is deliberate: the webhook path never live-deducts a wamid that has a hold row (dual-emit guard, T4), so draining via settlement is the only way that money moves. To stop settlement entirely, stop the cron entry.

SettleDaily itself is not flag-gated (drain semantics) — invoking it manually settles regardless of the flag.

Invariants (what "correct" looks like)​

Two ledgers, one per audience: batch amounts are Meta-base-only (reconciliation against Meta); hold settled_amount and log total_price/credit are the client charge (base + margin); the log's base_price bridges the two.

  • Per settled batch: settled_amount + unmatched_cost == meta_cost_total and Σ wa_conversation_logs.base_price (origin_type='reconciliation_settlement', batch's holds) == settled_amount.
  • Pool movement per batch == Σ wa_conversation_logs.credit == Meta base share + margin × holds.
  • A hold is settled at most once: state='settled' ⇒ reconciliation_batch_id set, and the (external_id, state) partial unique index blocks duplicates.
  • Re-running any day produces zero additional movement (Meta totals are frozen on batch creation; settled batches are skipped).
  • Settlement never touches wa_credit. COST==0 buckets settle their holds at 0 with billed_to='free', no pool movement and no margin.

Routine operations​

Re-run settlement for one organization​

Safe at any time (idempotent):

Billings::SettleWaHoldsWorker.perform_async(organization_id)
# or synchronously, one WABA, explicit day:
Billings::Repositories::V1::WaHoldSettlement::SettleDaily.new(
  organization_id: org_id, waba_id: waba_id, timezone: 'Asia/Jakarta'
).call

Inspect pending (unreconciled) batches​

-- chat_billing shard
SELECT waba_id, phone_recipient, category, meta_date, meta_volume, settled_count,
       meta_cost_total, settled_amount
FROM wa_reconciliation_batches
WHERE state = 'pending'
ORDER BY meta_date ASC;

A batch stays pending while settled_count < meta_volume (late deliveries). It is reopened by every subsequent run and force-closed at the 30-day horizon (SETTLEMENT_HORIZON_DAYS), where any uncovered residual is phantom-deducted (Enhancement B) so the closed batch has unmatched_cost == 0.

Verify a batch settled to the cent​

SELECT b.meta_cost_total, b.settled_amount, b.unmatched_cost,
       COALESCE(SUM(l.base_price), 0)  AS meta_base_total,   -- must == b.settled_amount
       COALESCE(SUM(l.credit), 0)      AS client_charged     -- base + margin (pool movement)
FROM wa_reconciliation_batches b
LEFT JOIN wa_balance_holds h ON h.reconciliation_batch_id = b.id
LEFT JOIN wa_conversation_logs l ON l.external_id = h.external_id
       AND l.origin_type = 'reconciliation_settlement'
WHERE b.id = :batch_id
GROUP BY b.id;

Expect meta_base_total == settled_amount and settled_amount + unmatched_cost == meta_cost_total; client_charged exceeds meta_base_total by the org's margin per message.

Alerts and what to do​

Rollbar: [wa_hold_settlement] reconciliation gap phantom-deducted at 30-day horizon​

A bucket hit the horizon with settled_count < meta_volume: Meta billed for messages we hold no matching reservation for. As of BIF-9080 Enhancement B the gap is deducted, not left as a shortfall — the residual is charged as one synthetic origin_type='reconciliation_gap_deduct' log row (client charge = Meta base + blended-country margin), unmatched_cost closes at 0, and Σ settled_amount == meta_cost_total. The alert is now observability, not a lost charge. Datadog counter: wa_hold_settlement_shortfall.

The pre-BIF-9080 behaviour — record unmatched_cost + a reconciliation_shortfall log, never deducted (RFC SC-4 bounded under-charge) — has been removed. There is no longer a reconciliation_shortfall origin type.

Triage (the charge is now correct; alert tells you why holds were missing):

  1. Compare hold count vs Meta volume for the bucket — a systematic gap usually means CreateHold was not wired for that send path or the wamid never reached the send subscriber (correlate with the T4 wa_hold_missing_on_webhook metric). Fix the hold-creation gap so real holds (with exact per-country margin) settle instead of the blended phantom.
  2. One-off small gaps are expected noise (Meta counts a message we never created a hold for, e.g. sent outside Qontak) — now billed via the phantom entry, no action needed.
  3. The batch is closed and fully billed. Do not manually reopen; a late hold for the slot is ignored (no double-charge).

Sidekiq: ActiveRecord::StaleObjectError re-enqueues​

Optimistic-lock conflict (webhook transition or a concurrent run touched the same hold / whatsapp_packages row). The worker swallows it and re-enqueues the whole org; each bucket commits in its own transaction, so the retry only picks up what did not commit. A tight re-enqueue loop (> ~10/min for one org) means something is hammering the same whatsapp_packages row — check for a stuck webhook replay before doing anything else.

Meta API failures​

A failed pricing_analytics fetch skips that Meta day only (logged as SettleDaily meta fetch failed); pending batches still process. A missed day has no batch rows, so the lazy window keeps retrying it on every run while inside the 7-day window (META_FETCH_WINDOW_DAYS). If Meta was down for longer than 7 days, run SettleDaily manually with today: set so the affected days fall inside the window — batch creation is idempotent. Note the lazy-window edge: a day that already produced some batch rows is never re-fetched, so a bucket Meta reports late for that day is not picked up — it surfaces as a shortfall at the horizon instead.

Known sharp edges​

  • Meta totals are frozen at first sight. If Meta restates a day's cost after the batch row was created, settlement will not pick up the new figure. Detect via the whatsapp_usage_comparison reports (daily-usage recorder); remediation is a manual finance adjustment, not a code path.
  • Buckets match holds on (waba_id, phone_recipient, category), normalization-tolerant. phone_recipient matches the normalized .to_phone key or its legacy +-prefixed twin, and the category match accepts hold-vocabulary aliases (MARKETING_LITE ⇄ marketing). If a channel's server_wa_id genuinely changes (different number), old holds keep the old number and will not match new-number buckets; they are covered by the phantom deduct on the new number at the horizon and eventually swept (T6).
  • Residual allocation is unit-price, not cost/consumed.size. On a partial (late-delivery) pass each hold settles at meta_cost_total / meta_volume; the batch-completing hold (real or the horizon phantom) takes the residual. Dividing by consumed.size would front-load the whole cost onto early holds and mis-price the per-message base.
  • Phantom margin is blended, not country-exact. The horizon gap-deduct prices its margin with the most common country among the batch's already-settled sibling holds (or an OTHER/ business-number fallback). Base is Meta-exact; only the smaller margin component is approximated (bounded error). Country-exact phantom margin is out of scope unless finance needs per-country phantom reporting.