Skip to main content

Deduction V2 — Test Case Gap Analysis (vs. PRD, RFC, Task Breakdown & JIRA)

Generated: 2026-07-21 16:43 WIB (UTC+7) · AUGFLOW-185

Scope analyzed: bifrost/deduction-v2/testcases/deduction-v2.md (TC-01–TC-38) against prds/deduction-v2.md (v1.6), rfcs/deduction-v2.md + rfcs/deduction-v2.task-breakdown.md, delivery/decisions/0001 + 0002, delivery/status/*, documentations/*.md (as-built runbooks, cross-checked against live hub_core source), and JIRA epic BIF-8644 (live state, 2026-07-21).

Status: synthesized from 5 parallel investigations + direct primary-source reads, then put through one adversarial verification pass (a fresh agent instructed to refute every load-bearing claim against primary text). All corrections from that pass are folded in below.

0. The one fact everything else depends on​

There have been two design pivots, and the test-case file was never updated for either:

  1. ~2026-06-29: PRD's core mechanism (debit-at-webhook: hold created on webhook receipt, reversed/reinserted daily, v2_effective_date/v2_end_date delayed local-midnight toggle) was replaced by the RFC's authorize-at-send + capture-at-EOD design (hold reserved at send once wamid is known; webhook only transitions hold state; EOD SettleDaily allocates Meta's actual cost FIFO across delivered holds; enablement is a plain on/off flag, no effective-date columns at all).
  2. 2026-07-15 → 2026-07-20: DED-S08's gating design lived through its own mini-pivot — ADR-0001 (2026-07-15) adopted a per-range activation-boundary ("historical range keeps per-message, spanning range forces daily"), then ADR-0002 (2026-07-20) replaced it with the exact previously-rejected alternative: a flat per-company rule, no boundary date, no per-message mode at all for V2 companies.

Current source of truth, confirmed from primary evidence (not inferred):

  • Part 1 (T1–T7, the hold/settlement engine) is merged and Done (BIF-8740…8746). Verified directly against live hub_core source, not just the runbooks.
  • Part 2 (M0–M5, the usage-history/download UI reinstating DED-S07/DED-S08) is not started (BIF-8959/8883/8884/8885/8886/8887/8958 — mostly To Do, two In Progress, one Need Feedback).
  • The PRD (last touched 2026-07-15, with a narrow §9.2.1 patch on 2026-07-20) still describes the original debit-at-webhook design as its core mechanism. The RFC itself tracks this as an open, unresolved blocker (OQ-1): "the source PRD contradicts this design (send-blocking). The code is executable today; the approval gate needs the PRD aligned." This has been open since 2026-06-29 and was still open as of the 2026-07-21 status report.
  • The test cases (testcases/deduction-v2.md) read as written against the stale PRD plus the since-superseded ADR-0001 — there is no evidence in any document read of the test-case file being revised after either pivot.

Everything below is organized against this ground truth.

1. Missing test cases​

Behavior that is built (or specified) and has no test case at all in the current file.

#GapWhy it mattersAnchor
1.1Send-gate blocking over-spend (Available − cost < 0 → send blocked)This is the headline feature of the entire redesign (RFC SC-2) and the single reason the PRD needs a rewrite (it directly reverses PRD Non-Goal #4). Verified against all 38 TCs individually: none exercise it, and TC-03 actively asserts the opposite ("the balance is allowed to go negative"). The shipped behavior is concrete and testable today — available-balance-gate-and-display.md:20-21 names the exact failure: Failure 'cannot send message, insufficient balance' when available − cost < 0. Zero test cases exercise this path or assert that message.RFC Decision 1/5, T3 (BIF-8742, shipped); PRD §5 Non-Goal #4
1.2Double-charge / mutual-exclusion invariant (a message is never both live-deducted and held+settled)RFC's top-listed threat (SC-3). No test asserts an org can't be on both paths at once, or that toggling mid-flight doesn't double-charge an in-flight hold.RFC §1 SC-3, Decision 6 predicate
1.3Refund path (failed/TTL-expiry webhook → held|delivered → refunded, reserve released, nothing charged)TC-01–TC-07 only cover hold creation; none cover a failed send releasing its reservation.T4 runbook (webhook-hold-transitions.md), state machine
1.4Missing-hold-on-webhook path (webhook arrives, no matching hold found → log + metric, explicitly no fallback deduction)A deliberate fail-safe with no legacy fallback — an easy thing to regress without a test.T4 runbook; RFC §5 monitoring
1.5Stale-hold sweeper (30-day expiry, T6)Entire task has zero test-case coverage in the PRD-derived file.T6 runbook (stale-hold-sweeper.md), BIF-8745
1.6FIFO settlement mechanics: unit-price allocation, residual-to-batch-completing-hold, partial/late-delivery reopen, idempotent re-run, 30-day shortfall force-closeTC-08–TC-11 test "reverse yesterday's holds, insert settled" as a whole-day operation — none of the actual per-bucket FIFO/partial/reopen/shortfall mechanics are covered (see §2 for why the existing TCs are also wrong, not just incomplete).T5 runbook (eod-settlement-settle-daily.md); RFC Decision 4
1.7Free/service conversations produce zero WaConversationLog rows under V2 (vs. one row/conversation today)Confirmed as currently-shipped but explicitly flagged in the runbook as pending stakeholder sign-off (BIF-8824/RFC OQ-7). No test case exists either asserting current behavior or covering the sign-off decision either way.T4 runbook; BIF-8824 (Done — sign-off itself is closed, but the product consequence table still reads "pending" in the runbook — worth reconciling)
1.8Enablement predicate itself (Services::Billing::FeatureFlag.enabled?(:wa_hold_settlement, unique_id: company_id))TC-05/TC-07 test the old per-CID deduction_v2_enabled flag; none test the real global-flag-or-per-company mechanism, and — see §3.3 — even the as-built documentation of this mechanism is stale, so this is doubly untested.helpers.rb:61-63 (verified against live source)
1.9Part 2 APIs entirely: M2 GET usage API (day × WhatsApp-Business-number × category, reading wa_reconciliation_batches), M3 CSV download (always daily, no per-message mode, no 422 case), M0's settled_charged_amount correctnessThese are net-new API contracts with zero existing test coverage — TC-25–TC-38 test the old contracts (see §2), not these. Flagged separately since Part 2 hasn't shipped — see the scope note in §5.task-breakdown Part 2, ADR-0002
1.10Authorization for the manual settlement backfill/re-triggerConfirmed to exist as-built (SettleWaHoldsWorker.perform_async / SettleDaily.new(...).call), but no CAN/CANNOT table exists anywhere (PRD, RFC, or runbook) for who may invoke it — it's a rake/worker call with no visible role check. No test case, because there's no spec to test against yet.RFC §3 Security Implications (silent on this); T5 runbook

2. Test cases that exist but are not complete enough (or now assert the wrong thing)​

TC rangeStoryProblem
TC-01, TC-03, TC-04, TC-06, TC-07DED-S01All assert the hold is created on webhook receipt. As-built, the hold is created at send, the instant wamid is known; the webhook only transitions held→delivered or →refunded. These test cases assert the wrong trigger point for the wrong mechanism entirely, not just an incomplete one.
TC-08–TC-11DED-S02Assert "all yesterday's holds reversed, settled records inserted" as a single whole-day operation. As-built, settlement is per-bucket (waba_id, phone_recipient, category, meta_date), FIFO over delivered holds up to Meta's reported volume, with partial-fill reopen and a 30-day shortfall horizon — a materially different mechanism that happens to produce a superficially similar end state on the easy path. None of partial-fill, late-delivery reopen, re-run idempotency, or shortfall recording are exercisable from these four cases.
TC-16–TC-18DED-S04Built around a date_of_origin/WABA-local-midnight stamping model. As-built, there is no date_of_origin field; settlement buckets by Meta's own reported meta_date plus a waba_timezone column resolved per WABA via WaTimezoneResolver. This is a semantic change (whose clock decides the day), not a rename — these cases need rewriting against the real mechanism, not just re-pointing at a renamed field.
TC-25DED-S07Asserts grouping by day/category/destination country (e.g. "May 18 – Marketing – ID: 500 conversations"). ADR-0002/M2 formally supersedes this: the real API groups by day × WhatsApp Business number × category, with no destination_country column. Engineering has already flagged this divergence twice (task-breakdown lines 795, 995) as needing a PRD refresh — the test case inherits the same staleness.
TC-32–TC-38DED-S08Mixed, not uniform — corrected after adversarial re-read of each TC body. Not all seven encode ADR-0001's per-range activation boundary; some already match ADR-0002's flat rule and only the references are stale:
• TC-33 literally asserts "only 'Daily aggregation' is available and no activation-boundary lookup occurs" for a historical range — this is ADR-0002's flat-rule behavior already, and it directly contradicts the PRD's own original DED-S08/AC-2 (which TC-33 still cites as its References column) that says historical ranges should keep both options.
• TC-35 ("rejects per-message export for any V2 company") is also flat-rule-shaped, with no date qualifier.
• TC-32 leans flat-rule but is ambiguous.
• TC-37 and TC-38 are the genuinely stale ones — "V2-era range" phrasing presupposes a range-based distinction ADR-0002 eliminates entirely, and TC-35/37 specifically test the 422 per_message_unavailable_for_v2 error path M3's spec says will not exist ("there is no aggregation_by param, no per-message mode, and therefore no 422/rejection case to build").
What's uniformly wrong across all seven: the References column cites composite AC ids (DED-S08/AC-2, /AC-3, /ERR-1) that task-breakdown:994 and PRD §9.2.1 both call superseded, and TC-32/35 still name the retired deduction_v2_enabled flag instead of wa_hold_settlement. Net: rewrite TC-37/38 and the 422 assertions in TC-35; re-point references across all seven; TC-33 may already be closer to correct than the PRD it cites.
TC-31DED-S07 (cross-tenant 403)Conceptually still valid, but not verified against the actual M2 endpoint's real oauth2 scope list (:modpanel, :admin, :owner, :supervisor, :agent, :member) — needs re-grounding once M2 ships, not a rewrite.

3. Gaps in the implementation (test cases/stories exist, but engineering's task breakdown missed them)​

#FindingEvidence
3.1DED-S03 (Manual Re-trigger, BIF-8983) and DED-S04 (Timezone, BIF-8984) have no dedicated engineering task anywhere — not in Part 1 (T1–T7), not in Part 2 (M0–M5), not in the task-breakdown's own "Skipped Stories" closing table. This is a silent gap, in contrast to DED-S05/S06 which were explicitly and reasonably dropped with a stated rationale. The underlying mechanisms (SettleWaHoldsWorker.perform_async, WaTimezoneResolver) exist as byproducts of T5, but were never scoped, acceptance-criteria'd, or tested against the specific PRD behaviors (e.g. "returns confirmation of dates modified" — unconfirmed whether the real call returns this).task-breakdown.md (confirmed absent across both parts + closing table); T5 runbook (mechanism exists, undocumented against DED-S03/S04 identifiers)
3.2JIRA stories for DED-S03/04/05/06 (BIF-8983–8986) are empty template stubs — identical placeholder text, all four created within a verified 8.19-second window (2026-07-21T00:41:12.030 → 00:41:20.221), citing "PRD §8" (the API/Webhook Behavior section) rather than "PRD §9.2" (User Stories) like every substantively-authored story in the epic. This means JIRA gives no signal distinguishing "still needed, not yet scoped" (S03/S04) from "should have been closed as dropped" (S05/S06) — both pairs look identical in the tracker.JIRA live read, BIF-8983/8984/8985/8986
3.3✅ Addressed 2026-07-22 — the runbook (wa_hold_settlement-flag-registration.md) was rewritten to the as-built Services::Billing::FeatureFlag mechanism, and as-built banners + corrections were applied to the RFC (§1 Overview, Decision 6, dependencies, decision summary) and the task-breakdown T1. Original finding retained below. Doc documentations/wa_hold_settlement-flag-registration.md (the "authoritative as-built runbook" for T1) is itself stale against shipped code — confirmed twice, independently, against live source. It documented the original T1 commit's enablement predicate (Services::Preference global flag OR organization_packages.extras['wa_hold_settlement']), replaced by commit 3df1264923 — whose message states verbatim: "Replace the two-tier gate (Services::Preference global flag + per-org package extras opt-in) with a single Services::Billing::FeatureFlag lookup." Current shipped code (hub_core/app/core/domains/repositories/billings/helpers.rb:61-63): Services::Billing::FeatureFlag.new.enabled?(:wa_hold_settlement, unique_id: organization_package&.company_id). A repo grep for extras['wa_hold_settlement'] / extras[:wa_hold_settlement] returns zero hits — that tier is dead code. Anyone following the runbook to enable V2 for a pilot org, or writing QA test setup against it, would be operating on a column the code no longer reads. Note this drift is not invisible org-wide — the task-breakdown's own "Verified repo facts" and ADR-0002 already state this exact fact ("the RFC's Decision 6 text... is stale against the shipped T1 code") — so it's specifically the runbook and the RFC's own §2 body text that were never updated, not a fact nobody on the team knows.Direct source + git commit-message verification (3df1264923), hub_core worktree; task-breakdown:600-610; ADR-0002 (0002-modpanel-v2-detection-and-usage-api-ownership.md):42-53
3.4No dedicated RFC exists for Part 2 (DED-S07/S08's hub_core GET/download APIs + moderator-be proxy). The task-breakdown says so explicitly. This means the new cross-service call (moderator-be → hub_core, M5 — "the first such HTTP client in this codebase") never went through the same HA/Security review T1–T7 got. Timeout/retry/failure-mode behavior is described only at task-breakdown granularity, not reviewed at RFC depth.task-breakdown.md Part 2 preamble
3.5REV-2 (open blocker): the Available-only balance-read endpoint has no pinned contract — no path, response field name, before/after example payloads, or read-failure error code defined anywhere. A concrete test case cannot yet be written against this endpoint without an RFC follow-up landing first.RFC §5 Concerns/Questions, REV-2
3.6REV-3 (open blocker, compliance): wa_balance_holds.customer_ref may store a customer phone number in plaintext, while the equivalent field elsewhere (wa_conversation_log.phone_number) is Lockbox-encrypted. Unresolved, and infosec is a named RFC approver — this could gate release entirely.RFC §5, REV-3
3.7M4 (client hub-app frontend for DED-S07) is explicitly ungrounded — the hub repo isn't checked out anywhere in this workspace; its file paths are carried from the PRD's own unverified repo-analysis note. The task-breakdown itself flags this as "low confidence" and instructs re-grounding before work starts. Any gap-analysis (or QA plan) treating M4's scope as fixed is building on an acknowledged unknown.task-breakdown.md M4 section
3.8Unresolved open questions with no implementation yet: Meta-currency conversion (OQ-3 — not implemented in SettleDaily despite being flagged since 2026-07-15) and held_at IS NULL holds being permanently skipped by the sweeper (an edge case no doc explains how could occur, or how such a hold would ever be released).RFC OQ-3; T6 runbook
3.9Authorization model for the manual re-trigger is unspecified at the implementation-planning level (see also §1.10) — this is as much a "task breakdown missed it" gap as a "missing test" gap: no task in T1–T7 or M0–M5 scopes adding a role check to the rake/worker invocation.RFC §3 (silent); T5 runbook

4. Test cases available, PRD needs update​

#PRD sectionWhat's staleStatus of the fix
4.1§5 Non-Goal #4 ("does not implement balance-based sending restrictions... the deduction engine only records charges")Directly reversed by the shipped send-gate (RFC SC-2). This is the RFC's own tracked OQ-1, open since 2026-06-29, still open in the 2026-07-21 status report ("Governance gate... PRD refresh for the authorize-at-send reversal of Non-Goal #4... still open, PM-owned, not blocking engineering").Open — no ETA visible in any document.
4.2§7/§8/§9.1 (hold created on webhook, whole system flow)Structural rewrite needed: hold creation moves from "on webhook" to "at send"; webhook becomes a pure state-transition step; the reconciliation flow becomes per-bucket FIFO, not whole-day reverse/insert.Same OQ-1 blocker — no separate tracking.
4.3§8 behavior #5/#6, §9.2 DED-S05/DED-S06 (v2_effective_date/v2_end_date, delayed local-midnight toggle activation/deactivation)Entire mechanism dropped in favor of a plain on/off flag (global FeatureFlag predicate). Task-breakdown records this as an intentional, reasoned drop.Also folded into OQ-1 per the 2026-07-21 status note ("...and now the dropped v2_effective_date/DED-S05-S06 delayed-activation") — i.e., the PRD refresh scope has grown since it was first opened.
4.4§7/§9.2 CHG-001, DED-S07 (grouping by day/category/destination-country)Superseded by day × WhatsApp Business number × category (ADR-0002/M2). Engineering has already formally flagged this twice in the task-breakdown as "needs a PRD refresh; flagged, not silently applied."Partially landed, informally — see nuance below.
4.5§9.2 DED-S08 (per-range activation-boundary gating: AC-2/AC-3 date-dependent per-message-vs-daily split, ERR-1 per_message_unavailable_for_v2 422)Superseded by a flat per-company rule with no date branching and no 422 case at all (ADR-0002/M3). Also already formally flagged in the task-breakdown as needing a PRD refresh.Same nuance as 4.4.
4.6(Contingent) free/service-conversation reporting + next-day (not same-day) revenue recognitionIf BIF-8824's sign-off (Done) settles this as permanent behavior, the PRD's reporting/observability sections should be updated to state it explicitly rather than leaving it implicit.Contingent on confirming BIF-8824's resolution reflects final product intent (the runbook still frames it as "pending sign-off" even though the ticket itself is closed — worth reconciling directly with the DRI).
4.7JIRA traceability hygiene (minor)BIF-8983–8986 cite "PRD §8" while every substantively-maintained story in the epic cites "PRD §9.2" — a small but real sign the PRD's own section numbering has drifted from what these stub tickets were generated against.Not a design issue, but worth a housekeeping fix when the PRD is refreshed.

Important nuance caught on verification, not in the original draft: the PRD already contains a section — §9.2.1 "Reporting amendment — 2026-07-20" (prds/deduction-v2.md:237-256) — that rewrites DED-S07/AC-1, AC-2 and DED-S08/AC-1, AC-2, AC-3, ERR-1 to match ADR-0002 (flat rule, rollup-based reads). So a PRD update has landed for 4.4/4.5, content-wise. But procedurally it's informal: the PRD's own header still reads Last Updated: 2026-07-15 (not bumped), and there's no corresponding row in the PRD CHANGELOG (which stops at v1.6). That's exactly why JIRA (BIF-8881/8882, both updated 2026-07-20: "PRD divergence flagged, not yet applied to the PRD file") and the task-breakdown still describe this as an open gap — the org's own tracking doesn't recognize the §9.2.1 side-channel as closing it. The actionable ask isn't "write the PRD update" — it's "formalize the update that already exists": bump the PRD version/changelog and have JIRA/task-breakdown point at it. Section 4.3 (Non-Goal #4 / OQ-1 / the dropped delayed-toggle) has no such partial fix — that's a genuinely unstarted rewrite of the PRD's core mechanism sections, not just a changelog/versioning gap.

Disambiguation: two different files are both informally called "ADR-0002" across these documents — delivery/decisions/0002-daily-rollup-api-ownership.md (DRI addo.hernando, PM-level, the one the PRD's §9.2.1 amendment cites) and delivery/decisions/0002-modpanel-v2-detection-and-usage-api-ownership.md (DRI hafriz.damarsidi, engineering-level, the one the task-breakdown and ADR-0001's superseded-banner cite). They agree on the outcome (flat rule, rollup-based reads) but are not the same document — cite the specific filename when following up, not just "ADR-0002."

What does not need a PRD update (to avoid over-claiming): §5 Non-Goal #1 ("no separate held/pending line item") is preserved as-is by the Available-only display; the plan-scope note (Billing V3.0 eligibility) was not contradicted by anything read, but it also wasn't confirmed present anywhere in the RFC/task-breakdown/runbooks — flagging this as an open question rather than a finding (see §5).

5. Open items for you to weigh in on (not silently resolved)​

  1. Is Part 2 (M0–M5, not started) in scope for "implementation gaps" (§3), or only merged Part 1? Default taken above: report both, labeled by status (Part 1 = shipped-vs-tested gaps; Part 2 = planned-but-unbuilt, so its "gaps" are really "no test coverage exists yet for work not yet done"). Say if you want Part 2 findings pulled out entirely.
  2. billing_version = 3.0 eligibility — the PRD's most recently and deliberately maintained scoping decision (2026-07-15, §15) — has zero hits (grep-confirmed) anywhere in the RFC or task-breakdown, and no runbook discusses it as an eligibility gate (there is one unrelated hit — available-balance-gate-and-display.md mentions "billing-v3 postpaid orgs" for the postpaid_limit ladder conditional, which is a different concept, not an enablement check). The shipped enablement mechanism is purely the wa_hold_settlement flag/company_id predicate, with no visible billing_version check anywhere in the code paths investigated. This could mean the constraint was silently dropped, or simply enforced somewhere not yet investigated (e.g. at the admin console that sets the flag). Worth a direct question to the DRI (addo.hernando@mekari.com) rather than treating either way as confirmed.
  3. BIF-8824 reconciliation: the ticket is closed (Done — sign-off obtained), but the T4 runbook still frames the free/service no-log-row consequence as "pending stakeholder sign-off." Worth confirming with the DRI whether the sign-off actually landed and the runbook is just unupdated (consistent with the doc-staleness pattern found elsewhere), or whether Done means something narrower than "product accepted this consequence permanently."

Traceability matrix (condensed, by story)​

StoryJIRA StoryTCsVerdict
DED-S01 (hold on webhook)BIF-8738 (In Progress)TC-01–07Wrong trigger point — rewrite against authorize-at-send
DED-S02 (daily reconciliation)BIF-8739 (In Progress)TC-08–11Wrong mechanism — rewrite against per-bucket FIFO settlement
DED-S03 (manual re-trigger)BIF-8983 (To Do, stub)TC-12–15No engineering task — mechanism exists incidentally via T5, unscoped
DED-S04 (timezone)BIF-8984 (To Do, stub)TC-16–18Wrong model (date_of_origin→meta_date) + no engineering task
DED-S05 (toggle on)BIF-8985 (To Do, stub)TC-19–21Design dropped — delete/replace with flag-mechanism tests
DED-S06 (toggle off)BIF-8986 (To Do, stub)TC-22–24Design dropped — delete/replace
DED-S07 (history UI)BIF-8881 (To Do, ADR-0002-updated)TC-25–31Grouping superseded; Part 2 not started — no impl to test yet
DED-S08 (MCC download)BIF-8882 (To Do, ADR-0002-updated)TC-32–38Gating model superseded (5-day-lived design replaced); Part 2 not started

Verification note​

Every numbered finding above was checked by a second, independent agent instructed to refute it against primary text (full 38-TC file, full RFC, full task-breakdown, both decision docs, all 6 runbooks, live hub_core source, and live JIRA). One finding (the TC-32–38 row) was corrected from a blanket claim to a per-TC breakdown after that pass showed it was overstated; one important nuance (PRD §9.2.1's informal partial fix) was added after the pass surfaced it. Everything else was confirmed with an exact quote or line citation.