Data & Decisions
Check A/A Test Results Before Trusting a New Experimentation Pipeline
Use an A/A test to examine a specific experimentation change. A completed reporting-migration plan and an executed local join-fanout case show why equal assignment counts cannot validate a binary metric, and why a repaired fixture leaves real pipeline checks open.
Use an A/A test to check whether a changed experimentation pipeline assigns, observes and analyzes two groups receiving the same experience consistently. Record what changed, verify the experiences are actually identical, and investigate assignment or measurement discrepancies before trusting later A/B results. One balanced split or nonsignificant comparison cannot certify the system permanently.
This LandOffer.ai guide includes a completed fictional validation plan and an executed local example in which an analytics join duplicates records for one group. No experiment, SDK, warehouse, dashboard, account, flag or live traffic was operated. The local checks demonstrate an authored failure and correction, not an actual A/A experiment or a validated statistical engine.

Give the A/A test a specific validation question
Amplitude's A/A documentation describes identical variations as a quality-assurance technique for experimentation infrastructure. It identifies initial setup, major infrastructure changes and suspicious results as useful reasons to investigate. Its guidance also discourages automatically adding an A/A test before every routine experiment.
For fictional RouteLedger, a maintenance-record portal, the change is a migration of experiment reporting from an older aggregation job to a new job with an account-dimension join. Analyst Ivonne wants to know whether the new job preserves assignment denominators and the meaning of “account completed an export.” Engineer Pavel owns that aggregation change.
Keep the validation question tied to the migrated reporting job. The team is not trying to discover whether one export page is better: both groups should receive the same export interface and configuration. Nor is this a performance test of the maintenance portal. Changed reporting logic can alter the displayed numerator even when the interface and completed tasks stay the same.
An A/A validation record should explain why these checks address the migration. An unchanged marketing page or unrelated metric may give a reassuring result while never exercising the changed join. Choose metrics and records that travel through the component whose trustworthiness is in question.
Complete the plan before inspecting the readout
The following is RouteLedger's finished fictional plan. Its traffic, timing, owners and acceptance rules are authored choices, not an experiment launched on customers or a recommended sample size for every organization.
| Plan field | Completed RouteLedger agreement |
|---|---|
| Change under review | New reporting job adds account-dimension enrichment before metric aggregation; old job remains the reference for the selected records. |
| Identical experience | Both labels point to layout=standard and export_version=v2; no variant-specific UI, export logic or intentional timing change. |
| Unit and allocation | One eligible operator account, stable assignment across visits, planned 50/50. Staff sharing an account do not become extra independent units. |
| Cohort and timing | Fixed seven-day enrollment, then allow the final account's 24-hour outcome window to mature. Preserve the original cutoff rather than ending at a pleasing p-value. |
| Primary validation metric | Unique assigned accounts with at least one completed export within 24 hours, divided by eligible assigned accounts with a mature window. Repeated events do not add binary successes. |
| Data-quality checks | Assignment counts versus plan; identifier stability; exposure linkage; join multiplicity; missing outcomes; raw-to-report reconciliation; identical configuration evidence. |
| Review method | Inspect predeclared metrics and critical paths; statistical reviewer agrees tests, thresholds and repeated-look handling before a real run. No rule that every displayed p-value must exceed 0.05. |
| Acceptance owner | Ivonne can qualify the reporting change only after discrepancies are explained and real implementation evidence is reviewed. Pavel supplies corrected job/version and replay evidence. |
| Stop or hold | Unexplained fanout, lost identities, divergent served configuration or unresolved metric definitions hold pipeline acceptance. Operational faults can trigger investigation without a claim of product efficacy. |
The seven-day window gives this invented team a concrete cutoff; it does not prove the sample will be sufficiently informative. Required traffic depends on the defect or discrepancy the team needs to detect, expected metric variation, clustering and the analysis method. The 8,000-account fixture below is deliberately constructed and is not a power calculation.
A real implementation also needs evidence that the intended values are served. Matching configuration entries alone cannot rule out cached values, conditional execution or different SDK paths. In this guide, the local dictionary comparison verifies only the authored configuration values, including a negative check that detects an intentionally changed value.
Check the measurement chain beyond allocation counts
Review the chain in order: eligible account, stable assignment, exposure, raw outcome record, enrichment join, aggregation and displayed metric. Preserve versioned inputs and outputs at each boundary so the first changed count can be traced to its producing step. If the same account acquires multiple variant labels, investigate the identifier or assignment path rather than treating those labels as independent observations.
Microsoft's during-experiment guidance, published January 2021, separates data-quality measures such as join rates, data loss and sample ratio mismatch from evaluation and guardrail metrics. That distinction helps RouteLedger: a tidy assignment table cannot validate a later join or the displayed success rate.
For the local worked example, the authored assignment list has 4,000 accounts in each group. The source outcome list contains 800 unique completed-export events per group, one for each of 800 accounts. All other accounts lack a completion record in this selected fixture. These exact equalities are chosen inputs, not evidence that a real randomizer produced unusually perfect balance.
The old reference calculation counts unique accounts with a completion and reports 800/4,000, or 20%, in each group. The faulty new enrichment step has two matching dimension rows for the first 200 completing accounts in group B. It produces two joined rows for each of those events. Group A has no such fanout.
If the reporting job counts joined rows as completing accounts, B's numerator becomes 1,000 and its displayed rate becomes 25%. Assignment counts remain 4,000/4,000. A count-only check therefore misses the measurement fault entirely.
Worked anomaly record: equal experience, unequal report
This completed evidence table comes from the saved Python fixture. It executes the constructed join and aggregations locally; it does not query a warehouse or reproduce a particular vendor's implementation.
| Evidence checkpoint | Group A | Group B | Completed interpretation |
|---|---|---|---|
| Authored assigned accounts | 4,000 | 4,000 | Equal chosen inputs; no allocation mechanism tested |
| Source completion events and completing accounts | 800 / 800 | 800 / 800 | One source event per completing account in this fixture |
| Rows after faulty enrichment | 800 | 1,000 | 200 B event IDs appear twice; no source IDs are lost |
| Faulty displayed binary numerator | 800 | 1,000 | Counting joined rows violates the agreed unique-account definition |
| Faulty displayed rates | 20% | 25% | Apparent difference of five percentage points is not a product improvement |
| Corrected unique-account numerator | 800 | 800 | Local correction matches the reference for these inputs |
| Corrected rates | 20% | 20% | The replayed aggregation is repaired; live pipeline acceptance remains open |
The explanation rests on record lineage, not on a significant-looking percentage. Pavel can point to 200 duplicated event IDs, each occurring twice after the join, all belonging to B. The source still contains 1,600 unique event IDs; the joined output has 1,800 rows. That evidence identifies the deliberately introduced fault in this fixture.
Counting unique completing accounts restores the binary metric here. It is not a universal instruction to apply DISTINCT until dashboards agree. A genuine count of export operations has a different unit, and blindly deduplicating can remove legitimate repeat work. The team must also repair or justify the join's multiplicity for other downstream measures.

Record the repair without overclaiming qualification
Completed fictional repair note. “Pavel preserves the faulty output as migration-v1 and records the duplicate dimension matches. In migration-v2, the binary export metric counts unique completing account IDs. Replaying the same source and joined records returns 800 in each group, matching the reference. Replaying duplicate copies of the joined rows leaves that binary count unchanged.”
Completed review decision. “Ivonne accepts the local numerator correction for the saved case. She keeps the real migration unqualified pending review of the dimension-table uniqueness contract, deployed aggregation version, exposure linkage, missing records and representative production evidence. No product rollout or general experimentation approval is granted by this replay.”
The fixture additionally removes one B account's completion record and observes that its counted numerator falls to 799. It cannot tell whether that account failed to complete or the telemetry was lost. A missing-event sensitivity check exposes dependence on the input; it does not establish an actual loss rate or recover the missing fact.
These decisions preserve useful partial progress. The local defect has a demonstrated correction, while the real implementation questions remain assigned and visible. Repeating the same fixture provides regression evidence for that case; it does not replace a representative A/A validation of the deployed system.
Interpret statistical alarms without requiring perfect sameness
In a real A/A experiment, random groups need not have exactly equal outcomes. A nonsignificant result can reflect limited sensitivity, while an occasional significant result can occur even under a correctly specified null. Interpret a flagged metric alongside record lineage, planned comparisons and the sensitivity of the check.
Multiplicity matters. Under the explicit mathematical assumptions of 20 independent tests, each with an exact 5% false-positive probability under its true null, the chance of at least one false positive is 1 − 0.95²⁰, approximately 64.15%. The companion fixture checks this arithmetic; it does not run 20 experiments or measure a platform's false-positive rate.
NIST's Bonferroni reference explains simultaneous comparison protection for a predefined family. As simple threshold arithmetic, allocating a 0.05 family error budget across 20 valid individual tests gives 0.0025 each. Bonferroni protection does not require independent tests, but it does require valid individual tests and a defined family. This fixed-family threshold alone does not handle arbitrary repeated dashboard checks or optional stopping. The statistical reviewer must define the actual family and method.
Microsoft's guidance specifically warns that repeated measurement needs appropriate handling of multiple testing and early peeking. An A/A test run until every alert disappears does not provide the same evidence as the predeclared analysis. Investigating a concrete logging failure is appropriate; changing the inferential plan afterward needs to remain visible in the record.
Repeated null experiments or carefully designed simulation can examine calibration across runs, but one clean comparison cannot estimate that distribution reliably. This guide's deterministic replay runs no stochastic A/A experiments and computes no metric p-values or confidence intervals. It demonstrates a semantic aggregation failure without pretending to validate inferential calibration.
Close with an evidence-based pipeline decision
The local fixture passes 24 assertions covering configurations, source and joined counts, duplicated IDs, reference and corrected binary aggregation, missing-event sensitivity and the stated probability arithmetic. No SDK, actual allocation persistence, served page, event transport, warehouse query or dashboard statistic is tested.
For more practical work guides, visit LandOffer.ai. Before trusting the next A/B readout, give the acceptance owner the validation plan, anomaly lineage, corrected replay and remaining deployed-system questions. Record which pipeline version, metric definitions and execution evidence the owner accepted. Keep untested paths open and revisit the affected checks when the implementation changes.
Sources and Further Reading
- Amplitude: Run an A/A test — identical experiences, targeted infrastructure checks and when routine repetition is unnecessary; freshly reviewed October 27, 2026.
- Microsoft Research: Patterns of Trustworthy Experimentation, During-Experiment Stage — January 25, 2021; data-quality taxonomy and repeated-measurement cautions.
- NIST: Bonferroni's method — simultaneous comparison context and family protection.


