1. LandOffer
  2. Blog
  3. All articles
  4. Sample Ratio Mismatch in A/B Testing: Check Observed Counts Against the Planned Split

Data & Decisions

Sample Ratio Mismatch in A/B Testing: Check Observed Counts Against the Planned Split

Compare observed account counts with the allocation planned before an A/B test. A completed 70/30 diagnosis and repair record show how a selective filter can create mismatch, why restored counts leave measurement questions open, and when effect analysis must remain on hold.

9 min readLandOffer team

Check sample ratio mismatch against the allocation planned before the experiment: a deliberate 70/30 split should be tested against 70/30, not against equal group sizes. Count the agreed randomization units, calculate expected counts for the observed total, and apply the planned statistical check before interpreting treatment effects. When a mismatch is flagged, investigate where the counted population changed; use record-level evidence to identify the component responsible.

This LandOffer.ai guide uses a completed fictional 70/30 diagnosis and an executed local arithmetic fixture. No experimentation platform, live traffic, account assignment, logging pipeline or allocation setting was accessed or changed. The account records, people and repair decisions below are authored teaching material.

A nursery worker holds a 70/30 planting card beside a larger A seedling tray and a smaller B tray in a greenhouse.

Establish which counts should follow the planned ratio

Before calculating SRM, write down exactly which population and unit each count represents. Identify the experiment version, eligible population, assignment unit, allocation probabilities, observation cutoff and duplicate rules. Then establish whether the records being counted represent assignment, exposure, a triggered subset or successful outcomes. Label each population separately so a later filter does not silently change the question your allocation check answers.

For fictional PlotOrder, a wholesale nursery ordering tool, analyst Mira checks a revised reorder landing page. The agreed unit is one eligible shop account, allocated once and kept in the same arm across visits. A receives 70% of assignments and B receives 30%. Multiple staff members or page views within an account do not become extra independent observations.

The example assumes independent account allocation and no interference between shops. The saved fixture constructs account IDs to reproduce specified counts; it does not implement or verify random assignment. A real team must validate its allocation mechanism and consider clustering, shared identifiers and dependence before treating these assumptions as established.

Mira's authored protocol specifies one count review at the fixed cohort cutoff, using a flag threshold of p < 0.001. It also permits operational investigation when instrumentation faults are reported. This illustrative threshold is agreed before these counts are examined; it is not a universal platform default or a reason to repeatedly stop and restart until a preferred result appears.

Microsoft Research's September 2020 SRM article describes checking sample balance before examining effects, and distinguishes detection from diagnosis. Its reported platform threshold is 0.0005. PlotOrder's separate example threshold should not be attributed to Microsoft.

Calculate expected counts from the actual total

The NIST chi-square goodness-of-fit reference describes comparing observed counts with counts expected under a specified distribution. For two fixed allocation probabilities, calculate each expected count as the observed total multiplied by its planned probability. Sum the squared observed-minus-expected differences divided by their expected counts.

For PlotOrder's first authored cohort, 10,000 eligible accounts appear in the assignment ledger: 7,040 in A and 2,960 in B. Expected counts are 7,000 and 3,000. The statistic is:

X² = (7,040 − 7,000)² / 7,000 + (2,960 − 3,000)² / 3,000 = 0.7619

With two arms, fixed probabilities and no probability estimated from these observations, the reference distribution has one degree of freedom. Its upper-tail probability is approximately 0.3827. Under the example's rule, this cohort is not flagged. The observed split need not land exactly on the configured percentages.

That result means the count discrepancy does not provide sufficient evidence against the planned ratio at the stated threshold. It does not prove the randomizer is correct, show that all accounts were measured, establish independence or validate an effect estimate. Balanced omissions from both arms could escape this check.

Comparing the percentages by eye leaves sample size out of the decision. The same percentage difference can have different statistical implications at different sample sizes. Another mistake is to test against 50/50 because the interface calls the variants A and B. That tests the wrong null hypothesis here and flags an intended allocation difference.

The approximation also has conditions. NIST notes the need for sufficient sample size. This fixture declines calculations with an expected cell count below five as a local guard, rather than treating that cutoff as a complete statistical adequacy rule. Sparse samples, clustered assignment or a different allocation design need an appropriate method and review.

Worked diagnosis: assignment counts pass, filtered counts fail

Mira compares account IDs and arm counts in assignment output v1 with analysis-inclusion output v2. Both refer to the same closed cohort and cutoff. The analysis mistakenly requires a legacy page event that 460 B accounts lack after the redesigned page changes the emitting path. In this constructed case, the predicate removes those accounts even though they remain eligible under the original definition.

This is an authored fault, not a cause inferred from a small p-value. The fixture explicitly creates the removed-ID set and checks that all 460 excluded IDs belong to B. In a real investigation, Mira would need equivalent evidence from authorized logs, query versions and account-level reconciliation before making that claim.

Count checkpoint A / B observed Total Expected at 70/30 X²; approximate p Completed interpretation
Original eligible assignment cohort 7,040 / 2,960 10,000 7,000 / 3,000 0.7619; 0.3827 No SRM flag under the stated rule; other validity checks remain
Incorrect legacy-event inclusion filter 7,040 / 2,500 9,540 6,678 / 2,862 65.4108; 6.08 × 10⁻¹⁶ SRM flag; hold effect interpretation and reconcile exclusions
Cohort restored to agreed eligibility 7,040 / 2,960 10,000 7,000 / 3,000 0.7619; 0.3827 Count qualification restored; outcome recovery still unresolved

Notice that the filtered row uses 9,540 as its total. Reusing 7,000/3,000 as its expected counts would compare the observed table with expectations that no longer sum to its denominator. The missing 460 accounts matter diagnostically, but the goodness-of-fit calculation still needs internally consistent counts.

The tiny filtered-cohort p-value is not the probability that the filter is broken, nor the size of treatment harm. It expresses how incompatible these count discrepancies are with the null model through the selected statistic. The account-ID difference and predicate evidence locate the authored failure; the statistical flag alone cannot.

Mira does not remove extra A accounts, change the allocation specification to fit the observations or weight the remaining B accounts and declare the problem solved. Those changes can conceal the symptom without recovering the intended population or its missing measurements.

A hand lifts a plant tag from a mesh screen above tray B while tray A remains unobstructed beside a 70/30 card.

Trace where the population first diverges

Microsoft's SRM taxonomy includes assignment, execution, log processing and analysis causes. Use those stages to organize the evidence. A mismatch already present in the assignment ledger points to a different investigation from one introduced by a later join or filter. An apparently clean upstream table is helpful only if its completeness and meaning are credible.

At assignment, confirm the configured ratio for that version and period, stable unit identity and whether accounts could choose or change arms. If the experiment ramped from one ratio to another, do not pool all accounts under a single guessed probability. Reconstruct the applicable allocation history and use an analysis suitable for that design.

At exposure and logging, compare delivery paths, event coverage, redirects and missing identifiers. Similar observed counts do not establish equal capture quality. A log can silently lose a class of accounts in both groups, and an assignment record can exist without proving that the intended experience was served.

At processing and analysis, inspect join multiplicity, deduplication, time boundaries and inclusion predicates. PlotOrder's concrete diagnostic question is: “Which eligible assigned accounts disappear between these two versioned outputs, and which rule excludes them?” That question gives engineer Jules a bounded artifact to examine instead of a request to explain an alarming screenshot.

Browser, region or date slices may help locate a discrepancy, but searching many segments creates more opportunities for misleading alarms. Preserve which slices were planned and which were exploratory. Bring the complete investigation to the statistical reviewer rather than presenting the smallest discovered p-value as an independently confirmed diagnosis.

Completed repair and data-qualification record

The following record is a finished fictional review, not evidence of a real deployment. Mira owns the analysis decision; Jules owns the inclusion change and evidence packet. Their immediate choice is whether the data can support effect interpretation, not which variant should ship.

Review item Completed PlotOrder record
Before repair Assignment cohort 7,040/2,960; legacy-event output 7,040/2,500; hold comparative effects. Preserve both output versions and the 460-ID difference.
Agreed correction Jules removes the accidental legacy-event requirement from cohort inclusion and restores the originally eligible assigned accounts. He does not change the planned 70/30 allocation.
Local recheck The restored ID set equals the original 10,000-account set; counts and SRM calculation return to the first row. This demonstrates the authored fixture correction only.
Missing evidence The count repair does not recreate missing reorder-outcome events for excluded accounts. Completeness and meaning of the outcome telemetry remain unverified.
Old-cohort decision Mira records “count check qualified; effect analysis not qualified.” Reanalysis requires a defensible recovery of required outcomes or a separately justified design; silently treating missing events as no reorder is rejected.
Separate post-fix example A distinct authored cohort contains 6,980 A and 3,020 B accounts. Its X² is 0.1905 and p is approximately 0.6625; it is not combined with the old cohort or offered as proof of actual logging repair.
Next owner and gate Jules supplies production evidence of corrected inclusion and measurement coverage through the authorized process. Mira reviews that evidence, maturity of outcome windows and remaining assumptions before approving effect analysis. No rollout decision is recorded.

The post-fix cohort is a separate example, not 10,000 accounts added to the original total. Its compatible counts are useful evidence about that count table only. They do not demonstrate corrected event collection, adequate power or a beneficial treatment. A team should preserve that distinction in the dashboard status and the written readout.

Keep the reproducible check and its limits together

The saved companion Python fixture executes 23 assertions. It checks the chosen counts, expected totals, statistics, threshold flags, exact restoration of the constructed cohort and the separate post-fix calculation. It also demonstrates the incorrect 50/50 comparison and rejects invalid counts or cases outside its small-count guard.

For one degree of freedom, the fixture computes the chi-square upper tail using erfc(sqrt(X² / 2)). NIST's distribution reference describes the relationship between chi-square and squared independent standard normal variables. The local implementation checks that a zero statistic has upper-tail probability one and that X² = 3.8414588 gives approximately 0.05. These sanity checks do not turn it into a general statistical package, sequential test or platform validation.

No treatment outcomes are generated or analyzed. The fixture does not execute SQL, exercise a randomizer, establish logging completeness or measure causal effects. A compatible SRM check is one qualification condition among several; an inconclusive product result and an invalid sample are different conclusions requiring different next steps.

For more work guides, visit LandOffer.ai. Before your next experiment readout, give the analysis owner the allocation contract, checkpoint counts, reconciled exclusions and a written decision stating which validity checks remain open. Keep an unresolved measurement gap visible even when the repaired sample ratio looks reasonable.

Sources and Further Reading