1. LandOffer
  2. Blog
  3. All articles
  4. Do Not Stop an A/B Test Just Because the Result Looks Significant Today

Data & Decisions

Do Not Stop an A/B Test Just Because the Result Looks Significant Today

Check the inference method and agreed stopping policy before ending an A/B test. Follow an actual synthetic repeated-look simulation and two complete stopping records that separate fixed-horizon decisions, valid sequential methods and operational halts.

10 min readLandOffer team

Do not stop a fixed-horizon A/B test because an ordinary p-value first falls below your threshold today. Check the statistical method and the stopping rule agreed before outcomes were inspected. Repeated unadjusted significance tests followed by significance-based stopping can increase false declarations. A properly designed sequential procedure allows monitoring under its own rules; simply opening a chart is not the statistical mistake.

This LandOffer.ai guide includes an actual offline simulation and completed fictional stopping records. We generated equal-probability binary outcomes, not customer results. No experiment platform, real assignment, release or user traffic was tested or changed. The simulation illustrates one mechanism; it does not validate a vendor engine, estimate your product's error rate or prescribe a sample size.

Two adults hold patterned wood panels while one reads a plain paper card in a marquetry workshop.

Find the method behind today's significant label

A fixed-horizon test is designed around a planned analysis opportunity, with sample and timing requirements established before examining outcomes. Its ordinary fixed-look p-value does not automatically account for a decision to keep checking and stop at the first attractive result. That decision uses information from the sequence, not just the final displayed row.

Optimizely's May 13, 2026 method comparison distinguishes fixed-horizon and sequential approaches and discusses repeated-analysis risk. Its current results-page documentation also says the engine is selected during setup. Read the saved engine setting; a platform name or significance badge does not identify this experiment's method.

Ask for the saved design and actual configuration: primary outcome, assignment unit, comparison, significance level, observation window and stopping criteria. Is the number an ordinary fixed-look p-value, a sequentially valid result, a Bayesian quantity or something else? If you cannot establish the method, leave the efficacy decision unresolved.

Do not interpret a p-value as the probability that the null hypothesis is true or that a rollout will succeed. Statistical evidence also does not settle whether an effect is large enough to matter, whether the metric is measured correctly or whether guardrails permit deployment.

Worked asset: an actual repeated-look simulation

We generated 4,000 independent synthetic trials using Python 3.12.14 and fixed seed 20261027. In every trial, each arm contains 1,000 independent binary draws with success probability 0.5. The true difference is zero by construction. These are deliberately simple assumptions; no seasonality, delayed outcomes, shared-account clustering or interference is simulated.

Every fifty observations per arm, the script calculates a two-sided pooled normal-approximation test for equal proportions. That creates twenty cumulative looks, from 50 through 1,000 per arm. The test statistic follows the large-sample form described in NIST's difference-of-proportions reference. This is an approximate test, not an exact finite-sample procedure.

The script evaluates two policies on the same complete generated path. Final-only declares a difference only if the p-value at 1,000 per arm is below 0.05. Repeated unadjusted declares one if any of the twenty p-values is below 0.05. Either direction counts as a declaration. The generator produces all 1,000 draws per arm for both policies; the repeated-stop decision is then evaluated on that saved cumulative path. Thus the final-only and repeated policies see the same generated outcomes.

At each look, the pooled success fraction is (control successes + treatment successes) / (2 × observations per arm). The standard error uses that pooled fraction and both arm sizes. The script converts the absolute z statistic to a two-sided normal-tail probability using Python's erfc. It retains unrounded values for the threshold comparison. This calculation assumes independent binary observations; repeated rows from one account would not supply the same independent evidence.

Policy over the same 4,000 null trials Trials declaring a difference Fraction of simulated trials
Final-only test 186 4.65%
Stop at any unadjusted crossing 1,013 25.325%

Because both generating probabilities are known to be equal, these declarations are false differences in this simulation. The observed 25.325% is a finite simulated fraction under this exact unadjusted rule, not a universal error rate for viewing results. It depends on the number and spacing of looks, sample sizes, test approximation, generator assumptions and finite simulation. No sequential method was implemented or assessed in these two rows; they cannot establish the performance of mSPRT or another monitoring-aware method.

The twenty looks are dependent because they reuse earlier observations. Do not calculate their combined error probability as though they were twenty independent experiments. The saved simulation estimates the behavior of its specific dependent sequence; it does not supply a general correction formula or replace an approved sequential analysis.

The retained evidence includes all 80,000 cumulative-look count records and one decision record per trial. The original script and seed make the exercise reproducible. Re-running with a different seed could change the observed fractions; selecting the most alarming run would add another selection problem.

Follow a transient crossing without treating it as typical

For illustration, the script selects the first generated trial whose initial significant crossing favors treatment but whose final test is not significant. That selection rule produces trial 20. It is deliberately selected to show the mechanism, not presented as a randomly representative path.

Observations per arm Control successes Treatment successes Treatment minus control Ordinary two-sided p-value
50 18 24 +12.0 percentage points 0.224114
150 64 78 +9.33 points 0.105472
300 135 159 +8.0 points 0.049998
500 231 261 +6.0 points 0.057748
1,000 479 511 +3.2 points 0.152386

The first crossing occurs at 300 observations per arm. The unrounded p-value is approximately 0.0499976803, barely below 0.05; rounding it to 0.05 would obscure the comparison. Stopping there would elevate a transient result into a declaration under the unadjusted rule.

The final result does not establish that treatment has no effect in a real experiment. We know there is no generating effect here only because we authored the simulator. In real data, a nonsignificant final readout can leave meaningful effects unresolved. Conversely, early significance is not guaranteed to disappear: the problem is the error behavior of the stopping procedure, not a rule that every early signal is false.

Keep operational monitoring separate from efficacy stopping

Teams still need to watch assignment, logging, missing outcomes, serious errors and customer harm. A fixed-horizon plan is not an instruction to ignore a broken experience until the sample target arrives. Define operational authority and incident criteria before launch, and record any intervention with its reason.

A safety halt and an efficacy win are different conclusions. If an operator stops exposure because records are being corrupted, preserve that incident and the interrupted study state. Do not rebrand the partial outcome as a valid planned efficacy analysis. Whether a statistical harm boundary is valid also depends on its own design; an operational stop does not create a calibrated p-value.

Check outcome maturity before counting a look. An account assigned an hour ago cannot yet contribute a complete 24-hour outcome window. Changes in eligible population, assignment persistence or event joins can invalidate an apparently significant result regardless of the chosen inference method.

One adult raises an open palm beside another setting down a chisel, with an open paper schedule and two trays of veneer pieces on the bench.

Worked asset: completed fixed and sequential stopping records

The fictional Pollen task-notes app team is considering clearer wording beside a completion control. Statistician Inez, analyst Ewan, product owner Livia and operator Tomas complete two alternative records before any launch. The primary outcome is whether an eligible assigned account has an authoritative completed-note event within 24 hours. Each account is assigned once, nominally 50:50, with stable treatment across visits.

Both records use one primary comparison and a two-sided 0.05 statistical error target. Livia sets a fictional two-percentage-point practical benefit criterion, separate from statistical significance and sample planning. The dates and caps below are completed planning choices for this specimen, not a powered product design. Baseline, effect-to-detect, power and usable traffic must be reviewed before launch.

Design field Completed fixed-horizon record F1 Completed sequential record S1
Inference method One planned fixed-horizon analysis; Inez selects and validates the appropriate fixed-look test before launch A documented mSPRT implementation reviewed by Inez before launch; ordinary repeated fixed-look p-values are excluded
Monitoring permission Ewan reviews assignment, missing events and incident indicators; interim outcome snapshots do not authorize an efficacy decision Ewan may inspect the engine's sequential output under its documented assumptions and configured boundary; no replacement calculator
Completion and timing Proposed enrollment November 2–16, 2026, UTC; close at the date or a 2,000-account total budget cap, whichever occurs first, then wait 24 hours after the last included assignment Proposed start November 2, two-week minimum to November 16 and maximum enrollment through November 30, UTC; use mature 24-hour outcomes and wait for the last included outcome before the final readout
Statistical stop Analyze once at the approved endpoint; no extension or early win based on a favorable interim value Stop for efficacy only at the approved sequential boundary and after the minimum duration; at the approved maximum without a crossing, record inconclusive evidence
Business and operational gate Livia reviews effect size, uncertainty and costs; Tomas can halt exposure for an incident Same owners and gates; crossing a sequential boundary is not automatic rollout approval
Current disposition Launch blocked: formal sizing, approval of the proposed dates, event contract and implementation review remain incomplete Launch blocked: exact engine configuration, version, settings and maximum duration have not been verified

These are completed governance decisions with explicit blockers, not claims that an unconfigured test is ready to run. Neither record retrospectively converts trial 20 into an approved product experiment. Livia retains the current wording while the missing design work is resolved; no actual experiment was created.

For F1, reaching the budget cap early does not prove that a full business cycle was observed or that the result has adequate power. For S1, a crossing before November 16 does not override the minimum duration, and an efficacy decision afterward still requires the approved sequential boundary to be met at that decision time. Tomas can halt either proposal for a serious operational incident, with the efficacy decision left unresolved. These calendar and budget requirements belong to this fictional team's policy, not to every statistical method.

Amplitude's sequential-inference documentation describes its mSPRT family as designed for sequential monitoring. That supports the distinction between a monitoring-aware method and repeated ordinary tests. It does not prove that Pollen selected a valid implementation or satisfied its assumptions. This article neither reproduces mSPRT nor validates an authenticated platform setting.

Other valid designs can use planned interim analyses or different monitoring procedures. Their boundaries and assumptions must match their design. You cannot obtain continuous-monitoring validity merely by calling a spreadsheet “sequential,” adding a minimum sample after seeing the data or switching methods because the current result is disappointing.

Preserve the rule when the result becomes inconvenient

If a stakeholder asks to stop today, record the exact request, current method and reason the proposed stop is or is not allowed. Keep the original design version so the next reviewer can distinguish a permitted stop from a rule changed after seeing outcomes. If the design needs to change, obtain a prospective reviewed plan and disclose what was already observed; do not silently rewrite the original stopping rule.

Do not rescue an inconclusive primary result by searching devices, regions, metrics or dates until one becomes significant. Those comparisons create additional selection questions. Label exploratory findings and plan new evidence rather than presenting them as the original confirmatory result.

Finish with an honest disposition: continue under the existing rule, stop for a documented incident, conclude at the planned endpoint or hold a decision because the method is unknown. For more practical work records, continue with LandOffer.ai's workplace guides; keep the method and stopping policy attached whenever the result is shared.

Sources and Further Reading