A/B Test Duration and Sample Size: How to Decide When a Conversion Experiment Is Reliable
A-B-testingCROexperimentationconversion-ratestatistics

A/B Test Duration and Sample Size: How to Decide When a Conversion Experiment Is Reliable

AAnalytics Editorial Team
2026-11-13
7 min read

Learn how to estimate A/B test sample size and duration, set stopping rules, check measurement quality, and judge whether a conversion result is reliable.

An A/B test is not reliable simply because a platform labels one variation a winner. This guide shows how to estimate sample size and test duration before launch, choose sensible assumptions, interpret early results, and document whether a conversion experiment supports a decision.

Overview

A/B test duration is the time required to collect enough valid observations for the result to be useful. The correct duration depends on traffic, baseline conversion rate, the improvement you want to detect, traffic allocation, statistical power, and the quality of your measurement.

The goal is not to run every test for the same number of days. A low-traffic site may need several weeks for one experiment, while a high-traffic site may collect the required sample quickly. Even then, ending a test immediately after an apparently positive result can be risky. Early results are more sensitive to random variation, unusual traffic sources, technical problems, and novelty effects.

Before launching, define four things:

  • Primary metric: the conversion event that determines the decision, such as completed checkout, qualified lead, or signup.
  • Baseline rate: the current conversion rate for the population and period relevant to the test.
  • Minimum detectable effect: the smallest improvement worth acting on.
  • Stopping rule: the planned sample, duration, and decision criteria.

A useful result is both statistically credible and commercially meaningful. A tiny improvement may be unlikely to be random but still too small to justify development or implementation costs. Conversely, a large observed lift from a small sample may look attractive while remaining too uncertain to support a permanent rollout.

How to estimate A/B test sample size and duration

Most A/B test duration calculators ask for a baseline conversion rate, an expected or minimum detectable lift, the number of variants, traffic allocation, statistical power, and a significance threshold. Use the result as a planning estimate rather than a guarantee.

1. Establish the baseline conversion rate

Calculate the rate as conversions divided by eligible visitors or users. Use a consistent denominator. For example, do not compare purchases per session in one period with purchases per user in another. The baseline should represent the audience and page experience that the experiment will actually affect.

2. Choose the minimum detectable effect

Express the target as either a relative or absolute change. If the baseline is 5% and the minimum worthwhile result is a rise to 6%, the absolute change is one percentage point and the relative lift is 20%. These are not interchangeable inputs. Decide which interpretation matches the business decision before using a calculator.

Choose an effect that matters financially or operationally, not an optimistic effect chosen to shorten the test. If a smaller improvement would not change the decision, there is little value in powering the experiment to detect it.

3. Set power, significance, and allocation

Statistical power describes the chance of detecting the chosen effect if it is genuinely present. A significance threshold defines how much evidence is required under the selected testing method. Your testing platform may use different terminology or a model-based approach, so document the method rather than treating every probability display as equivalent.

A 50/50 allocation generally gathers data most efficiently for a two-variant test. Unequal allocation may be appropriate when exposure to a new experience carries operational risk, but it usually increases the time needed to reach the same precision. Tests with multiple variants also require more traffic and a clear plan for comparisons.

4. Convert sample size into calendar time

Estimate eligible traffic per day, then divide the required observations by expected daily traffic. If a test requires 20,000 eligible users per variant and each variant is expected to receive 2,000 users per day, the arithmetic estimate is 10 days. Treat that as a minimum planning estimate, not an automatic stopping date.

Allow time for normal traffic cycles. A test that runs only during weekdays may not represent weekend behavior, while a test launched during a promotion may capture demand that will not persist. A practical schedule should cover the meaningful operating pattern of the site and avoid ending because one favorable day changed the average.

Inputs and assumptions that affect reliability

Primary and secondary metrics

Choose one primary conversion metric before launch. Secondary metrics can explain why a result occurred, such as add-to-cart rate, form completion, revenue per visitor, or content engagement. They should not quietly become the reason for declaring a winner after the primary metric fails to support the hypothesis.

Unit of analysis

Decide whether randomization and analysis use users, sessions, accounts, or another unit. A returning user who sees both experiences can contaminate the comparison if the design assumes persistent assignment. Record how returning visitors, logged-out devices, and cross-device activity are handled.

Instrumentation quality

A larger sample cannot repair broken conversion tracking. Check that exposure, assignment, eligibility, and conversion events are recorded consistently. Validate the experiment event in the browser and analytics platform, inspect duplicate conversions, and compare totals with a trusted business system where possible.

Privacy controls, browser restrictions, consent choices, ad blockers, and server-side or client-side implementation differences can affect observed counts. These issues do not automatically invalidate a test, but they should be documented and monitored. A tracking plan template can help connect the experiment hypothesis to the events and properties required for analysis.

Novelty, seasonality, and exposure quality

Users may initially respond to a new layout, message, or feature because it is unfamiliar. Results can also vary by device, geography, acquisition channel, returning status, or campaign mix. Avoid changing targeting or traffic sources mid-test unless the change is part of the design. If a major external event changes demand, record it and consider whether the original assumptions still apply.

Stopping rules

Predefine when you will inspect and stop the test. Repeatedly checking results and stopping at the first favorable reading can increase the chance of a misleading conclusion. If your platform supports sequential testing, use its stated method and follow its rules. Otherwise, set a planned sample or duration and resist unplanned early stopping except for safety, tracking failure, or clear operational harm.

Worked examples

Example 1: Moderate traffic, modest improvement

Suppose a landing page converts at 4%. The team considers a change worthwhile only if it raises conversion to at least 4.8%, an absolute increase of 0.8 percentage points. Traffic is split evenly between control and variation, and the team selects its power and significance settings before launch.

The calculator output should be interpreted as the required eligible users per variant under those assumptions. If the forecast is 1,500 eligible users per variant per day, divide the required sample per variant by 1,500 to estimate calendar time. Then add enough time to cover the site’s normal weekly pattern and check whether traffic forecasts are realistic.

Example 2: Early apparent winner

After two days, a variation shows a large lift, but each variant has received only a small fraction of the planned sample. The correct response is not automatically to publish the variation. First check assignment, exposure, conversion logging, traffic composition, and implementation errors. If those checks pass, continue under the original stopping rule unless the test was designed for sequential decision-making.

Example 3: Traffic forecast changes

A test was planned for 14 days, but a campaign sends less traffic than expected. Recalculate the finish date using actual eligible users per day. Do not compensate by lowering the minimum detectable effect after seeing the results. If the business decision has changed, document the new decision threshold and treat it as a revised experiment plan.

When to recalculate

Revisit the estimate before every new experiment and whenever an important input changes. Recalculate when:

  • the baseline conversion rate has shifted because of seasonality, pricing, traffic mix, or a redesigned funnel;
  • forecast traffic is materially higher or lower than the launch plan;
  • the number of variants, allocation, audience, or primary metric changes;
  • the minimum worthwhile effect changes after financial or product review;
  • tracking, consent behavior, tagging, or the conversion definition changes;
  • an outage, promotion, campaign, or other event makes part of the sample unrepresentative.

Use this pre-launch checklist:

  1. Write the hypothesis and primary metric in one sentence.
  2. Record the baseline rate and its measurement window.
  3. Define the minimum detectable effect in absolute and relative terms.
  4. Choose the test design, allocation, power, and significance approach.
  5. Estimate eligible traffic per day and the calendar duration.
  6. Confirm assignment, exposure, and conversion tracking in a QA environment.
  7. Set the stopping rule and list conditions that would invalidate the test.
  8. Document the result, including exclusions, segments reviewed, and the final decision.

For implementation checks, pair the experiment record with a documented GA4 audit workflow or relevant conversion-tracking QA. Recalculate whenever the underlying inputs move, but do not rewrite the rules simply to fit an appealing result. Reliable experimentation comes from consistent design, clean measurement, and decisions made against criteria defined before the outcome is known.

Related Topics

#A-B-testing#CRO#experimentation#conversion-rate#statistics
A

Analytics Editorial Team

Senior SEO Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.