A B Testing
Use when asked to design or automate an A/B test — random traffic allocation, statistical significance, and multivariate testing — as distinct from black/white-box testing technique (see black-box-testing, white-box-testing) or UI/UX evaluation broadly (see ui-ux-testing).
A/B testing compares two versions of an application or feature against each other to determine which performs better on a specific, predefined metric — a data-driven alternative to deciding by opinion.
How it works
- Define the goal metric before starting (conversion rate, click-through, engagement time) — chosen after the fact, a metric can always be found that makes either variant look like the winner.
- Randomly split traffic between variant A (control) and variant B (treatment), so any observed difference is attributable to the variant, not to a systematic difference in who saw which one.
- Run until reaching statistical significance — enough samples that an observed difference is unlikely to be random noise — not just until one variant happens to be ahead.
- Multivariate testing extends this to multiple simultaneous changes, testing combinations rather than one isolated variable at a time; it needs proportionally more traffic to reach significance for each combination.
Automation
Modern A/B test automation integrates with CI/CD and production traffic: frameworks handle random assignment, collect metrics (conversion, load time, error rate) continuously, and can automatically flag a statistically significant result or a performance regression rather than waiting for a human to notice.
Common pitfalls
- Stopping early because one variant looks ahead — checking results repeatedly and stopping the moment a difference appears inflates the false-positive rate; decide the sample size (or a proper sequential testing method) in advance.
- No pre-registered goal metric — picking the metric after seeing the data invites cherry-picking whichever number happened to favor the preferred variant.
- Sample pollution — a user seeing both variants (e.g. across devices, or due to a caching bug) breaks the random-assignment assumption the whole method depends on.
- Ignoring interaction effects in multivariate tests — two changes that each help individually can combine to hurt, and vice versa; a multivariate design accounts for this, a series of separate A/B tests doesn't.
Learn more
- UI UX Testing for the broader design-evaluation methods A/B testing is one technique within.
- Performance Testing for the load/response-time metrics an A/B test sometimes tracks alongside conversion.