---
title: "A/B Testing"
description: "A controlled experiment in which two or more versions of a web page, email, or element are shown to different audience segments to determine which performs better against a defined metric."
url: "https://skribra.com/glossary/a-b-testing"
type: "glossary-term"
category: "Analytics & Measurement"
updated: "2026-06-08"
---

# A/B Testing

**Definition.** A controlled experiment in which two or more versions of a web page, email, or element are shown to different audience segments to determine which performs better against a defined metric.

**In short.** A/B testing removes guesswork from optimization. Instead of assuming what works, you show two variations to real users simultaneously and let the data decide — eliminating opinion-based debates.

## Key points

- A/B tests require statistical significance to be valid — running a test until you see a result you like (known as 'peeking') produces false positives
- Only change one variable at a time in an A/B test — testing multiple changes simultaneously makes it impossible to attribute results
- SEO A/B testing (testing page elements that affect ranking, not just conversion) requires specialized tools because splitting traffic between variants can confuse crawlers
- Minimum sample sizes for statistical significance at 95% confidence typically require thousands of conversions — small-traffic pages cannot be reliably A/B tested

## How A/B Testing Works

In a standard A/B test, traffic is randomly split between two variants: the control (version A, the current design) and the challenger (version B, the variant being tested)[1]. Both variants run simultaneously to eliminate time-based variables (seasonal changes, weekly traffic patterns, etc.). A testing tool assigns users to variants via cookies, ensuring the same user always sees the same variant. After a predetermined run time or sample size is reached, statistical analysis determines whether the observed difference in [CTR](https://skribra.com/glossary/click-through-rate), [bounce rate](https://skribra.com/glossary/bounce-rate), or conversion rate is likely to be a real effect or random noise. Only test one variable at a time — changing multiple elements simultaneously makes attribution impossible.

## A/B Testing for SEO

SEO-focused A/B testing tests elements that affect [organic performance](https://skribra.com/glossary/organic-traffic) — primarily [title tags](https://skribra.com/glossary/title-tag), [meta descriptions](https://skribra.com/glossary/meta-description), and [structured data](https://skribra.com/glossary/schema-markup) — rather than on-page conversion elements[1]. This is trickier than standard CRO testing because you can't split-test the same URL (Google sees only one version). Instead, SEO tests use matched page testing: identify a set of similar pages, randomly assign some to the control (no change) and some to the variant (changed element), then measure ranking/[CTR](https://skribra.com/glossary/click-through-rate) differences between groups after sufficient time. Pages showing high [Impressions](https://skribra.com/glossary/impressions) but low CTR in Google Search Console are ideal A/B test candidates — the ranking exists, only the snippet needs improvement.

## Common A/B Testing Mistakes

The most costly mistake is 'peeking' — ending a test as soon as results look favorable, before statistical significance is reached. This systematically biases results toward false positives[1]. Other common errors: testing too many things simultaneously (making attribution impossible), running tests for too short a time (missing weekly traffic variation), testing on low-traffic pages (insufficient sample for significance), and ignoring the novelty effect (visitors may behave differently with a new design simply because it's unfamiliar, not because it's better). Always calculate required sample size before starting a test using a power calculator, and commit to running it to completion. Improvements in [engagement rate](https://skribra.com/glossary/bounce-rate) or [organic traffic](https://skribra.com/glossary/organic-traffic) after an A/B test should be monitored for at least 4 weeks before being declared permanent wins.

## Related terms

- [Click-Through Rate (CTR)](https://skribra.com/glossary/click-through-rate): The percentage of users who click on a search result after seeing it in the SERP, calculated as (clicks ÷ impressions) × 100.
- [Bounce Rate](https://skribra.com/glossary/bounce-rate): The percentage of sessions in which a user visits only one page on your website and leaves without interacting further, as measured by Google Analytics.
- [Impressions](https://skribra.com/glossary/impressions): The number of times a URL from your website appeared in a Google search result, regardless of whether the user scrolled to see it or clicked on it.
- [Organic Traffic](https://skribra.com/glossary/organic-traffic): Website visitors who arrive through unpaid search engine results, as opposed to paid ads, social media, direct visits, or referral links.

## Sources

1. [VWO — The Complete Guide to A/B Testing](https://vwo.com/ab-testing/)
2. [Google Search Central — Monitor and Debug with Search Console](https://developers.google.com/search/docs/monitor-debug/search-console-start)

## Machine-readable index

- HTML version of this page: https://skribra.com/glossary/a-b-testing
- Site index for agents: https://skribra.com/llms.txt
- Sitemap: https://skribra.com/sitemap.xml
- Public REST API: https://skribra.com/api/v1
- OpenAPI specification: https://skribra.com/openapi.json