A/B Testing

A/B testing is the practice of comparing two versions of a page, message, or experience by showing each to a roughly equal slice of your audience and seeing which one performs better. The "A" and "B" are the two variants. The variant with the better result — more sign-ups, more purchases, more time on page — wins, and you ship it.

For a concrete example, an online clothing store can't decide whether their spring sale should lead with "30% off everything" or "free shipping on every order." Instead of arguing in a meeting, they put both versions live, split the incoming traffic 50/50, and let the conversion numbers settle the question after a week. That's an A/B test.

The mechanics

Every A/B test has the same three pieces:

1. A traffic split. Roughly half the visitors see version A; the other half see version B.
2. A measurement. What you're comparing — conversion rate, click-through rate, revenue per visitor.
3. A decision rule. When do you stop the test, and what counts as a winner?

The split itself is the easy part. The hard parts are picking the right metric and waiting long enough for the result to mean something.

The lightweight pattern: A/B testing with a single link

Most articles on A/B testing assume you've already invested in a dedicated experimentation platform. For a lot of small marketing tests, that's overkill — the lightest-weight pattern is to test variants behind a single short link that splits its traffic.

You create one short URL — acme.shop/sale — configured to split incoming clicks evenly between two destinations: one variant of the landing page, the other variant. Share the single URL across email, social, ads, SMS. The conversion data from each destination tells you which variant won.

The naive alternative is to share two different URLs — one per variant — and split your audience manually. The link-based approach has three real advantages over that:

• Apples-to-apples splits. A 50/50 split removes the bias that creeps in when you let the audience self-select between variants (e.g., posting variant A on Twitter and variant B in email — different audiences, different baselines).
• One source of truth. Click counts on the single URL give you total reach. The variant-level analytics give you the comparison.
• Same UTM hygiene. Pasting one URL — acme.shop/sale?utm_source=newsletter — means the UTM parameters are consistent across both variants. Manual splits often produce mismatched tagging.

Redirect Evenly handles the 50/50 split as a smart link — every click is sent to the next variant in turn, so the cohorts stay balanced. Redirect Randomly is the close cousin: each click is assigned independently at random, which is statistically purer but produces variance from a perfect even split. Redirect by Weight handles the same pattern with non-equal splits — useful when you want to expose a new variant to only a small fraction of traffic.

What you actually need to make it work

A/B testing is a statistics problem dressed up as a marketing problem. Three things separate a useful test from a meaningless one:

• A defensible sample size. A 4% vs 5% conversion-rate difference at 200 visitors per variant is noise. Most legitimate calls require thousands of conversions per variant — set a target before you start.
• The right metric. Click-through rate tells you which variant got clicked; conversions tell you which one paid off; retention and revenue tell you which produced lasting value. Pick the metric that aligns with the business outcome before you start.
• One variable at a time. If variant B differs from variant A in both headline and image, you don't know which change drove the difference. Test one variable per test, or use a multivariate testing tool.

If the test you want to run requires per-visitor consistency (the same person always sees the same variant, across multiple visits), multi-step funnels, or cohort analysis — you've outgrown the link-based pattern. A dedicated experimentation tool is the right fit.

Common mistakes

• Stopping the test early. Three days in, every comparison looks decisive. Set a sample-size target before starting and stick to it.
• Optimising for the wrong metric. A high-CTR variant whose traffic doesn't convert is the wrong answer. So is a high-conversion variant whose customers churn faster.
• Testing more than one change at a time. You can't disentangle joint effects from a single test.
• Mixing tracking parameters between variants. If variant A's destination has different UTMs than variant B's, your downstream analytics will be a mess.
• Testing things that don't matter. A 0.1% lift on a button colour at the cost of two weeks of attention isn't worth it. Reserve testing for changes likely to move the needle.