A/B testing means putting two versions of something in front of similar people and measuring which one gets more of the action you want. Version A is what you have today. Version B changes one thing: the headline, the price, the button text, the cover image. You let real visitors decide, then keep the winner.

It concerns anyone who sells online and wants to stop guessing. A coach deciding between two sales page headlines, a designer testing a template at $19 or $24, a newsletter writer comparing two subject lines. The method is the same. What changes is how much traffic you have, and that decides how you should test.

What is A/B testing?

A/B testing is a controlled experiment. You split an audience into two random groups, show each group a different version and compare one metric, usually the conversion rate. Because the groups are random and exposed at the same time, the only systematic difference between them is the change you made. That is what lets you say the change caused the result.

It is not the same as "we changed the page and sales went up". Before-and-after comparisons mix your change with everything else that moved that week: a viral post, a holiday, a newsletter send. They are useful as rough signals, but they are not A/B tests.

Related vocabulary you will meet:

  • Variant: each version in the test. A is often called the control, B the challenger.
  • Multivariate test: testing several elements and their combinations at once. It needs far more traffic.
  • Split URL test: two separate pages at different addresses, with traffic divided between them.
  • Statistical significance: how confident you can be that the difference is not luck.
  • Sample size: the number of visitors or recipients each variant needs before you read the result.

A/B testing is one tool inside conversion rate optimization. CRO also covers research, user feedback and fixing obvious problems, which often pay off before any test does.

Why it matters

Small changes on a page that already gets traffic compound. Consider a course creator selling a $197 program from a sales page that gets 3,000 visits a month and converts at 1.5%. That is 45 sales, or $8,865 a month.

Say a new headline lifts conversion to 1.8%. That is 54 sales, or $10,638 a month. Nine extra buyers, $1,773 more per month, about $21,000 over a year, with no extra ad spend and no new content. The page did the work.

The opposite matters just as much. Many "obvious" improvements do nothing, and some hurt. A longer page, a countdown timer, a cheaper price: each can lower revenue for your audience even if it worked for someone else. Testing protects you from shipping changes that feel right and quietly cost money.

How it works

A clean test follows the same steps every time:

  • Pick one goal metric. Purchases, email signups, clicks on the buy button. Decide before you start. Revenue per visitor is often better than conversion rate when you test prices.
  • Write a hypothesis. "Showing the three outcomes students get, instead of the module list, will raise purchases because visitors buy results, not content." A hypothesis tells you what to learn even if B loses.
  • Change one thing. If you change the headline and the price together, you will not know which one moved the number.
  • Split traffic randomly. Each visitor sees one version and keeps seeing it. For emails, split the list randomly before sending.
  • Estimate the sample size. As a rough rule, to detect a lift from 2% to 2.5% with reasonable confidence, you need around 14,000 visitors per variant. To detect 2% to 3%, around 3,800 per variant. Big changes need fewer visitors than small ones.
  • Run full weeks. Buying behavior differs between Monday and Saturday. Run at least one full week, ideally two, even if the numbers look decided earlier.
  • Read the result once. Check at the planned end, not every morning. Stopping the first time B looks ahead is the fastest way to crown a false winner.
  • Record it. Keep a simple log: date, hypothesis, variants, traffic, result, decision.

Benchmarks and examples

Most tests do not produce a winner. Across teams that test seriously, a common experience is that one test in four or five gives a clear, lasting improvement. Treat a flat result as information, not failure.

Typical situations:

  • A newsletter with 2,500 subscribers can test subject lines. Send A to 20% of the list and B to another 20%, wait a few hours, then send the better one to the remaining 60%. With open rates around 40%, each group of 500 gives about 200 opens, which is enough to spot a gap of several points.
  • A coach with 4,000 followers and a landing page that gets 600 visits a month cannot detect a 10% lift. It would take a year. Test bold differences instead: a completely different offer angle, a video versus no video, a $97 versus $147 price.
  • A small physical-product store with 20,000 monthly sessions can test product page elements, shipping thresholds and call to action wording, one at a time, every two to three weeks.
  • Price tests often move revenue more than design tests. A template priced at $24 instead of $19 may sell 10% fewer copies and still earn 14% more.

Common mistakes

  • Stopping early. Early results swing wildly. A variant that leads by 40% after two days often ends flat.
  • Testing tiny details on low traffic. A button color test on 300 visits a month will never reach a conclusion. Test big ideas until your traffic grows.
  • Changing several things and crediting one. You learn nothing reusable.
  • Ignoring the downstream metric. A headline that doubles signups but attracts people who never buy is not a win. Follow the test to revenue.
  • Running tests during unusual periods. A launch, a sale or a viral week distorts behavior. The result will not hold afterwards.

Best practices

  • Start with research, not ideas. Read buyer emails, refund reasons and questions. Test answers to real objections.
  • Prioritize by impact and traffic. Test the pages that see the most visitors and sit closest to the payment: the sales page, the product page, the checkout.
  • Go bold when traffic is small. Test different offers, prices and page structures rather than wording tweaks.
  • Use sequential tests when you cannot split traffic. Run version A for two weeks, then B for two weeks, with similar promotion. It is weaker evidence, so only trust large differences.
  • Measure revenue per visitor. It combines conversion and order value, so it catches price and bundle effects.
  • Keep a test log. After a year, it becomes the most useful document you own about your audience.
  • Retest winners occasionally. Audiences change. What worked for your first 1,000 followers may not work for the next 10,000.

In Roctify

Roctify does not include a built-in A/B testing tool today. You can still test in a disciplined way. Create two versions of a product page or offer, promote them in separate periods of equal length, and compare sales in your orders. Discount codes help tell sources apart: give each variant its own code and count redemptions. On the Pro plan, audience analytics, reports and exports let you compare visits and sales across periods, or export the data to a spreadsheet for a closer look.

Because every plan has 0% transaction fees, a price test on Roctify shows you your real margin difference, not one blurred by a platform cut that grows with the price. For email, the email marketing tools on the Creator plan and up show opens and clicks per campaign, so you can compare two subject lines or two offers sent in similar conditions and keep what your list responds to.

FAQ

How much traffic do I need for A/B testing?

It depends on how big a difference you want to detect. Spotting a lift from 2% to 3% needs roughly 3,800 visitors per variant, while a lift from 2% to 2.2% needs tens of thousands. Under 1,000 visitors a month, test bold changes or use sequential tests.

How long should an A/B test run?

At least one full week, and ideally two, so every day of the week is represented. Decide the sample size in advance and stop when you reach it, not when the result looks good.

Is A/B testing worth it for a small creator?

Yes for email subject lines and big offer or price decisions, which need little traffic to show a clear gap. For small page details, your time is better spent talking to buyers and fixing obvious friction until your traffic grows.