CRO without theatre: how to run experiments properly

Conversion rate optimisation has become a buzzword that covers everything from changing button colours to complete page redesigns. Most of what agencies sell as CRO is poorly structured, statistically invalid and commercially pointless. Here is how to do it properly.

The Problem with Most CRO

The typical agency CRO process looks like this: install a heatmap tool, identify "friction points" based on gut feeling, make changes, declare victory when the conversion rate happens to be higher that week.

This is not optimisation. It is guesswork with a retrospective narrative.

Building a Proper Hypothesis

Every experiment needs a structured hypothesis before you touch any code:

Format: "If we [change], then [metric] will [improve/decrease] by [amount], because [reasoning based on data]."

Example: "If we move the size selector above the fold on mobile product pages, then add-to-cart rate will increase by 0.3 percentage points, because 34% of mobile users are not scrolling past the product images."

The key elements:

• A specific, measurable change
• A specific metric to measure
• A predicted magnitude of effect
• Evidence-based reasoning for why it should work

Sample Size and Duration

This is where most CRO programmes fall apart. You need enough data to reach statistical significance:

• Minimum 1,000 conversions per variation (not visitors, conversions)
• Run for at least 2 full business cycles (usually 2-4 weeks)
• Do not peek at results and stop early when they look good
• Account for day-of-week and time-of-day variations

For a store converting at 2% with 50,000 monthly visitors, a single A/B test needs roughly 4-6 weeks to reach significance.

What to Test

Focus on high-impact, high-confidence changes:

• Product page layout: Image size, information hierarchy, CTA placement
• Collection page density: Products per row, filter visibility, sort defaults
• Cart and checkout flow: Shipping calculator visibility, trust signals, payment options
• Mobile-specific changes: Sticky add-to-cart bars, image gallery behaviour, form inputs

Do not test:

• Button colours (the effect size is too small to measure reliably)
• Copy changes on low-traffic pages
• Multiple changes simultaneously (you cannot attribute the result)

Measuring Commercial Impact

The goal of CRO is not a higher conversion rate. It is more revenue. Track:

• Revenue per visitor (not just conversion rate)
• Average order value changes
• Return rate changes (sometimes higher conversion = more impulse purchases = more returns)
• Customer lifetime value for test cohorts