TL;DR

You don’t need perfect experimentation to improve conversion.
You need a consistent baseline, a repeatable method, and a habit of documenting confounders.

This is the measurement approach I use when shipping Shopify UX improvements at a weekly cadence.

Step 1: Pick 3 core metrics (and stop adding more)

  • Conversion rate (primary)
  • Add-to-cart rate (supporting)
  • Checkout initiation rate (supporting)

Optional: AOV, refund rate, support tickets, page speed signals.

Step 2: Define a baseline and a post-change window

Be explicit:

  • baseline window: X days/weeks before changes
  • post window: X days/weeks after changes

If you ship weekly, use rolling windows and track change dates.

Step 3: Segment by device

Many Shopify issues are mobile-only:

  • variant selection
  • sticky bars
  • image/gallery behavior
  • page speed and layout shifts

Always check mobile vs desktop.

Step 4: Track confounders like a responsible adult

Write down:

  • campaigns (email, paid, promos)
  • pricing changes / discounts
  • stockouts or “not shippable” surprises
  • fulfillment/carrier changes
  • seasonality spikes

You can’t control everything, but you can stop lying to yourself.

Step 5: Prefer time series + sanity checks

If A/B testing is not possible:

  • use time-series comparison
  • compare to similar prior periods (if seasonal)
  • check that supporting metrics move in the same direction

Step 6: Publish a short weekly report

Keep it short:

  • what shipped
  • what moved (metrics)
  • what we think caused it
  • what we’ll do next

This is how you build compounding improvements.

Example structure (copy/paste)

  • Change: what shipped
  • Hypothesis: why it should help
  • Baseline: dates + core metrics
  • After: dates + core metrics
  • Confounders: campaigns/stockouts/etc
  • Decision: keep / iterate / revert