A/B test results & reporting
ShopGuide's built-in A/B testing splits traffic across variants and ties the outcome to real Shopify orders. Here's exactly what's in the results.
How visitors get assigned
A test has a name, a start and end date, and a traffic split across up to three variants — say, control, chat, and highlights — each with a target percentage. For every visitor, ShopGuide records which variant they landed in, whether they viewed the chat widget, and whether they interacted with it.
What the results actually contain
For each variant:
- Users — unique visitors assigned to it
- Chat Interactions — how many actually sent a message
- Orders and revenue — pulled directly from Shopify orders tagged with the variant (via an
ab_test_variantorder attribute), filtered to your online sales channel and the test's date range - Revenue / User, Avg Order Value, Conversion Rate (orders divided by users) and Add to Cart Rate
A ★ marks the variant with the highest revenue per visitor, and a line below the table names the leader and compares it with the runner-up. Revenue is shown in your shop's currency.
You can read results while a test is still running (View Results recalculates them live), or after it's finished: results are frozen when you click Stop. ShopGuide recommends running a test for at least 21 days and tells you when a test is shorter than that.
What's not calculated
No p-values, no confidence intervals, no statistical significance shown. If a difference between variants looks real, that's a judgment call you're making, not a claim ShopGuide is making for you — treat the numbers as directional, especially with a small number of users or orders in either variant, and apply your own significance testing if you need to be sure before acting. There's also no geographic segmentation of results and no automated phased rollout.