A/B Tests Analytics

FunnelFox gives you analytics on A/B tests so you can make more confident, data-driven decisions.

A/B Test Metrics

Conversions

Conversions metric presents you with the number of users who completed a purchase.

---

Conversion Rate (CR)

Conversion Rate (CR) presents you with the percentage of users who completed a purchase.
Formula:
CR = Conversions / Users
Example: 100 users, 7 purchases → CR = 7%.

---

CR Confidence Level

What it is:

The probability that the difference between A and B is statistically significant — not random.

Thresholds:

• ≥ 95% - high confidence, variation is likely better or worse
• < 80% - low confidence, more data is recommended

How it’s calculated:

Based on standard error, z-score, and p-value.
Takes into account the size of the difference between A and B, and the amount of data collected.

---

CR Confidence Interval

What it is:
A range that likely contains the true conversion rate, with a specified confidence level (e.g. 95%).

How to read it:

• Interval is positive if the lower bound is above the control’s CR
• Interval is negative if the upper bound is below the control’s CR

This helps understand not just if the variation is better, but by how much.

---

Observed Power

What it is:
The probability that the test will detect a real difference, if it exists.

Thresholds:

• ≥ 80% — strong test, high chance of detecting meaningful differences
• < 80% — test may miss actual effects

Calculated based on sample size, CR difference, and standard error.

---

P-value

What it is:
The probability of seeing results this extreme (or more) if the null hypothesis is true (i.e. there’s no real difference between A and B).

Thresholds:

• < 0.05 — statistically significant, variation difference is likely real
• ≥ 0.05 — not enough evidence to say variations are different

---

📘

Example Analysis:

Variation B has a 9.1% conversion rate, compared to 7.5%
Confidence interval: 8.5% – 9.7%
CR Confidence Level: 98.4% → statistically significant
Power: 86% → high reliability
P-value: 0.015 → low probability that results are due to chance (random)