Evaluating a Homepage Redesign Using A/B Testing (Statistical Approach)

🧩 Title

Evaluating a Homepage Redesign Using A/B Testing (Statistical Approach)

🏢 Business Context

At EduPrime (EdTech platform), the product team launched a new homepage design aimed at improving user conversions.

🎯 Objective

Determine whether the new homepage (Variant B) significantly improves conversion rate compared to the
old version (Variant A).

📊 Experiment Setup

Group A (Control) B (Test)

Users 10,000 10,000

Conversions 1,200 1,260

Conversion Rate 12.0% 12.6%

📈 Initial Observation

• Absolute Lift = +0.6 percentage points

• Relative Lift ≈ +5%

👉 At first glance, Variant B appears better.

⚠️ Problem
Is this improvement real, or just due to random chance?

🧪 Statistical Approach

Since:

• Outcome = binary (conversion: 0/1)

• Two independent groups

👉 I used a two-proportion z-test

🧠 Hypotheses

• H₀ (Null Hypothesis): No difference in conversion rates

• H₁ (Alternative): Variant B performs better

📉 Key Insight

The observed lift was NOT statistically significant

Meaning:

• The improvement could be due to random variation

• No strong evidence that B is actually better

🔍 Additional Analysis

1. Segmentation

To uncover hidden patterns, data was segmented by:

• Device type

• User type (new vs returning)

• Acquisition channel

👉 Result:

• No segment showed strong, consistent improvement

2. Experiment Evaluation

• Sample size may be insufficient for small lift

• Effect size (0.6%) is relatively small

🚀 Business Decision

Do NOT launch Variant B

💡 Recommendations

Short-Term:

• Continue experiment with larger sample size

• Investigate specific user journeys (heatmaps, session recordings)

Long-Term:

• Test more impactful design changes

• Focus on hypotheses with higher expected lift

📈 Key Learning

“Not every positive-looking result is actionable. Statistical validation is critical before making product decisions.”