AB Test/ Statistical Analysis

Project Overview

Problem Statement: From the funnel analysis, I discovered that over 50% of customer drop-offs occurred on the page following the home screen (as shown on the right). The product team wanted to test whether users would prefer sending gifts via email ID as an alternative for those without contact numbers. Consequently, it was decided, in collaboration with stakeholders (PMs, engineers, designers, and senior leaders), to design and run an A/B experiment before rolling out the new email ID feature in production. GrabGifts (GG) is available in all SEA countries where Grab operates, including Singapore, Malaysia, the Philippines, Indonesia, Thailand, Myanmar, and Cambodia.

Null Hypothesis:   New screen with feature of Email will attract more users and their will be more user comparison as compared to current screen (with no email option)

IF P-value is less than 5% or 0.05 for P0 metric we accept null hypothesis else we reject Null hypothesis. Also, Guardrail metric should not fail.

Countries chosen: Singapore (which was top performing country) and Indonesia (which was low performing country)

*Success Metrics chosen: 

Primary Metric (P0): Conversion Rate= total users transacted/total users engaged on the platform

Secondary Metric (P1): Gross merchandise value (GMV)= total sales generated/total users

Guardrail Metric: Funnel drop off should not go below existing numbers

Approach: Immediate focus - Took random users from target audience 

Steps Taken:

1. Designed the experiment: Used Grab's in-house A/B experimentation tool to design the variables of experiment. Using evanmiller website I calculated sample size considering current baseline conversion rate, power as 80%, significance level as 5% ,splitted total sample size into 50-50 as control and treatment users. Planned the experiment to run for 2 weeks in Singapore (SG) and Indonesia (ID), taking into account higher weekend traffic compared to weekdays.

2. Run and monitoring the experiment: Continuously monitored the traffic during the experiment period to ensure data integrity and consistency.

3. Perform Statistical Analysis on experiment data: Upon completion of the experiment, performed statistical analysis using Proportional Test and Z Test. Calculated the P-value for both control and treatment groups for P0 and P1 metrics. Found that the P-value was less than 0.05 for the P0 metric, indicating statistical significance.

4. Bias Check and Conclusion: Checked additional parameters to eliminate potential biases.

5. Result: Concluded that the null hypothesis could be accepted, meaning that introducing the new email feature would not disrupt the flow. Determined that users would send more gifts when given both phone and email options compared to only having the phone option.

6. Presentation and Stakeholder Communication: Created a detailed presentation including all findings, visualizations, and recommendations. Shared the presentation with stakeholders to communicate the results effectively.

7. Rolled Out the Feature: Decided to roll out the new email feature to the full audience.

8. Monitoring post Roll out: Monitored the feature’s performance post-launch to check for any errors or exceptions and ensure smooth operation.

*Due to security concerns I am not allowed to share screenshot or deck of my presentation or code. I would love to share more details should you be interested to know more