A/B Testing

New Member Enrollment Rate

To test the impact of the tiered membership plan on the number of new members who sign up, we will compare the average number of new members signing up across the three months before the change and the three months after implementing the change using an independent t-test. This will allow us to determine if there is a significant change in average enrollment numbers due to the new membership plan, and if so, whether average enrollment has increased or decreased (posthoc test). 

Impact on Revenue

A similar independent t-test comparing average revenue across the three months before vs. after implementing the new membership plan will allow us to determine if there is a significant change in revenue. If a significant change is found, a posthoc test will inform whether revenue has increased or decreased on average after implementing the tiered membership plan

Enrollment numbers in Basic vs. Premium Membership

Finally, we plan to compare enrollment numbers between Basic and Premium Membership, looking at the total number of members enrolled in each tier during the initial three-month period using the chi-square test for independence. Enrollment numbers between the two tiers can also be cross-referenced with demographic info to provide further insight into whether the different tiers appeal to different demographic groups. 

Post Hoc Tests

Depending on the results of our planned tests, there are few additional post hoc tests that may provide further insights. 

Significant differences in new membership enrollment rate month-to-month 

Assuming a significant difference in the number of new members enrolling after implementing the tiered membership plan, a post hoc test comparing the rate of enrollment between the three months post-implementation can help determine whether the significant difference is due to: 

1.  a short-term spike in enrollment in response to the new membership plan

2.  a general increase in the average number of new enrollments per month

3.  a gradual increase in enrollments across the three months, signaling the potential for continued growth

Significant differences in revenue rate month-to-month

Assuming a significant difference in average revenue after implementing the tiered membership plan, a similar post hoc test can be conducted to determine if changes in quarterly revenue are due to a short-term spike in revenue following the change to the tiered membership, an increase in the average revenue per month, or a gradual increase in revenue across the three months following the implementation of the tiered-membership plan.