What is GAE/Bingo?
GAE/Bingo is a lightweight A/B testing library. It can start tests and track conversion events on the server and from the client.
What is the basic workflow for running and A/B test?
- Add an A/B test in your code
- Monitor results, have fun looking at graphs, try not to make up fake stories about the data
- Choose an alternative using the GAE/Bingo interface
- Delete the A/B test from your code
- Archive (save results for future reference) or delete (completely remove history of) the experiment from GAE/Bingo.
Adding an A/B test
webapp/experiments.py and create a new class for your test, its alternatives, conversion events, and call to
View test results
When logged in as an admin, visit
/gae_bingo/dashboard. Click on your test to view the results.
Not seeing your test results? There are two gotchas to watch out for:
A call to gae_bingo.ab_test("my test name") must happen for your test to show up.
- Tests and results live temporarily in memcache before being persisted to the datastore with a cron job. During development visit
/gae_bingo/persist to see the results immediately.
Choosing an A/B test alternative with GAE/Bingo
There is a cookie you can set locally that will set the A/B test alternative for you, if you have admin permissions. The easiest way to set the cookie is through the admin console, using the "Preview this" button next to the alternative you want to see:
Reset test results during development
During development it's useful to clear running tests and start them over, particularly if you're modifying test metadata like conversions. First, clear out any current test data by flushing memcache. You can do this from the App Engine admin interface. Second, delete existing instances of your tests from the datastore. There are two ways to do this:
When running A/B tests, there are certain core metrics you may wish to include. To do so, simply
include experiments and specify the metrics (or metric categories) you wish to measure when running your A/B test, and call the
ab_test function provided there. There's no need to add
bingo() calls for the core metrics, as they're already all there.. If there are conversions you want to have in addition to the core ones provided in
experiments.py, you can add them as you normally would. For example:
experiment_one = experiments.CoreMetrics.ab_test("chimpanzee", core_categories=["exercises", "videos", "retention"])
experiment_two = experiments.CoreMetrics.ab_test("monkey", core_categories="all")
experiment_three = experiments.CoreMetrics.ab_test("zebra", conversion_name=["eat", "run"], core_categories=["retention"])
core_categories must be either "all" or a list of strings.
If you think a metric you're adding for your experiment would be useful to other experiments, feel free to add it to
core_metrics. GAE/Bingo will take care of adding it to the experiment--and any already running experiments will now be able to see results from that metric as well.
The current categories and metrics (as of 7/3/2012--check experiments.py for the latest version) are as follows (Note: binary metrics indicate whether the user in question ever converted in the given metric, while count metrics indicate the number of times they converted in that metric.):
|Convert when the user logs in (at most once every 3 hours)
||Indicates whether the user in question registered.
|Convert when the user gains a new proficiency
|Convert when the user attempts any problem.
|Convert when the user answers a problem correctly.
|Convert when a user answers a problem incorrectly
|Convert when a user changes their avatar
|Convert when a user edits the contents of their display case
||Convert when a user updates their nickname/display name
|Convert when the user updates anything on their profile
||Convert when the user changes the public/private status of their profile
|Convert when a logged-in user returns to the site after having been away for some amount of time (currently 3 hours)
|Convert when a phantom user returns to the site after having been away for some amount of time (currently 3 hours)
|Convert when a pre-phantom user returns to the site after having been away for some amount of time (currently 3 hours)
|Convert when any user returns to the site after having been away for some amount of time (currently 3 hours)
|Convert when a user finishes watching a video
|Convert when a user starts to watch a video