A/B Testing with GAE/Bingo

GAE/Bingo is no more. See A/B testing with BigBingo for working with the new A/B testing framework.

What is GAE/Bingo?

GAE/Bingo is a lightweight A/B testing library. It can start tests and track conversion events on the server and from the client.

What is the basic workflow for running and A/B test?

  1. Add an A/B test in your code
  2. Monitor results, have fun looking at graphs, try not to make up fake stories about the data
  3. Choose an alternative using the GAE/Bingo interface
  4. Delete the A/B test from your code
  5. Archive (save results for future reference) or delete (completely remove history of) the experiment from GAE/Bingo.

Adding an A/B test

Open webapp/experiments.py and create a new class for your test, its alternatives, conversion events, and call to ab_test().

View test results

When logged in as an admin, visit /gae_bingo/dashboard. Click on your test to view the results.

Not seeing your test results? There are two gotchas to watch out for:

  1. A call to gae_bingo.ab_test("my test name") must happen for your test to show up.
  2. Tests and results live temporarily in memcache before being persisted to the datastore with a cron job. During development visit /gae_bingo/persist to see the results immediately.

Choosing an A/B test alternative with GAE/Bingo

There is a cookie you can set locally that will set the A/B test alternative for you, if you have admin permissions. The easiest way to set the cookie is through the admin console, using the "Preview this" button next to the alternative you want to see:

Reset test results during development

During development it's useful to clear running tests and start them over, particularly if you're modifying test metadata like conversions. First, clear out any current test data by flushing memcache. You can do this from the App Engine admin interface. Second, delete existing instances of your tests from the datastore. There are two ways to do this:

  • Option 1: From the GAE/Bingo dashboard, open your test and click "End experiment, picking this". Then, click "Delete this experiment".
  • Option 2: From the App Engine admin interface's interactive console, clear all GAE/Bingo tests from the datastore with this script:
    from gae_bingo import models
    from google.appengine.ext import db


Core metrics

When running A/B tests, there are certain core metrics you may wish to include.  To do so, simply include experiments and specify the metrics (or metric categories) you wish to measure when running your A/B test, and call the ab_test function provided there. There's no need to add bingo() calls for the core metrics, as they're already all there.. If there are conversions you want to have in addition to the core ones provided in experiments.py, you can add them as you normally would.  For example:

import experiments

experiment_one = experiments.CoreMetrics.ab_test("chimpanzee", core_categories=["exercises", "videos", "retention"])
experiment_two = experiments.CoreMetrics.ab_test("monkey", core_categories="all")
experiment_three = experiments.CoreMetrics.ab_test("zebra", conversion_name=["eat", "run"], core_categories=["retention"])

core_categories must be either "all" or a list of strings.

If you think a metric you're adding for your experiment would be useful to other experiments, feel free to add it to core_metrics. GAE/Bingo will take care of adding it to the experiment--and any already running experiments will now be able to see results from that metric as well.

The current categories and metrics (as of 7/3/2012--check experiments.py for the latest version) are as follows (Note: binary metrics indicate whether the user in question ever converted in the given metric, while count metrics indicate the number of times they converted in that metric.):
Category Conversion name Notes/description
accounts login_binary
Convert when the user logs in (at most once every 3 hours)
registration_binary Indicates whether the user in question registered.
exercises new_proficiency_binary
Convert when the user gains a new proficiency
Convert when the user attempts any problem.
Convert when the user answers a problem correctly.
Convert when a user answers a problem incorrectly
profile avatar_update_binary
Convert when a user changes their avatar
Convert when a user edits the contents of their display case
nickname_update_binary Convert when a user updates their nickname/display name
Convert when the user updates anything on their profile
public_update_binary Convert when the user changes the public/private status of their profile
retention logged_in_return_visit_binary
Convert when a logged-in user returns to the site after having been away for some amount of time (currently 3 hours)
Convert when a phantom user returns to the site after having been away for some amount of time (currently 3 hours)
Convert when a pre-phantom user returns to the site after having been away for some amount of time (currently 3 hours)
Convert when any user returns to the site after having been away for some amount of time (currently 3 hours)
videos video_completed_binary
Convert when a user finishes watching a video
Convert when a user starts to watch a video