Backend instances are a relatively new feature of App Engine. They are different from normal frontend instances in several ways. We use dynamic, private backend instances to process MapReduce jobs, which relieves the frontend instances of that duty and provides an across-the-board speed improvement due to the elimination of resource contention between MapReduce and non-MapReduce requests. 

The deploy script handles deploying to frontends and backends simultaneously. By default it will only deploy to backends during standard daily deploys. If you want to deploy non-default code to backends (e.g. to run a one-off backfill), add the --force-backends option when running the deploy script. Additionally, you must use the script to set the default version in a way that keeps frontends and backends in sync. (This is all covered in the deploy documentation.)

Some important points to be aware of:
  • Old backend versions have to be manually cleaned up just like old frontend versions. You can do this from the Backends tab of the App Engine admin interface. If there are too many old versions of backends, the deploy could fail because you have too many backend versions (limit is 5) or because there is only a limited amount of memory that can be configured across all backends. (The limit is 40 GiB, which the App Engine team raised for us from the default limit of 10 GiB.)
  • If you deploy non-default frontends and backends using --force-backends, you are updating queue.yaml to point all MapReduce jobs on all versions to your new non-default backends. Make sure this doesn't break anything.
  • The App Engine SDK's local development server supports backends via the --backends flag in theory, but I found that the dev server ignores this flag and sends all requests to the frontend. Chris talked to a GAE team member at Google I/O about this bug, and they are aware of it.