Performance

Performance is a feature at Khan Academy. Focusing on both frontend (see Best practices for speeding up your website) and backend perf is a priority.

App Engine Performance

Google briefly describes a good set of practices for App Engine performance in Best practices for writing scalable applications. It's a five-part series, the most useful of which are:
To get started using Google's BigQuery, see the instructions in the BigQuery Cookbook.

To experiment on logs-based queries, it can be handy to create a temporary table with the subset of interesting rows. For example, when analyzing out-of-memory errors, I created a table containing only logs with out-of-memory issues. This table was a few MB in contrast to the original tens of GBs of data:
SELECT module_id, resource, latency, app_logs.message
  FROM [logs.requestlogs_20140211]
  WHERE app_logs.message CONTAINS 'Exceeded soft private memory limit'

Out-of-memory errors

-- Find URLs for requests where an instance is forced to shutdown due
-- to exceeding the soft memory limit. Note this counts URLs and not
-- URL routes, which is what you really want.
--
-- For example, the top 10 entries from Feb 11, 2014:
--
-- Row count_ module_id resource_  
-- 1 2900 mapreduce /_ah/stop  
-- 2 2434 mapreduce /admin/calculate_next_tasks_for_user  
-- 3 476 default         /_ah/pipeline/run  
-- 4 395 default         /api/labs/show_scratchpad  
-- 5 260 default         /api/v1/user/students/progress/summary  
-- 6 202 default         /cs  
-- 7 166 default         /_ah/queue/deferred  
-- 8 136 default         /api/v1/topictree  
-- 9 132 default         /_ah/queue/deferred_log_summary  
-- 10 115 default         /api/labs/scratchpads
--
SELECT COUNT(*) as count_,
       module_id,
       REGEXP_EXTRACT(resource, r'(^[^?]+)') as resource_
  FROM [logs.requestlogs_20140211]
  -- Out-of-memory errors look like:
  -- Exceeded soft private memory limit with 260.109 MB after servicing 2406 requests total
  WHERE app_logs.message CONTAINS 'Exceeded soft private memory limit'
  GROUP BY module_id, resource_
  ORDER BY count_ DESC

-- Find modules experiencing the most out-of-memory errors and report
-- on how many requests instances are serving before shutting down.
-- 
-- For example, from the Feb 11, 2014 logs:
--
-- Row count_ module_id numserved_10th numserved_50th numserved_90th  
-- 1 5421 mapreduce 25              121                 427  
-- 2 5082 default     605             2568        7403  
--
SELECT COUNT(module_id) AS count_,
       module_id,
       NTH(10, QUANTILES(INTEGER(REGEXP_EXTRACT(app_logs.message, r'after servicing (\d+) requests')), 101)) as numserved_10th,
       NTH(50, QUANTILES(INTEGER(REGEXP_EXTRACT(app_logs.message, r'after servicing (\d+) requests')), 101)) as numserved_50th,
       NTH(90, QUANTILES(INTEGER(REGEXP_EXTRACT(app_logs.message, r'after servicing (\d+) requests')), 101)) as numserved_90th
  FROM [logs.requestlogs_20140211]
  -- Out-of-memory errors look like:
  -- Exceeded soft private memory limit with 260.109 MB after servicing 2406 requests total
  WHERE app_logs.message CONTAINS 'Exceeded soft private memory limit'
    AND module_id IS NOT NULL
  GROUP BY module_id
  ORDER BY count_ DESC