Updating the source from upstream
The phabricator source lives (as sub-repos) in the Khan/internal-webserver repo.
git clone firstname.lastname@example.org:Khan/internal-webserver
To update our version of phabricator with the latest changes from upstream, run
If the pull commands show merges or other problems, resolve them manually before continuing with this script! How will you know? The script will stop after showing you a git status and then ask you
Does everything look ok? (y/N)
if you affirm this, the script will then proceed to push all the changes to phabricator.khanacademy.org and then it will stop all the phab daemons and _then_ it will restart them. the last sixty or so lines of output will let you know if anything has gone disastrously wrong as a result of the update process (i.e. lighty not waking up, the phab daemons going on strike, etc). In all likelihood, it looks like
Already on 'master'
ssh to toby/phabricator, look at
/var/tmp/phd/log/daemons.log for anything weird, and go to bed.
Handling error conditions
Errors in the daemon logs
The daemon logs live on the internal-webserver machine (toby) at
/var/tmp/phd/log/daemons.log. If the logs show recent messages of the form
EXCEPTION: (Exception) Attempting to archive a task which hasn't been save()d! at [/home/ubuntu/internal-webserver/phabricator/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:75]
you can fix it by running:
sudo mysql phabricator_worker -e 'truncate table worker_archivetask;' && cd ~/internal-webserver && PHABRICATOR_ENV=khan phabricator/bin/phd restart
Phabricator is acting slow
You can debug this using xhprof.
- At http://phabricator.khanacademy.org/config/edit/debug.profile-rate/, set the value to 1
- Do normal things until you hit some slow requests.
- Go look in http://phabricator.khanacademy.org/xhprof/ for the slow profiles.
- If there are profiles there but no slow ones, it's probably
somewhere before we get to handle the request (e.g., maybe blocking in
php5-fpm because there aren't enough child processes available or
something like that).
- If there are slow profiles, that will hopefully make it clear what's slow.
- Turn off after about 15 minutes or so. Profiling slows down all requests by 100% or so (so twice as slow), so you don't want to leave this on longer than necessary.
Sometimes if you have to re-create a repo on phabricator, or create a branch, phabricator will re-create a bunch of audits associated with that repo. You can delete them manually, though the tool is pretty crude:
e.g. Delete everything before Sept 5:
internal_webserver/phabricator$ ./bin/config set phabricator.base-uri http://phabricator.khanacademy.org/
internal_webserver/phabricator$ ./bin/audit delete --max-commit-date="5 sep 2013" --repositories WA --dry-run
Restarting the phabricator daemons
Sometimes a phabricator deamon gets stuck on a
-- this happens a lot when github is having trouble -- and you need to kick it to get it started again. Here is the command you can run to do that (from your local machine no less!) Note it is always safe to restart daemons; they were written to be interruptible.
ssh email@example.com -i "$HOME/.ssh/internal_webserver.pem" "\
cd internal-webserver; \
git checkout master; \
git pull; \
PHABRICATOR_ENV=khan phabricator/bin/phd stop; \
sudo service nginx stop; \
sudo service php5-fpm stop; \
PHABRICATOR_ENV=khan phabricator/bin/storage upgrade --force; \
sudo service php5-fpm start; \
sudo service nginx start; \
PHABRICATOR_ENV=khan phabricator/bin/phd start; \
You may need to tweak this depending on what you called your
.pem file. Alternately, you can just log into the phabricator machine and run the stuff within the
This command also make sure the phabricator machine is up to date with code and storage. If you want to skip those parts, just run the
cd and the two
This command is taken verbatim from the end of