For phabricator admins

Updating the source from upstream

The phabricator source lives (as sub-repos) in the Khan/internal-webserver repo.


git clone


To update our version of phabricator with the latest changes from upstream, run ./ from the internal-webserver directory.
If the pull commands show merges or other problems, resolve them manually before continuing with this script! How will you know? The script will stop after  showing you a git status and then ask you

Does everything look ok? (y/N)

if you affirm this, the script will then proceed to push all the changes to and then it will stop all the phab daemons and _then_ it will restart them. the last sixty or so lines of output will let you know if anything has gone disastrously wrong as a result of the update process (i.e. lighty not waking up, the phab daemons going on strike, etc). In all likelihood, it looks like

Launching 'PhabricatorRepositoryPullLocalDaemon'...
Launching 'PhabricatorGarbageCollectorDaemon'...
Launching 'PhabricatorTaskmasterDaemon'...
Launching 'PhabricatorTaskmasterDaemon'...
Launching 'PhabricatorTaskmasterDaemon'...
Launching 'PhabricatorTaskmasterDaemon'...
Already on 'master'

ssh to toby/phabricator, look at /var/tmp/phd/log/daemons.log for anything weird, and go to bed.

Handling error conditions

Errors in the daemon logs

The daemon logs live on the internal-webserver machine (toby) at /var/tmp/phd/log/daemons.log. If the logs show recent messages of the form

EXCEPTION: (Exception) Attempting to archive a task which hasn't been save()d! at [/home/ubuntu/internal-webserver/phabricator/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:75]
you can fix it by running:
sudo mysql phabricator_worker -e 'truncate table worker_archivetask;' && cd ~/internal-webserver && PHABRICATOR_ENV=khan phabricator/bin/phd restart

Phabricator is acting slow

You can debug this using xhprof.

  1. At, set the value to 1
  2. Do normal things until you hit some slow requests.
  3. Go look in for the slow profiles.
    1. If there are profiles there but no slow ones, it's probably somewhere before we get to handle the request (e.g., maybe blocking in php5-fpm because there aren't enough child processes available or something like that).
    2. If there are slow profiles, that will hopefully make it clear what's slow.
  4. Turn off after about 15 minutes or so.  Profiling slows down all requests by 100% or so (so twice as slow), so you don't want to leave this on longer than necessary.

Common tasks

Deleting audits

Sometimes if you have to re-create a repo on phabricator, or create a branch, phabricator will re-create a bunch of audits associated with that repo.  You can delete them manually, though the tool is pretty crude:

e.g. Delete everything before Sept 5:
internal_webserver/phabricator$ ./bin/config set phabricator.base-uri
internal_webserver/phabricator$ ./bin/audit delete --max-commit-date="5 sep 2013" --repositories WA --dry-run

Restarting the phabricator daemons

Sometimes a phabricator deamon gets stuck on a git pull -- this happens a lot when github is having trouble -- and you need to kick it to get it started again.  Here is the command you can run to do that (from your local machine no less!)  Note it is always safe to restart daemons; they were written to be interruptible.

ssh -i "$HOME/.ssh/internal_webserver.pem" "\
   cd internal-webserver; \
   git checkout master; \
   git pull; \
   PHABRICATOR_ENV=khan phabricator/bin/phd stop; \
   sudo service nginx stop; \
   sudo service php5-fpm stop; \
   PHABRICATOR_ENV=khan phabricator/bin/storage upgrade --force; \

   sudo service php5-fpm start; \
   sudo service nginx start; \
   PHABRICATOR_ENV=khan phabricator/bin/phd start; \

You may need to tweak this depending on what you called your .pem file.  Alternately, you can just log into the phabricator machine and run the stuff within the ssh.

This command also make sure the phabricator machine is up to date with code and storage.  If you want to skip those parts, just run the cd and the two phd commands.

This command is taken verbatim from the end of