Creating git submodules / mercurial subrepos

We use several mercurial subrepos in the 'stable' tree. This gives us a hermetic build, and makes it easy(ish) to update third-party libraries we use.

To add a third-party library as a subrepo to 'stable', follow these steps:

1. Create a fork of the third-party library in github

We try never to mirror something directly. There are two reasons for this: first, it allows us to easily make local-only changes to the third-party library if we want, and second, it means our sub-repos only depend on github being up. If you have some subrepos from github, some from bitbucket, some from google code, than if any of them is down nobody will be able to do a git push (git push talks to all subrepos to see if it needs to push any edits to them).

Case 1: third-party library is in github

The easy case: use the 'fork' feature of github to fork it to Khan/<library-name>.

Then follow the instructions for setting up permissions and hipchat notifications for the new group.

Case 2: third-party is not in github

  1. Create a new github repo. Make sure the owner is Khan. The repository name should be the name of the library you're cloning. Description should be A fork of <url of third-party library> with modifications required by Khan Academy. The repository should be public. Do not initialize the repository with a README, or add a .gitignore
  2. In your terminal, cd <root-of-stable>/third-party. Run the appropriate command as follows:
    1. library is in git: git clone <url of third-party library> <library-name>-khansrc
    2. library is in hg: git hg clone <url of third-party library> <library-name>-khansrc (download git-hg if needed).
    3. library is in svn: git svn clone <url of third-party library> <library-name>-khansrc (download git-svn if needed).
  3. cd <library-name>-khansrc && git remote add origin git@github.com:Khan/<library-name>.git && git push -u origin master
  4. cd .. && rm -rf <library-name>-khansrc           # we are going to 'submodule-add' it, in step 2 below
  5. Then follow the instructions for setting up permissions and hipchat notifications for the new group.

2. Add the fork as a subrepo

For git: Run this command:

git submodule add https://github.com/Khan/<library-name>.git third_party/<library-name>-khansrc

Add javascript repos to third_party/javascript-khansrc!

For Mercurial: Edit the .hgsub file (in root-of-stable) to add a line like this:

third_party/<library-name>-khansrc = [git]https://github.com/Khan/<library-name>.git

3. Create a 'use' symlink

Look at third_party/<library-name>-khansrc, and find the subdirectory that has the actual files to use. If this is a python library, this will be the subdirectory with the actual .py files in it. You will want to create a symlink to that directory, that folks will use.  For instance:

cd third_party
ln -s <library-name>-khansrc/trunk/src <library_name>    # this is an example subdir
git add <library-name>

Now when folks do import <library-name> they have direct access to the python files.

4. Document how the library is used in app.yaml

We need to know whether this third-party library should be uploaded to appengine or not.  So take a moment to add this directory to the 'third-party' section of app.yaml -- you'll see examples there of what to do.

Note: you can skip this step if you've added a javascript library under third_party/javascript-khansrc: we automatically figure out what to upload for these libraries via our javascript packaging system.

5. Add to the i18n whitelist if needed

This is rare, but important when it applies.  If you are checking third-party javascript code (into third_party/javascript-khansrc), and you intend to edit this third-party javascript to use khan academy-provided translations (via $._), add this directory to intl/english_only.py:_THIRD_PARTY_JS_NLTEXT_WHITELIST.

6. Double-check and check in!

Run git commit -a (or hg commit). It should show 4 things (3 if you didn't update app.yaml).

#    modified:   .gitmodules
#    modified:   app.yaml
#    new file:   third_party/<library-name>
#    new file:   third_party/<library-name>-khansrc

If you see this, feel free to commit! Your work here is done.

Comments