UCSC genome browser

Kent source tree

Important README files: kent/src/product/README.xxx

System design

The genome browser is composed by following parts: 
  • For service
    • server-side CGI binaries and HTML files
    • hg.conf configuration file
  • For administration
    • utility tools
    • .hg.conf file
  • data
    • stored in Mysql
    • stored in external files (optional, such as bigBed, bigWig, ...)
Design principle:
  • Each species will occupy a Mysql database. Inside each database:
    • each track correspond to one table, and each track belongs to a group
    • one trackDb table exists for one species, used to describe tracks
      • each record of the data in trackDb defines one track
    • one grp table to hold group info
    • one hgFindSpec table
  • Apart from databases for individual species, special databases exist:
    • hgFixed
    • hgcentral

Constructing new database

  1. Download kent source and compile everything. See src/product/README.trackDb on how to compile everything.
  2. Create hg.conf file at /cgi-bin/ directory. Sample file can be found at: http://genome-test.cse.ucsc.edu/~kent/src/unzipped/product/ex.hg.conf
  3. Create .hg.conf file at your home directory:
    $ cat > ~/.hg.conf
    $ chmod 600 ~/.hg.conf
  4. Create mysql database, make sure all folders at mysql directory are with user/grp of mysql.mysql. If not, will throw "errno: 13" when trying to modify it!!

To add new tracks:

Refer to general guide: kent/src/product/README.trackDb
  1. generate track data in appropriate format (http://genome.ucsc.edu/FAQ/FAQformat.html)
  2. Create a table to hold the track data. Need to identify the format of the track data file, and use corresponding loader program to load it. Following example is for bed format:
    1. $ hgLoadBed dbName trackTableName file.bed
    2. Realistic example: $ hgLoadBed -noBin -bedGraph=4 hg19 track_name data.bedGraph
    3. Loader program source locates at: kent/src/hg/makeDb/
  3. Create/update the trackDb:
    1. Compose a new trackDb.ra file (by editing the old one) with configuration section for the new track
      1. Example *.ra files: src/hg/makeDb/trackDb/[organism name]/[genome version]
      2. Information on trackDb options: src/hg/makeDb/trackDb/README. Also see next section.
    2. Compose a makefile at the place where trackDb.ra file resides. Could be like:
      trackDbSql=/home/cgs/twlab/xzhou/kent/src/hg/lib/trackDb.sql DB=hg19 all:: ~/latest/hgTrackDb . ${DB} trackDb ${trackDbSql} .
      Run $ make all to update the trackDb table.

Also tracks could be in bigBed or bigWig formats. See: http://www.mail-archive.com/genome@lists.soe.ucsc.edu/msg00924.html

hgsql hg19 -e 'drop table if exists myLocalBigWig; create table myLocalBigWig (fileName varchar(255) not null); insert into myLocalBigWig values ("/gbdb/hg19/bbi/myLocalBigWig.bw");'

About trackDb.ra files:

  • Example files for human hg19 can be found at: kent/src/hg/makeDb/trackDb/human/hg19/
  • use blank lines to separate tracks
  • each line begins with an attribute name and value, separated by space
  • Configurations
    • field: track
      • track trackName [override]
    • field: type
      • a lot of types there...
    • Track height: use maxHeightPixels 128:32:16
    • Track color: color r,g,b
    • Sub groups: subGroup1 sampleType Sample_Type fetalK=fetalK CD34=CD34 ....