README.trackDb

Creating tracks in the browser.

See also:
http://genome.ucsc.edu/goldenPath/help/trackDb/trackDbDoc.html
http://genomewiki.ucsc.edu/index.php/Local_tracks_at_mirror_sites

A track needs two items to make it exist in the browser:

1.  A database table with the track data
2.  An entry in a database table: trackDb_localTracks
	Built from track specifications in your trackDb.ra file.
	Please note the description of trackDb.ra entries in the
	source tree: src/hg/makeDb/trackDb/README
	The correspondence between the database table and the trackDb.ra
	definition is in the name used on the 'track' line in the
	trackDb.ra file.  Your database table name is used on the 'track'
	definition line.

Almost all of the database tables have specific loader
programs to load the track data.  The loader programs
also verify the data before it is added to the table,
and they create the proper indexes on the table to allow
efficient display by the genome browser.

By far the most common format of track data is the BED format.
See also: http://genome.ucsc.edu/FAQ/FAQformat.html#format1
for a description of BED file formats.

A typical BED file format is loaded into a database table with
the loader: hgLoadBed
For example, to load the data from the file: data.bed into
the table named: bedExample

$ hgLoadBed hg17 bedExample data.bed

There are a variety of file formats: GFF, GTF, PSL, WIG, MAF as well as
a variety of specialized data types.  All the loader programs can be seen
in the source tree as subdirectories in: src/hg/makeDb/

$ cd src/hg/makeDb
$ ls -d hg*

The build instructions for the browser code do not include
instructions for building all of the loaders, or other utilities
in the kent source tree.  This is because there are literally
hundreds of utilities,  345 at last count, that are not needed
for ordinary browser development.  In most cases a developer will
need only a couple of the loaders and utilities.  Since the libraries
were built for the CGI binaries, to build any utility or
loader, simply go into its directory and run a 'make'

For our purposes here, we need for example, for BED format tracks:
1. hgLoadBed
2. hgTrackDb
3. hgFindSpec

To build the three loaders mentioned, go to the three directories:
src/hg/makeDb/hgTrackDb/
src/hg/makeDb/hgFindSpec/
src/hg/makeDb/hgLoadBed/

And run a 'make' in each one.  The resulting binary is placed
in: $HOME/bin/$MACHTYPE
This binary directory should be in your PATH, or make this directory
be a symlink to some binary directory that is in your PATH
and you have write permission to.

With those three loader programs built, you can now load BED
format tracks, and build the trackDb_localTracks table as
mentioned next.

The hgTrackDb and hgFindSpec loaders are used to build the trackDb and
hgFindSpec tables in the database.  Older instructions used to mention
using the trackDb file hierarchy in the source tree.  This is no longer
necessary and is not recommended.  You can certainly obtain example
trackDb entries from the source tree hierarchy: src/hg/makeDb/trackDb/
in any of the *.ra files.  And you will need to refer to the README
file in that directory for information about options you can use with
each track type.  To work independently of the UCSC source tree,
establish your own trackDb.ra files outside the UCSC source tree in
a directory of your choice under your control.  Then, to load them
into the database, run the hgTrackDb command with this
simple makefile in the directory where your .ra file exists:
	
trackDbSql=/path/to/kent/source/tree/src/hg/lib/trackDb.sql
DB=hg19

all::
	hgTrackDb . ${DB} trackDb_localTracks ${trackDbSql} .

This hgTrackDb command reads your trackDb.ra file and converts it
into row entries for each track specified in it into row contents
in this new table trackDb_localTracks.

The DB= specification is your database of interest, this example: hg19
This loads your local specific table trackDb_localTracks in the database.
This name trackDb_localTracks is not special, just different than
the ordinary trackDb table.  It should have some meaning to anyone
in your environment and not be the same name as any UCSC database
table.  The two '.' arguments in the command above refer
to directory names.  Since you have no hierarchy of levels in this
single directory, unlike in the source tree trackDb hierarchy, the
'.' arguments refer to the current directory.

To direct the genome browser to this table to use as extra trackDb
definitions, add to the specification in your cgi-bin/hg.conf file:

db.trackDb=trackDb_localTracks,trackDb

Beware of the specified order of the tables if there are tracks
by the same name in each table.  Any definitions for tracks
in trackDb_localTracks will override any definitions for the
same named tracks in trackDb.  You could thus override the
standard definitions for tracks from the trackDb table.
Your usual case will be that your tracks are unique to your
local installation.

========================================================================
Extra Credit:

See also: new assistant scripts as of March 2010 in the ./scripts/
	directory here to fetch and build the source tree.

Older instructions about building the source tree remain valid:

If you really do want to build all the utilities and all database
loaders, perform the following 'make' commands in your source tree:

$ cd src
$ make clean
$ make libs
$ cd hg
$ make
$ cd ../utils
$ make

This builds everything cleanly, all CGI binaries, all database
loaders, all utilities.  Perform this sequence each time you
do a 'git pull' on your source tree.  The 'make clean' step
is especially important since the makefile hierarchy does not
have built in dependencies and will not rebuild items that
depend upon each other.  The traditional dependency on the
source tree libraries is taken care of because a make in any
directory that produces a binary will always re-link the
binary every time, thus always picking up any potentially new
library.

====================================================================
