Source tree compilation on Debian/Ubuntu: Difference between revisions
m (add category tag) |
(changed cse to soe and genome-source fixes) |
||
(48 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
The UCSC source tree includes tools for the command line and the CGIs for a local UCSC genome browser mirror. The installation is not easy, see below for a script that automates the process. | |||
Please note the instructions and scripts in the source tree that can build the kent source tree on | |||
any Linux system: | |||
[http://genome-source.soe.ucsc.edu/gitlist/kent.git/tree/master/src/product;hb=HEAD src/product] | |||
These commands compile the UCSC genome browser and all source tools. They have been tested on Debian Lenny (64 and 32bit), Ubuntu Maverick and Ubuntu 13.04. They will put all binaries in /usr/local, all cgi-bins in /usr/lib/cgi-bin and all html-like things into /var/www/genome. | |||
In theory, once you have installed the samtools library, you can just copy-paste this script into a command line | |||
<pre> | <pre> | ||
# we need to be root to install packages and put stuff into /usr/local | |||
su | su | ||
# install required packages from repository | |||
# please not that these won't install the samtools library to handle bam files | |||
apt-get install mysql-server-5.0 apache2 libmysqlclient15-dev libpng12-dev libssl-dev openssl mysql-client-5.5 mysql-client-core-5.5 | |||
# set variables for compilation | |||
export MACHTYPE=$(uname -m) | export MACHTYPE=$(uname -m) | ||
export MYSQLINC=/ | export MYSQLINC=`mysql_config --include | sed -e 's/^-I//g'` | ||
DIRS='SCRIPTS=/usr/local/bin CGI_BIN=/usr/lib/cgi-bin DOCUMENTROOT=/var/www/genome BINDIR=/usr/local/bin ENCODE_PIPELINE_BIN=/usr/local/bin' | export MYSQLLIBS=`mysql_config --libs` | ||
DIRS='SCRIPTS=/usr/local/bin CGI_BIN=/usr/lib/cgi-bin DOCUMENTROOT=/var/www/genome BINDIR=/usr/local/bin' | |||
# this does not seem to be necessary anymore | |||
#ENCODE_PIPELINE_BIN=/usr/local/bin' | |||
# download | # download | ||
cd /usr/local | cd /usr/local | ||
wget http://hgdownload. | wget http://hgdownload.soe.ucsc.edu/admin/jksrc.zip | ||
unzip jksrc.zip | unzip jksrc.zip | ||
mkdir -p /var/www/genome/ | mkdir -p /var/www/genome/ | ||
# compile libraries | # compile libraries | ||
cd kent/src/ | cd kent/src/lib | ||
make | make | ||
cd ../jkOwnLib | |||
make | |||
# compile browser | # compile browser | ||
cd .. | cd .. | ||
make $DIRS | make $DIRS | ||
# set permissions | |||
cd /home/data/www | |||
chown apache:apache -R * | |||
</pre> | </pre> | ||
Add the parameter HG_WARN=-w to DIRS if you don't want the compilation to stop on error messages. Dirty, but it works. :-) | |||
== The fastest way ever to get a genome browser up and running on Ubuntu, Fedora, Centos, OSX == | |||
Just tested this with ubuntu 14 LTS, centos 6, centos 7, fedora 20 and OSX 10.10 (on OSX, requires Xcode and macports): | |||
wget https://github.com/maximilianh/browserInstall/blob/master/browserInstall.sh | |||
bash installBrowser.sh | |||
Then connect with a firefox to localhost. Done. | |||
If you look into the /usr/local/apache/cgi-bin/hg.conf file, you can see that we use the public mysql server through the internet. This will be very slow unless you're in San Francisco. So let's copy the complete genome of C. Briggsae + all annotations: | |||
bash browserInstall.sh cb1 | |||
Now select the nematodes clade, the C. Briggsae 2002 Assembly. It should be very fast. | |||
You can mirror hg19 like this ("bash browserInstall.sh hg19"), but remember that it will take a lot of space, ~2.2 TB. | |||
== Run a genome browser mirror on your local machine, compiled from source == | |||
I prefer to place the CGIs in the same location where UCSC has them, that's /usr/local/apache. This path does not conform to Ubuntu's specifications but has the advantage that we don't need to change any make variable in the UCSC tree. | |||
sudo apt-get install ptrace apache2 | |||
sudo mkdir -p /usr/local/apache/{htdocs,cgi-bin,trash} | |||
# apache needs to be able to create images | |||
sudo chown www-data:www-data /usr/local/apache/trash | |||
# the cgis create links to images in /trash | |||
cd /usr/local/apache/htdocs | |||
sudo ln -s ../trash | |||
# setup the custom trash dir in the same way | |||
sudo mkdir -p /usr/local/apache/trash/customTrash | |||
sudo chown www-data:www-data /usr/local/apache/trash/customTrash | |||
# make cgi-bin writable by ourselves | |||
sudo chown $USER:$USER /usr/local/apache/cgi-bin | |||
# setup apache | |||
sudo a2enmod include | |||
wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/000-default -O /etc/apache2/sites-enabled/000-default | |||
sudo service apache2 restart | |||
# setup CGIs | |||
# setup mysql, set the root password to "browser" | |||
apt-get install mysql-server | |||
mysql -u root -p (you chose the password during installation of the package) -e 'create database customTrash; create database hgcentral; | |||
# configure the CGIs | |||
cd /usr/local/apache/cgi-bin | |||
wget 'http://genome-source.soe.ucsc.edu/gitlist/kent.git/raw/master/src/product/ex.hg.conf' -O hg.conf | |||
vim hg.conf | |||
Set the mysql connection to the public mysql server: | |||
db.host=genome-mysql.soe.ucsc.edu | |||
db.user=genomep | |||
db.password=password | |||
customTracks.host=localhost | |||
customTracks.user=root | |||
customTracks.password=<enter the password you defined during mysql installation here> | |||
customTracks.useAll=yes | |||
customTracks.tmpdir=/usr/local/apache/trash/customTrash | |||
central.db=hgcentral | |||
central.host=localhost | |||
central.user=root | |||
central.password=<your password again> | |||
central.domain=<you internet domain> | |||
# alternative: use the public mysql server and my configuration, should work if your root password is "browser" | |||
wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/hg.conf.public -O /usr/local/apache/cgi-bin/hg.conf | |||
# copy just the cgi bin scripts | |||
cd /usr/local/apache/cgi-bin | |||
wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/cgiBinUbuntu1304_64.tgz -O - | tar xvz | |||
OR: compile everything: | |||
# set variables | |||
export MYSQLLIBS="-lmysqlclient -lz" | |||
export MACHTYPE=$(uname -m) | |||
export MYSQLINC=/usr/include/mysql | |||
cd ~/kent | |||
make alpha | |||
Download Hiram's install scripts to some place... how?... and run them. | |||
[[Category:User Developed Scripts]] | [[Category:User Developed Scripts]] | ||
[[Category:Installation]] |
Latest revision as of 07:32, 1 September 2018
The UCSC source tree includes tools for the command line and the CGIs for a local UCSC genome browser mirror. The installation is not easy, see below for a script that automates the process.
Please note the instructions and scripts in the source tree that can build the kent source tree on any Linux system: src/product
These commands compile the UCSC genome browser and all source tools. They have been tested on Debian Lenny (64 and 32bit), Ubuntu Maverick and Ubuntu 13.04. They will put all binaries in /usr/local, all cgi-bins in /usr/lib/cgi-bin and all html-like things into /var/www/genome.
In theory, once you have installed the samtools library, you can just copy-paste this script into a command line
# we need to be root to install packages and put stuff into /usr/local su # install required packages from repository # please not that these won't install the samtools library to handle bam files apt-get install mysql-server-5.0 apache2 libmysqlclient15-dev libpng12-dev libssl-dev openssl mysql-client-5.5 mysql-client-core-5.5 # set variables for compilation export MACHTYPE=$(uname -m) export MYSQLINC=`mysql_config --include | sed -e 's/^-I//g'` export MYSQLLIBS=`mysql_config --libs` DIRS='SCRIPTS=/usr/local/bin CGI_BIN=/usr/lib/cgi-bin DOCUMENTROOT=/var/www/genome BINDIR=/usr/local/bin' # this does not seem to be necessary anymore #ENCODE_PIPELINE_BIN=/usr/local/bin' # download cd /usr/local wget http://hgdownload.soe.ucsc.edu/admin/jksrc.zip unzip jksrc.zip mkdir -p /var/www/genome/ # compile libraries cd kent/src/lib make cd ../jkOwnLib make # compile browser cd .. make $DIRS # set permissions cd /home/data/www chown apache:apache -R *
Add the parameter HG_WARN=-w to DIRS if you don't want the compilation to stop on error messages. Dirty, but it works. :-)
The fastest way ever to get a genome browser up and running on Ubuntu, Fedora, Centos, OSX
Just tested this with ubuntu 14 LTS, centos 6, centos 7, fedora 20 and OSX 10.10 (on OSX, requires Xcode and macports):
wget https://github.com/maximilianh/browserInstall/blob/master/browserInstall.sh bash installBrowser.sh
Then connect with a firefox to localhost. Done.
If you look into the /usr/local/apache/cgi-bin/hg.conf file, you can see that we use the public mysql server through the internet. This will be very slow unless you're in San Francisco. So let's copy the complete genome of C. Briggsae + all annotations:
bash browserInstall.sh cb1
Now select the nematodes clade, the C. Briggsae 2002 Assembly. It should be very fast.
You can mirror hg19 like this ("bash browserInstall.sh hg19"), but remember that it will take a lot of space, ~2.2 TB.
Run a genome browser mirror on your local machine, compiled from source
I prefer to place the CGIs in the same location where UCSC has them, that's /usr/local/apache. This path does not conform to Ubuntu's specifications but has the advantage that we don't need to change any make variable in the UCSC tree.
sudo apt-get install ptrace apache2 sudo mkdir -p /usr/local/apache/{htdocs,cgi-bin,trash} # apache needs to be able to create images sudo chown www-data:www-data /usr/local/apache/trash # the cgis create links to images in /trash cd /usr/local/apache/htdocs sudo ln -s ../trash # setup the custom trash dir in the same way sudo mkdir -p /usr/local/apache/trash/customTrash sudo chown www-data:www-data /usr/local/apache/trash/customTrash # make cgi-bin writable by ourselves sudo chown $USER:$USER /usr/local/apache/cgi-bin # setup apache sudo a2enmod include wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/000-default -O /etc/apache2/sites-enabled/000-default sudo service apache2 restart # setup CGIs
# setup mysql, set the root password to "browser" apt-get install mysql-server mysql -u root -p (you chose the password during installation of the package) -e 'create database customTrash; create database hgcentral;
# configure the CGIs cd /usr/local/apache/cgi-bin wget 'http://genome-source.soe.ucsc.edu/gitlist/kent.git/raw/master/src/product/ex.hg.conf' -O hg.conf vim hg.conf
Set the mysql connection to the public mysql server:
db.host=genome-mysql.soe.ucsc.edu db.user=genomep db.password=password customTracks.host=localhost customTracks.user=root customTracks.password=<enter the password you defined during mysql installation here> customTracks.useAll=yes customTracks.tmpdir=/usr/local/apache/trash/customTrash central.db=hgcentral central.host=localhost central.user=root central.password=<your password again> central.domain=<you internet domain>
# alternative: use the public mysql server and my configuration, should work if your root password is "browser" wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/hg.conf.public -O /usr/local/apache/cgi-bin/hg.conf
# copy just the cgi bin scripts cd /usr/local/apache/cgi-bin wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/cgiBinUbuntu1304_64.tgz -O - | tar xvz
OR: compile everything: # set variables export MYSQLLIBS="-lmysqlclient -lz" export MACHTYPE=$(uname -m) export MYSQLINC=/usr/include/mysql cd ~/kent make alpha
Download Hiram's install scripts to some place... how?... and run them.