Gbib development

This page is for developers, it's not relevant for QA. The QA page is Genome_Browser_in_a_Box_config.

Setup your laptop

to connect to the gbib running on hgwdev, add this to .ssh/config on your laptop:

 Host devtunnel
 Hostname hgwdev.gi.ucsc.edu
 User YOURUSERNAME
 # local forward to gbib running on hgwdev, using local port 4321 to avoid clashing with the gbib running on your own machine
 LocalForward 127.0.0.1:4321 127.0.0.1:1234
 # local forward to devbox running on hgwdev
 LocalForward 127.0.0.1:1236 127.0.0.1:1236

Converting your gbib into a machine for development

The following constructions convert any gbib (running on hgwdev or on your laptop) into a machine for development similar to hgwdev.

if yo have a multicore machine, you might want to give the browserbox more of your CPUs so you can do "make -j2" or "make -j4". In virtualbox GUI, right click browserbox and increase the CPU slider.

On your local machine, make sure you have the file ~/.ssh/id_dsa.pub or ~/.ssh/id_rsa.pub. If not, run ssh-keygen (do not define a password).

It's better to connect from your terminal than to work in the strange virtualbox console that can't paste.

We first create the same username as on hgwdev and a symlink /cluster/home so you don't have to change your login scripts.

 ssh browser@localhost -p 1235
 # password is "browser"
 MYUSER=<YOURHGWDEVUSERNAME>
 # <YOURUSERNAME> is preferably your hgwdev username, so all your shell scripts continue to work
 sudo useradd $MYUSER -G sudo -m -s /bin/bash
 # let's create a /cluster/home like on hgwdev
 sudo mkdir /cluster
 sudo ln -s /home /cluster/home 

Now we get ssh publib key logins to work, so we don't need to type in this password anymore:

 sudo su - $MYUSER
 mkdir ~/.ssh
 chmod 700 ~/.ssh
 sudo cp /home/browser/ssh/config ~/.ssh/
 vim ~/.ssh/authorized_keys
 # paste the file ~/.ssh/id_dsa.pub (or id_rsa.pub) from your OSX filesystem into this file and save
 logout

Now from your local terminal, there is no more need for a password

 ssh $MYUSER@localhost -p 1235

(you can even add lines like "Hostname box\NHost localhost\NPort 1235" to your .ssh/config to shorten this to "ssh box")

Your hgwdev login shell scripts should work now to some extent in the gbib:

 scp $MYUSER@hgwdev:.bashrc ./
 scp $MYUSER@hgwdev:.vimrc ./
 bash
 # does this look familiar?  

Let's start the browser install then. Get the full software package repos:

 sudo vim /etc/apt/sources.list
 # remove the comments around the two lines that end with "trusty main" and "trusty-updates main"
 # line should look like "deb http://us.archive.ubuntu.com/ubuntu trusty main universe multiverse"

pull the software lists

 sudo apt-get update

vim users might want to upgrade the tiny vim to the real thing:

 sudo apt-get remove vim-tiny
 sudo apt-get install vim
 unalias vim
 vim /etc/profile
 # remove the line "alias vim=vim.tiny"

emacs users might want to install emacs:

 sudo apt-get install emacs

csh fanatics might want to add csh:

 sudo apt-get install csh

Most people will like at least one of these two development tools:

 sudo apt-get install ctags cscope sshfs

get the libs

 sudo apt-get install gcc libssl-dev git libpng-dev  libmysqlclient-dev libstdc++-4.8-dev # not sure why libstdc++-4.8-dev
 cd /data
 sudo mkdir samtabix
 sudo chmod a+rwx samtabix
 git clone http://genome-source.soe.ucsc.edu/samtabix.git samtabix
 cd samtabix
 make

add the vars

 vim /etc/profile
 # add these two lines at the end
 export SAMTABIXDIR=/data/samtabix
 export USE_SAMTABIX=1
 source /etc/profile

The VM separates the basic linux system (in "/") from your data ("/data"), so your data is separated from the rest of the VM.

So we clone the tree into a directory under /data and link to it:

 cd /data  
 sudo mkdir kent
 sudo chmod a+rwx kent 
 cd 
 ln -s /data/kent
 git clone $USER@hgwdev.gi.ucsc.edu:/data/git/kent.git kent

do it:

 sudo chmod -R a+wX /usr/local/apache/cgi-bin # make sure we can write in the cgi-bin directory
 sudo chmod -R a+wX /usr/local/apache/htdocs
 sudo ln -s /usr/local/apache/cgi-bin /usr/local/apache/cgi-bin-$USER  # so we don't have to do "make alpha" all the time
 cd kent/src
 make cgi

to feel a bit more at home, let's also get /hive:

 sudo mkdir /hive
 sudo chmod a+rwx /hive
 sshfs hgwdev.gi.ucsc.edu:/hive /hive -oauto_cache,reconnect

It's pretty slow. Maybe this makes it slightly faster?

 fusermount -u /hive
 sshfs hgwdev.gi.ucsc.edu:/hive /hive -oauto_cache,reconnect,compression=yes

Development on the devbox (not recommended anymore)

The above paragraph describes how to develop on the GBiB itself. This is why the paragraph below is not recommended anymore.

Box-specific development can happen on the devbox, a development version of the browserbox with git, gcc etc. For testing during development, a special command on the box can pull in the CGIs that were compiled on the devbox. The devbox is necessary as otherwise the gbib image has to allow development which adds around 2gb of packages most users don't ever need. In addition, it avoids the potential problem of our developer passwords leaking out through tempfiles or the swapspace.

The browserbox-dev VM, alias "the devbox", listens to http on 1236 and for ssh on 1237.

On hgwdev, all virtual machines live under ~qateam/VirtualBox VMs/. The central virtualbox xml config file where they are registered is ~/.config/VirtualBox/VirtualBox.xml

To start the devbox, you need to login as qateam onto hgwdev.

From hgwdev as qateam, you can then connect to the box with "ssh devbox". This works, because of qateam's .ssh/config file:

 Host devbox
 Hostname localhost
 Port 1237
 User browser

You can start and stop browserbox-dev with the aliases devStart and devStop as qateam.

Commented walkthrough of a GBib specific bugfix using the devbox

open a new terminal on your laptop and login to hgwdev with the tunnel:

 ssh devtunnel

Now start browserbox-dev as qa:

 ssh qa
 devStart

Wait a little bit then ssh into the devbox:

 ssh devbox
 cd kent/src
 git remote set-url origin ssh://YOURUSERNAME@hgwdev.gi.ucsc.edu/data/git/kent.git/
 git pull
 git checkout <THE BRANCH YOU WANT TO WORK ON, e.g. beta or master>
 make docs
 make cgi -j8 COREDUMP=1

setup the tunnel on port 1236 to hgwdev from your laptop (do this in a 2nd terminal):

 ssh devtunnel

You can now test the CGIs on the devbox by going to this URL

 http://127.0.0.1:1236

You can browse around on this site. It's slightly less configured than gbib (colors, icons etc) but the CGIs are the same. As soon as you go to the URL that fails in your browser e.g.:

 http://127.0.0.1:1236/cgi-bin/hgFileUi?db=hg19&g=wgEncodeAwgSegmentation

.. there will be no error message, but the CGI crashed with "abort()" and it will core dump.

This should produce a new core file on the devbox:

 ll /data/trash/core/core
 -rw------- 1 www-data www-data 73338880 Oct  3 08:07 /data/trash/core/core

Run gdb with the core file (the sudo password is "browser"):

 sudo gdb /usr/local/apache/cgi-bin/hgFileUi /data/trash/core/core

A typical solution is to replace cloneString with hReplaceGbdb at places where files in /gbdb are opened. Fix the bug and test after recompile.

Then don't forget to checkout the master branch and commit there (!):

 git checkout master
 git add hg/lib/fileUi.c
 git commit
 git push
 

Now you can file a build patch with this commit id.

testing of alpha binaries on a real gbib

To do a more realistic test, the alpha binaries can also be put into the final browserbox.

To copy the alpha binaries into the browserbox: Copy them from the devbox to alpha distribution rsync server directory (this command is defined in qateam's .bashrc) by running this command as qateam on hgwdev while the devbox is running (this will ssh into the devbox and then copy the binaries onto hgwdev):

 boxUpdateAlpha

Now start the gbib running on hgwdev and pull the alpha binaries from the rsync server:

 boxStart # will show an error message if already running
 ssh box
 updateBrowser alpha

Or you can run on the gbib on your own machine and type:

 updateBrowser alpha

You can then test the alpha CGIs in the "real" gbib environment via 127.0.0.1:4321 (gbib on hgwdev) or 127.0.0.1:1234 (your own gbib).

Don't forget to start the ssh tunnel to hgwdev (ssh devtunel) if you're using the gbib on hgwdev.

Gbib performance and internet latency

The Gbib is running on users' machines and users can be very far from UCSC, up to 200msec in TCP time

To simulate this situation, you can try the "netSlow" command on the gbib. The apache error log /var/log/apache2/error.log should tell you what the slow statements are, as the JKSQL_PROF profiling is enabled in hg.conf

If a track does a SELECT statment for each feature, this will make it very slow. As a remedy, you can

• copy the mysql table onto the gbib (mysqldump + scp + hgsql hg19< dumpfile)
  • weekly rsync will keep it updated then in the future, see the /root/updateBrowser shell script
• or modify the source code to reduce the number of select statements.

Pushing your changes to hgwdev from the devbox

Earlier on this page, we did the git pull/push directly from the devbox onto hgwdev. Here are some more details on this.

Direct git, as above - easier: set username for kent repo:

 cd kent
 git remote set-url origin ssh://YOURUSERNAME@hgwdev.gi.ucsc.edu/data/git/kent.git/
 git push
 # You can also pull from the public repo
 git remote set-url origin git://genome-source.soe.ucsc.edu/kent.git

Indirect: simply rsync or scp to copy into your own homedir on hgwdev and commit there:

 rsync file USER@hgwdev:kent/PATH/FILE
 logout

intensive box cleaning (optional)

Each time a new file is created on the box, the virtual disk file gets filled with the data, even if the file is deleted. The browserbox zipfile will therefore grow. If the zipfile is getting very big, it might be good to overwrite the empty space on the virtual disk with zeros.

To do this, install an RDP client on your OSX machine (Microsoft RDP client for OSX, in the app store, free), uncomment the RDP 3389 port forward in your .ssh/config and do an "ssh hgwdevtunnel" your desktop.

Start your RDP client, connect to localhost. You should see the console of the browserbox. Type:

 sudo reboot

• Keep shift pressed while the box is rebooting.
• Select the Recovery mode, type "enter".
• Type "r" to select "root". Type "enter".
• Password is "browser".
• Run the script "./zeroVm.sh". run "sudo reboot".