CGI Build Process

This page explains the process we use for building and releasing our CGIs. This is done on a three-week schedule.

The builds are built into sandboxes which are located here:

hgwbeta:/data/releaseBuild
titan:/scratch/releaseBuild

Older hgwbeta builds are periodically relocated here to save space on hgwbeta which has a small SSD disk:

 /hive/groups/browser/build

Setting Up the Environment for the Build

NOTE: The actions in this section are a one-time only set up performed by the new "build-meister".

Becoming the Build-Meister

All build scripts are now run by the "build" user. This user should already have it's environment properly configured. However, the build-meister will need to be able to log in (though ssh) as the build user, and the build user will need to know where to send mail.

• Set up .ssh/authorized_keys so that you can log in as the build user. Seek assistance from cluster-admin if you need it.

• Assign the build user's BUILDMEISTER environmental variable to equal the user name of the new build meister

hgwdev> ssh -X build@hgwdev     # the optional '-X' allows X-windows support
<build@hgwdev> edit .tcshrc     # use your preferred editor
# alter the following line:
< setenv BUILDMEISTER tdreszer
> setenv BUILDMEISTER chinli

Remember you will need to log out and log back in for the changes to take affect.

• Make sure the build user's cron jobs send you mail. Unlike the various build scripts which can use the BUILD_MEISTER environmental variable to find you, cron runs without access to that variable. Instead, you should add yourself to cron's MAILTO variable:

hgwdev> ssh -X build@hgwdev            # the optional '-X' allows X-windows support
<build@hgwdev> crontab -l > cron.txt
<build@hgwdev> edit cron.txt           # use your preferred editor
< MAILTO=rhead,tdreszer,braney,cricket
> MAILTO=rhead,chinli,braney,cricket
# and
< MAILTO=tdreszer,braney
> MAILTO=chinli,braney
<build@hgwdev> crontab cron.txt
<build@hgwdev> crontab -l        # verify what you have done.

Optionally set up your own build environment

NOTE: The remainder of this section is historical. Nevertheless, it is worth keeping these details here, especially if the build-meister wishes to try experimental changes under their own identity.

• Before running build scripts as yourself, you will need to set up the following in your log in file:

 hgwdev> cd ~  # go to your home directory as yourself.
 hgwdev> edit .tcshrc  # use your preferred editor

 # add (or update) the following lines:
 > umask 002
 > source /cluster/bin/build/scripts/buildEnv.csh

 # To be able to run the Java robot programs, add the following to the top of your path setting:
 > set path = ( /usr/java/default/bin $path ... )

 # optionally add helper aliases:
 > # wb gets you to the scripts dir.
 > alias wb 'cd $WEEKLYBLD'     
 > # cd $hg gets you to the latest build sandbox
 > if (( "$HOST" == "$BOX32" ) || ( "$HOST" == "hgwbeta" )) then
 >     setenv hg $BUILDDIR/v${BRANCHNN}_branch/kent/src/hg
 > endif

Remember you will need to log out and log back in for the changes to take affect.

NOTE: For those more comfortable with other shells (e.g. bash), it should be possible to run build scripts from another shell. However, the main limitation is the buildEnv.csh file which is edited and checked in every week, then sourced by .tcshrc. Without changes it cannot be sourced by .bashrc.

• Set up autologin among the general cluster machines

# On your local cse box (i.e. screech, pfft, whatever)
screech> ssh-keygen -t dsa  (use enter for all defaults)
screech> cd ~/.ssh,
# add yourself to the authorized keys
screech> cp id_dsa.pub authorized_keys

Also put these in your home directory on hgwdev:

screech> scp -r .ssh/ hgwdev:
# Permissions on .ssh should be 700.
# Permissions on files in .ssh/ should be 600 or 640.

• Set up autologin for hgdownload and hgdownload-sd by copying your public key to the list of authorized keys on those machines. You may need assistance from someone already authorized to login to hgdownload and hgdownload-sd:

hgwdev> edit ~/.ssh/id_dsa.pub     # copy the public key into the clipboard and then log into hgdownload as user qateam
hgwdev> ssh qateam@hgdownload
hgdownload> cd ~/.ssh
hgdownload> edit authorized_keys   # paste the key to the authorized_keys file

• You will also need a copy of .hg.conf.beta in your $HOME directory. This should be obtained from /cluster/home/build/.hg.conf.beta.

• Build Symlinks. These are critical for building 32 bit and 64 bit utilities

 
 hgwdev> cd ~/bin

 # Make sure you have $MACHTYPE directories
 hgwdev> mkdir i386     
 hgwdev> mkdir x86_64

 # Create a symlink for each $MACHTYPE
 hgwdev> ln -s /cluster/bin/i386 i386.cluster
 hgwdev> ln -s /cluster/bin/x86_64 x86_64.cluster

The symtrick.csh uses these automatically. If a script crashes and leaves the symlinks in an incorrect state, use unsymtrick.csh to restore. Build scripts check to see if unsymtrick.csh should be executed.

Preview1 Day Build : Day 2

This is day 2 in the schedule.

Run Git Reports

• Connect as "build" to hgwbeta. Then go to the weekly build dir on beta

hgwdev> ssh -X build@hgwbeta          # the optional '-X' allows X-windows support
<build@hgwbeta> cd $WEEKLYBLD

• make sure you are on master branch

<build@hgwbeta> git checkout master

• edit buildEnv.csh: change the 5th line then the 4th line

<build@hgwbeta> edit buildEnv.csh     # use your preferred editor
< setenv LASTREVIEWDAY 2012-06-12     # v269 preview
> setenv LASTREVIEWDAY 2012-07-03     # v270 preview
#   and
< setenv REVIEWDAY  2012-07-03        # v270 preview
> setenv REVIEWDAY  2012-07-24        # v271 preview

• re-source buildEnv.csh and check that vars are correct

<build@hgwbeta> source buildEnv.csh   # or just restart your shell windows
<build@hgwbeta> env

• commit the changes to this file to Git:

<build@hgwbeta> @ NEXTNN = ( $BRANCHNN + 1 ) ; git commit -m "v$NEXTNN preview1" buildEnv.csh; git pull
<build@hgwbeta> git push

• run doNewReview.csh

<build@hgwbeta> screen                # use screen if you wish
<build@hgwbeta> ./doNewReview.csh     # review the variables

• run for real and direct output to a log file (this takes about 2 minutes - it runs git reports by ssh'ing to hgwdev)

<build@hgwbeta> ./doNewReview.csh real >& doNewRev.log 
<build@hgwbeta> ctrl-a, d             # to detach from screen
<build@hgwbeta> tail -f doNewRev.log  # see what happens

Check the reports

• The reports are automatically built by the script into this location.

Briefly review the reports quickly as a sanity check.

Generate summary and review pairings

(Ann takes care of this)

• Summarize the code changes that were committed during the past week.
• Update this page with the summary.
• Assign code-review partners.
• Send an email to browser-staff with the summary and code-review pairings.

Preview2 Day Build : Day 9

This is day 9 in the schedule.

Run Git Reports

• Connect as "build" to hgwbeta. Then go to the weekly build dir on beta

hgwdev> ssh -X build@hgwbeta          # the optional '-X' allows X-windows support
<build@hgwbeta> cd $WEEKLYBLD

• make sure you are on master branch

<build@hgwbeta> git checkout master   # or just: git branch

• edit buildEnv.csh: change the 7th line then the 6th line

<build@hgwbeta> edit buildEnv.csh     # use your preferred editor
< setenv LASTREVIEW2DAY 2012-06-19    # v269 preview2
> setenv LASTREVIEW2DAY 2012-07-10    # v270 preview2
#   and
< setenv REVIEW2DAY 2012-07-10        # v270 preview2
> setenv REVIEW2DAY 2012-07-31        # v271 preview2

• re-source buildEnv.csh and check that vars are correct

<build@hgwbeta> source buildEnv.csh   # or just restart your shell windows
<build@hgwbeta> env

• commit the changes to this file to Git:

<build@hgwbeta> @ NEXTNN = ( $BRANCHNN + 1 ) ; git commit -m "v$NEXTNN preview2" buildEnv.csh; git pull
<build@hgwbeta> git push

• run doNewReview2.csh

<build@hgwbeta> screen                # use screen if you wish
<build@hgwbeta> ./doNewReview2.csh    # review the variables

• run for real and direct output to a log file (this takes about 2 minutes - it runs git reports by ssh'ing to hgwdev)

<build@hgwbeta> ./doNewReview2.csh real >& doNewRev2.log 
<build@hgwbeta> ctrl-a, d             # to detach from screen
<build@hgwbeta> tail -f doNewRev2.log # see what happens

Check the reports

• The reports are automatically built by the script into this location.

Briefly review the reports quickly as a sanity check.

Generate summary and review pairings

(Ann takes care of this)

• Summarize the code changes that were committed during the past week.
• Update this page with the summary.
• Assign code-review partners.
• Send an email to browser-staff with the summary and code-review pairings.

Final Build : Day 14

This is day 14 in the schedule.

Do the Build

• Connect as "build" to hgwbeta. Then go to the weekly build dir on beta

hgwdev> ssh -X build@hgwbeta          # the optional '-X' allows X-windows support
<build@hgwbeta> cd $WEEKLYBLD

• make sure you are on master branch and there is no uncommitted script changes

<build@hgwbeta> git checkout master
<build@hgwbeta> git status

• edit the buildEnv.csh file

<build@hgwbeta> edit buildEnv.csh     # use your preferred editor
< setenv LASTWEEK 2012-06-26          # v269 final
> setenv LASTWEEK 2012-07-17          # v270 final
#   and
< setenv TODAY 2012-07-17             # v270 final
> setenv TODAY 2012-08-07             # v271 final
#   and the big one:
< setenv BRANCHNN 270
> setenv BRANCHNN 271

• re-source buildEnv.csh and check that vars are correct

<build@hgwbeta> source buildEnv.csh   # or just restart your shell windows
<build@hgwbeta> env

• commit the changes to this file to Git:

<build@hgwbeta> git commit -m "v$BRANCHNN final build" buildEnv.csh;git pull
<build@hgwbeta> git push

• do a dummy-login to the 32-bit box (titan) so that the script will be more likely not to hang later on building 32-bit cgis.

<build@hgwbeta> ssh $BOX32 "ls -l" <SKIPPED for the time being>

Note that the login sometimes demands a password. If it does, logout/re-login to hgwbeta as build and try again!

• run doNewBranch.csh

<build@hgwbeta> screen                # NOTE: screen is recommended this time!
<build@hgwbeta> ./doNewBranch.csh     # review the variables

• run for real, send the output to a file and review while it is written (takes ~1 hour)

<build@hgwbeta> ./doNewBranch.csh real >& doNewBranch.log
<build@hgwbeta> ctrl-a, d             # to detach from the screen
<build@hgwbeta> tail -f doNewBranch.log     # follow what happens

• look for files that tell you it was successful (script will report whether these files were created):

<build@hgwbeta> ls -l /cluster/bin/build/scripts/32bitUtils.ok  <SKIPPED for the time being>
<build@hgwbeta> ls -l /cluster/bin/build/scripts/GitReports.ok

• Check timestamp of CGIs in hgwbeta:/usr/local/apache/cgi-bin and the version number in the browser title header.

• If you get errors it might be because the script is wrong, rather than there actually be an error. For example, to check for errors, the 'make' log file is grepped for 'error|warn' so any new 'C' file with error or warn in its name will show up as an error whether or not it compiled cleanly. You might need to change the script to remove references to files like this, eg edit buildBeta.csh to ignore references to files like gbWarn.c and gbWarn.o in the log:

<build@hgwbeta> edit buildBeta.csh
...
make alpha >& make.alpha.log
# These flags and programs will trip the error detection
sed -i -e "s/-DJK_WARN//g" make.alpha.log
sed -i -e "s/-Werror//g" make.alpha.log
#-- report any compiler warnings, fix any errors (shouldn't be any)
#-- to check for errors:
set res = `/bin/egrep -i "error|warn" make.alpha.log | /bin/grep -v "gbWarn.o -c gbWarn.c" | /bin/grep -v "gbExtFile.o gbWarn.o gbMiscDiff.o"`

• What the doNewBranch.csh script does:

1. edits the versionInfo.h file
2. makes tags (takes 1 minute)
3. builds Git reports (takes 1 minute)
4. does build (takes 5-10 minutes)
   1. builds utils (of secondary importance)
   2. builds CGIs (most important)

Check the reports

• The reports are automatically built by the script into this location.
• Briefly review the reports quickly as a sanity check.

Run the Robots

• [build-meister] run doRobots.csh, and watch the log if you are interested (most log messages go to the logs/ dir mentioned below)

<build@hgwbeta> screen                # startup a new screen 
<build@hgwbeta> ./doRobots.csh >& $BRANCHNN.robots.log 
<build@hgwbeta> ctrl-a, d             # detach from screen
<build@hgwbeta> tail -f $BRANCHNN.robots.log

• What the doRobots.csh script does:

1. runs robots one at a time
   1. hgNear (20 min)
   2. hgTables (several hours)
   3. TrackCheck (several hours)
   4. LiftOverTest (quick)

• [build shepherd] Review the error logs for the robots:

error logs located here: hgwbeta:/cluster/bin/build/scripts/logs

• hgNear -- sends email with results
• hgTables -- send email with results
• TrackCheck -- must check by hand: grep -i "error" logs/TrackCheck-v$BRANCHNN.log (TrackCheck person does this)
• LiftOverTest -- must check by hand: cat logs/LiftOverTest-v$BRANCHNN.log

Test on hgwbeta

• Wait to hear from QA about how their CGIs look on hgwbeta. QA members should update the CGI build chatter ticket in Redmine with a "done testing" message or, if applicable, "not following issues for this release" message. Each member of the QA team has testing responsiblities.

Make changes to code base as necessary

This happens on days 15, 16, 17, and 18 in the schedule.

• If there are problems with the build a developer will fix the code. This fix needs to be patched into the build on hgwbeta. This page explains how to do a Cherry Pick on hgwbeta.

Fixing problems in the Build

This usually happens between days 16 and 19.

QA advises buildmeister to cherry pick

• see these instructions.

Push the CGIs

This is day 23 in the schedule.

The day before the push (day 22 in the schedule) send email notice

Send email to all of browser-staff (which includes cluster-admin) letting them know that tomorrow is a push day. Something along these lines:

Just a heads up that tomorrow is a CGI push day. If you have big code changes included in 
this release please be available in case something goes wrong with the push of your changes. 
QA typically starts the push around 1:30pm.

Push to hgw0 only

• hgw0 is identical to the RR machines but not actually in the RR (i.e. changes there are not seen by the public).
• QA will send an email to push-request the morning of the push letting the pushers know that today is a CGI push day (this is their notice to be vigilant about pushing quickly).
• QA will ask for push of CGIs from hgwbeta to hgw0 only. If there is a NEW CGI or file(s) going out this week, be sure to make a prominent note of it in your push request. The admins push from a script, and they will need to add your new CGI to the script. (the build-meister should not be cc'd on this email.)

As of October 2011, here's a list of the CGIs and data files we push. Note: CGIs and data files may have been added since this list was created -- this is meant to be a starting point.

cartDump cartReset das hgc hgPal hgEncodeVocab
hgBlat hgConvert hgCustom hgGateway hgGene hgGenome
hgLiftOver hgNear hgPcr hgSession hgTables hgTracks hgTrackUi
hgVisiGene hgEncodeDataVersions pbGateway pbGlobal
pbTracks phyloGif hgSuggest hgApi hgFileUi hgFileSearch
hgHubConnect hgRenderTracks hgEncodeApi hgLogin

and these configuration files:

/usr/local/apache/cgi-bin/all.joiner
/usr/local/apache/cgi-bin/galaAvail.tab
/usr/local/apache/cgi-bin/hgNearData/*
/usr/local/apache/cgi-bin/hgGeneData/*
/usr/local/apache/cgi-bin/hgcData/*
/usr/local/apache/cgi-bin/hgCgiData/*
/usr/local/apache/cgi-bin/loader/*
/usr/local/apache/cgi-bin/visiGeneData/*
/usr/local/apache/cgi-bin/lsSnpPdbChimera.py
/usr/local/apache/cgi-bin/greatData/* 

For these directories we request an rsync --delete (from hgwbeta to the RR)

/usr/local/apache/htdocs/js/*  
/usr/local/apache/htdocs/style/*  

• Run TrackCheck on hgw0. This is the responsibility of the QA person who tests hgTracks.

1. make a props file which specifies the machine/db to check. Set zoomCount=1 and it will only check the default position for each assembly. Example props file for hgw0:

machine mysqlbeta.soe.ucsc.edu #This is where TrackCheck checks for active databases (active=1). These databases maynot be on the the RR and it will give errors which can be ignore.
server hgw0.soe.ucsc.edu # this is the machine that you are testing.
quick false 
dbSpec all #You can list just one database here.
table all  #You can list one table if need be.
zoomCount 1 # if number is greater than one it will check links at higher zoom levels.

1. run it from hgwdev: nohup TrackCheck hgw0.props > & $WEEKLYBLD/logs/TrackCheck-hgw0.07-13-2006
   1. run in the background if desired by typing Ctrl-Z then "bg", to check status type "jobs" or "ps -ef | grep TrackCheck".
2. examine the file for errors.

• Monitor Apache Error Log (QA does this):

hgw0:/usr/local/apache/logs/error_log

To watch the log without line wraps, type "less -S error_log." Typing capital "F" will all you to follow incoming errors. When errors arrise you can type Ctrl-C and use the right arrow to scroll the window over to see the entire message.

• Wait to hear from QA about how their CGIs look on hgw0. Each member of the QA team has testing responsiblities. Check also that TrackCheck ran successfully.

Push to hgwN only

• hgwN is one of the RR machines, hgw1-8. Each build, rotate to the next machine in numeric order i.e. hgw1 then hgw2 etc. so that one machine is not being worked more than the others.
• Once the new CGIs are on hgwN the push shepherd will watch the error logs for a short while to make sure no new errors occur under load.

Push to the rest of the RR and hgwbeta-public and euronode

• QA will ask for push from hgwbeta to the rest of the hgwN machines, as well as hgwbeta-public and euronode. The js and style directory files should also go to /usr/local/apache/htdocs/js-public/* (or style-public/*) on hgwbeta ONLY in order to keep the javascript the same on the RR and hgwbeta-public. So, in addition to asking for the rsync --delete of the directories from hgwbeta to the RR machines, we also need to ask for an rsync --delete:

from
/usr/local/apache/htdocs/js/*
/usr/local/apache/htdocs/style/*  (on hgwbeta)
to
/usr/local/apache/htdocs/js-public/*
/usr/local/apache/htdocs/style-public/*  (on hgwbeta ONLY)

• QA will send email to the build-meister to let him/her know that the CGIs are on the RR.

Remember to keep track of new features

Anyone can add to this list at any time, but if no notes for this release have been made on the new features page, now is a good time to add some.

Final Build Wrap-up

This is day 23 in the schedule.

The buildmeister should do these steps once QA has notified you that all RR machines have been updated.

Normally this is run once at the end of the cycle. However, occassionally it is necessary to patch a build after it is already released on the RR. Depending upon the extent of the patch, it may be desirable or even necessary to rerun the wrap-up. All of these scripts can be safely rerun into the next build is made (until BRANCHNN is updated in buildEnv.csh).

• Connect as "build" on hgwdev this time. Then go to the weekly build dir on beta

hgwdev> ssh -X build@hgwdev            # the optional '-X' allows X-windows support
<build@hgwdev> cd $WEEKLYBLD

• build and push hgcentral IF there are any changes

<build@hgwdev> ./buildHgCentralSql.csh
<build@hgwdev> ./buildHgCentralSql.csh real

• check that the hgcentral.sql has been updated: http://hgdownload.soe.ucsc.edu/admin/ and http://hgdownload-sd.soe.ucsc.edu/admin/

• Now connect to hgwbeta.

hgwdev> ssh -X build@hgwbeta           # the optional '-X' allows X-windows support
<build@hgwbeta> cd $WEEKLYBLD
<build@hgwbeta> screen                 # if desired

• do a dummy-login to the 32-bit box (titan) so that the script will be more likely not to hang later

<build@hgwbeta> ssh $BOX32 "ls -l"

Note that the login sometimes demands a password. If it does, logout/re-login to hgwbeta as build and try again!

• build 'userApps' target (various utilities) on hgwbeta and scp them to hgdownload and hgdownload-sd

<build@hgwbeta> ./doHgDownloadUtils.csh
# patch as needed
# pipe output to log file by "./doHgDownloadUtils.csh >& doHgDownloadUtils.log" if desired.

• check that the utils version is right: http://hgdownload.soe.ucsc.edu/admin/exe and http://hgdownload-sd.soe.ucsc.edu/admin/exe

• build 32bit build and push to hgdownload and hgdownload-sd

<build@hgwbeta> cd $WEEKLYBLD
<build@hgwbeta> ./doBuildCgi32.csh
# patch as needed
# pipe output to log file by "./doBuildCgi32.csh >& doBuildCgi32.log" if desired

• check that the source file version is right on both download machines:

<build@hgwbeta> ssh qateam@hgdownload "ls -ltr /mirrordata/apache/cgi-bin-i386"
<build@hgwbeta> ssh qateam@hgdownload-sd "ls -ltr /mirrordata/apache/cgi-bin-i386"

• update the beta tag to match the release:

<build@hgwbeta> cd $WEEKLYBLD
<build@hgwbeta> env             # (just to make sure it looks right)
<build@hgwbeta> ./tagBeta.csh
<build@hgwbeta> ./tagBeta.csh real >& tagBeta.log

• tag the official release

<build@hgwbeta> cd $WEEKLYBLD
<build@hgwbeta> git fetch
<build@hgwbeta> git tag | grep "v${BRANCHNN}_branch"
# Note: use .1 or .2 or whatever is the next unused subversion number
<build@hgwbeta> git push origin origin/v${BRANCHNN}_branch:refs/tags/v${BRANCHNN}_branch.1
<build@hgwbeta> git fetch

• zip the source code

<build@hgwbeta> cd $WEEKLYBLD
<build@hgwbeta> ./doZip.csh  # (this is automatically pushed to hgdownload)

• check that the source file version is right: http://hgdownload.soe.ucsc.edu/admin/ and http://hgdownload-sd.soe.ucsc.edu/admin/

• send email to genome-mirror@soe.ucsc.edu.

Include this link to latest source: http://hgdownload.soe.ucsc.edu/admin/jksrc.zip. Use the last email as a template (see https://www.soe.ucsc.edu/pipermail/genome-mirror). If you push the hgcentral.sql, make sure to mention this has also changed in the email.

Example:

To: genome-mirror@soe.ucsc.edu
Subject: v273 Genome Browser Available

The source is now available at:
      http://hgdownload.soe.ucsc.edu/admin/jksrc.zip

or labeled with source number:
      http://hgdownload.soe.ucsc.edu/admin/jksrc.v273.zip

A license is required for commercial download and/or installation of
the Genome Browser binaries and source code. No license is needed for
academic, nonprofit, and personal use.

Summaries of changes can be found here:
http://genecats.soe.ucsc.edu/builds/versions.html

The following CGIs were updated:

cartDump cartReset das hgc hgPal hgEncodeVocab
hgBlat hgConvert hgCustom hgFileSearch hgFileUi hgGateway
hgGene hgGenome hgHubConnect hgLiftOver hgLogin hgNear hgPcr
hgSession hgTables hgTracks hgTrackUi hgVisiGene mkEncodeFrameset
hgEncodeDataVersions phyloGif hgSuggest hgApi hgEncodeApi

 and these configuration files:

/usr/local/apache/cgi-bin/all.joiner
/usr/local/apache/cgi-bin/galaAvail.tab
/usr/local/apache/cgi-bin/hgNearData/*
/usr/local/apache/cgi-bin/hgGeneData/*
/usr/local/apache/cgi-bin/hgcData/*
/usr/local/apache/cgi-bin/hgCgiData/*
/usr/local/apache/cgi-bin/loader/*
/usr/local/apache/cgi-bin/visiGeneData/*
/usr/local/apache/cgi-bin/lsSnpPdbChimera.py
/usr/local/apache/cgi-bin/greatData/*
/usr/local/apache/cgi-bin/encode/cv.ra

Please rsync --delete these directories:

/usr/local/apache/htdocs/js/*
/usr/local/apache/htdocs/style/*

Please rsync this directory:
/usr/local/apache/htdocs/images/*

A new hgcentral.sql file should now be present at:
  http://hgdownload.cse.ucsc.edu/admin/

If you have any questions or concerns, please feel free to write back
to this mail list.

Thanks,
{buildmeister}