Make your own virtual space: Difference between revisions

From Genecats
Jump to navigationJump to search
No edit summary
 
(18 intermediate revisions by 9 users not shown)
Line 1: Line 1:
===So you want to have your own copies of the binaries and your own hgwdev-user.cse.ucsc.edu browser?  Here is a walkthrough:===
===So you want to have your own copies of the binaries and your own sandbox or hgwdev-user.soe.ucsc.edu browser?  Here is a walkthrough:===




Line 7: Line 7:


  <VirtualHost 128.114.xx.yyy>
  <VirtualHost 128.114.xx.yyy>
   ServerName hgwdev-myusername.cse.ucsc.edu
   ServerName hgwdev-myusername.sdsc.edu
   ServerAlias hgwdev-myusername
   ServerAlias hgwdev-myusername hgwdev-myusername.soe.ucsc.edu hgwdev-myusername.gi.ucsc.edu
   ScriptAlias /cgi-bin/ "/usr/local/apache/cgi-bin-myusername/"
   ScriptAlias /cgi-bin/ "/usr/local/apache/cgi-bin-myusername/"
  </VirtualHost>
  </VirtualHost>


====Populate my /usr/local/apache/cgi-bin-myusername/ directory for the first time by doing the following:====
====Prepare for the making of your CGIs====
 
* If you don't already have one, set up your own personal Kent repository by following the instructions here: http://genomewiki.ucsc.edu/index.php/Getting_Started_With_Git#Setting_Up_Your_Own_Personal_Git_Kent_Repository
 
* Make sure your .bashrc or .tcshrc file (in your home directory) sets the [http://genomewiki.ucsc.edu/index.php/Build_Environment_Variables build environment variables].  It should include something like the following:
 
export MACHTYPE=`uname -m`
 
* Copy an hg.conf file from another user and place it here '/usr/local/apache/cgi-bin-username/'. It should contain at a minimum:
include ../cgi-bin/hg.conf
db.trackDb=trackDb_''myusername''
<span style="color: gray"># Angie recommends that you do <B>not</B> add these lines unless you are using
# your own central database instead of hgcentraltest.
# Assuming you are using the default central.db (hgcentraltest):
# * If you use the default central.* settings from ../cgi-bin/hg.conf, then you can
#  seamlessly switch between genome-test and hgwdev-$USER -- your cookie session
#  will be the same on both.
# * If you change the central.* options, you will have separate cookies between
#  genome-test and hgwdev-$USER, which means your settings from one won't carry
#  over to the other unless you use the same hgsid when switching hosts.
central.host=localhost
central.domain=hgwdev-''myusername''.soe.ucsc.edu
central.cookie=hguid.hgwdev-''myusername''
backupcentral.domain=hgwdev-''myusername''.ucsc.edu </span>


* Obtain a copy of the file .bashrc from another QA engineerCopy this file into your home directory at /cluster/home/myusername
Note that you can define your own background image by putting it in ~/public_html/, adding a short .css file (e.g., myusername.css) with the following lines
body.hgTracks {
    background: url("http://genome-test.gi.ucsc.edu/~myusername/yourimage.jpg");
  }
to your /usr/local/apache/htdocs-myusername/style/ directory, and adding the following line to your /usr/local/apache/cgi-bin-myusername/hg.conf file
browser.style=/style/myusername.css


* Set up your own personal Kent repository by following the instructions here: http://genomewiki.ucsc.edu/index.php/Getting_Started_With_Git#Setting_Up_Your_Own_Personal_Git_Kent_Repository
Here is a generic [[http://hgwdev.gi.ucsc.edu/images/floret.jpg background image]] file you can download and edit.


* Set your environment variables by typing the following:
* Please note, '''in order for your browser to be fully functional''', there needs to be '''a symlink to trash in your htdocs-''myusername''''':


  [myusername@hgwdev ~]$ MYSQLINC=/usr/include/mysql
  [''myusername''@hgwdev ~]$ cd /usr/local/apache/htdocs-''myusername''
[myusername@hgwdev ~]$ MYSQLLIBS="/usr/lib64/mysql/libmysqlclient.a -lz"
  [''myusername''@hgwdev htdocs-''myusername'']$ ln -s ../trash
[myusername@hgwdev ~]$ MACHTYPE=x86_64
  [myusername@hgwdev ~]$ HGCGI=/usr/local/apache/cgi-bin
[myusername@hgwdev ~]$ export MYSQLINC MYSQLLIBS MACHTYPE HGCGI


(For more on environment variables, see the article at http://genomewiki.ucsc.edu/index.php/Build_Environment_Variables)
* In order to load database tables and run CGIs on the command line for debugging, you will also need to create your own ~/.hg.conf .  It must be readable and writable only by you because it contains a username and password that enables write access to the database.  Here is one way to create it:
touch ~/.hg.conf
chmod 600 ~/.hg.conf
cat > ~/.hg.conf <<EOF
include /usr/local/apache/cgi-bin-''myusername''/hg.conf
db.user=''uuuu''
db.password=''pppp''
EOF
Instead of ''uuuu'' and ''pppp'', use the actual write-access username and passwords -- ask cluster-admin or a colleague (best not to email them).
 
====Populate my /usr/local/apache/cgi-bin-myusername/ directory for the first time by doing the following:====
 
* Before continuing forward, make sure user account is included in the genecats group. This can be done with '$ groups'. If genecats is not present, message cluster admins with a request at cluster-admin@soe.ucsc.edu.


* From your home directory on hgwdev /cluster/home/myusername type the following things:
* From your home directory on hgwdev /cluster/home/myusername type the following things:
Line 36: Line 74:
  [myusername@hgwdev src]$ cd hg
  [myusername@hgwdev src]$ cd hg
  [myusername@hgwdev hg]$ make cgi
  [myusername@hgwdev hg]$ make cgi
  [myusername@hgwdev hg]$ cd makeDb/trackDb
  [myusername@hgwdev hg]$ cd htdocs
[myusername@hgwdev htdocs]$ make user
[myusername@hgwdev htdocs]$ cd ../makeDb/trackDb
  [myusername@hgwdev trackDb]$ make update
  [myusername@hgwdev trackDb]$ make update


You now have your own compiled version of the CGIs.
You now have your own compiled version of the CGIs.


* Please note, in order for your browser to be fully functional, there needs to be a symlink to trash and to the javascript in your htdocs-myusername:
====Keep it current====


  [myusername@hgwdev ~]$ cd /usr/local/apache/htdocs-myusername
'''Note''': When we used CVS for revision control, it made sense to automate a nightly job that merged in everybody else's changes with our own work-in-progress and rebuilt the tree, so that we would keep up with others' changes and discover any conflicts sooner rather than later. Now that we use git, the workflow is a little different because git will refuse to merge if you have made changes but have not yet committed them to your local repository. '''If''' you always commit your changes at the end of the day, and never leave any work-in-progress edits overnight, then the script below will work.  In Angie's opinion it is better to use a more git-compatible workflow, for example using branches for projects that take longer than a day and being more deliberate about pulling and merging. See http://genomewiki.ucsc.edu/index.php/Getting_Started_With_Git and http://genomewiki.ucsc.edu/index.php/Working_with_branches_in_Git .
  [myusername@hgwdev htdocs-myusername]$ ln -s ../trash
[myusername@hgwdev htdocs-myusername]$ cd js
[myusername@hgwdev js]$ ln -s ../../htdocs/js/myusername


==== Create /usr/local/apache/cgi-bin-myusername/hg.conf:====
Here is a script (makeBinaries.csh) that you can copy and paste into your ~myusername/bin directory to make your binaries and trackDb_''myusername'' every night. You may or may not want to keep the git commands: they will fail if you have made edits but have not yet committed them to your local repository. Make sure to change the permissions (chmod 775 ~/bin/makeBinaries.csh) and if the script doesn't run check that the export PATH is set in your .bashrc file:
* Create a hg.conf file in '/usr/local/apache/cgi-bin-username/' with these lines using your user name:
 
include ../cgi-bin/hg.conf
#!/bin/sh
  db.trackDb=trackDb_myusername
# Script to rebuild the current user's sandbox
# Set environment
umask 002
MACHTYPE=`uname -m`
PATH=/usr/local/bin:/bin:/usr/bin:/cluster/bin/${MACHTYPE}:${HOME}/bin/${MACHTPE}:/cluster/bin:/cluster/bin/scripts:${HOME}/bin:${HOME}/bin/scripts/
export MACHTYPE PATH
  if [ -z "${USER}" ]; then
USER=${LOGNAME}
export USER
fi
   
   
  <span style="color: gray"># Angie recommends that you do <B>not</B> add these lines unless you are using
  # Clean up the source tree
# your own central database instead of hgcentraltest.
  cd $HOME/kent/src && make clean > /dev/null 2> /dev/null && rm -f tags
# Assuming you are using the default central.db (hgcentraltest):
  # * If you use the default central.* settings from ../cgi-bin/hg.conf, then you can
#  seamlessly switch between genome-test and hgwdev-$USER -- your cookie session
#  will be the same on both.
# * If you change the central.* options, you will have separate cookies between
#  genome-test and hgwdev-$USER, which means your settings from one won't carry
#  over to the other unless you use the same hgsid when switching hosts.
central.host=localhost
central.domain=hgwdev-myusername.cse.ucsc.edu
central.cookie=hguid.hgwdev-myusername
backupcentral.domain=hgwdev-myusername.ucsc.edu </span>
* You can define your own background image by putting it in ~/public_html/ and adding
browser.background=http://hgwdev.cse.ucsc.edu/~myusername/yourimage.jpg 
* Here is a generic [[http://hgwdev.cse.ucsc.edu/images/floret.jpg background image]] file you can download and edit.
 
====Keep it current:====
 
Here is a script (makeBinaries.csh) that you can copy and paste into your ~myusername/bin directory to make your binaries every night. Make sure to change the permissions (chmod 775) and if the script doesn't run check that the export PATH is set in your .bashrc file:
 
#!/bin/tcsh
   
   
  if ($HOST != hgwdev) then
# Update the source tree
  echo "\n  this must be run from hgwdev!\n"
cd $HOME/kent
  endif
git pull > $HOME/git.pull.output 2>&1
  if [ $? -ne 0 ]; then
echo "git pull failed, see $HOME/git.pull.output" | \
mail -s 'Git pull error on kent tree' ${LOGNAME}
  fi
   
   
# Build the CGIs
  cd ~/kent/src
  cd ~/kent/src
  make clean > /dev/null
  make -j 40 cgi > ~/daily.out 2>&1
  cd ~/kent
# If there are any errors, they will come via separate email from cron
git pull | mail -s 'Git pull report on kent tree' $USER
  egrep -y "erro|warn" ~/daily.out | grep -v "\-Werror" | \
  cd ~/kent/src
grep -v "gbWarn.o" | grep -v "disabling jobserver mode"
  make libs
  make tags-all >/dev/null 2>&1
  cd ~/kent/src/hg
  make cgi
  # Update htdocs
  cd ~/kent/src/hg/makeDb/trackDb
  cd ~/kent/src/hg/htdocs
  make update
  make user >/dev/null 2>&1
# Update trackDb
# Errors with context will be reported by cron
  cd ~/kent/src/hg/makeDb/trackDb/
  make update |& grep -i -B20 error


====Automate it:====
====Automate it:====
Line 135: Line 172:
for its output.
for its output.


==== A quicker list of the steps as of 2024 ====
<code>
# Ask Erich to create browser sandbox:
# hgwdev-USERNAME.gi.ucsc.edu loads from:
# hgwdev:/usr/local/apache/htdocs-USERNAME
# hgwdev:/usr/local/apache/cgi-bin-USERNAME
# Create ~/.hg.conf (get from someone in the group, e.g. max); replace "max" with your username in in ~/.hg.conf (and remove all commented lines)
# create /usr/local/apache/cgi-bin-USERNAME/hg.conf (get from Max). Replace "max" with your username in this file! (optionally remove all commented lines)


# Run the following commands:
chmod 600 ~/.hg.conf
cd ~ && git clone USERNAME@hgwdev.gi.ucsc.edu:/data/git/kent.git
export PATH=$PATH:/cluster/home/USERNAME/kent/src/utils/  # Need gitFiles from this directory
cd ~/kent/src && make cgi -j 30
cd ~/kent/src && make doc && make doc-install  # This needs the gitFiles script
~/kent/src/hg/makeDb/trackDb && make DBS=hg38
cd /usr/local/apache/htdocs-USERNAME && ln -s ../trash trash


[[Category:Browser Development]]
# Then your browser should be ready at https://hgwdev-USERNAME.gi.ucsc.edu/cgi-bin/hgTracks?db=hg38
[[Category:Browser QA]]
</code>
[[Category:Browser QA Training]]

Latest revision as of 11:06, 25 June 2024

So you want to have your own copies of the binaries and your own sandbox or hgwdev-user.soe.ucsc.edu browser? Here is a walkthrough:

Request a "virtual space" from the cluster-admins:

This includes, among other things, adding a block to the file /usr/local/apache/conf/httpd.conf

<VirtualHost 128.114.xx.yyy>
 ServerName hgwdev-myusername.sdsc.edu
 ServerAlias hgwdev-myusername hgwdev-myusername.soe.ucsc.edu hgwdev-myusername.gi.ucsc.edu
 ScriptAlias /cgi-bin/ "/usr/local/apache/cgi-bin-myusername/"
</VirtualHost>

Prepare for the making of your CGIs

  • Make sure your .bashrc or .tcshrc file (in your home directory) sets the build environment variables. It should include something like the following:
export MACHTYPE=`uname -m`
  • Copy an hg.conf file from another user and place it here '/usr/local/apache/cgi-bin-username/'. It should contain at a minimum:
include ../cgi-bin/hg.conf
db.trackDb=trackDb_myusername

# Angie recommends that you do not add these lines unless you are using
# your own central database instead of hgcentraltest.
# Assuming you are using the default central.db (hgcentraltest):
# * If you use the default central.* settings from ../cgi-bin/hg.conf, then you can
#   seamlessly switch between genome-test and hgwdev-$USER -- your cookie session
#   will be the same on both.
# * If you change the central.* options, you will have separate cookies between
#   genome-test and hgwdev-$USER, which means your settings from one won't carry
#   over to the other unless you use the same hgsid when switching hosts.
central.host=localhost
central.domain=hgwdev-myusername.soe.ucsc.edu
central.cookie=hguid.hgwdev-myusername
backupcentral.domain=hgwdev-myusername.ucsc.edu 

Note that you can define your own background image by putting it in ~/public_html/, adding a short .css file (e.g., myusername.css) with the following lines

body.hgTracks {
    background: url("http://genome-test.gi.ucsc.edu/~myusername/yourimage.jpg");
}

to your /usr/local/apache/htdocs-myusername/style/ directory, and adding the following line to your /usr/local/apache/cgi-bin-myusername/hg.conf file

browser.style=/style/myusername.css 

Here is a generic [background image] file you can download and edit.

  • Please note, in order for your browser to be fully functional, there needs to be a symlink to trash in your htdocs-myusername:
[myusername@hgwdev ~]$ cd /usr/local/apache/htdocs-myusername
[myusername@hgwdev htdocs-myusername]$ ln -s ../trash
  • In order to load database tables and run CGIs on the command line for debugging, you will also need to create your own ~/.hg.conf . It must be readable and writable only by you because it contains a username and password that enables write access to the database. Here is one way to create it:
touch ~/.hg.conf
chmod 600 ~/.hg.conf
cat > ~/.hg.conf <<EOF
include /usr/local/apache/cgi-bin-myusername/hg.conf
db.user=uuuu
db.password=pppp
EOF

Instead of uuuu and pppp, use the actual write-access username and passwords -- ask cluster-admin or a colleague (best not to email them).

Populate my /usr/local/apache/cgi-bin-myusername/ directory for the first time by doing the following:

  • Before continuing forward, make sure user account is included in the genecats group. This can be done with '$ groups'. If genecats is not present, message cluster admins with a request at cluster-admin@soe.ucsc.edu.
  • From your home directory on hgwdev /cluster/home/myusername type the following things:
[myusername@hgwdev ~]$ cd kent
[myusername@hgwdev ~/kent]$ git pull
[myusername@hgwdev ~/kent]$ cd src
[myusername@hgwdev src]$ make libs
[myusername@hgwdev src]$ cd hg
[myusername@hgwdev hg]$ make cgi
[myusername@hgwdev hg]$ cd htdocs
[myusername@hgwdev htdocs]$ make user
[myusername@hgwdev htdocs]$ cd ../makeDb/trackDb
[myusername@hgwdev trackDb]$ make update

You now have your own compiled version of the CGIs.

Keep it current

Note: When we used CVS for revision control, it made sense to automate a nightly job that merged in everybody else's changes with our own work-in-progress and rebuilt the tree, so that we would keep up with others' changes and discover any conflicts sooner rather than later. Now that we use git, the workflow is a little different because git will refuse to merge if you have made changes but have not yet committed them to your local repository. If you always commit your changes at the end of the day, and never leave any work-in-progress edits overnight, then the script below will work. In Angie's opinion it is better to use a more git-compatible workflow, for example using branches for projects that take longer than a day and being more deliberate about pulling and merging. See http://genomewiki.ucsc.edu/index.php/Getting_Started_With_Git and http://genomewiki.ucsc.edu/index.php/Working_with_branches_in_Git .

Here is a script (makeBinaries.csh) that you can copy and paste into your ~myusername/bin directory to make your binaries and trackDb_myusername every night. You may or may not want to keep the git commands: they will fail if you have made edits but have not yet committed them to your local repository. Make sure to change the permissions (chmod 775 ~/bin/makeBinaries.csh) and if the script doesn't run check that the export PATH is set in your .bashrc file:

#!/bin/sh
#	Script to rebuild the current user's sandbox

#	Set environment 
umask 002
MACHTYPE=`uname -m`
PATH=/usr/local/bin:/bin:/usr/bin:/cluster/bin/${MACHTYPE}:${HOME}/bin/${MACHTPE}:/cluster/bin:/cluster/bin/scripts:${HOME}/bin:${HOME}/bin/scripts/
export MACHTYPE PATH
if [ -z "${USER}" ]; then
	USER=${LOGNAME}
	export USER
fi

#	Clean up the source tree
cd $HOME/kent/src && make clean > /dev/null 2> /dev/null && rm -f tags

#	Update the source tree
cd $HOME/kent 
git pull > $HOME/git.pull.output 2>&1
if [ $? -ne 0 ]; then
	echo "git pull failed, see $HOME/git.pull.output" | \
		mail -s 'Git pull error on kent tree' ${LOGNAME}
fi

#	Build the CGIs
cd ~/kent/src
make -j 40 cgi > ~/daily.out 2>&1
#	If there are any errors, they will come via separate email from cron
egrep -y "erro|warn" ~/daily.out | grep -v "\-Werror" | \
	grep -v "gbWarn.o" | grep -v "disabling jobserver mode"
make tags-all >/dev/null 2>&1

#	Update htdocs
cd ~/kent/src/hg/htdocs
make user >/dev/null 2>&1

#	Update trackDb
#	Errors with context will be reported by cron
cd ~/kent/src/hg/makeDb/trackDb/
make update |& grep -i -B20 error

Automate it:

Create a crontab file. Note that you may have numerous crontab files. Each machine that you want cron jobs running on will need its own crontab file (assuming the jobs are different). Example, name your crontab file: hgwdev.cron for your hgwdev crontab. With your editor, add a line to your crontab file to run the command at the time you desire. For example, to run your daily build at 04:42 every weekday, the line would read:

42 04 * * 1-5 bin/makeBinaries.csh

See also: 'man 5 crontab' for a description of the fields used in these crontab lines.

When your crontab file is set correctly, hand it off to the cron system with the command:

$ crontab <yourCrontabFile>

Example: 'crontab hgwdev.cron'

To see what you have submitted to the cron system, ask it to display what it knows about your cron jobs with the -l argument:

$ crontab -l

Cron job tips: It is better to not run your jobs exactly at the top of the hour. They would be competing with system cron jobs that are running at times like the top of the hour. Better to run your job at odd minutes during the hour. Don't use the example time listed here, otherwise everyone's job will be running at 04:42 every day and competing with each other.

Any output to stdout or stderr by your cron job will be emailed to you via the cron system as it runs your job. If you would rather not see that output in email, construct your cron job shell script in such a way that it takes care of all of its stdout and stderr output. For example, it could create an organized by date hierarchy of log files for its output.

A quicker list of the steps as of 2024

  1. Ask Erich to create browser sandbox:
  2. hgwdev-USERNAME.gi.ucsc.edu loads from:
  3. hgwdev:/usr/local/apache/htdocs-USERNAME
  4. hgwdev:/usr/local/apache/cgi-bin-USERNAME
  1. Create ~/.hg.conf (get from someone in the group, e.g. max); replace "max" with your username in in ~/.hg.conf (and remove all commented lines)
  2. create /usr/local/apache/cgi-bin-USERNAME/hg.conf (get from Max). Replace "max" with your username in this file! (optionally remove all commented lines)
  1. Run the following commands:

chmod 600 ~/.hg.conf cd ~ && git clone USERNAME@hgwdev.gi.ucsc.edu:/data/git/kent.git export PATH=$PATH:/cluster/home/USERNAME/kent/src/utils/ # Need gitFiles from this directory cd ~/kent/src && make cgi -j 30 cd ~/kent/src && make doc && make doc-install # This needs the gitFiles script ~/kent/src/hg/makeDb/trackDb && make DBS=hg38 cd /usr/local/apache/htdocs-USERNAME && ln -s ../trash trash

  1. Then your browser should be ready at https://hgwdev-USERNAME.gi.ucsc.edu/cgi-bin/hgTracks?db=hg38