ENCODE QA

From Genecats
Revision as of 21:30, 10 March 2011 by Vanessa (talk | contribs)
Jump to navigationJump to search

Getting Started

  • Choose a track from the encodePushQ (sub pushQ accessed from pushQ Gateway).
  • Also claim the Redmine Issue of that track & change status from "Approved" to "Reviewing"
    • Add yourself as a watcher to the redmine ticket (that way when you assign it back to the developer you will get updates)
    • Make a question in the redmine ticket asking Kate to make a determination about the composite and subtrack labels (may not be necessary for subsequent releases)
    • email Kate when you claim a track (cc Katrina)
    • email wrangler telling them to update the ENCODE status to "Reviewing"
  • Check out the developer's notes file to get a feel for what the track consists of.
    • the path to the notes file will be in the "Notes" section of the pushQ entry
    • trust the notes file over the pushQ entry table/files information
  • If this is a subsequent release, see #Subsequent Release of Data (e.g. Release 2) first.
  • Use the % Done on Redmine Issue to track your QA progress. Kate uses this to check status.

run qaEncodeTracks.csh

You will need to dump the list of tables (just the new tables if this is a releaseN) from the pushQ (or developer's notes file if this a releaseN) to a file (i.e. tableList in the usage statement). Then run qaEncodeTracks.csh, which does:

  • countPerChrom
  • check for entry in tableDescriptions table
  • check that shortLabel does not exceed 17 characters
  • check that longLabel does not exceed 80 characters
  • check that there are no underscores in the table names
  • check for indices on the tables
  • check that positional tables are sorted
  • checkTableCoords (checks for any illegal coordinates)

Staging on hgwbeta

  1. Make a list of all tables (new & updated that need to be pushed to beta)
  2. In trackDb, change 'release alpha' lines to 'release alpha,beta' lines and 'release beta,public' to 'release public' and then check in these changes. (Make sure there are release tags for tracks in super tracks! See http://genomewiki.ucsc.edu/index.php/ThreeStateTrackDb#Existing_super-tracks_MUST_use_release_tags )
    • A quick way to replace these line in vi is ":#,## s/release alpha/release alpha,beta/" where # = from start line and ## = to ending line
  3. If this is a subsequent release (an update to an existing track), you may want to hold off on the next step so that you can compare old and new tracks on hgwdev and hgwbeta:
    • Open the track on hgwbeta before staging it to make sure that the update won't cause a cart clash for users currently looking at the track (as evidenced by a completely blank screen, for instance). If you need to do a cartReset to get the track to show up correctly, something is wrong.
  4. Do bigPush.csh using list created above
  5. Push any new /gbdb files (e.g. .wib or .bb files) from hgwdev to hgnfs1 if applicable
  6. On hgwbeta in trackDb: make beta DBS=dbName

Staging metaDb on hgwbeta

Starting from /cluster/home/$usr/trackDb/$species/$db/

  1. copy metaDb .ra file from ~metaDb/alpha -> metaDb/beta
  2. add .ra file name to the makefile in ~metaDb/beta
  3. From trackDb on hgwbeta: make beta DBS=<$db>

Other things to check by hand

  1. For hg19 check that labels follow this new convention: http://encodewiki.ucsc.edu/EncodeDCC/index.php/ENCODE_track_settings_style_guide
  2. Run genePredCheck/pslCheck if applicable. (i.e. if your track is a gene prediction track)
  3. make sure there is a link to the help doc (in the config section: "Select views (help)")
  4. check that metadata is present by clicking on "..." link in tables list on details page, spot check a few to make sure they are correct
  5. read description page (if the track is part of a super track, make sure to QA the super track description)
    • is it detailed enough, especially Methods
    • Is the text consistent with the rest of the site:
      • Make sure that any references to "data" are plural
      • Make sure that all units have a space in between their quantity ie. 50 bp and not 50bp
    • are the citations in correct format (CBSE_citation_format)
    • does the "Display Conventions and Configuration" section cover all track types
    • test all hyper-links
    • releaseN tracks should contain a section called "Release Notes" which should state the release# and provide a description of the changes in that particular release. See this page for an example.
    • Check for lab contact (sanitize email addresses using encodeEmail.pl script)
  6. Release log (look in PushQ): must contain "ENCODE", usually it is just be the shortlabel, if it is a subsequent release it should have "(release #)". If there is something weird about the data that needs to be noted, make sure it fits in nicely with the current release log entries. (url should be of the format: ../../cgi-bin/hgTrackUi?db=hg18&g=wgEncodeAffyRnaChip )
  7. configuration section (does it work?)
    • Check the views are working & that the settings work
    • Here are some additional specific guidelines when checking the Signal track default settings:
      • auto-scale shouldn't be used unless a lab insists; should be a fixed range
      • mean + whiskers should be the default setting unless there is a good reason
      • check signal in dense view for the whole chrom to make sure the fixed range allows for nice pattern of dark bands (we don't want to see all light gray across), wrangler should fix if need be
      • if there are multiple signal tracks in a track, their settings should be independent
  8. multi-view config: matrix etc.
    • By default, matrix boxes should be fully checked or fully unchecked (not grayed), if not, this is trackDb setting issue the wrangler should fix.
    • Tier 1 and Tier 2 cell lines should be labeled as such in bold in the matrix.
  9. check "reset to defaults" button (does it work)
  10. Make sure there's a link to the ENCODE Data Release Policy (at the bottom of the description page).
  11. Make sure the tracks in the Tier1 and Tier2 Cell Lines are properly colored (no black, and all tracks from one Cell Line have the same color).
  12. Tier 1 and 2 cell lines should be displaying first by default when viewed in the browser.
  13. For supertracks, by default the supertrack control should be on hide, and the tracks within should be on dense

Testing in the Browser

  1. test one point from table to view in GB (pick a point which can be obtained by clicking on "schema" from the track configuration page)
    1. If they are bam files, the schema will only give you a filePath. You will need to use SamTools to obtain a point to test.
      1. add /hive/data/outside/samtools/svn_${MACHTYPE}/samtools to $PATH in your .bashrc
      2. You can run the command line using the fileName found in the schema:
        samtools view -x filePath chrx:xxxx-xxxxxx | head
        samtools view -x /data/NT/gbdb/hg18/neandertal/seqAlis/Feld1-hg18.sorted.bam chr1:2000000-3000000 | head
      3. The output will give you the start position and then it gives you read length. Add the read length to the start position to get the end position. This will give the point needed to put into the browser for testing purposes.
    2. If they are bigWig files, you can use the utility: bigWigInfo file.bw - make sure they are V4!
    3. If they are bigBed files, you can use the utility: bigBedInfo file.bb
  2. If hg19, compare that same point to the equivalent position in hg18 by doing a "convert" from hg19 to see the equivalent position in hg18. In your hg19 window, go back to your region, open new window and paste in hg18 equivalent position and compare hg19 to hg18. (*Notee: Comparisons to hg18 should be very cursory. Any differences should be noted in the redmine ticket, but not necessarily investigated unless a user also brings up an issue. The thinking behind this is that when there are differences, it is most likely an error with hg18, not hg19 and we are unnecessarily holding up hg19).
  3. zoom into base level (at different visibilities)
  4. zoom way out 1million bps (at different visibilities)
  5. searching: should items be searchable
  6. default visibility: should this track be on by default?

Does the data make Sense?

  1. Compare related subtracks of related Views to each other. For example:
    • Does the All Signal Raw Signal really seem to comprise of the data in the Plus Raw Signal & Minus Raw Signal?
    • Do Peaks really represent the high Signal areas of the Signal View subtracks?
  2. Do the data make sense Biologically? Turn on other tracks to compare. For example:
    • RNA-seq data should correlate with the exons in a genes track
    • TFBS tracks should correlate with the beginning of gene transcripts

Performance Tests

  1. Does the first 'Signal' subtrack pass the chr1 test (chr1 loads in less than one minute)
  2. Do all views for one experiment pass chr1 test (e.g. Pol2 in GM12878 cells)?
  3. A user-oriented test would be to test the performance in a gene-size region of the track with just the default-on subtracks (for the Yale track, and many other ENCODE tracks, default-on subtracks will be all experiments in the GM12878 cell lines, Signal view only -- this should be the configuration you see after a cart reset, then turning the overall track vis to full).
  4. Note that ENCODE tracks can have any number of subtracks, and will continue to grow with time. We should definitely assure that useful subsets can be displayed in user-friendly time.

Files

First, a note about finding the files. One of the most time-consuming things we do is track down items that should have been placed in the "Files" section of the pushQ entries but weren't. It takes us a long time to (a) figure out what's missing, and (b) find it. If developers can ensure that both the /gbdb and /goldenPath files are there, it would be a huge help!

Download File specifics

  1. Check the Index page (e.g. this page ), which created from two files, preamble.html & index.html) in the downloads directory:
    • preamble.html - the top description part of the index page (not the list of files)
      • an introductory paragraph with a brief description, and link to the trackUi.
      • should include a link to the files.txt and md5sum.txt files
      • releaseN: should also include the release number at end of description (e.g. "This is Release 2 of this track. Release notes are included in the track description.")
      • If you need to edit the preamble.html (not the list of files), follow these steps:
        1. Edit the preamble.html file (note that it is not in git) in the downloads directory (/usr/local/apache/htdocs-hgdownload/goldenPath/$db/encodeDCC/) on hgwdev
        2. Regenerate index.html by running the script: encodeDownloadsPage.pl index.html (I think the preamble.html needs to be in the dir where you run the script)
        3. Look at results on genome-test. If necessary, go back to step 1.
    • index.html - the list of files on the index page (not the description on top)
      • should contain the name of each file being released (and only the name of those files in this release). A good way to spot check this is to make sure the number of files at the bottom of the list is correct.
        • new track: use the PushQ file list (shouldn't be files with V2 or V3, etc.) and your best judgment to determine that all the appropriate files (and only those files are listed).
        • releaseN: make sure the right # of files are there. Check some of the removed files to make sure they were in fact removed from the list. If the list doesn't seem right, run the encodeDownloadsPage.pl script (at the prompt type: encodeDownloadsPage.pl index.html) directly in the /releaseN directory (hgwdev: /kent/src/hg/htdocs/goldenPath/<db>/encodeDCC/<trackName>/releaseN) to generate a new index.html page that you know contains all the files in that directory. Then, copy the /releaseN/index.html to the main track directory (hgwdev: /kent/src/hg/htdocs/goldenPath/<db>/encodeDCC/<trackName>/) as that is the index.html used on the site. Don't forget to commit your changes.
      • make sure this file is executable (because of the way the links are created)
  2. Downloads directory also needs to contain the following:
    • files.txt - plaintext version of index.html; lists files with metadata
    • md5sum.txt - checksum of all files in download directory
  3. When you are ready to release make sure your track is listed on the downloads page - if it isn't listed, go /kent/src/hg/htdocs/ENCODE/downloads.html to add a line for your track, and push the following from hgwdev -> hgwbeta, RR:
 /usr/local/apache/htdocs-hgdownload/ENCODE/downloads.html

Also, if the new track is part of a super-track, when you add the super-track category, please make the title a non-underlined link to the super-track page, for example: 

 <A style="text-decoration:none" HREF="http://genome.ucsc.edu/cgi-bin/hgTrackUi?db=hg19&g=wgEncodeRnaSeqSuper" TARGET=_blank>RNA-seq</A>

Pushing Files

Pushing the three main types of files involved in ENCODE tracks.

  • gbdb Files

Files of this form get pushed hgwdev -> hgnfs1 

 /gbdb/hg18/wib/wgEncode*.wib
  • Other Files

Files of this form get pushed hgwdev -> hgwbeta & RR (they are quite often accidentally omitted from the pushQ entry -- you will need these types of protocol PDF files, if this is the first subtrack released for this cell line from this lab) 

 /usr/local/apache/htdocs/ENCODE/protocols/cell/*.pdf
  • Download Files

Download files for an original release get pushed hgdownload-test -> hgdownload (list the entire file path as usual) 

 /usr/local/apache/htdocs-hgdownload/goldenPath/hg18/encodeDCC/wgEncode*/index.html
 /usr/local/apache/htdocs-hgdownload/goldenPath/hg18/encodeDCC/wgEncode*/wgEncode*.[bed/wig].gz

When pushing download files for a subsequent release track (e.g. release 2), push files as follows (but in your request, list the from/to paths at the top followed by a list of the file names without the full path) 

 from hgwdev: /usr/local/apache/htdocs-hgdownload/goldenPath/<db>/encodeDCC/<trackName>/releaseN/
 to hgdownload: /usr/local/apache/htdocs/goldenPath/<db>/encodeDCC/<trackName>/
 (Note: no releaseN directory on hgdownload)
  • After Pushing Files:

Once the files have been pushed you can check to see if the push was successful using this script: checkPushedFiles.csh

validateFiles

  • No longer run, here are Tim's comments about QA running validateFiles: "To get these things through the pipeline, we run them through validateFiles, so I think your running them through again is one time too many. But if you are going to, then each lab and each file type may have negotiated limits (which may change between submissions). These limits are found in the relevant submission directory DAF files."
  • Old validateFiles process:

Test a smattering of different file types using this tool: validateFiles (type the program name without arguments to see the usage statement). If there are no errors, there will be no output. For example, for files of type tagAlign, invoke the tool like this: 

validateFiles -type=tagAlign -genome=/gbdb/hg18/hg18.2bit /usr/local/apache/htdocs/goldenPath/hg18/encodeDCC/wgEncodeHudsonalphaChipSeq/wgEncodeHudsonalphaChipSeqAlignmentsRep1Gm12878Control.tagAlign.gz

For tagAligns there are several relevant validateFiles options: 

  mismatches  - frequently 2 but negotiated for each lab.  Set this to 5 to be tolerant
  matchFirst - negotiated.  You should set this to 25 and even then you may need to adjust it
  nMatch - negotiated, but you should always have this parameter set.

If you want to be exact, then the metadata as seen on the downloads page tells which submission directory the file belongs to, and the most recent *.DAF (or *.daf) will have a validationSettings line in it which will include the settings that belong to each file type. Example: 

  /hive/groups/encode/dcc/pipeline/encpipeline_prod/773/UtaChIPseqBOonlyV1.DAF

has the line: 

  validationSettings allowReloads;validateFiles.tagAlign:mmCheckOnInN=100,mismatches=3,nMatch,matchFirst=25

This means that the tag aligns were validated with -mismatches=3 -nMatch -matchFirst=25

Subsequent Release of Data (e.g. Release 2)

Periodically, the existing ENCODE tracks will be augmented with new data as labs complete experiments on new cell lines, etc. The new data will come in various formats: some will replace existing data, some will be brand new, some old data will be eliminated, etc.

Notes file

The data wrangler will create a text document, check it into git, and place it here: kent/src/hg/makeDb/doc/encodeDccHg18/*.txt

This document should contain complete lists of each table and file and what its disposition is. The tables and files will fall into categories similar to this:

  • A) Untouched - are on public browser and should remain
  • B) Deprecated - are currently on RR but will no longer be needed and should not be referenced by the public site.
    • NOTE: NO FILES SHOULD BE REMOVED from the downloads directory on hgdownloads (RR).
    • This list is provided for completeness. Any files marked here as in gbdb may be eliminated.
  • C) New - are only currently on test but will need to be pushed to the RR.
  • D) Additional items of note

This document may not match reality. It may be the case that some of the tables/files do not exist at all, the names are incorrect, they are not present on the machine as listed in the file, they do not match the list that is in the pushQ. The first challenge in QAing a subsequent ENCODE release is to determine if/how the file diverges from reality. To do this, compare the file to the "snapshot" of what is included in the release (and what you should QA), which can be found in the "release2" list in the downloads directory (hgwdev: /data/apache/htdocs/goldenPath/<db>/<trackName>/release<x>/). If the file differs from the downloads directory, then send that information to the data wrangler and pop the track into the B-queue while they sort it out. Otherwise, QA spends far too much time figuring out exactly what they are expected to QA.

Once the list is finalized, proceed with the QA work as outlined above. Note the additional steps in the #Files section for how to handle the /releaseN directory.

MetaDb changes

There also may be some metadata changes to fix errors or add information such as the GEO Series and GEO Sample.

  • Be sure to QA these metadata changes
  • Check for related redmine issues (addition of GEO accessions should definitely have a related issue for the addition of this info)
  • You can also do a diff to check for any other metadata changes.From kent/src/hg/makeDb/trackDb/<org>/<db>/metaDb, do the following diff:
diff beta/wgEncodeYourTrack alpha/wgEncodeYourTrack

Releasing to RR

Note: Cc the data wrangler for this track on all your pushes Cc encode@soe.ucsc.edu on your final push.

  1. Check release log field in PushQ...needs to start with ENCODE
  2. If this is a first release, skip this step and go straight to Step#3. If this is a subsequent release, do the following:
    • Remove the 'release public' block (including sub-blocks) of your track from trackDb.wgEncode.ra.
    • Remove the 'release alpha,beta' lines from the release alpha blocks (including sub-blocks), and then on parent and view-in-the-middle blocks if applicable:
      • also note: removeAlphas script may not run if your table list file has lines that begin with a tab/space (remove these in vi with :%s/^ *//)
    1. git pull
    2. run removeAlphaBetas script (> to a file)
    3. diff between file & trackDb.wgEncode.ra (diff file1 file2)
      • double check # of release alpha,betas matches # of tables (diff file1 file2 | grep release | wc --lines)
    4. copy file over trackDb.wgEncode.ra
    5. git diff to check new copy against repository copy
    6. If necessary, remove release alphas from the parent block and "view-in-the-middle" sub-blocks and git diff again
    7. make alpha on db to make sure everything looks good on dev
    8. commit change
    9. git diff again to make sure they are the same
    10. remove file
    11. from trackDb on hgwbeta: make beta DBS=<db>
  3. Releasing the metaDb (most tracks will require this)
    • Starting from /cluster/home/$usr/trackDb/$species/$db/
    1. copy metaDb .ra file from ~metaDbeta-> metaDb/public
    2. add .ra file name to the makefile in ~metaDb/public
  4. Do a make public DBS=<db> (from trackDb on hgwbeta) and announce it to QA
  5. If this is a subsequent release, check track on http://hgwbeta-public.cse.ucsc.edu
  6. Run comparePublic.csh to check differences between trackDb_public and RR and hgwbeta.
  7. Push track tables from mysqlbeta -> mysqlrr (not trackDb_public yet)
  8. Drop tables from hgwbeta that need to be removed (being replaced by V# tables)
  9. Drop tables being removed from the RR
  10. Push trackDb and friends (Pushing_trackDb) and tableDescriptions from mysqlbeta -> mysqlrr
  11. Push download files, index.html, files.txt and md5sum.txt (from hgwdev to hgdownload)
    • If this is a releaseN, even though there is a releaseN directory on hgwdev, do not create one on hgdownload (see the Download Files section of #Pushing Files for specifics)
  12. Drop .wib files that need to be dropped (from hgnfs1)
  13. Check the ENCODE/downloads.html page to see if your track is listed. If not (mostly for first releases), edit and push ENCODE/downloads.html from hgwdev -> hgwbeta & RR (a special Encode download release log)
  14. This step is now done when you request a push of trackDb and friends so this step can be skipped: Push cv.ra file (only if there is a matrix) located here: /usr/local/apache/cgi-bin/encode
  15. Click "push requested" in the pushQ record and then click "done" after verification on the RR. Then transfer pushQ entry from the the L queue to the main pushQ.
Retrieved from "https://genomewiki.ucsc.edu/genecats/index.php?title=ENCODE_QA&oldid=102"