Monitoring Tasks Notes
This page is intended to document procedures and notes for various QA monitoring tasks.
Cronjob: Results from checkGbibMd5.sh
Hiram generally requests the push of vXXX gBiB Store Push (push request example). This push is usually done soon after the Tuesday release of CGIs to the RR/world. However, if this push is not done before this script runs, then you may see the md5 mismatch. It is recommended to wait until the end of the week (and possibly wait until the end of Friday following the release) to give Hiram time to do the push. Check the push request group for the latest store push. If the md5sums still do not match after the push of the GBiB to the store, then let Hiram know.
The /cluster/home/qateam/bin/scripts/checkGbibMd5.sh script checks for matching 'modify' times between beta's hgTracks and RR's hgTracks:
ssh qateam@hgwbeta stat -c %y /usr/local/apache/cgi-bin/hgTracks 2017-03-20 10:31:02.000000000 -0700
ssh qateam@hgw1 stat -c %y /usr/local/apache/cgi-bin/hgTracks 2017-03-20 10:31:02.000000000 -0700
If those dates match, the md5sums are compared between the local gbibBeta with the version on the store, which should also match, but in this case they don't match.
md5sum /usr/local/apache/htdocs/gbib/gbibBeta.zip | awk '{print $1}'
14e5f65fd19ecf43af05a313f884d26c
curl -s https://genome-store.ucsc.edu/media/products/gbib.zip | md5sum | awk '{print $1}'
0a90536be4f4e2fdb3e2d865762db818
Cronjob: Results from checkMetaAday.csh
This is one of the monitoring tasks that looks at metaData such as in hgcentral on hgwdev, hgwbeta, and the RR and notifies people with an email that they must investigate to see if there are differences. Differences are then corrected. The output is put into genecats: http://genecats.cse.ucsc.edu/qa/test-results/metadata/ where each database gets an entry.
Email Example
checkMetaAday.csh mm8
database = mm8
0 dbDb.mm8.hgcentralbetaOnly 0 dbDb.mm8.hgcentralOnly 1 dbDb.mm8.common
0 blatServers.mm8.hgcentralbetaOnly 0 blatServers.mm8.hgcentralOnly 2 blatServers.mm8.common
0 defaultDb.mm8.hgcentralbetaOnly 0 defaultDb.mm8.hgcentralOnly 0 defaultDb.mm8.common
0 genomeClade.mm8.hgcentralbetaOnly 0 genomeClade.mm8.hgcentralOnly 1 genomeClade.mm8.common
0 liftOverChain.mm8.hgcentralbetaOnly
1 liftOverChain.mm8.hgcentralOnly <---This should be a zero. There is "1" row difference. Need to fix.
52 liftOverChain.mm8.common
details in http://genecats.cse.ucsc.edu/qa/test-results/metadata/details
Process Example
Problem: The file below is in rr but not in beta
ornAna1 mm8 /gbdb/ornAna1/liftOver/ornAna1ToMm8.over.chain.gz 0.1 0 0 Y 1 N
Solution: Add row to beta liftOverChain table
First, double check: rr
hgsql -h genome-centdb -Ne "SELECT * FROM liftOverChain WHERE fromDb = 'ornAna1' AND toDb = 'mm8'" hgcentral +---------+-----+---------------------------------------------------+-----+---+---+---+---+---+ | ornAna1 | mm8 | /gbdb/ornAna1/liftOver/ornAna1ToMm8.over.chain.gz | 0.1 | 0 | 0 | Y | 1 | N | +---------+-----+---------------------------------------------------+-----+---+---+---+---+---+
beta
hgsql -h mysqlbeta -Ne "SELECT * FROM liftOverChain WHERE fromDb = 'ornAna1' AND toDb = 'mm8'" hgcentralbeta
No results. No entries match. Need to move the row that exists in the rr table into the beta table.
First, make a file of the row which is on rr but not on beta.
hgsql -h genome-centdb -Ne "SELECT * FROM liftOverChain WHERE fromDb = 'ornAna1' AND toDb = 'mm8'" hgcentral > chain.dev
Next, load the file into beta:
hgsql -h mysqlbeta -e "LOAD DATA LOCAL INFILE 'chain.dev' INTO TABLE liftOverChain" hgcentralbeta
Look in beta to see if the contents of the file make it into the table:
hgsql -h mysqlbeta -Ne "SELECT * FROM liftOverChain WHERE fromDb = 'ornAna1' AND toDb = 'mm8'" hgcentralbeta +---------+-----+---------------------------------------------------+-----+---+---+---+---+---+ | ornAna1 | mm8 | /gbdb/ornAna1/liftOver/ornAna1ToMm8.over.chain.gz | 0.1 | 0 | 0 | Y | 1 | N | +---------+-----+---------------------------------------------------+-----+---+---+---+---+---+
Looks good. CheckMetaAday again, for both ornAna1 and mm8
checkMetaAday.csh mm8
database = mm8
0 dbDb.mm8.hgcentralbetaOnly 0 dbDb.mm8.hgcentralOnly 1 dbDb.mm8.common
0 blatServers.mm8.hgcentralbetaOnly 0 blatServers.mm8.hgcentralOnly 2 blatServers.mm8.common
0 defaultDb.mm8.hgcentralbetaOnly 0 defaultDb.mm8.hgcentralOnly 0 defaultDb.mm8.common
0 genomeClade.mm8.hgcentralbetaOnly 0 genomeClade.mm8.hgcentralOnly 1 genomeClade.mm8.common
0 liftOverChain.mm8.hgcentralbetaOnly 0 liftOverChain.mm8.hgcentralOnly <---Looks good! 53 liftOverChain.mm8.common <--up by 1, good!
SLA Monitoring & Reporting
In Jan 2017, these data were migrated to a Google Spreadsheet:
- SLA Outage Report Worksheet.
- Current manager of this spreadsheet: Cath
- Please contact the current manager of the spreadsheet to enter/modify outages.
- The spreadsheet has a "README" tab to explain the spreadsheet format and best practices.
The old (unused) way to report outages was by adding it to an html table, SLA.html.
Cronjob: Results from realTime.csh (previously known as gbLoaded)
Previously, this job checked table times for the table gbLoaded in the database of the day. The problem with that is that the genome-asia machine doesn't like the pushed gbLoaded table because there's something different about the timestamp field type on that machine. No other genbank table uses that field type, and nothing uses gbLoaded except for this job. Braney proposed a switch to using xenoRefGene instead of gbLoaded for this check.
- Job is tracked in the genecats repository under qa/crontabs).
- Brian Lee updated this on hgwdev 3/30/17 and in the repository with commit 795b903 changing gbLoaded to xenoRefGene
- Note that there are a few assemblies that don't have this table (xenTro3, fr1, fr2, fr3, eboVir3, dm2). When the database of the day is on those assemblies, the check will provide no output.
- QA should check that the update times for each server are all close in time, within a week. Set 1 (dev & beta) are usually the same times, and Set 2 (rr/euro/asia) are usually on the same day, and the two sets are generally vary under a week.
Example output:
[qateam@hgwdev /cluster/home/qateam] /cluster/bin/scripts/realTime.csh `/cluster/bin/scripts/databaseAday.csh today` xenoRefGene verbose xenoRefGene ============= dev 2017-03-28 09:26:59 beta 2017-03-28 09:26:59 rr 2017-03-22 22:49:49 euro 2017-03-23 06:49:49 asia 2017-03-23 14:49:49
