Monitoring Tasks: Difference between revisions

From Genecats
Jump to navigationJump to search
(Adding link for notes on realTime (previously gbLoaded))
(Adding the Hub Public Coordinator role to Monitoring Tasks)
 
(53 intermediate revisions by 7 users not shown)
Line 1: Line 1:
Please add to the documentation page, [[Monitoring_Tasks_Notes|Monitoring Tasks Notes]] to add procedures and notes for specific jobs, then link back to the job in the list below.  
Please add to the documentation page, [[Monitoring_Tasks_Notes|Monitoring Tasks Notes]] to add procedures and notes for specific jobs, then link back to the job in the list below.
 
<table border="1">
<table border="1">


Line 6: Line 5:
<th>task</th>
<th>task</th>
<th>frequency</th>
<th>frequency</th>
<th>NOW</th>
<th> Until Aug '18</th>
<th>who April '16 -Feb '17</th>
<th>Until Sep '19</th>
<th>previous</th>
<th>Now</th>
<th>previous</th>
<th></th>
</tr>
</tr>


Line 20: Line 17:
<td>Respond to items on the broken links report: Check to make sure that broken links reported are actually broken.  If so, fix the link if it is clear what the new link should be and push to the RR.  If not, contact the appropriate person at the web site in question to find out what the new link should be.</td>
<td>Respond to items on the broken links report: Check to make sure that broken links reported are actually broken.  If so, fix the link if it is clear what the new link should be and push to the RR.  If not, contact the appropriate person at the web site in question to find out what the new link should be.</td>
<td>daily</td>
<td>daily</td>
<td>ChrisV</td>
<td>Jairo</td>
<td>Cath, Matt</td>
<td>Jairo</td>
<td>Matt</td>
<td>Jairo</td>
<td>Matt</td>
<td></td>
</tr>
</tr>


Line 31: Line 26:
<td>weekly</td>
<td>weekly</td>
<td>Jairo</td>
<td>Jairo</td>
<td>Cath</td>
<td>Conner</td>
<td>Matt</td>
<td>Lou</td>
<td>Luvina</td>
 
<td></td>
</tr>
</tr>




<tr>
<tr>
<td>receive output from syncAday.csh; alert; if anything is out of sync (checks that tables are in sync between beta and RR for database of the day) </td>
<td>'''Retired (See #23998)''' receive output from syncAday.csh; alert; if anything is out of sync (checks that tables are in sync between beta and RR for database of the day) </td>
<td>daily</td>
<td>daily</td>
<td>Cath</td>
<td>Cath</td>
<td>ChrisV</td>
<td>Lou</td>
<td>Brian</td>
<td>Retired</td>
<td>Brian</td>
<td></td>
</tr>
</tr>


<tr>
<tr>
<td>receive output from [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#Cronjob:_Results_from_checkMetaAday.csh checkMetaAday.csh]; alert;  if anything is out of sync (compares hgcentralbeta to hgcentral for db of the day) </td>
<td>'''Retired (See #23998)''' receive output from [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#Cronjob:_Results_from_checkMetaAday.csh checkMetaAday.csh]; alert;  if anything is out of sync (compares hgcentralbeta to hgcentral for db of the day) </td>
<td>daily</td>
<td>daily</td>
<td>Cath</td>
<td>Cath</td>
<td>Cath, Matt</td>
<td>Lou</td>
<td>Matt</td>
<td>Retired</td>
<td>Matt</td>
<td></td>
</tr>
</tr>


<tr>
<tr>
<td>receive output from [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#Cronjob:_Results_from_realTime.csh_.28previously_known_as_gbLoaded.29 realTime.csh] (previously gbLoaded) cron job and alert braney if anything seems very outdated</td>
<td> Receive output of checkSyncAndMetaDataAll.py which is a combination of the two previous cronjobs syncAday.csh and checkMetaAday.csh. This cron refines the hgcentralbeta/hgcentral sync output as well as the hgwbeta/hgw1/hgw2 table syncs, and runs on all databases. See [http://redmine.soe.ucsc.edu/issues/23998 #23998] for more info. </td>
<td>daily, problems every 3-6 months</td>
<td>weekly</td>
<td>Jairo</td>
<td>Cath, Luvina</td>
<td>Steve, Jonathan</td>
<td>Steve, Jonathan</td>
<td></td>
<td></td>
<td>Lou, Conner</td>
<td>Lou</td>
</tr>
</tr>


<tr>
<tr>
<td>receive output from showProcesslist.csh</td>
<td>receive output from [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#Cronjob:_Results_from_realTime.csh_.28previously_known_as_gbLoaded.29 realTime.csh] (previously gbLoaded) cron job and alert braney if anything seems very outdated <br> No output for xenTro3, fr1, fr2, fr3, eboVir3, dm2</td>
<td>daily, no action</td>
<td>daily, problems every 3-6 months</td>
<td>Cath, Luvina</td>
<td>Jairo</td>
<td>Jairo, Matt </td>
<td>Jairo</td>
<td>Steve, Jonathan</td>
<td>Daniel</td>
<td>Steve, Jonathan</td>
<td></td>
</tr>
</tr>


<tr>
<tr>
<td>get output from checkTableStatus.csh on Mondays (this is a report on whether the genbank table dumps are up-to-date...if we could get rid of our dependence on those, we could get rid of this cron job)</td>
<td>get output from checkTableStatus.csh on Mondays (this is a report on whether the genbank table dumps are up-to-date...{Read how this cronjob is important for updateTimes.csh [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#Cronjob:_Results_from_checkTableStatus.csh_.22_TABLE_STATUS_dump.22_emails here]</td>
<td>weekly</td>
<td>weekly</td>
<td>Cath</td>
<td>Cath</td>
<td>Brian, Jonathan</td>
<td>Lou</td>
<td>Steve, Jonathan</td>
<td>Gerardo</td>
<td>Steve, Jonathan</td>
<td></td>
</tr>
</tr>


Line 91: Line 75:
<td>get an email from backupCentral.csh each month that keeps a dump of hgcentral tables (there's a redmine ticket on whether we should change this one: http://redmine.soe.ucsc.edu/issues/7834)</td>
<td>get an email from backupCentral.csh each month that keeps a dump of hgcentral tables (there's a redmine ticket on whether we should change this one: http://redmine.soe.ucsc.edu/issues/7834)</td>
<td>monthly</td>
<td>monthly</td>
<td>Luvina</td>
<td>Brian</td>
<td>Matt, Brian </td>
<td>Daniel</td>
<td>Steve, Jonathan</td>
<td>Gerardo</td>
<td>Steve, Jonathan</td>
<td></td>
</tr>
</tr>


Line 101: Line 83:
<td>receive an email on additions and subtractions to files in /gbdb on hgnfs1</td>
<td>receive an email on additions and subtractions to files in /gbdb on hgnfs1</td>
<td>daily</td>
<td>daily</td>
<td>Matt</td>
<td>Brian</td>
<td>ChrisL, Luvina </td>
<td>Conner</td>
<td>Steve, Jonathan</td>
<td>Lou</td>
<td>Steve, Jonathan</td>
<td></td>
</tr>
</tr>


<tr>
<tr>
<td>receive an email from cron if qateam DailySandboxUpdate fails</td>
<td>receive an email from cron if qateam DailySandboxUpdate fails, '''note:''' emails defined in /cluster/home/qateam/bin/scripts/DailySandboxUpdate</td>
<td>daily</td>
<td>daily</td>
<td>Matt, Brian</td>
<td>ChrisL, Brian</td>
<td>Matt, Brian</td>
<td>Jairo Brian Lou</td>
<td>Steve, Jonathan</td>
<td>Jairo, Brian, Lou</td>
<td>Steve, Jonathan</td>
<td></td>
</tr>
</tr>


<tr>
<tr>
<td>email request to build [http://genomewiki.ucsc.edu/genecats/index.php/Public_Hub_QA#Push_to_the_RR doPublicCrawl] public.ix and public.ixx hub search files and push them.</td>
<td>email request to check [https://en.wikipedia.org/wiki/UCSC_Genome_Browser#Genomes Genomes section of the Wikipedia Page] and make other changes or updates if needed.</td>
<td>runs quarterly</td>
<td>runs quarterly</td>
<td>Jairo</td>
<td>ChrisL</td>
<td></td>
<td></td>
<td>Bob,Conner</td>
<td>Bob,Gerardo</td>
</tr>
<tr>
<td>email request to check trackDbLibrary.shtml and trackDbHub and run hubCheck -settings -level=required</td>
<td>runs monthly</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>BrianL</td>
</tr>
<tr>
<td>'''Retired (See #27425)''' email request to check [http://genomewiki.ucsc.edu/genecats/index.php/Public_Hub_QA#Push_to_the_RR doPublicCrawl] public.ix and public.ixx hub search files and push them.</td>
<td>runs quarterly</td>
<td>Jairo</td>
<td>Daniel</td>
<td>Gerardo,Brian</td>
</tr>
</tr>


<tr>
<tr>
<td> increment databaseAday, /cluster/bin/scripts/databaseAday.csh </td>
<td> increment databaseAday, /cluster/bin/scripts/databaseAday.csh </td>
<td>problems only</td>
<td>problems only</td>
<td>ChrisL, Matt, Brian</td>
<td>ChrisL, Cath, Brian</td>
<td>Brian, Jonathan</td>
<td>Jairo, Brian, Lou</td>
<td>Steve, Jonathan</td>
<td>Jairo, Brian, Lou</td>
<td>Steve, Jonathan</td>
<td></td>
</tr>
</tr>


Line 141: Line 132:
<td>checkHgFindSpec -checkTermRegex,  /cluster/bin/scripts/runCheckHgFindSpec go</td>
<td>checkHgFindSpec -checkTermRegex,  /cluster/bin/scripts/runCheckHgFindSpec go</td>
<td>problems only</td>
<td>problems only</td>
<td>ChrisL, Matt, Brian</td>
<td>ChrisL, Brian, Cath</td>
<td>Brian, Jonathan</td>
<td>Jairo, Brian, Lou</td>
<td>Steve, Jonathan</td>
<td>Jairo, Brian, Lou</td>
<td>Steve, Jonathan</td>
<td></td>
</tr>
</tr>


Line 151: Line 140:
<td>check to see if there are new files to go with the CGI push /cluster/home/qateam/bin/scripts/findCgiFiles.csh go</td>
<td>check to see if there are new files to go with the CGI push /cluster/home/qateam/bin/scripts/findCgiFiles.csh go</td>
<td>problems only</td>
<td>problems only</td>
<td>ChrisL, Matt, Brian</td>
<td>ChrisL, Cath, Brian</td>
<td>Brian, Jonathan</td>
<td>Jairo, Brian, Lou</td>
<td>Steve, Jonathan</td>
<td>Jairo, Brian, Lou</td>
<td>Steve, Jonathan</td>
<td></td>
</tr>
</tr>




<tr>
<tr>
<td>get output from checkGenomeMysql</td>
<td>get output from checkGenomeMysql (discovers corrupt tables)</td>
<td>daily</td>
<td>daily</td>
<td>Luvina</td>
<td>Brian</td>
<td>ChrisL, Matt</td>
<td>Brian, Daniel</td>
<td>Steve, Jonathan</td>
<td>Daniel, Gerardo</td>
<td>Steve, Jonathan</td>
<td></td>
</tr>
</tr>


Line 173: Line 158:
<td>runs weekly</td>
<td>runs weekly</td>
<td>Jairo, Cath</td>
<td>Jairo, Cath</td>
<td>Brian, Cath, Jonathan</td>
<td>Jairo,Conner,Daniel</td>
<td>Brian, Steve, Jonathan</td>
<td>Lou,Gerardo,Daniel</td>
<td></td>
<td></td>
</tr>
 
<tr>
<td>Check for unigene update for human SAGE track</td>
<td>daily</td>
<td>Cath, Braney</td>
<td>Luvina, Braney</td>
<td>Steve, Braney</td>
<td></td>
<td></td>
</tr>
</tr>


Line 195: Line 168:
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td></td>
<td></td>
</tr>
</tr>


Line 203: Line 174:
<td>twice yearly</td>
<td>twice yearly</td>
<td>ChrisL</td>
<td>ChrisL</td>
<td>Cath, Galt</td>
<td>Conner</td>
<td>Steve, Galt</td>
<td>Brian</td>
<td></td>
<td></td>
</tr>
</tr>


Line 215: Line 184:
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td></td>
<td></td>
</tr>
</tr>


Line 225: Line 192:
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td></td>
<td></td>
</tr>
</tr>


<tr>
<tr>
<td>Check to see what kind of browsers and o/s people are using to access the public website</td>
<td>sends a reminder to check the list of folks who have mirror sites. This list is created by a script of Hiram's and located here: http://genome-test.gi.ucsc.edu/admin/tools/useCount.html. The information comes from the phoneHome program in the CGIs (see #22593 for login/password and discussion about how an initial 'registration.txt' file is created then moved to trash, where presence of that file prevents the request being repeated so only an initial phone call is made). Sometimes companies set up mirrors, which is not OK.</td>
<td>weekly</td>
<td>monthly</td>
<td>Ann, Jairo, ChrisL</td>
<td>Ann [was: Matt, Brian]</td>
<td>Ann, ChrisL, Brian</td>
<td>Ann,Lou,Hiram</td>
<td>Ann, Steve</td>
<td>Ann,Jairo,Hiram</td>
<td></td>
<td></td>
</tr>
</tr>


<tr>
<tr>
<td>sends a reminder to check the list of folks who have mirror sites. This list is created by a script of Hiram's and located here: http://genome-test.cse.ucsc.edu/admin/tools/useCount.html. The information comes from the phoneHome program in the CGIs. Sometimes companies set up mirrors, which is not OK.</td>
<td>Monthly reminder to check for stale New pennantIcon tags</td>
<td>monthly</td>
<td>monthly</td>
<td>Matt, Luvina</td>
<td>Brian, Luvina</td>
<td>Matt</td>
<td>Pauline</td>
<td></td>
<td></td>
<td>Lou</td>
<td>Lou</td>
</tr>
</tr>


<tr>
<tr>
<td>run /cluster/bin/scripts/sessionStats.csh</td>
<td>run /cluster/bin/scripts/sessionStats.csh NOTE: this is separate to each machine (genome-asia & genome-euro) thus cronjobs are specific to each machine and jonathan owns asia</td>
<td>monthly</td>
<td>monthly</td>
<td>Angie, Brian, Bob</td>
<td>Angie, Brian, Bob</td>
<td>Angie, Brian, Bob</td>
<td>Angie, Brian, Bob</td>
<td>Angie, Brian, Bob, Steve</td>
<td>Angie, Lou, Bob</td>
<td></td>
<td></td>
</tr>
</tr>


Line 262: Line 222:
<td>ask Galt/Jim about blat updates</td>
<td>ask Galt/Jim about blat updates</td>
<td>3x per year</td>
<td>3x per year</td>
<td>ChrisV,Cath</td>
<td>ChrisL,Cath</td>
<td>Cath, Brian </td>
<td>Daniel, Conner</td>
<td>Steve, Jonathan</td>
<td>Daniel, Gerardo</td>
<td>Pauline, Steve, Jonathan</td>
<td></td>
</tr>
</tr>


Line 273: Line 231:
<td>daily</td>
<td>daily</td>
<td>Jairo</td>
<td>Jairo</td>
<td>Cath</td>
<td>Conner</td>
<td>Ann</td>
<td>Daniel</td>
<td>Ann</td>
<td></td>
</tr>
</tr>


<tr>
<tr>
<td>check that blat servers are running ok</td>
<td>check that blat servers are running ok [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#check_that_blat_servers_are_running_ok link to error log checking]</td>
<td>ongoing</td>
<td>ongoing</td>
<td>Ann, Brian, Donna, Galt, Luvina, Matt, Erich, ChrisL</td>
<td>Ann, Brian, Galt, Matt, Erich, ChrisL, Jairo, Cath</td>
<td>Ann, Brian, Donna, Galt, Luvina, Matt, Gary,</td>
<td>Ann, Galt, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo</td>
<td>Ann, Brian, Donna, Galt, Luvina, Steve, Gary</td>
<td>Clay, Galt, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo</td>
<td></td>
<td></td>
</tr>
</tr>


Line 292: Line 246:
<td>receive output from hgTracksRandom cron; alert admins to problems</td>
<td>receive output from hgTracksRandom cron; alert admins to problems</td>
<td>as needed</td>
<td>as needed</td>
<td>many: cath, chmalee, kuhn, brianlee, braney, luvina, gary, ann,, jcasper, matt, jairo, chrisv</td>
<td>many: cath, chmalee, kuhn, brianlee, braney, ann,, jcasper, matt, jairo</td>
<td>many: cath, chmalee, kuhn, brianlee, braney, luvina, gary, ann,, jcasper, matt</td>
<td>Bob,Ann, Galt, Braney, Jonathan, Christopher, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo</td>
<td>many: kuhn, katrina, brianlee, braney, luvina, gary, ann, steve, jcasper, matt</td>
<td>Clay, Bob, Braney, Jonathan, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo</td>
<td>many: kuhn, pauline, katrina, brianlee, braney, luvina, gary, ann, steve, jcasper, matt</td>
<td></td>
</tr>
</tr>


Line 302: Line 254:
<td>check that beta builds alright</td>
<td>check that beta builds alright</td>
<td>3x week</td>
<td>3x week</td>
<td>Luvina, Matt</td>
<td>Brian, ChrisL</td>
<td>Brian, ChrisL,Braney</td>
<td>Brian, Braney, Jairo, Lou</td>
<td>Steve, Jonathan, Braney</td>
<td>Brian, Braney, Jairo, Lou</td>
<td></td>
<td></td>
</tr>
</tr>


<tr>
<tr>
<td>hubPublicCheck complains when the hubPublic table doesn't have the same strings as the actual hub.txt file. '' (NOTE: To run only on Beta hubs requires these declarations: '''export HGDB_CONF=$HOME/.hg.conf.beta'''  and '''SHELL=/bin/sh''')''.</td>
<td>'''Retired (See #27779)''' hubPublicCheck complains when the hubPublic table doesn't have the same strings as the actual hub.txt file. '' (NOTE: To run only on Beta hubs requires these declarations: '''export HGDB_CONF=$HOME/.hg.conf.beta'''  and '''SHELL=/bin/sh''')''.</td>
<td>runs daily, complains rarely</td>
<td>runs daily, complains moderately</td>
<td>Matt</td>
<td>ChrisL</td>
<td>Matt</td>
<td>Daniel, Braney, Christopher</td>
<td>Brian</td>
<td>Daniel, Braney, Christopher</td>
<td>Brian</td>
<td></td>
</tr>
</tr>


Line 322: Line 270:
<td>Curl of http://genomewiki.ucsc.edu/index.php/Assembly_Hubs to test Public Wiki is accessible, output only if no response.</td>
<td>Curl of http://genomewiki.ucsc.edu/index.php/Assembly_Hubs to test Public Wiki is accessible, output only if no response.</td>
<td>hourly</td>
<td>hourly</td>
<td>ChrisV</td>
<td>Brian</td>
<td>Brian</td>
<td>Brian</td>
<td>Conner</td>
<td>Daniel</td>
</tr>
<tr>
<td>qaAutoTrack -script that does some auto-QA on the autopushed tracks.</td>
<td>daily</td>
<td>Jario</td>
<td>Jairo</td>
<td>Jairo</td>
</tr>
<tr>
<td>checkAllStaticLinks.csh script that checks static pages.</td>
<td>weekly</td>
<td></td>
<td></td>
<td>Daniel,Conner,Brian</td>
<td>Daniel,Gerardo,Brian</td>
</tr>
<tr>
<td>assemblyStatsCron.py pulls and reports GB usage stats over the last month from the RR+mirrors.</td>
<td>monthly</td>
<td></td>
<td></td>
<td>QA Team</td>
<td>QA Team</td>
</tr>
</tr>


<tr>
<tr>
<td>Cronjob reports checkUrlsInTable hg38 wgEncodeCell  and cv.ra (likely infrequent results) </td>   
<td>HTTPs expiration check runs genecats/qa/testTools/checkHTTPSdir/checkHTTPS to report any expiring certificates</td>
<td>weekly</td>
<td>weekly on Tuesday</td>
<td>ChrisV</td>
<td></td>
<td>ChrisL</td>
<td>Brian</td>
<td>Brian</td>
<td></td>
<td></td>
<td>QA Team, cluster admin</td>
</tr>
</tr>


<tr>
<tr>
<td>qaAutoTrack -script that does some auto-QA on the autopushed tracks.</td>
<td>Hub Public Coordinator maintains error-free Public Hubs hubs and keeps the hubs updated. Fully automated to update labels/titles, the search index, and contact hub provider when hub is down. Monitor the three cronjobs: hubPublicAutoUpdate, hubSearchUpdate, and hubPublicMail. More details can be found [http://genomewiki.ucsc.edu/genecats/index.php/Public_Hub_QA#Hub_Public_Coordinator_Role here].</td>
<td>daily</td>
<td>daily</td>
<td>Matt</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>Daniel, Gerardo</td>
</tr>
</tr>


<tr>
<tr>
<td colspan="7" style="background-color:gray"><b>from build cron</b></td>
<td colspan="7" style="background-color:gray"><b>from build cron </b></td>
</tr>
</tr>




<tr>
<tr>
<td>receive cron output for tableDescriptions build errors</td>
<td>receive cron output for tableDescriptions build errors, AutoPush completed</td>
<td>rare</td>
<td>rare</td>
<td>Matt</td>
<td>Cath</td>
<td>Luvina</td>
<td>QA Team</td>
<td>Steve, Luvina</td>
<td>QA Team</td>
<td>Pauline, Matt</td>
<td></td>
</tr>
</tr>


Line 375: Line 338:
<td>Galt</td>
<td>Galt</td>
<td>Galt</td>
<td>Galt</td>
<td>Galt</td>
<td></td>
</tr>
<tr>
<td>back up the qapushq</td>
<td>every 30 minutes M-F</td>
<td>Galt</td>
<td>Galt</td>
<td>Galt</td>
<td>Galt</td>
<td></td>
</tr>
</tr>


Line 397: Line 348:
<td>weekly</td>
<td>weekly</td>
<td>Cath</td>
<td>Cath</td>
<td>Luvina</td>
<td>Lou</td>
<td>Luvina</td>
<td>Daniel</td>
<td>Luvina</td>
<td></td>
</tr>
</tr>


<tr>
<tr>
<td>Monitor & report UCSC Genome Browser outages. [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#SLA_Monitoring_.26_Reporting See wiki procedures]</td>
<td>Monitor & report UCSC Genome Browser outages (SLA). [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#SLA_Monitoring_.26_Reporting See wiki procedures]</td>
<td>hopefully, not much!</td>
<td>hopefully, not much!</td>
<td>Alexa/ChrisV</td>
<td>Cath</td>
<td>Cath, Luvina</td>
<td>Conner</td>
<td>Steve, Jonathan</td>
<td>Gerardo</td>
<td>Steve, Jonathan</td>
<td></td>
</tr>
</tr>


Line 418: Line 365:
<td>Brian</td>
<td>Brian</td>
<td>Brian</td>
<td>Brian</td>
<td>Jonathan</td>
<td>Brian</td>
<td>Pauline, Jonathan</td>
<td></td>
</tr>
</tr>


Line 426: Line 371:
<td>update UCSC Entrez LinkOut when necessary. Entrez LinkOut sends requests for changes (and statistics) to the browser-qa email address</td>
<td>update UCSC Entrez LinkOut when necessary. Entrez LinkOut sends requests for changes (and statistics) to the browser-qa email address</td>
<td>rarely</td>
<td>rarely</td>
<td>ChrisL</td>
<td>ChrisL,Matt</td>
<td>Matt</td>
<td>Daniel</td>
<td>Matt</td>
<td>Gerardo</td>
<td>Bob</td>
<td></td>
</tr>
</tr>


<tr>
<tr>
<td>Check genome-preview machine is still running</td>   
<td>Check genome-preview machine is still running well</td>   
<td>biweekly</td>
<td>weekly</td>
<td>Jairo</td>
<td>Jairo</td>
<td>Brian</td>
<td>Conner</td>
<td>Brian</td>
<td>Daniel</td>
<td>Brian</td>
<td></td>
</tr>
</tr>


Line 446: Line 387:
<td>Updated Facebook with Announcements</td>     
<td>Updated Facebook with Announcements</td>     
<td>As needed</td>
<td>As needed</td>
<td>Alexa/Team</td>
<td>Team</td>
<td>Brian/Alexa/Team</td>
<td>Team</td>
<td>Brian</td>
<td>Team</td>
<td>Brian</td>
<td></td>
</tr>
</tr>


Line 456: Line 395:
<td>Update Twitter with Announcements</td>       
<td>Update Twitter with Announcements</td>       
<td>As needed</td>
<td>As needed</td>
<td>Alexa/Team</td>
<td>Team</td>
<td>Brian/Alexa/Team</td>
<td>Team</td>
<td>Ann</td>
<td>Team</td>
<td></td>
<td></td>
</tr>
</tr>


Line 470: Line 407:
<td>runs daily, complains bi-weekly</td>
<td>runs daily, complains bi-weekly</td>
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>ChrisL</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
<td></td>
</tr>
</tr>


Line 480: Line 415:
<td>runs daily, complains rarely</td>
<td>runs daily, complains rarely</td>
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>ChrisL</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
<td></td>
</tr>
</tr>


Line 490: Line 423:
<td>runs daily, complains monthly</td>
<td>runs daily, complains monthly</td>
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>ChrisL</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
<td></td>
</tr>
</tr>


Line 500: Line 431:
<td>runs daily, complains monthly</td>
<td>runs daily, complains monthly</td>
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>ChrisL</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
<td></td>
</tr>
</tr>


Line 510: Line 439:
<td>runs weekly, complains quarterly</td>
<td>runs weekly, complains quarterly</td>
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>ChrisL</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
<td></td>
</tr>
</tr>


Line 520: Line 447:
<td>runs weekly, complains quarterly</td>
<td>runs weekly, complains quarterly</td>
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>ChrisL</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
<td></td>
</tr>
</tr>


Line 533: Line 458:
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>nobody</td>
<td>jcasper</td>
<td>nobody</td>
<td></td>
</tr>
</tr>


Line 543: Line 466:
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>nobody</td>
<td>jcasper</td>
<td>nobody</td>
<td></td>
</tr>
</tr>



Latest revision as of 20:59, 25 March 2022

Please add to the documentation page, Monitoring Tasks Notes to add procedures and notes for specific jobs, then link back to the job in the list below.

task frequency Until Aug '18 Until Sep '19 Now
from hgwdev qateam cron
Respond to items on the broken links report: Check to make sure that broken links reported are actually broken. If so, fix the link if it is clear what the new link should be and push to the RR. If not, contact the appropriate person at the web site in question to find out what the new link should be. daily Jairo Jairo Jairo
Respond to the trackDbGlobal report: Make public and request a push of trackDb and friends for any assemblies that show up as changed in the trackDbGlobal report. weekly Jairo Conner Lou
Retired (See #23998) receive output from syncAday.csh; alert; if anything is out of sync (checks that tables are in sync between beta and RR for database of the day) daily Cath Lou Retired
Retired (See #23998) receive output from checkMetaAday.csh; alert; if anything is out of sync (compares hgcentralbeta to hgcentral for db of the day) daily Cath Lou Retired
Receive output of checkSyncAndMetaDataAll.py which is a combination of the two previous cronjobs syncAday.csh and checkMetaAday.csh. This cron refines the hgcentralbeta/hgcentral sync output as well as the hgwbeta/hgw1/hgw2 table syncs, and runs on all databases. See #23998 for more info. weekly Lou, Conner Lou
receive output from realTime.csh (previously gbLoaded) cron job and alert braney if anything seems very outdated
No output for xenTro3, fr1, fr2, fr3, eboVir3, dm2
daily, problems every 3-6 months Jairo Jairo Daniel
get output from checkTableStatus.csh on Mondays (this is a report on whether the genbank table dumps are up-to-date...{Read how this cronjob is important for updateTimes.csh here weekly Cath Lou Gerardo
get an email from backupCentral.csh each month that keeps a dump of hgcentral tables (there's a redmine ticket on whether we should change this one: http://redmine.soe.ucsc.edu/issues/7834) monthly Brian Daniel Gerardo
receive an email on additions and subtractions to files in /gbdb on hgnfs1 daily Brian Conner Lou
receive an email from cron if qateam DailySandboxUpdate fails, note: emails defined in /cluster/home/qateam/bin/scripts/DailySandboxUpdate daily ChrisL, Brian Jairo Brian Lou Jairo, Brian, Lou
email request to check Genomes section of the Wikipedia Page and make other changes or updates if needed. runs quarterly Bob,Conner Bob,Gerardo
email request to check trackDbLibrary.shtml and trackDbHub and run hubCheck -settings -level=required runs monthly BrianL
Retired (See #27425) email request to check doPublicCrawl public.ix and public.ixx hub search files and push them. runs quarterly Jairo Daniel Gerardo,Brian
increment databaseAday, /cluster/bin/scripts/databaseAday.csh problems only ChrisL, Cath, Brian Jairo, Brian, Lou Jairo, Brian, Lou
checkHgFindSpec -checkTermRegex, /cluster/bin/scripts/runCheckHgFindSpec go problems only ChrisL, Brian, Cath Jairo, Brian, Lou Jairo, Brian, Lou
check to see if there are new files to go with the CGI push /cluster/home/qateam/bin/scripts/findCgiFiles.csh go problems only ChrisL, Cath, Brian Jairo, Brian, Lou Jairo, Brian, Lou
get output from checkGenomeMysql (discovers corrupt tables) daily Brian Brian, Daniel Daniel, Gerardo
[checkGbibMd5.sh] compares the md5sum of gbibBeta.zip on dev with the sum for gbib.zip on genome-store, complaining if they don't match. runs weekly Jairo, Cath Jairo,Conner,Daniel Lou,Gerardo,Daniel
look for broken symlinks in /gbdb: daily Browser Staff Browser Staff Browser Staff
look for broken symlinks in in (very large) visiGene directory twice yearly ChrisL Conner Brian
look for broken symlinks in htdocs-hgdownload/goldenpath daily Browser Staff Browser Staff Browser Staff
Check the source tree for duplicate case-independent filenames. These wreck havoc with git. daily Browser Staff Browser Staff Browser Staff
sends a reminder to check the list of folks who have mirror sites. This list is created by a script of Hiram's and located here: http://genome-test.gi.ucsc.edu/admin/tools/useCount.html. The information comes from the phoneHome program in the CGIs (see #22593 for login/password and discussion about how an initial 'registration.txt' file is created then moved to trash, where presence of that file prevents the request being repeated so only an initial phone call is made). Sometimes companies set up mirrors, which is not OK. monthly Ann [was: Matt, Brian] Ann,Lou,Hiram Ann,Jairo,Hiram
Monthly reminder to check for stale New pennantIcon tags monthly Lou Lou
run /cluster/bin/scripts/sessionStats.csh NOTE: this is separate to each machine (genome-asia & genome-euro) thus cronjobs are specific to each machine and jonathan owns asia monthly Angie, Brian, Bob Angie, Brian, Bob Angie, Lou, Bob
ask Galt/Jim about blat updates 3x per year ChrisL,Cath Daniel, Conner Daniel, Gerardo
check that the ga4gh beacon server is running. Email Max if there is an issue. daily Jairo Conner Daniel
check that blat servers are running ok link to error log checking ongoing Ann, Brian, Galt, Matt, Erich, ChrisL, Jairo, Cath Ann, Galt, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo Clay, Galt, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo
receive output from hgTracksRandom cron; alert admins to problems as needed many: cath, chmalee, kuhn, brianlee, braney, ann,, jcasper, matt, jairo Bob,Ann, Galt, Braney, Jonathan, Christopher, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo Clay, Bob, Braney, Jonathan, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo
check that beta builds alright 3x week Brian, ChrisL Brian, Braney, Jairo, Lou Brian, Braney, Jairo, Lou
Retired (See #27779) hubPublicCheck complains when the hubPublic table doesn't have the same strings as the actual hub.txt file. (NOTE: To run only on Beta hubs requires these declarations: export HGDB_CONF=$HOME/.hg.conf.beta and SHELL=/bin/sh). runs daily, complains moderately ChrisL Daniel, Braney, Christopher Daniel, Braney, Christopher
Curl of http://genomewiki.ucsc.edu/index.php/Assembly_Hubs to test Public Wiki is accessible, output only if no response. hourly Brian Conner Daniel
qaAutoTrack -script that does some auto-QA on the autopushed tracks. daily Jario Jairo Jairo
checkAllStaticLinks.csh script that checks static pages. weekly Daniel,Conner,Brian Daniel,Gerardo,Brian
assemblyStatsCron.py pulls and reports GB usage stats over the last month from the RR+mirrors. monthly QA Team QA Team
HTTPs expiration check runs genecats/qa/testTools/checkHTTPSdir/checkHTTPS to report any expiring certificates weekly on Tuesday QA Team, cluster admin
Hub Public Coordinator maintains error-free Public Hubs hubs and keeps the hubs updated. Fully automated to update labels/titles, the search index, and contact hub provider when hub is down. Monitor the three cronjobs: hubPublicAutoUpdate, hubSearchUpdate, and hubPublicMail. More details can be found here. daily Daniel, Gerardo
from build cron
receive cron output for tableDescriptions build errors, AutoPush completed rare Cath QA Team QA Team
from galt's cron
run ReleaseLog script (this requires a copy of the kent repository) daily Galt Galt Galt
non-cron jobs
Check for MLQs that may have slipped through the cracks with Google Groups to check for unanswered Q's. Check to make sure that the last response for every ticket came from one of us. If not, look it up in redmine to ensure that A) there is indeed a redmine ticket for it, and B) it has been responded to. If not, notify the appropriate MLM with as much information about the MLQ as possible. weekly Cath Lou Daniel
Monitor & report UCSC Genome Browser outages (SLA). See wiki procedures hopefully, not much! Cath Conner Gerardo
update crontab in genecats/qa/crontabs/*.crontab (and log in as qateam and restart the qateam crontab with updates) as needed Brian Brian Brian
update UCSC Entrez LinkOut when necessary. Entrez LinkOut sends requests for changes (and statistics) to the browser-qa email address rarely ChrisL,Matt Daniel Gerardo
Check genome-preview machine is still running well weekly Jairo Conner Daniel
Updated Facebook with Announcements As needed Team Team Team
Update Twitter with Announcements As needed Team Team Team
otto cron jobs (/hive/data/outside/otto)
omimWrapper.sh checks for new OMIM data and downloads and loads it into hgwdev if so runs daily, complains bi-weekly jcasper ChrisL ChrisL
omimUploadWrapper.sh uploads the OMIM tables to OMIM runs daily, complains rarely jcasper ChrisL ChrisL
decipherWrapper.sh checks for new DECIPHER data and downloads and loads it into hgwdev if so runs daily, complains monthly jcasper ChrisL ChrisL
iscaWrapper.sh checks for new ISCA data and downloads and loads it into hgwdev if so runs daily, complains monthly jcasper ChrisL ChrisL
gwasWrapper.sh checks for new GWAS data and downloads and loads it into hgwdev if so runs weekly, complains quarterly jcasper ChrisL ChrisL
geneReviewsWrapper.sh checks for new GeneReviews data and downloads and loads it into hgwdev if so runs weekly, complains quarterly jcasper ChrisL ChrisL
otto cron jobs (/hive/data/outside/otto)
~jcasper/bin/scripts/updateHgConf.sh - checks for changes to hg.conf files and checks them in to the kent tree runs weekly jcasper jcasper jcasper
~jcasper/bin/backup_hgconfprivate.sh - checks most recent versions of hg.conf.private files into a private repository (~jcasper/hgconf_private/) runs twice a week jcasper jcasper jcasper