Emergency Backup BLAT Servers
Overview
In case certain blat servers go down, there is a a backup blat server ("blatx") that has ports pre-configured to host the "top 5" blatted assemblies (hg38, hg19, hg18, mm10, mm9). If the current blat server(s) hosting any of these 5 assemblies goes down, manually switch to the specified fail-over server in the list below by updating the hgcentral.blatServers table. A common solution is for the admins to restart the blat server that has gone down.
List of current backup "blatx" server ports
- hg38
- host: blatx
- trans port: 17786 isTrans=1, canPcr=0
- untrans port: 17787 isTrans=0, canPcr=1
- hg19
- host: blatx
- trans port: 17778 isTrans=1, canPcr=0
- untrans port: 17779 isTrans=0, canPcr=1
- hg18
- host: blatx
- trans port: 17782 isTrans=1, canPcr=0
- untrans port: 17783 isTrans=0, canPcr=1
- mm10
- host: blatx
- trans port: 17780 isTrans=1, canPcr=0
- untrans port: 17781 isTrans=0, canPcr=1
- mm9
- host: blatx
- trans port: 17784 isTrans=1, canPcr=0
- untrans port: 17785 isTrans=0, canPcr=1
"mm10 blat is down!" Example for QA to follow when moving a down-blat-server assembly to emergency blatx
How is mm10 blat currently configured?
hgsql -h genome-centdb -e "select * from blatServers where db='mm10'" hgcentral +------+--------+-------+---------+--------+ | db | host | port | isTrans | canPcr | +------+--------+-------+---------+--------+ | mm10 | blat1d | 17779 | 0 | 1 | | mm10 | blat1d | 17778 | 1 | 0 | +------+--------+-------+---------+--------+
Update the blatServers table to point mm10 to the backup "blatx" server:
hgsql -h genome-centdb -e "update blatServers set host='blatx' where db='mm10'" hgcentral hgsql -h genome-centdb -e "update blatServers set port='17780' where db='mm10' and isTrans='1'" hgcentral hgsql -h genome-centdb -e "update blatServers set port='17781' where db='mm10' and isTrans='0'" hgcentral
Check the change:
hgsql -h genome-centdb -e "select * from blatServers where db='mm10'" hgcentral +------+-------+-------+---------+--------+ | db | host | port | isTrans | canPcr | +------+-------+-------+---------+--------+ | mm10 | blatx | 17781 | 0 | 1 | | mm10 | blatx | 17780 | 1 | 0 | +------+-------+-------+---------+--------+
Mm10 blat should now be fixed (is on the emergency server), go to the RR and do a blat test to be sure.
When the failed blat server has been fixed, roll back to it. In this example, move mm10 from blatx back to blat1d:
hgsql -h genome-centdb -e "update blatServers set host='blat1d' where db='mm10'" hgcentral hgsql -h genome-centdb -e "update blatServers set port='17778' where db='mm10' and isTrans='1'" hgcentral hgsql -h genome-centdb -e "update blatServers set port='17779' where db='mm10' and isTrans='0'" hgcentral
mm10 blat should now be fixed on the original blat server, be sure to go to the RR and do another blat test.
Test with gfServer
A quick command-line test example:
gfClient blatx.soe.ucsc.edu 17845 /gbdb/strPur4/ test.fa dnaTestOut.psl
About gfServer logs
As of 1/25/18: Haifang says: I just upgraded gfServer on blatx and added two new options, -debugLog -seqLog.
>ssh qateam@blatx.soe.ucsc.edu
Chris Lee says:
debugLog and seqLog are options on gfServer, not gfClient. You can see the additional lines in the gfServer logs on the blatx machine:
[qateam@blatx.soe.ucsc.edu /home/qateam] tail -15 /scratch/strPur4/gfServer.log >query gcccagctaaaaccaatcaccttcaagggataaaggttcatccaaatatg taaaaaccgactgaagacagcttagttagaaacgtgtgtgggatcaacta tggcgttgccgttgactagatctagatccctacttttttttgttctcaat acggtatgtagcttggtatccgcctctgacaatgatcccagccctagctt ctgatcatgatctggcacaaaatcaatgtaggtctagatctagatctatc cgacttcgtaaacaacatagcctatttctttatcgttgaactatgagttt atgactaaaacattcttcttcttcttcttctt 2018/01/25 09:58:52: debug: 1711519 101 clumps, 36593 hits 2018/01/25 09:59:18: info: gfServer version 36x2 on host blat4d, port 17845 connection from 132.249.245.79 2018/01/25 09:59:18: warn: Zero sized query 2018/01/25 10:00:12: info: gfServer version 36x2 on host blat4d, port 17845 connection from 132.249.245.79 2018/01/25 10:00:12: debug: 0ddf270562684f29pcr cctacattgattttgtgccagatcatgatcagaagctagggctgggatca gcccagctaaaaccaatcaccttcaagggataaaggttcatccaaat 4000 2018/01/25 10:00:12: debug: 1791616 PCR cctacattgattttgtgccagatcatgatcagaagctagggctgggatca gcccagctaaaaccaatcaccttcaagggataaaggttcatccaaat 63 clumps Compared to a gfServer running without those options, like on blat1a: [qateam@blat1a.cse.ucsc.edu /scratch/bosTau7] tail gfServer.log 2018/01/25 07:24:13: info: gfServer version 34x12 on host blat1a, port 17845 connection from 128.114.119.132 2018/01/25 08:00:02: info: gfServer version 34x12 on host blat1a, port 17845 connection from 132.249.245.79 .... # more of the same
