Athurva Gore:LabNotes/ExomePipeline

From ZhangLabWiki
Jump to navigation Jump to search

Human Genome Resequencing Pipeline[edit]

  • Pipeline can be found in:
    • genome-miner:
      • Human: /home/kunzhang/bin/variantCallerBwaGATK_05012015.pl
      • Mouse: /home/ajgore/AG_Ext12T/GATK_01022012/variantCallerBwaGATK-latest/variantCallerBwaGATK_mouse.pl
    • tscc-login:
      • Human: /home/k4zhang/bin/variantCallerBwaGATK_05012015.pl
      • Mouse: /projects/zhang-lab/softwares/GenomeAnalysisTK_01022012/variantCallerBwaGATK-latest/variantCallerBwaGATK_mouse.pl
    • This link will always point to the updated version
  • Updated on 07/11/2012

Changelog[edit]

05/01/2015[edit]

  • Updated GATK to v3.3, replace UnifiedGenotyper with HaplotypeCaller.
  • Updated resource bundle to v2.8
  • Replaced the human genome with human_g1k_v37_decoy.fasta

7/11/2012[edit]

  • Added support for Mouse genome using new GATK.
  • Still need to implement upgrade to GATK 1.5 from 1.2. Should be done by next week.

6/21/2012[edit]

  • Now using phase1 to store resources folder on Triton. This will stop many of the stalling issues, and greatly increase execution speed.

3/28/2012[edit]

  • Improved input checking. Software now verifies that input files exist.
  • Number of cores to use is now a variable in Perl script; 4 for genome-miner, 8 for triton

Requirements[edit]

  • All required files are currently on genome-miner and triton cluster. The below paths list genome-miner locations for human files.
  • NCBI Human Genome Reference Sequence, v37 (/home/ajgore/AG_Ext12T/GATK_01022012/resources_1.2/human_g1k_v37.fasta)
  • CCDS BED file (/home/ajgore/AG_Ext12T/GATK_01022012/resources_1.2/CCDS.20120105.UCSC.nochr.bed)
  • dbSNP in VCF format (/home/ajgore/AG_Ext12T/GATK_01022012/resources_1.2/dbsnp_132.b37.vcf)
  • GATK from 01022012 (/home/ajgore/AG_Ext12T/GATK_01022012/GenomeAnalysisTK-1.4-1-g1b6d528/)
  • BWA 0.5.8 or later (/home/kunzhang/softwares/bwa-latest)
  • Samtools 0.1.7 or later (/home/kunzhang/softwares/samtools-latest)
  • Picard-tools 1.38 or later (/home/kunzhang/softwares/picard-tools-latest)

Instructions[edit]

  • For mouse data, follow procedure as below, but append "_mouse.pl" to variantCallerBwaGATK command.

Generate data annotation file (.info file)[edit]

  • First, generate a text file containing your data information. This file should look like the following:
genome-miner
1
exome
SampleID	LibraryID	FlowCell	Lane	Barcode	File1	File2
CV-iF	AL-Exome-CV-iF-PP	HL022	s5	NA	/media/Syn_15T/Ext9T_Backup/SeqStore2009/091109_HL022/s_5_sequence.txt.gz	
CV-iF	AL-Exome-CV-iF-PP	HL022	s6	NA	/media/Syn_15T/Ext9T_Backup/SeqStore2009/091109_HL022/s_6_sequence.txt.gz	
CV-iF	AL-Exome-CV-iF-PP	HL022	s7	NA	/media/Syn_15T/Ext9T_Backup/SeqStore2009/091109_HL022/s_7_sequence.txt.gz	
CV-iF	AL-Exome-CV-iF-PP	HL025	s7	NA	/media/Syn_15T/Ext9T_Backup/SeqStore2009/091123_HL025/s_7_sequence.txt.gz	
CV-iF	AL-Exome-CV-iF-PP	HL026	s5	NA	/media/Syn_15T/Ext9T_Backup/SeqStore2009/091127_HL026/s_5_sequence.txt.gz	
CV-iF	AL-Exome-CV-iF-NG	HL053	s2	NA	/media/SeqStore2/100503_HL053/s_2_1_sequence.txt.gz	/media/SeqStore2/100503_HL053/s_2_2_sequence.txt.gz
  • This file is a tab-delimited text file containing 7 columns
  • The first three lines list script parameters.
    • Line 1: Please write either "genome-miner" or "tscc" based on the system of choice
    • Line 2: Please write "1" to remove clonal reads or "0" to keep clonal reads
    • Line 3: Please write "exome" to limit realignment and variant calling to coding regions, or "genome" to look at all regions
    • Line 4: MUST contain the header line listed above.
  • Each line represents one SE fastq file or a pair of PE fastq files.
  • Detailed description of each column:
    • Column 1: SampleID
      • This column should contain a short identifier describing your sample(s).
      • If multiple samples are provided, a multi-sample BAM file will be generated and multi-sample variant calling will be performed.
    • Column 2: LibraryID
      • This column should describe details of your library.
      • Generally, you should include the initials of the person making the library, the type (Exome, Genome, BSPP, etc), and the sampleID.
      • If you generated multiple sequencing libraries for your samples, please provide them unique identifiers here to facilitate proper removal of clonal reads
      • For example, here I have denoted that one library was made using padlock probes and one was made using NimbleGen Exome Capture.
    • Column 3: FlowCell
      • This column should contain the flow cell in which the fastq file was sequenced.
    • Column 4: Lane
      • This column should contain the letter "s" followed by the lane number.
    • Column 5: Barcode
      • This column should list the barcode used for the library.
      • Examples: "NA" for no barcode or "Indx7" for barcode #7, etc.
    • Column 6: First End FASTQ file
      • This file should list the full path to the first end FASTQ file.
      • For Single-End data, simply provide the path to the FASTQ file.
    • Column 7: Second End FASTQ File
      • This column should list the full path to the second end FASTQ file
      • For single-end data, this column should be left blank.
  • Save this file with a .info extension. I will call the file "prefix.info" in this example.
  • Upload this file to genome-miner to your folder of choice. For this example, we will use /home/ajgore/TestRun.

Run pipeline (Genome-Miner)[edit]

  • To run the pipeline, issue the following terminal command from the folder you uploaded the .info file to (in this case /home/ajgore/TestRun):
nohup /home/ajgore/AG_Ext12T/GATK_01022012/variantCallerBwaGATK-latest/variantCallerBwaGATK.pl prefix.info &> prefix.status &
  • You can monitor the status of the pipeline by looking at the "prefix.status" file using less:
less prefix.status
  • The output of the pipeline will be two files:
    • prefix.fixed.bam
    • prefix.snp.raw.vcf

Run pipeline (Triton)[edit]

  • On Triton, you will need to make a job file that submits the pipeline to the cluster.
    • A template job file can be found at: triton-login:/projects/zhang-lab/softwares/GenomeAnalysisTK_01022012/variantCallerBwaGATK_03282012/prefix.job
    • For this case, the job file (named prefix.job) should look like:
#!/bin/csh
#PBS -q small
#PBS -l walltime=24:00:00
#PBS -o variantCallerBwaGATK.outputfile.txt
#PBS -e variantCallerBwaGATK.errorfile.txt
#PBS -N prefix.info
#PBS -V
#PBS -M ajgore@eng.ucsd.edu
#PBS -m abe
#PBS -A zhang-lab
cd /home/ajgore/TestRun
/projects/zhang-lab/softwares/GenomeAnalysisTK_01022012/variantCallerBwaGATK-latest/variantCallerBwaGATK.pl prefix.info >& prefix.status
  • NOTE: The ampersand in the pipeline command has been moved after the redirect due to differences between bash and csh.
  • To run the pipeline, issue the following terminal command from the folder you uploaded the .info file to (in this case /home/ajgore/TestRun):
qsub prefix.job
  • You can monitor the status of the pipeline by looking at the "prefix.status" file using less:
less prefix.status
  • The output of the pipeline will be two files:
    • prefix.fixed.bam
    • prefix.snp.raw.vcf

Exome Variant Calling Pipeline[edit]

  • Pipeline can be found in:
    • genome-miner:
      • Human: /home/ajgore/AG_Ext12T/GATK_01022012/variantFiltration-latest/variantFiltration.pl
      • Mouse: /home/ajgore/AG_Ext12T/GATK_01022012/variantFiltration-latest/variantFiltration_mouse.pl
    • triton:
      • Human: /projects/zhang-lab/softwares/GenomeAnalysisTK_01022012/variantFiltration-latest/variantFiltration.pl
      • Mouse: /projects/zhang-lab/softwares/GenomeAnalysisTK_01022012/variantFiltration-latest/variantFiltration_mouse.pl
    • This link will always point to the updated version
  • Updated on 07/11/2012

Requirements[edit]

  • All required files are currently on genome-miner and triton cluster. The below paths list genome-miner locations.
  • GATK from 01022012 (/home/ajgore/AG_Ext12T/GATK_01022012/GenomeAnalysisTK-1.4-1-g1b6d528/)
  • Samtools 0.1.7 or later (/home/kunzhang/softwares/samtools-latest)
  • NCBI Human Genome Reference Sequence, v37 (/home/ajgore/AG_Ext12T/GATK_01022012/resources_1.2/human_g1k_v37.fasta)
    • Note: Due to bug in samtools pileup, needed to create a separate index file on a symlinked genome.
    • This can be found here (also present on triton): /home/ajgore/AG_Ext12T/GATK_01022012/resources_1.2/human_g1k_v37_samtools7.fasta

Instructions[edit]

Gather necessary files[edit]

  • Two output files from mapping pipeline are required:
    • .fixed.bam file
    • .snp.raw.vcf file
  • Either run the pipeline in the same directory as these files or use the full paths to these files in the command below.

Run pipeline[edit]

  • To run the pipeline, issue the following terminal command from the folder you uploaded the .info file to (in this case /home/ajgore/TestRun):
nohup /home/ajgore/AG_Ext12T/GATK_01022012/variantFiltration-latest/variantFiltration.pl prefix.snp.raw.vcf prefix.fixed.bam genome-miner &> prefix.filter.status &
  • The first argument should be the .vcf file, the second argument should be the .fixed.bam file, and the third argument should be the system (either genome-miner or triton)
  • You can monitor the status of the pipeline by looking at the "prefix.status" file using less:
less prefix.filter.status
  • The output of the pipeline will be five files:
    • prefix.ind.filtered.vcf - VCF file containing filtered indel calls both passing and failing filters (GATK Best Practices Exome v3 Filters)
    • prefix.snp.filtered.GATK_New.vcf - VCF file containing snp calls both passing and failing filters (GATK Best Practices Exome v3 Filters)
    • prefix.snp.GATK_New.pileup - Pileup file containing filtered snp calls ONLY (GATK Best Practices Exome v3 Filters)
    • prefix.snp.filtered.GATK_Old.vcf - VCF file containing snp calls both passing and failing filters (GATK Best Practices Exome v1 Filters)
    • prefix.snp.GATK_Old.pileup - Pileup file containing filtered snp calls ONLY (GATK Best Practices Exome v1 Filters)