Chris:LabNotes/Microbiome/2013/2013-5-14: Difference between revisions

From ZhangLabWiki
Jump to navigation Jump to search
>Cjwei
(Created page with "=Moleculo Data Analysis Part 1= ==Background== *We are integrating Illumina (short read) and Moleculo (long read) data to assemble the oncomelania genome *The overall plan tha...")
 
>Cjwei
 
(5 intermediate revisions by the same user not shown)
Line 3: Line 3:
*We are integrating Illumina (short read) and Moleculo (long read) data to assemble the oncomelania genome
*We are integrating Illumina (short read) and Moleculo (long read) data to assemble the oncomelania genome
*The overall plan that we are using is:
*The overall plan that we are using is:
Test assembly of small Illumina dataset
Test assembly of small Illumina dataset
           |
           |
           V
           V
Validation of assembly (using assembly statistics and core genes search)
Validation of assembly (using assembly statistics and core genes search)
           |
           |
           V
           V
Assembly of whole Illumina dataset
Assembly of whole Illumina dataset
           |
           |
           V
           V
Integration of Moleculo with Illumina reads
Integration of Moleculo with Illumina reads
           |
           |
           V
           V
Validation of hybrid assembly (assembly statistics, core genes search, assembly errors)
Validation of hybrid assembly (assembly statistics, core genes search, assembly errors)
*A more detailed plan can be found in: [[File:Moleculo_Data_Analysis_Plan_5_14_2013.docx]]
*A more detailed plan can be found in: [[File:Moleculo_Data_Analysis_Plan_5_14_2013.docx]]
==Procedure==
===0_Illumina_Test_correction===
*Triton Folder: </projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Hybrid_Assembly_5_14_2013/0_Illumina_Test_correction/>
*In order to first start with the test assembly of small Illumina dataset, we first want to perform correction of reads and test whether correction will help improve assembly
*We are using the correction software used in Assemblathon2 by SOAPdenovo: (found in </home/cjwei/software/SOAPdenovo2-src-r223/SOAPdenovoPipeline4BirdInAssemblathon2/>
**kmerfreq
**correct_error
*Steps for error correction:
0) Need to gzip read files prior to error correction (can also use .gz files for assembly in SOAPdenovo)
1)Create read list for correction <reads2cor.lst>, which is just the file location of reads.  If have paired read files, must put each file consecutively.
/projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Illumina_raw/HB_gDNA200b_121224L1-4_R1_filtered.fastq.gz
/projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Illumina_raw/HB_gDNA200b_121224L1-4_R2_filtered.fastq.gz
/projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Illumina_raw/HB_gDNA2kb_NoIndex_L005_R1_001_filtered.fastq.gz
/projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Illumina_raw/HB_gDNA2kb_NoIndex_L005_R2_001_filtered.fastq.gz
2) Run commands (need to run kmerfreq then correct_error scripts in order) <1_correction>
*Parameters set based on those used in assemblathon2:
**kmerfreq
***k=k-mer size, 13-27 (used 27)
***l=input read file list
***p=output prefix
***t=cores used (32 cores per node)
***i=set initial size of hash table (used 400000000)
***L=max read length (used 200 in order to ensure enough memory usuage set aside)
**correct_error
***k=k-mer size, must be same as kmerfreq (used 27)
***l=low frequency cutoff (used 2)
***a=set whether remain all the data or not (used 1)
***e=set whether trim suspicious region at end of read according to Q-value or not (used 1)
***w=set whether trim error bases with Q-value <=2 instead of correct (used 1, yes)
***q=set quality threshold (35)
***t=number of cores (used 32)
***j=set whether convert read1 and read 2 corrected file into pair-end file (used 1, yes)
/home/cjwei/software/SOAPdenovo2-src-r223/SOAPdenovoPipeline4BirdInAssemblathon2/kmerfreq -k 27 -l /projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Hybrid_Assembly_5_14_2013/1_Illumina_Test_Assembly/reads2cor.lst -p /projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Hybrid_Assembly_5_14_2013/1_Illumina_Test_Assembly/test_assembly -t 32 -i 400000000 -L 200 >/projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Hybrid_Assembly_5_14_2013/1_Illumina_Test_Assembly/kmerfreq.log
/home/cjwei/software/SOAPdenovo2-src-r223/SOAPdenovoPipeline4BirdInAssemblathon2/correct_error -k 27 -l 2 -a 0 -e 1 -w 1 -q 35 -t 32 -j 1 /projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Hybrid_Assembly_5_14_2013/1_Illumina_Test_Assembly/test_assembly.freq.gz /projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Hybrid_Assembly_5_14_2013/1_Illumina_Test_Assembly/reads2cor.lst >/projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Hybrid_Assembly_5_14_2013/1_Illumina_Test_Assembly/correct.log
3) Output can be found in original raw read folder as .cor.pair files because program combines the paired read files into one </projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Illumina_raw/>
HB_gDNA200b_121224L1-4_R1_filtered.fastq.gz.cor.pair
HB_gDNA2kb_NoIndex_L005_R1_001_filtered.fastq.gz.cor.pair
===1a-c_Illumina_Test_Assembly===
*Continued on <http://genome-tech.ucsd.edu/LabNotes/index.php/Chris:LabNotes/Microbiome/2013/2013-5-15>

Latest revision as of 19:00, 15 May 2013

Moleculo Data Analysis Part 1[edit]

Background[edit]

  • We are integrating Illumina (short read) and Moleculo (long read) data to assemble the oncomelania genome
  • The overall plan that we are using is:
Test assembly of small Illumina dataset
         |
         V
Validation of assembly (using assembly statistics and core genes search)
         |
         V
Assembly of whole Illumina dataset
         |
         V
Integration of Moleculo with Illumina reads
         |
         V
Validation of hybrid assembly (assembly statistics, core genes search, assembly errors)

Procedure[edit]

0_Illumina_Test_correction[edit]

  • Triton Folder: </projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Hybrid_Assembly_5_14_2013/0_Illumina_Test_correction/>
  • In order to first start with the test assembly of small Illumina dataset, we first want to perform correction of reads and test whether correction will help improve assembly
  • We are using the correction software used in Assemblathon2 by SOAPdenovo: (found in </home/cjwei/software/SOAPdenovo2-src-r223/SOAPdenovoPipeline4BirdInAssemblathon2/>
    • kmerfreq
    • correct_error
  • Steps for error correction:

0) Need to gzip read files prior to error correction (can also use .gz files for assembly in SOAPdenovo)

1)Create read list for correction <reads2cor.lst>, which is just the file location of reads. If have paired read files, must put each file consecutively.

/projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Illumina_raw/HB_gDNA200b_121224L1-4_R1_filtered.fastq.gz
/projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Illumina_raw/HB_gDNA200b_121224L1-4_R2_filtered.fastq.gz
/projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Illumina_raw/HB_gDNA2kb_NoIndex_L005_R1_001_filtered.fastq.gz
/projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Illumina_raw/HB_gDNA2kb_NoIndex_L005_R2_001_filtered.fastq.gz

2) Run commands (need to run kmerfreq then correct_error scripts in order) <1_correction>

  • Parameters set based on those used in assemblathon2:
    • kmerfreq
      • k=k-mer size, 13-27 (used 27)
      • l=input read file list
      • p=output prefix
      • t=cores used (32 cores per node)
      • i=set initial size of hash table (used 400000000)
      • L=max read length (used 200 in order to ensure enough memory usuage set aside)
    • correct_error
      • k=k-mer size, must be same as kmerfreq (used 27)
      • l=low frequency cutoff (used 2)
      • a=set whether remain all the data or not (used 1)
      • e=set whether trim suspicious region at end of read according to Q-value or not (used 1)
      • w=set whether trim error bases with Q-value <=2 instead of correct (used 1, yes)
      • q=set quality threshold (35)
      • t=number of cores (used 32)
      • j=set whether convert read1 and read 2 corrected file into pair-end file (used 1, yes)
/home/cjwei/software/SOAPdenovo2-src-r223/SOAPdenovoPipeline4BirdInAssemblathon2/kmerfreq -k 27 -l /projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Hybrid_Assembly_5_14_2013/1_Illumina_Test_Assembly/reads2cor.lst -p /projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Hybrid_Assembly_5_14_2013/1_Illumina_Test_Assembly/test_assembly -t 32 -i 400000000 -L 200 >/projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Hybrid_Assembly_5_14_2013/1_Illumina_Test_Assembly/kmerfreq.log 
/home/cjwei/software/SOAPdenovo2-src-r223/SOAPdenovoPipeline4BirdInAssemblathon2/correct_error -k 27 -l 2 -a 0 -e 1 -w 1 -q 35 -t 32 -j 1 /projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Hybrid_Assembly_5_14_2013/1_Illumina_Test_Assembly/test_assembly.freq.gz /projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Hybrid_Assembly_5_14_2013/1_Illumina_Test_Assembly/reads2cor.lst >/projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Hybrid_Assembly_5_14_2013/1_Illumina_Test_Assembly/correct.log

3) Output can be found in original raw read folder as .cor.pair files because program combines the paired read files into one </projects/zhang-lab/cjwei/130419_Oncomelania_Ilumina_Moleculo/Illumina_raw/>

HB_gDNA200b_121224L1-4_R1_filtered.fastq.gz.cor.pair
HB_gDNA2kb_NoIndex_L005_R1_001_filtered.fastq.gz.cor.pair

1a-c_Illumina_Test_Assembly[edit]