Sam:LabNotes/Microbiome-new/2011-4-25: Difference between revisions

From ZhangLabWiki
Jump to navigation Jump to search
>Sam Chiang
>Sam Chiang
 
(22 intermediate revisions by the same user not shown)
Line 1: Line 1:
='''Taxonomic analysis on proportioned raw reads and failed-to-align(bowtie) reads on E.coli SAG Illumina and Nextera libraries'''=
='''1,000 reads BLASTn taxonomic analysis on raw reads and failed-to-align(bowtie)reads from E.coli SAG Illumina and Nextera libraries'''=


==Background==
==Background==
Line 11: Line 11:
*Test if Nextra library generate more no-hit or contamination then standard Illumina library.
*Test if Nextra library generate more no-hit or contamination then standard Illumina library.
**Compare the taxonomic distribution on (1)raw reads (2) failed-to-align reads
**Compare the taxonomic distribution on (1)raw reads (2) failed-to-align reads
*'''NOTE: Since I didn't use the same source of the amplified MDA sample to generate Ill and NX library for a side-by-side comparison, it is not convincing to judge which library construction protocol is "cleaner". However if the different is on the abundance of the "no hit", then it is not due to the contamination issue.
*'''NOTE: Since I didn't use the same source of the amplified MDA sample to generate Ill and NX library for a side-by-side comparison, it is not convincing to judge which library construction protocol is "cleaner". However if the different is located on the "no hit" catagory, then it is possible not due to the contamination issue or could be due to the low-sequencing quality or lack of NCBI-nt reference.
 
==Results==


==Procedures==
==Procedures==
Line 27: Line 25:


*Perform the blastn (against nt database) using the sampled 10,000 and 1,000 reads (fasta format).
*Perform the blastn (against nt database) using the sampled 10,000 and 1,000 reads (fasta format).
  samchiang@genemapster:/media/disk-2/ncbi-blast-2.2.23+/bin$ ./legacy_blast.pl blastall -i /media/disk-1/samchiang/Sub_tests_folder_imp/6_10000reads_subset_analysis/Ecoli_SAG_Ill_failed-to-aligned-reads/fasta_sequences_10000reads/Ecoli_SAG_ILL_failed_10000read.ID1.fa.fasta  -d /media/disk-2/data/nt -K 1 -m 0 -a 4 -v 1 -b 1 -o /media/disk-1/samchiang/Sub_tests_folder_imp/6_10000reads_subset_analysis/Ecoli_SAG_Ill_failed-to-aligned-reads/fasta_sequences_10000reads/Ecoli_SAG_ILL_failed_10000read.ID1.fa.blastn.txt --p /media/disk-2/ncbi-blast-2.2.23+/bin/blastn &
   
 
*Using MEGAN to parse blastn results and draw the pie-chart.
*Using MEGAN to parse blastn results and draw the pie-chart.
*Using MEGAN "compare" function is easier to interpret all of the results generated.
*Using MEGAN "compare" function is easier to interpret all of the results generated.
Line 35: Line 32:
#Select the most specific node on taxa rank for plotting.
#Select the most specific node on taxa rank for plotting.
#Plot the result in stacked bar chart and present in % ratio.
#Plot the result in stacked bar chart and present in % ratio.
   
 
==Results==
==Results==
*I also did 16 samples MEGAN analysis using 10,000 reads to check if there any bias due to the sampling size. The results suggest that the bias variation is within 1~3%. Sampling 1,000 reads or 10,000 drew very similar taxonomic conclusions.
*I also did 16 samples MEGAN analysis using 10,000 reads to check if there any bias due to the sampling size (10,000 reads vs 1,000 reads). The results suggest that the bias variation is within 1~3%. Sampling 1,000 reads or 10,000 drew very similar taxonomic conclusions.
**I can't test more than 10,000 reads because the limit the BLASTN efficiency.
 
'''Previous bowtie mapping results summary'''
[[File:050311_Ecoli_SAG_bowtie_mapping_summary.png|1100px]]


'''1000reads BLASTn + MEGAN results'''
  [[File:Ecoli_SAG_raw_1000reads_all_ILL_labeled.png|900px]]
  [[File:Ecoli_SAG_raw_1000reads_all_ILL_labeled.png|900px]]
   
   
Line 47: Line 49:
  [[File:Ecoli_SAG_failed2align_1000reads_all_NX_labeled.png|900px]]
  [[File:Ecoli_SAG_failed2align_1000reads_all_NX_labeled.png|900px]]


[[File:050311_Ecoli_SAG_bowtie_mapping_summary.png|1100px]]
==Discussions==
*I found that there is a inconsistency between the bowtie mapping and 1000-raw_read_blastn results.
**It is weird that in some libraries the E.coli read ration(%) were even lower then that in bowtie mapping results.
**It is unlikely due to the BLASTN stringency. BLASTN should be less rigid than bowtie since BLASTn reports any E.coli hit even it is a not a strain-specific E.coli read (e.g. E.coli DH1..).
 
*I doubt if the sample size plays a factor on this matter. I noticed that in the Illumina library ID3 and ID8, there are very high proportion of E.coli reads identified by blastn but the reads were fallen in the failed-to-align category. However, the original failed-to-align reads (in bowtie mapping) are much fewer in ID3 and ID8 libraries.
*I probably should not use the fixed sample size for taxonomic analysis on raw reads and failed-to-aligned reads.
 
==Next==
*It is possible the current conclusion is biased by my sampling approach.
*I will re-do this analysis with proportional sampling size for each of libraries.
 
==Follow up - validation of blastn settings==
*I used the default setting on blastn, which has E-value =10
**The common setting is -e 0.01 or 1Xe-5 suggested by Dr. Zhang
*Comparison
#blastn setting 1(default)                        : -K 1 -m 0 -a 4 -v 1 -b 1
#blastn setting 2(default w/ E=0.01)              : -K 1 -m 0 -a 4 -v 1 -b 1 -e 0.01
#blastn setting 3(default w/ E=0.01 plus filter on): -K 1 -F T -m 0 -a 4 -v 1 -b 1 -e 0.01
 
==Results==
*Even the default E value (10) was used, the reported alignments still have very low (Expect = 1e-31 ~ 9e-33)E-value and high identity (98~100%) of aligned region. It is possible we demand blastn to report only one best output (-K =1)
*The is no difference if I change to -e 0.01 or not. Same result came out.
*With low-complexity filter turned off (the default is on) I got a little bit more hits. I check the sequence on filtered out hits and found that they should be accurate hits.
**Same taxa are found in other non-filtered out hit.
**The E-value and Identity rate are high in these filter-out hits.
 
==Conclusion==
*I will use -e 0.01 to decrease the possibility of reporting bad alignment.
*I will turn off the query filtering.
**Information about DUST filter http://www.ncbi.nlm.nih.gov/blast/blast_help.shtml

Latest revision as of 01:12, 12 August 2011

1,000 reads BLASTn taxonomic analysis on raw reads and failed-to-align(bowtie)reads from E.coli SAG Illumina and Nextera libraries[edit]

Background[edit]

  • Since the blastn + MEGAN works very well on failed-to-align reads taxonomic analysis (Test did on E.coli SAG Nextera library). I am planing to extend the same method to characterize the raw reads.
    • Compare the the results with the result from bowtie mapping, then quantify how many false negative E.coli reads are mis-called by bowtie mapping.
  • Also, I will compare the different sampling size (10,000 reads vs. 1000 reads) and see if there is bias existed due to my sampling threshold.

Objectives[edit]

  • Analyze the taxonomic distribution on raw reads for E.coli SAGs Ill and NX library data. Compare the conclusion with bowtie's conclusion.
  • Test if 10000-read sample's conclusion is different from 1000-read sampling.
  • Test if Nextra library generate more no-hit or contamination then standard Illumina library.
    • Compare the taxonomic distribution on (1)raw reads (2) failed-to-align reads
  • NOTE: Since I didn't use the same source of the amplified MDA sample to generate Ill and NX library for a side-by-side comparison, it is not convincing to judge which library construction protocol is "cleaner". However if the different is located on the "no hit" catagory, then it is possible not due to the contamination issue or could be due to the low-sequencing quality or lack of NCBI-nt reference.

Procedures[edit]

  • Sample 1M reads from raw reads and failed-to-align reads for both Ecoli SAG Illumina (101123_HL083_s4_Ecoli_IllLib) and Nextera(110120_HL087_s4_Ecoli_NexLib) sequencing data.
    • Using the new perl script: fastq-data-sampler-v2-hc
samchiang@genemapster:~/Scripts$ perl fastq-data-sampler-v2-hc /media/disk-1/samchiang/Raw-Read-Complete/101123_HL083/HC-MDA-Ecoli-SAGs-Ind1to8-PE/s_4_1and2_ID2.txt /media/disk-1/samchiang/Raw-Read-Complete/101123_HL083/HC-MDA-Ecoli-SAGs-Ind1to8-PE/s_4_1and2_ID2.1Mreads 1000000 &
  • Deposit the sampled 10,000 reads(fastq) into galaxy server.
    • Use the workflow "Failed2align_1m_reads_analysis_new-v3-042311" to
  1. Original 1M reads to construct boxplot using quality score
  2. Random sampling 10,000 reads, extracted as fasta file.
  3. Random sampling 1,000 reads, extracted as fasta file.
  • Perform the blastn (against nt database) using the sampled 10,000 and 1,000 reads (fasta format).
  • Using MEGAN to parse blastn results and draw the pie-chart.
  • Using MEGAN "compare" function is easier to interpret all of the results generated.
  1. Import all of the .rma files (MEGAN output format)for comparison
  2. Use normalized count setting.
  3. Select the most specific node on taxa rank for plotting.
  4. Plot the result in stacked bar chart and present in % ratio.

Results[edit]

  • I also did 16 samples MEGAN analysis using 10,000 reads to check if there any bias due to the sampling size (10,000 reads vs 1,000 reads). The results suggest that the bias variation is within 1~3%. Sampling 1,000 reads or 10,000 drew very similar taxonomic conclusions.
    • I can't test more than 10,000 reads because the limit the BLASTN efficiency.
Previous bowtie mapping results summary 
File:050311 Ecoli SAG bowtie mapping summary.png
1000reads BLASTn + MEGAN results
File:Ecoli SAG raw 1000reads all ILL labeled.png

File:Ecoli SAG raw 1000reads all NX labeled.png

File:Ecoli SAG failed2align 1000reads all ILL labeled.png

File:Ecoli SAG failed2align 1000reads all NX labeled.png

Discussions[edit]

  • I found that there is a inconsistency between the bowtie mapping and 1000-raw_read_blastn results.
    • It is weird that in some libraries the E.coli read ration(%) were even lower then that in bowtie mapping results.
    • It is unlikely due to the BLASTN stringency. BLASTN should be less rigid than bowtie since BLASTn reports any E.coli hit even it is a not a strain-specific E.coli read (e.g. E.coli DH1..).
  • I doubt if the sample size plays a factor on this matter. I noticed that in the Illumina library ID3 and ID8, there are very high proportion of E.coli reads identified by blastn but the reads were fallen in the failed-to-align category. However, the original failed-to-align reads (in bowtie mapping) are much fewer in ID3 and ID8 libraries.
  • I probably should not use the fixed sample size for taxonomic analysis on raw reads and failed-to-aligned reads.

Next[edit]

  • It is possible the current conclusion is biased by my sampling approach.
  • I will re-do this analysis with proportional sampling size for each of libraries.

Follow up - validation of blastn settings[edit]

  • I used the default setting on blastn, which has E-value =10
    • The common setting is -e 0.01 or 1Xe-5 suggested by Dr. Zhang
  • Comparison
  1. blastn setting 1(default)  : -K 1 -m 0 -a 4 -v 1 -b 1
  2. blastn setting 2(default w/ E=0.01)  : -K 1 -m 0 -a 4 -v 1 -b 1 -e 0.01
  3. blastn setting 3(default w/ E=0.01 plus filter on): -K 1 -F T -m 0 -a 4 -v 1 -b 1 -e 0.01

Results[edit]

  • Even the default E value (10) was used, the reported alignments still have very low (Expect = 1e-31 ~ 9e-33)E-value and high identity (98~100%) of aligned region. It is possible we demand blastn to report only one best output (-K =1)
  • The is no difference if I change to -e 0.01 or not. Same result came out.
  • With low-complexity filter turned off (the default is on) I got a little bit more hits. I check the sequence on filtered out hits and found that they should be accurate hits.
    • Same taxa are found in other non-filtered out hit.
    • The E-value and Identity rate are high in these filter-out hits.

Conclusion[edit]