Sam:LabNotes/Microbiome-new/2011-5-8: Difference between revisions

From ZhangLabWiki
Jump to navigation Jump to search
>Sam Chiang
>Sam Chiang
 
(One intermediate revision by the same user not shown)
Line 41: Line 41:


===blstn with -e 0.01 and FT settings against nt===
===blstn with -e 0.01 and FT settings against nt===
*blastn setting -K 1 -F T -m 0 -a 2 -v 1 -b 1 -e 0.01
*blastn setting -K 1 -F T -m 0 -a 2 -v 1 -b 1 -e 0.01
*blastn setting -K 1 -F T -m 0 -a 2 -v 1 -b 1 -e 0.01
  samchiang@genemapster:/media/disk-2/ncbi-blast-2.2.23+/bin$ ./legacy_blast.pl blastall -i /media/disk-1/samchiang/Sub_tests_folder_imp/6_10000reads_subset_analysis/2nd_analysis_with_propotional_sampling/2nd_test_new/NX_lib_failed/s_4_pe_ID8.failed.txt.shuf-1265reads.fa -d /media/disk-2/data/nt -K 1 -m 0 -a 2 -v 1 -b 1 -e 0.01 -o /media/disk-1/samchiang/Sub_tests_folder_imp/6_10000reads_subset_analysis/2nd_analysis_with_propotional_sampling/2nd_test_new/NX_lib_failed/s_4_pe_ID8.failed.txt.shuf-1265reads.fa.blastn.FT.txt -p /media/disk-2/ncbi-blast-2.2.23+/bin/blastn &
  samchiang@genemapster:/media/disk-2/ncbi-blast-2.2.23+/bin$ ./legacy_blast.pl blastall -i /media/disk-1/samchiang/Sub_tests_folder_imp/6_10000reads_subset_analysis/2nd_analysis_with_propotional_sampling/2nd_test_new/NX_lib_failed/s_4_pe_ID8.failed.txt.shuf-1265reads.fa -d /media/disk-2/data/nt -K 1 -m 0 -a 2 -v 1 -b 1 -e 0.01 -o /media/disk-1/samchiang/Sub_tests_folder_imp/6_10000reads_subset_analysis/2nd_analysis_with_propotional_sampling/2nd_test_new/NX_lib_failed/s_4_pe_ID8.failed.txt.shuf-1265reads.fa.blastn.FT.txt -p /media/disk-2/ncbi-blast-2.2.23+/bin/blastn &

Latest revision as of 01:21, 1 July 2011

Repeat of 1,000 reads BLASTn taxonomic analysis from 04-25-'11 test[edit]

Background[edit]

  • The taxonomic analysis on 4-25-'11 showed some inconsistent conclusions between blastn and bowtie mapping.
  • Although these could be due to many differences between two approaches, I am wondering if the sampling approach would affect the results too.
  • Dr. Zhang point out the concerns of -e setting in blastn, in which the default -e (10) setting might not be rigid enough.

Objectives[edit]

  • Test on proportional sampling size
  1. Sampling 1000 reads for the lowest number group.
  2. Sampling the proportional number of reads for other groups based on fold variance.
  • Test different blastn parameter settings:
  1. Expectation value 10(default) => -e 0.01
  2. blastn DUST filter T(default) => F

Test of -e parameters[edit]

  • No bad alignment structure was found using default -e 10 setting.
  • No difference on output results was observed by switching between -e 0.01 and -e 10.
    • It is probably due to that I used the blastn setting that only report the best hits.
    • I will switch to use -e 0.01 anyway for future blastn.

Test of -F parameters[edit]

  • Turning off DUST-filter(FF) increase 10% hit compared to filter-on setting(FT)
  • However some data have problems to be run with FF setting. Error message: "Program failed, try executing the command manually.".
  • The problem sometime could be solved by changing the file name.
  • I need to do more tests to figure out why, but current I decide to keep using FT(default) for all of my blastn analysis.

Real tests of ILL- and NX-library data[edit]

Decide sampling size[edit]

px300

Shuffle sampling the fastq reads with proprotional sample size[edit]

  • Use the perl program Chris help me to modify to do sampling. The new program fastq2Tab2fas_shuf_sampling-cw3 need to input the number of total fastq read and desired sample-size.
  • Fastq-> Tabular -> Shuffling -> Downsampled -> Fasta -> (ready for blastn)
Command example:
samchiang@genemapster:~/Scripts$ perl fastq2Tab2fas_shuf_sampling-cw3 /media/disk-1/samchiang/Raw-Read-Complete/101123_HL083/HC-MDA-Ecoli-SAGs-Ind1to8-PE/s_4_1and2_ID1.txt 15836502 2984 &
  • In total 8 (ID1~ID8) x 2 (raw reads, failed-to-align reads) x 2 (ILL lib, NX lib) = 32 fastq data were re-sampled. The output

blstn with -e 0.01 and FT settings against nt[edit]

  • blastn setting -K 1 -F T -m 0 -a 2 -v 1 -b 1 -e 0.01
  • blastn setting -K 1 -F T -m 0 -a 2 -v 1 -b 1 -e 0.01
samchiang@genemapster:/media/disk-2/ncbi-blast-2.2.23+/bin$ ./legacy_blast.pl blastall -i /media/disk-1/samchiang/Sub_tests_folder_imp/6_10000reads_subset_analysis/2nd_analysis_with_propotional_sampling/2nd_test_new/NX_lib_failed/s_4_pe_ID8.failed.txt.shuf-1265reads.fa -d /media/disk-2/data/nt -K 1 -m 0 -a 2 -v 1 -b 1 -e 0.01 -o /media/disk-1/samchiang/Sub_tests_folder_imp/6_10000reads_subset_analysis/2nd_analysis_with_propotional_sampling/2nd_test_new/NX_lib_failed/s_4_pe_ID8.failed.txt.shuf-1265reads.fa.blastn.FT.txt -p /media/disk-2/ncbi-blast-2.2.23+/bin/blastn &

Using MEGAN analysis[edit]

  • Using MEGAN "compare" function to parse blastn taxonomic results.
  1. Run MEGAN analysis on individual data
  2. Import all of the .rma files (MEGAN output format) for comparison
  3. Use normalized count setting (actually it won't matter if we chose to use % presentation later)
  4. Select the most specific node on taxa rank for plotting.
  5. Plot results in stacked bar chart and present in % ratio.

Results[edit]

Previous bowtie mapping results summary 
File:050311 Ecoli SAG bowtie mapping summary.png

File:Ecoli SAG raw 1000reads up ILL lib-a.png

File:Ecoli SAG raw 1000reads up NX lib-a.png

File:Ecoli SAG failed2align 1000reads up ILL lib-a.png

File:Ecoli SAG failed2align 1000reads up NX lib-a.png

Discussion[edit]

  • Questions need to verify
  1. Difference between fixed-sampling vs. proportional sampling
  2. Difference between proportional sampling vs. bowtie mapping
  • Be careful that the colors represents E.coli and No hit are reversed in raw-reads chart and in failed-to-align reads charts.
  • I found that the libraries with higher E.coli abundance also showed higher E.coli abundance in the failed-to-align reads(bowtie mapping).
    • I am wondering why? But this phenomena implies that bowtie missed a lot of E.coli reads in all libraries evenly, so the missed showed same abundance of E.coli reads in failed-to-align pools.
    • Unfortunately, it's not working very well using blastn to estimate how many E.coli reads failed in bowtie since the blastn(using nt reference) can't be as specific as bowtie (using E.coli K-12 template).
    • However, the E.coli reads identified by blastn in failed-to-align pools showed about
File:051211-Ecoli abundance ILL.PNG  File:051211-Ecoli abundance NX.PNG

Next[edit]

  1. Find out a way to improve genome coverage by decrease the failed-to-align E.coli reads:
    1. Increase error tolerance in bowtie.
    2. Test BWA alignment. Check genome coverage. Play with parameters settings.
    3. Redo genome coverage calculation using -m 10 bowtie
  1. Is it necessary to do taxanomic analysis on unknown reads that fallen in no-hit category?
    1. I can blast them to customized database?
  1. Test if the missed E.coli reads are critical on genome coverage. (Probably not. Since the genome coverage is associated with amplification bias. Libraries with higher E.coli reads don't necessary showed higher genome coverage).
  2. Since the E.coli SAG still contains lots of non-specific contamination.

Additional idea[edit]

  • The E.coli SAG data could be a good dataset for testing binning accuracy.
  1. Objective: demonstrate genome binning could be used and necessary to remove non-specific contamination in SAG samples.It will be great if genome binning could be done on raw-reads without de novo assembly.
    1. E.coli SOAPdenovo (K27M3) -> validation by mapping to E.coli-12 reference
    2. Contigs tetramer feq. count
    3. Clustering -> Treeview building
    4. blastn validation -> grow binned contigs
    5. Validation of clustered contigs by mapping to E.coli-K12 reference.