Sam:LabNotes/Microbiome-new/2010-12-10: Difference between revisions

From ZhangLabWiki
Jump to navigation Jump to search
>Sam Chiang
(Created page with '='''Data analysis- HC-MDA-Ecoli-SAGs-Ind1to8(SE - down-sampled reads) - Coverage-vs-input reads plot'''= ==Background== *Construct Genome coverage-vs-input reads plot using uniq…')
 
>Sam Chiang
 
(60 intermediate revisions by 2 users not shown)
Line 1: Line 1:
='''Data analysis- HC-MDA-Ecoli-SAGs-Ind1to8(SE - down-sampled reads) - Coverage-vs-input reads plot'''=
='''Read distribution analysis (unique mapplabe reads) - HC-MDA-Ecoli-SAGs-Ind1to8(SE - down-sampled reads)'''=


==Background==
==Background==
*Construct Genome coverage-vs-input reads plot using unique mappable reads.
*DATA source: HL083 (HC-MDA-Ecoli-SAGs(Ind1to8)-Nov15)
**Y-axis is total genomic regions covered (bps)
*Genome coverage is just one indication to assess the amplification bias.
**X-axis is the total amount of sequences generated (or mapped).
*It is more informative to evaluate the mappable reads distribution from each library. A more even distribution (across E.coli genome) of mappable reads should be expected to see on samples with less amplification bias.
*An example of this plot could be found in Rax's paper
 


==Analysis==
*Running bowtie alignment on previous down-sampled datasets and transform the output to sam-format 
**Using samtools to convert the output from sam-format to bam-format and then to pileup-format.
**Script: fastq2pileup-hc2
**Data resource: down-sampled raw reads (meet 1,000,000 unique mappable reads): e.g. s_4_ID1_cut_2262955.txt
**Template:E.coli-k12-mg1655 template
**Bowtie setting: -k 1 -l 28 -n 2 -m 1 --best --best --phred64-quals
**Output the data: e.g. s_4_ID1_cut_2262955.bowtie.pileup


*In UNIX, extract data from column 1, 2 and column9 with "^" symbol ("^" read starting sign in pileup-format) from each line
**Output the data: e.g. s_4_ID1_cut_2262955.bowtie.pileup.readstart


*Using perl to calculate "^" symbol in defined bin size (10,000 bp)
**Script: CountReadStart-hc2
**Output the data: e.g. s_4_ID1_cut_2262955.bowtie.pileup.readstart.count


*Use excel to plot the Read-start counts vs. bin numbers (genome position)


==Plotting==
*Since the bin number data is not continuous (the bin number containing no reads will not be reported in previous step).
*I need to create a bin-ladder (from bin 0 to 463) and pool my data set with bin-ladder by matching bin number.
**Without doing this step will cause Excel draw the plot with discontinuous bin interval (since Excel regard bin number as a new data set but not X-axis).
*I pooled the data set using Excel Add-Ins function "Join (merge)Talbes"(from DigDB).
*Take the log10 value of total reads for Y-axis.
*'''IMPORTANT:''' Since log10 value can't be taken from no read (count=0/10,000bp), I manually assign the value of '''log10 of "no read data"''' as '''"-1"'''.
**The reason to do so is that there are lots of bins containing "0 read" exclusively in long-amplified SAGs' data sets.
*'''X-Y scatter plot analysis:'''
**Purpose: to compare if libraries made from the same MDA amplicon resource (e.g. ID1-ID3) behave more similar than libraries made from different MDA amplicons (e.g. ID1-ID2).


==Results==
[[File:sam121510-read-distribution - ID1 - ID3.jpg|600px]]  [[File:sam022011-scattor-plot ID1-ID3-new3.png|400px]]
[[File:sam121510-read-distribution - ID2 - ID4.jpg|600px]]  [[File:sam022011-scattor-plot ID2-ID4-new3.png|480px]]
[[File:sam121510-read-distribution - ID5 - ID7.jpg|600px]]  [[File:sam022011-scattor-plot ID5-ID7-new3.png|400px]]
[[File:sam121510-read-distribution - ID6 - ID8.jpg|600px]]  [[File:sam022011-scattor-plot ID6-ID8-new3.png|400px]]


'''Excel files:'''
[[Media:Bias-plot-all-pooled ID1-ID3-office2003.xls]]
[[Media:Bias-plot-all-pooled ID2-ID4-office2003.xls]]
[[Media:Bias-plot-all-pooled ID5-ID7-office2003.xls]]
[[Media:Bias-plot-all-pooled ID6-ID8-office2003.xls]]
'''*For unknown reason the wikinote doesn't allow me to upload Excel 2007 files(.xlsx), so I saved as Excel 2003 file format(.xls)'''


==Discussion==
*Short-amplified and long-amplified libraries showed '''very high overlapped''' on reads distribution.
*'''Reads from short-amplified libraries are more evenly distributed''' than that from long-amplified libraries.
**Some peaks are missing in long-amplified libraries but still shown in short-amplified libraries (see pooled figure).
*'''A significant difference between short and long-amplified libraries was shown at the Y-axis range -1 ~ 0''', where it showed one read(Y=0) in short-amplified libraries but no read (Y=-1) in long amplified libraries at the same bin.




In order to compare the amplification bias between short-amplified and long amplified MDA, we can compare genome-coverage to assess amplification bias.
='''Read distribution plot and X-Y scatter plot with 1Kbp bin'''=
**Hypothesis: Given a same number of unique mappable reads, the higher genome coverage means less (position)bias.
*I was wonder if using smaller bin size will show better difference between limited MDA and regular MDA results
*We need to use the ratio of unique mappable read/total reads to decided how much total read we should down-sample to get
*The number of unique mappable reads was decided on 1,000,000 reads (since the lowest number of unique mappable reads is '''1,192,051''' in ID-4 library.
**Ref. link: http://genome-tech.ucsd.edu/LabNotes/index.php/Sam:LabNotes/Microbiome-new/2010-12-8#Discussions
*The required total reads for getting '''1,000,000 unique mappable reads''':
[[File:sam120810-number of need to downsampled reads.jpg|700px]]


*Using the down-sampling script to get desired number of raw reads that meet 1,000,000 unique mappable reads).
==Analysis procedure==
**Script: Data-sampler.pl (= randomizor by CW)
*Started from .pileup file
**Data resource: HC-MDA-Ecoli-SAGs-Ind1to8 (FASTQ format)
*Only use ILL_ID6 and ILL_ID8 data for testing.
***NOTE: '''Actually this script only split the data from the beginning but not randomly sampling'''.
*Use the modified script '''CountReadStart-hc3_1kb_bin''' to count read in 1kb-bin size
**Data output:
samchiang@genemapster:~/Scripts$ perl CountReadStart-hc3_1kb_bin /media/disk-1/samchiang/Raw-Read-Complete/101123_HL083/HC-MDA-Ecoli-SAGs-Ind1to8-SE/s_4_ID6_cut_1680672.bowtie.pileup.readstart
  /media/disk-2/samchiang/Raw-Read-backup/101123_HL083/HC-MDA-Ecoli-SAGs-Ind1to8/
 
  s_4_ID1_cut_2262955.txt  s_4_ID2_cut_1929385.txt
samchiang@genemapster:~/Scripts$ perl CountReadStart-hc3_1kb_bin /media/disk-1/samchiang/Raw-Read-Complete/101123_HL083/HC-MDA-Ecoli-SAGs-Ind1to8-SE/s_4_ID8_cut_1422273.bowtie.pileup.readstart
  s_4_ID3_cut_1150351.txt  s_4_ID4_cut_1478415.txt
  s_4_ID5_cut_2042067.txt  s_4_ID6_cut_1680672.txt
  s_4_ID7_cut_1898254.txt  s_4_ID8_cut_1422273.txt


*Perform down-sampled reads alignment using Bowtie, followed by genome coverage calculation.
==Resutls==
**Script: Read2coverage-kz-3
[[File:ILL_L4_ID6_log10_1kbbin.jpg|600px]]
**Data resource: down-sampled raw reads (meet 1,000,000 unique mappalbe reads)
**Template:E.coli-k12-mg1655 template
[[File:ILL_R4_ID8_log10_1kbbin.jpg|600px]]
**Bowtie setting: -k 1 -l 28 -n 2 -m 1 --best --best --phred64-quals
**Downstream: Data was used for genome coverage calculation.
[[File:ILL_L4_R4_ID6and8_log10_1kbbin.jpg|600px]]
===Results===
  [[File:sam121410-genome coverage on down-sampled reads-3.jpg|900px]]
  [[File:ILL_L4_R4_ID6and8_log10_1kbbin_XY_scatter.jpg|400px]]


==Discussion==
==Discussion==
*Under a fair comparison using same number of unique mappable reads, '''short-amplified SAGs showed higher (2~3 fold) genome coverage than their long-amplified SAG mates''' (ID1 vs ID3, ID2 vs ID4, ID5 vs ID7, ID6 vs ID8).
*It seems like using 10k bin size shows more obvious of missing coverage in regular MDA.  
*'''Higher genome coverage suggested a less overlapped of sequencing reads, which also suggested a less amplification bias  in short-amplified MDA products'''.
*I will stick with 10k bin size plots for preparing manuscript.

Latest revision as of 07:51, 7 June 2011

Read distribution analysis (unique mapplabe reads) - HC-MDA-Ecoli-SAGs-Ind1to8(SE - down-sampled reads)[edit]

Background[edit]

  • DATA source: HL083 (HC-MDA-Ecoli-SAGs(Ind1to8)-Nov15)
  • Genome coverage is just one indication to assess the amplification bias.
  • It is more informative to evaluate the mappable reads distribution from each library. A more even distribution (across E.coli genome) of mappable reads should be expected to see on samples with less amplification bias.

Analysis[edit]

  • Running bowtie alignment on previous down-sampled datasets and transform the output to sam-format
    • Using samtools to convert the output from sam-format to bam-format and then to pileup-format.
    • Script: fastq2pileup-hc2
    • Data resource: down-sampled raw reads (meet 1,000,000 unique mappable reads): e.g. s_4_ID1_cut_2262955.txt
    • Template:E.coli-k12-mg1655 template
    • Bowtie setting: -k 1 -l 28 -n 2 -m 1 --best --best --phred64-quals
    • Output the data: e.g. s_4_ID1_cut_2262955.bowtie.pileup
  • In UNIX, extract data from column 1, 2 and column9 with "^" symbol ("^" read starting sign in pileup-format) from each line
    • Output the data: e.g. s_4_ID1_cut_2262955.bowtie.pileup.readstart
  • Using perl to calculate "^" symbol in defined bin size (10,000 bp)
    • Script: CountReadStart-hc2
    • Output the data: e.g. s_4_ID1_cut_2262955.bowtie.pileup.readstart.count
  • Use excel to plot the Read-start counts vs. bin numbers (genome position)

Plotting[edit]

  • Since the bin number data is not continuous (the bin number containing no reads will not be reported in previous step).
  • I need to create a bin-ladder (from bin 0 to 463) and pool my data set with bin-ladder by matching bin number.
    • Without doing this step will cause Excel draw the plot with discontinuous bin interval (since Excel regard bin number as a new data set but not X-axis).
  • I pooled the data set using Excel Add-Ins function "Join (merge)Talbes"(from DigDB).
  • Take the log10 value of total reads for Y-axis.
  • IMPORTANT: Since log10 value can't be taken from no read (count=0/10,000bp), I manually assign the value of log10 of "no read data" as "-1".
    • The reason to do so is that there are lots of bins containing "0 read" exclusively in long-amplified SAGs' data sets.
  • X-Y scatter plot analysis:
    • Purpose: to compare if libraries made from the same MDA amplicon resource (e.g. ID1-ID3) behave more similar than libraries made from different MDA amplicons (e.g. ID1-ID2).

Results[edit]

File:Sam121510-read-distribution - ID1 - ID3.jpg  File:Sam022011-scattor-plot ID1-ID3-new3.png 

File:Sam121510-read-distribution - ID2 - ID4.jpg  File:Sam022011-scattor-plot ID2-ID4-new3.png 

File:Sam121510-read-distribution - ID5 - ID7.jpg  File:Sam022011-scattor-plot ID5-ID7-new3.png 

File:Sam121510-read-distribution - ID6 - ID8.jpg  File:Sam022011-scattor-plot ID6-ID8-new3.png
Excel files:
Media:Bias-plot-all-pooled ID1-ID3-office2003.xls
Media:Bias-plot-all-pooled ID2-ID4-office2003.xls
Media:Bias-plot-all-pooled ID5-ID7-office2003.xls
Media:Bias-plot-all-pooled ID6-ID8-office2003.xls

*For unknown reason the wikinote doesn't allow me to upload Excel 2007 files(.xlsx), so I saved as Excel 2003 file format(.xls)

Discussion[edit]

  • Short-amplified and long-amplified libraries showed very high overlapped on reads distribution.
  • Reads from short-amplified libraries are more evenly distributed than that from long-amplified libraries.
    • Some peaks are missing in long-amplified libraries but still shown in short-amplified libraries (see pooled figure).
  • A significant difference between short and long-amplified libraries was shown at the Y-axis range -1 ~ 0, where it showed one read(Y=0) in short-amplified libraries but no read (Y=-1) in long amplified libraries at the same bin.


Read distribution plot and X-Y scatter plot with 1Kbp bin[edit]

  • I was wonder if using smaller bin size will show better difference between limited MDA and regular MDA results

Analysis procedure[edit]

  • Started from .pileup file
  • Only use ILL_ID6 and ILL_ID8 data for testing.
  • Use the modified script CountReadStart-hc3_1kb_bin to count read in 1kb-bin size
samchiang@genemapster:~/Scripts$ perl CountReadStart-hc3_1kb_bin /media/disk-1/samchiang/Raw-Read-Complete/101123_HL083/HC-MDA-Ecoli-SAGs-Ind1to8-SE/s_4_ID6_cut_1680672.bowtie.pileup.readstart
 
samchiang@genemapster:~/Scripts$ perl CountReadStart-hc3_1kb_bin /media/disk-1/samchiang/Raw-Read-Complete/101123_HL083/HC-MDA-Ecoli-SAGs-Ind1to8-SE/s_4_ID8_cut_1422273.bowtie.pileup.readstart

Resutls[edit]

File:ILL L4 ID6 log10 1kbbin.jpg

File:ILL R4 ID8 log10 1kbbin.jpg

File:ILL L4 R4 ID6and8 log10 1kbbin.jpg

File:ILL L4 R4 ID6and8 log10 1kbbin XY scatter.jpg

Discussion[edit]

  • It seems like using 10k bin size shows more obvious of missing coverage in regular MDA.
  • I will stick with 10k bin size plots for preparing manuscript.