Rui:Data Analysis: Difference between revisions

From ZhangLabWiki
Jump to navigation Jump to search
>RuiLiu
>RuiLiu
 
(23 intermediate revisions by the same user not shown)
Line 1: Line 1:
==Examples==
==Examples==
KUN's pipeline:
http://genome-tech.ucsd.edu/LabNotes/index.php/RNAseq_pipeline [http://genome-tech.ucsd.edu/LabNotes/index.php/RNAseq_pipeline]
  Jeff's samples:
  Jeff's samples:
  Kun:LabNotes/Haplotyping/2010-9-17 [http://genome-tech.ucsd.edu/LabNotes/index.php/Kun:LabNotes/Haplotyping/2010-9-17]
  Kun:LabNotes/Haplotyping/2010-9-17 [http://genome-tech.ucsd.edu/LabNotes/index.php/Kun:LabNotes/Haplotyping/2010-9-17]
Line 12: Line 15:
*[[Rui:Haplotyping 6.21.11|Haplotyping 6.21.11]]
*[[Rui:Haplotyping 6.21.11|Haplotyping 6.21.11]]
*[[Rui:RNAseq analysis from 7.18.11|RNAseq analysis from 7.18.11]]
*[[Rui:RNAseq analysis from 7.18.11|RNAseq analysis from 7.18.11]]
*[[Rui:RNAseq analysis on HL098l RNAseq analysis on HL098]]
*[[Rui:RNAseq analysis on HL098|RNAseq analysis on HL098]]
*[[Rui:RNAseq analysis on HL099|RNAseq analysis on HL099]]
*[[Rui:RNAseq analysis on HL099|RNAseq analysis on HL099]]
*[[Rui:RNAseq analysis on HL101|RNAseq analysis on HL101]]
*[[Rui:RNAseq analysis on Hiseq111005|RNAseq_PGCsamples on Hiseq111005]]
*[[Rui:Hap analysis on HL108|Hap analysis on HL108]]
*[[Rui:RNAseq analysis on HL109|RNAseq analysis on HL109]]
*[[Rui:DNAseq analysis on HL114|DNAseq analysis on HL114]]
*[[Rui:DNAseq analysis on Hiseq120313|DNAseq analysis on Hiseq120313]]
*[[Rui:DNAseq analysis on HL118|DNAseq analysis on HL118]]
*[[Rui:totoRNAseq on 121226_HL140|totoRNAseq on 121226_HL140]]
*[[Rui:totoRNAseq on 130120_HL141|totoRNAseq on 130120_HL141]]
*[[Rui:totoRNAseq on 130220_HL144|totoRNAseq on 130220_HL144]]
*[[Rui:totoRNAseq on BL-UHRR-IVT-May28_2013|totoRNAseq on BL-UHRR-IVT-May28_2013]]
*[[Rui:totoRNAseq on 131009_HL161|totoRNAseq on 131009_HL161]]


==RNAseq analysis on HL098==
==RNAseq analysis key notes==
===Database download===
* iGenome packages [http://tophat.cbcb.umd.edu/igenomes.html]
iGenome collection [http://cufflinks.cbcb.umd.edu/igenomes.html], save in ~/iGenome/
# UCSC genome package: [http://genome-tech.ucsd.edu/LabNotes/index.php/Rui:Tophat_mapping_10.27.11]
 
# Ensemble genome package: [http://genome-tech.ucsd.edu/LabNotes/index.php/Rui:Tophat_11.21.11#Re-download_Ensemble_package_from_iGenome_.5B1.5D]
* wget ftp://igenome:G3nom3s4u@ftp.illumina.com/Homo_sapiens/UCSC/hg19/Homo_sapiens_UCSC_hg19.tar.gz
* wget ftp://igenome:G3nom3s4u@ftp.illumina.com/Mus_musculus/UCSC/mm9/Mus_musculus_UCSC_mm9.tar.gz
* tar -zxvf Homo_sapiens_UCSC_hg19.tar.gz
* tar -zxvf Mus_musculus_UCSC_mm9.tar.gz
 
* BowtieIndex: ~/iGenome/Homo_sapiens/UCSC/hg19/Sequence/BowtieIndex | ~/iGenome/Mus_musculus/UCSC/mm9/Sequence/BowtieIndex
* Annotation: ~/iGenome/Homo_sapiens/UCSC/hg19/Annotation/Genes/genes.gtf
 
===Tophat===
nohup tophat -p 6 --solexa1.3-quals -o ./tophat-g/ -r 250 --mate-std-dev 80 -G ~/iGenome/Homo_sapiens/UCSC/hg19/Annotation/Genes/genes.gtf ~/iGenome/Homo_sapiens/UCSC/hg19/Sequence/BowtieIndex/genome ./seq/s_1_1_Indx1.txt ./seq/s_1_2_Indx1.mod.txt
 
nohup tophat -p 6 --solexa1.3-quals -o ./tophat-g/ -r 250 --mate-std-dev 80 -G ~/iGenome/Homo_sapiens/UCSC/hg19/Annotation/Genes/genes.gtf ~/iGenome/Homo_sapiens/UCSC/hg19/Sequence/BowtieIndex/genome ./seq/s_1_1_Indx2.txt ./seq/s_1_2_Indx2.mod.txt
 
===Samtools and clonalreads removal===
samtools flagstat accepted_hits.bam
 
~/RNAtools/removalclonalreads.pl accepted_hits.bam
 
 
{| {{table}} border=1
| align="center" style="background:#f0f0f0;"|'''Pair-end mapping'''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|'''Tophat'''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|'''removeClonalHits.pl'''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
|-
| ||||file size||fastq reads in||reads out||%||options||accepted hits||reads hits||pair/single||Properly paired||% of total||mapped reads||% of total||pro-rm clonal reads||% of total||% of mappable reads||then uniquely mapped||% of total||potential coverage
|-
| s_1_Indx1||s_1_1_Indx1||left read||2,307,029||2,300,738||0.9973||r:250||1,705,993||734,602||1,122,540||||0.657998011||1,615,384||0.3501||1,191,029||0.2581||0.7373||1,160,079||0.2514||0.04640316
|-
| ||s_1_2_Indx1||right read||2,307,029||2,304,795||0.9990||STD:80||||971,391||583,453||941,458||0.551853378||||||||||||||||
|-
| s_1_Indx1||s_1_1_Indx1||left read||2,307,029||2,300,738||0.9973||(+g)||1,733,904||746,989||1,156,628||||0.667065766||1,643,007||0.3561||1,218,033||0.2640||0.7413||1,186,899||0.2572||0.04747596
|-
| ||s_1_2_Indx1||right read||2,307,029||2,304,795||0.9990||UCSC||||986,915||577,276||965,638||0.556915492||||||||||||||||
|-
| ||||||||||||||||||||||||||||||||||||||
|-
| s_1_Indx2||s_1_1_Indx2||left read||3,078,335||3,069,878||0.9973||r:250||3,882,019||1,679,990||2,750,090||||0.70841745||3,678,987||0.5976||2,642,026||0.4291||0.7181||2,585,255||0.4199||0.1034102
|-
| ||s_1_2_Indx2||right read||3,078,335||3,075,318||0.9990||STD:80||||2,202,029||1,131,929||2,293,768||0.590869854||||||||||||||||
|-
| s_1_Indx2||s_1_1_Indx2||left read||3,078,335||3,069,878||0.9973||(+g)||3,913,815||1,694,018||2,792,224||||0.713427691||3,711,732||0.6029||2,674,065||0.4343||0.7204||2,617,136||0.4251||0.10468544
|-
| ||s_1_2_Indx2||right read||3,078,335||3,075,318||0.9990||UCSC||||2,219,797||1,121,591||2,344,834||0.59911723||||||||||||||||
|}
 
===Cufflinks and cuffdiff===
samtools view -h ../accepted_hits.bam > accepted_hits.sam (have to convert to sam first, i can't feed cufflinks with bam file --- error: segment fault...)
 
cufflinks -g ~/iGenome/Homo_sapiens/UCSC/hg19/Annotation/Genes/genes.gtf -m 250 -s 80 accepted_hits.sam (-g option is probably unnecessary...)
 
 
==RNAseq analysis on HL099==
===Download files from SeqStore===
* transfer lane 2 to 6 files from HL099_2 directory in SeqStore/genome-minor
* cat s_*_Indx1.txt > Indx1.txt
* cat s_*_Indx2.txt > Indx2.txt
 
Libraries (Indx1-6) in HL098 lane 1 and HL099 lane 2 are same. Libraries in HL099 lane 3 to 6 are same, re-adjusted ratio based on Indx2 (set as 1)
 
{| {{table}} border=1
| align="center" style="background:#f0f0f0;"|'''ID'''
| align="center" style="background:#f0f0f0;"|'''Sample'''
| align="center" style="background:#f0f0f0;"|'''file size'''
| align="center" style="background:#f0f0f0;"|'''relative ratio'''
|-
| HL098||s_1_Indx1.txt||1,063,078,988||0.75
|-
| HL098||s_1_Indx2.txt||1,418,503,096||1.00
|-
| HL098||s_1_Indx3.txt||1,126,757,636||0.79
|-
| HL098||s_1_Indx4.txt||1,773,739,660||1.25
|-
| HL098||s_1_Indx5.txt||4,002,812,404||2.82
|-
| HL098||s_1_Indx6.txt||4,927,760,680||3.47
|-
| ||||||10.09
|-
|
|-
| ||||||
|-
| HL099||s_2_Indx1.txt||799,803,430||0.73
|-
| HL099||s_2_Indx2.txt||1,088,795,544||1.00
|-
| HL099||s_2_Indx3.txt||816,515,216||0.75
|-
| HL099||s_2_Indx4.txt||668,702,922||0.61
|-
| HL099||s_2_Indx5.txt||1,263,996,032||1.16
|-
| HL099||s_2_Indx6.txt||1,152,064,234||1.06
|-
| ||||||5.32
|-
| ||||||
|-
| HL099||s_3_Indx1.txt||2,055,529,230||1.58
|-
| HL099||s_3_Indx2.txt||1,300,750,376||1.00
|-
| HL099||s_3_Indx3.txt||1,054,455,616||0.81
|-
| HL099||s_3_Indx4.txt||1,205,324,668||0.93
|-
| ||||||
|-
| ||||||
|-
| HL099||s_4_Indx1.txt||2,075,122,264||1.59
|-
| HL099||s_4_Indx2.txt||1,308,387,872||1.00
|-
| HL099||s_4_Indx3.txt||1,057,664,912||0.81
|-
| HL099||s_4_Indx4.txt||1,212,653,382||0.93
|-
| ||||||
|-
| ||||||
|-
| HL099||s_5_Indx1.txt||1,942,428,848||1.58
|-
| HL099||s_5_Indx2.txt||1,225,815,974||1.00
|-
| HL099||s_5_Indx3.txt||984,527,600||0.80
|-
| HL099||s_5_Indx4.txt||1,129,401,814||0.92
|-
| ||||||
|-
| ||||||
|-
| HL099||s_6_Indx1.txt||2,046,395,112||1.58
|-
| HL099||s_6_Indx2.txt||1,292,688,682||1.00
|-
| HL099||s_6_Indx3.txt||1,044,933,252||0.81
|-
| HL099||s_6_Indx4.txt||1,189,162,084||0.92
|}
 
 
===Tophat===
* nohup tophat -p 6 --solexa1.3-quals -G ~/iGenome/Homo_sapiens/UCSC/hg19/Annotation/Genes/genes.gtf ~/iGenome/Homo_sapiens/UCSC/hg19/Sequence/BowtieIndex/genome ./seq/Indx1.txt
* nohup tophat -p 6 --solexa1.3-quals -G ~/iGenome/Homo_sapiens/UCSC/hg19/Annotation/Genes/genes.gtf ~/iGenome/Homo_sapiens/UCSC/hg19/Sequence/BowtieIndex/genome ./seq/Indx2.txt
* I also tried to feed tophat with the junctions file generated in HL098 (w/ PE mapping strategy) to improve accuracy. However, the bed_to_juncs < junctions.bed > new_list.juncs command didn't work at all! (http://tophat.cbcb.umd.edu/manual.html)
* samtools flagstat accepted_hits.bam --> count the total hits
* ~/RNAtools/removalClonalhits.pl accepted_hits.bam --> total reads based on hits, reads after clonal reads removal, and reads after clonal reads removal plus uniquely mapped in genome (considered as mappable reads)
* potential coverage is based on estimation in the ENCODE standard: mappable reads (uniquely mapped un-clonal reads)/ (2X10^9 mRNAs/cell X ~1500bp/mRNA)
* Big problem in this run (HL099) is clonal reads -- more than 50% of total mapped reads! (the one in HL098 is roughly 28-30%)
 
 
{| {{table}} border=1
| align="center" style="background:#f0f0f0;"|'''run'''
| align="center" style="background:#f0f0f0;"|'''Indx'''
| align="center" style="background:#f0f0f0;"|'''file size'''
| align="center" style="background:#f0f0f0;"|'''QC'''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|'''Tophat'''
| align="center" style="background:#f0f0f0;"|'''removeClonalHits.pl'''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|''''''
|-
| ||||||reads-in||reads-out||% ||accepted hits||mapped reads||% of reads-in||pro-rm clonal reads||% of reads-in||% of mapped reads||then uniquely mapped||% of reads-in||potential coverage
|-
| HL099||Indx1||8,919,278,884||46,366,051||46,356,341||0.9998||23,635,554||20,388,632||0.44||8,637,058||0.19||0.42||8,355,371||0.18||0.33421484
|-
| HL099||Indx2||6,216,438,448||32,314,674||32,306,989||0.9998||27,114,017||23,424,639||0.72||10,904,543||0.34||0.47||10,567,772||0.33||0.42271088
|}
 
===Cuffdiff===
* I never get the original command (listed in cufflinks website) worked! Just as when I run cufflinks, I have to convert bam to sam before running, otherwise, it will show segmentation fault.
 
The original command
* Run Cuffdiff: Take the annotated transcripts for your genome (as GFF or GTF) and provide them to cuffdiff along with the BAM files from TopHat for each replicate: cuffdiff annotation.gtf mock_rep1.bam,mock_rep2.bam \ knockdown_rep1.bam,knockdown_rep2.bam
Run with accepted_hits.bam from tophat
* nohup cuffdiff ~/iGenome/Homo_sapiens/UCSC/hg19/Annotation/Genes/genes.gtf ../Indx1/tophat_out/accepted_hits.bam ../Indx2/tophat_out/accepted_hits.bam &
* [1]+  Segmentation fault      nohup cuffdiff ~/iGenome/Homo_sapiens/UCSC/hg19/Annotation/Genes/genes.gtf ../Indx1/tophat_out/accepted_hits.bam ../Indx2/tophat_out/accepted_hits.bam
 
Run with sorted.acceptedHits.sam from removalClonalhits.pl
* nohup cuffdiff ~/iGenome/Homo_sapiens/UCSC/hg19/Annotation/Genes/genes.gtf ../Indx1/tophat_out/sorted.acceptedHits.sam ../Indx2/tophat_out/sorted.acceptedHits.sam &
* Error: this SAM file doesn't appear to be correctly sorted!
        current hit is at chrX:64289, last one was at chrM:16509
Cufflinks requires that if your file has SQ records in
the SAM header that they appear in the same order as the chromosomes names
in the alignments.
If there are no SQ records in the header, or if the header is missing,
the alignments must be sorted lexicographically by chromsome
name and by position.


Convert bam to sam with header, then cuffdiff
* Comparison of RNAseq analysis programs: [http://genome-tech.ucsd.edu/LabNotes/index.php/Rui:RNAseq_analysis_on_Hiseq111005#Comparison_of_program]
* samtools view -h accepted_hits.bam > accepted_hits.sam
* nohup cuffdiff ~/iGenome/Homo_sapiens/UCSC/hg19/Annotation/Genes/genes.gtf ~/HL099/Indx1/tophat_out/cuffcompare/accepted_hits.sam ~/HL099/Indx2/tophat_out/cuffcompare/accepted_hits.sam &
* gene_exp.diff file has ~1700 genes with significant... repeat direct-cuffdiff on HL098 to compare [636 genes with significance as same as ones using tophat-cufflinks-cuffcompare-cuffdiff methods]
* Why no NGN3????

Latest revision as of 21:02, 31 October 2013

Examples[edit]

KUN's pipeline:
http://genome-tech.ucsd.edu/LabNotes/index.php/RNAseq_pipeline [1]
Jeff's samples:
Kun:LabNotes/Haplotyping/2010-9-17 [2]
Transcriptome:
Kun:LabNotes/SingleCellExpr/2011-6-13 [3]
Haplotyping:
Kun:LabNotes/Haplotyping/2011-6-14 [4]

Data analysis[edit]

RNAseq analysis key notes[edit]

  • iGenome packages [5]
  1. UCSC genome package: [6]
  2. Ensemble genome package: [7]
  • Comparison of RNAseq analysis programs: [8]