Rui:RNAseq analysis on Hiseq111005: Difference between revisions
Jump to navigation
Jump to search
>RuiLiu m (→Matrix) |
>RuiLiu m (→Tophat mapping) |
||
Line 240: | Line 240: | ||
* genome-miner | * genome-miner | ||
Need to | Need to download UCSC dataset from iGenome to genome-miner | ||
command: [http://genome-tech.ucsd.edu/LabNotes/index.php/Rui:RNAseq_analysis_on_HL098#Database_download] |
Revision as of 19:33, 27 October 2011
RNAseq analysis on Hiseq111005
Data
- RNA libraries were used for HL101 run, Indx3-15, [http://genome-tech.ucsd.edu/LabNotes/index.php/Rui_Liu#RNA_libraries_for_Dr._Yi_Zhang_lab
- Libraries to focus on is Indx9|10 (wt_m), Indx11|12 (wt_f), Indx13 (KO_m), Indx14|15 (KO_f)
- Data was stored in genome-miner:/media/Ext6T/111005_SN853/RNAseq
- make shortcut for input data:
ln -s /media/Ext6T/111005_SN853/RNAseq Hiseq111005/
- make a shorcut for output data:
ln -s /media/Ext4T/DataDrive.backup/RL_Scratch/RNAseq RNAseq/
- make copy of sequence file
cp Hiseq111005/* RNAseq/
- combine different files in different lanes into one file for each index:
less *Indx3.txt > Indx3.txt
- Transfer files to meangenemachine, as genome-miner is currently busy
ssh ruiliu@meangenemachine.dynamic.ucsd.edu scp ruiliu@genome-miner:~/RNAseq/Hiseq111005/Indx_seq/* ./
Tophat w/o G correction
Tophat mapping
genome-miner
- PATH issue from Athurva's note:
- Need PATH for bowtie, samtools, as well as GenomeDB??
ruiliu@genome-miner:~/RNAseq/Hiseq111005/tophat$ ~kunzhang/softwares/bowtie-latest/ bowtie bowtie-build-debug bowtie-inspect doc/ indexes/ scripts/ bowtie-build bowtie-debug bowtie-inspect-debug genomes/ reads/ ruiliu@genome-miner:~/RNAseq/Hiseq111005/tophat$ bowtie ^C ruiliu@genome-miner:~/RNAseq/Hiseq111005/tophat$ ls nohup.out tophat_Mm_Indx3 ruiliu@genome-miner:~/RNAseq/Hiseq111005/tophat$ echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games ruiliu@genome-miner:~/RNAseq/Hiseq111005/tophat$ PATH=/home/kunzhang/softwares/ abyss-1.2.5/ Dindel/ QuEST_2.4/ audy-stitch-db9e338/ dindel-1.01-linux.tar.gz QuEST_2.4.tar.gz audy-stitch-db9e338.tar.gz dindel-1.01-python/ README.txt beagle.jar GenomeAnalysisTK-1.0.3864/ samtools-0.1.12a/ blast-2.2.20/ GenomeAnalysisTK-1.0.4905/ samtools-0.1.7_x86_64-linux/ blast-2.2.20-x64-linux.tar.gz GenomeAnalysisTK-1.0.5083/ samtools-0.1.7_x86_64-linux.tar blat_34/ GenomeAnalysisTK-latest/ samtools-latest/ bowtie-0.12.7/ GenomeAnalysisTK-latest.tar SegSeq_1.0.1/ bowtie-latest/ gm_key_64.tar SegSeq_1.0.1.tar.gz brat-1.2.2/ greatTools/ SHERA/ brat-1.2.2.tar.gz greatTools.tar.gz SHERA_files.tar.gz bwa-0.5.8c/ Homo_sapiens_UCSC_hg19.tar.gz SNVMix2-0.11.8-r3/ bwa-0.5.9/ human_empty.bam SNVMix2-0.11.8-r3.tar.gz bwa-0.5.9.tar impute_v2.1.2_x86_64_static.tgz soap2.20release/ bwa-latest/ jksrc/ SOAPdenovo_Release1.04/ cgatools-1.3.0.9-docs/ jksrc.zip sratoolkit.2.1.6-centos_linux64/ cgatools-1.3.0.9-docs.tar.gz macs_1.4.1.deb sratoolkit.2.1.6-centos_linux64.tar.gz cgatools-1.3.0.9-linux-x86_64/ metaGene/ stampy-1.0.8/ cgatools-1.3.0.9-linux-x86_64.tar.gz MetaGeneMark_linux64/ stampy-latest.tgz cnver-0.7.2/ MetaGeneMark_linux64.tar.gz taoliu-MACS-7268e40/ cnver-0.7.2.tar.gz mga_ia64.tar taoliu-MACS-v2.0.7-11-g7268e40.tar.gz CNVnator/ ncbi-blast-2.2.24+/ tophat-1.2.0.Linux_x86_64/ CNVnator_v0.2.2.zip OLB-1.8.0/ tophat-1.3.1.Linux_x86_64/ cufflinks-1.0.3.Linux_x86_64/ OLB-1.9.3/ tophat-1.3.1.Linux_x86_64.tar.gz cufflinks-1.0.3.Linux_x86_64.tar.gz OLB-1.9.3.tar.tar.gz tophat-latest/ cufflinks-1.1.0.Linux_x86_64/ Phrap/ trinityrnaseq_r2011-07-13/ cufflinks-1.1.0.Linux_x86_64.tar.gz Phrap-distrib.tar.Z trinityrnaseq_r2011-07-13.tgz cufflinks-latest/ phred-dist-020425.c-acd.tar.Z velvet_1.0.18/ Cython-0.15/ picard-tools-1.38/ Cython-0.15.tar.gz picard-tools-latest/ ruiliu@genome-miner:~/RNAseq/Hiseq111005/tophat$ PATH=/home/kunzhang/softwares/^C ruiliu@genome-miner:~/RNAseq/Hiseq111005/tophat$ PATH=/home/kunzhang/softwares/bowtie-latest:$PATH /home/kunzhang/softwares/tophat-latest/tophat -p 8 --solexa1.3-quals -o tophat_Mm_Indx3 -G /GenomeDB/UCSC/Mus_musculus/mm9/Annotation/Genes/genes.gtf /GenomeDB/UCSC/Mus_musculus/mm9/Sequence/BowtieIndex/genome ../Indx_seq/Indx3.txt
- Add both bowtie and samtools to PATH:
PATH=/home/kunzhang/softwares/bowtie-latest:$PATH PATH=/home/kunzhang/softwares/samtools-latest:$PATH /home/kunzhang/softwares/tophat-latest/tophat -p 6 --solexa1.3-quals -o tophat_Indx9 /GenomeDB/UCSC/Mus_musculus/mm9/Sequence/BowtieIndex/genome ../Indx_seq/Indx9.txt
- Dr. Zhang fixed the problem of PATH, then simply run tophat under my directory:
tophat -p 6 --solexa1.3-quals -o tophat_Indx9 /GenomeDB/UCSC/Mus_musculus/mm9/Sequence/BowtieIndex/genome ../Indx_seq/Indx9.txt
meangenemachine
- tophat with G correction for major mRNA, based on data analysis on HL098, mapped reads from G correction are more (~100k) than ones w/o G correction
- Tophat without G correction for non-coding RNAs, plan as follows:
Here is my recommendation for your analysis: (1) Perform tophat mapping without any gene model. (2) Perform cuffdiff analysis using the UCSC gene model (like you did before) to look at protein coding genes. (3) Perform cuffdiff analysis using the Ensembl gene model to look at both coding and noncoding genes (you can compare the coding ones between (2) and (3) to check the consistency. (4) For functional annotation of LincRNAs that we don’t know too much about, perhaps you can use the GREAT analysis (http://great.stanford.edu/public/cgi-bin/greatWeb.php) , because most of the LincRNAs act in a cis- manner.
Ensemble reference /GenomeDB/MmGenome/Mus_musculus.NCBIM37.64.gtf
mkdir tophat_Indx15 nohup tophat -p 6 --solexa1.3-quals ~/iGenome/Mus_musculus/UCSC/mm9/Sequence/BowtieIndex/genome ../Indx15.txt & mkdir tophat_Indx15-g nohup tophat -p 6 --solexa1.3-quals -G ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf ~/iGenome/Mus_musculus/UCSC/mm9/Sequence/BowtieIndex/genome ../Indx15.txt &
bowtie.left_kept_reads.fixmap.log
- As proxy of mapping rate.
- Two issues: 1. reads spinning multiple exons can not be included 2. clonal reads included
- Total alignment can be calculated:
samtools flagstat accepted_hits.bam awk '{print $1}' accepted_hits.bam | wc -l
- Total reads can be estimated as
awk '{print $1}' accepted_hits.bam | sort | uniq | wc -l
- However, total reads reach to 95% of total processed reads???
' | ' | bowtie.left_kept_reads.fixmap.log | ' | ' | ' | ' | report.log | ' |
processed | aligned reads | percentage | failed reads | reads sup. -m | happy splice reads | percentage | ||
E13.5 wt_m1 | Indx5 | 32,990,203 | 20,649,504 | 0.6259 | 12,115,555 | 225,144 | 134,070 | 0.6300 |
E13.5 wt_f1 | Indx6 | 27,141,595 | 16,807,921 | 0.6193 | 10,178,028 | 155,646 | 139,151 | 0.6244 |
E13.5 wt_m1 | Indx9 | 46,375,424 | 29,145,806 | 0.6285 | 16,865,642 | 363,976 | 143,742 | 0.6316 |
E13.5 wt_m2 | Indx10 | 36,895,622 | 23,534,232 | 0.6379 | 13,046,273 | 315,117 | 144,857 | 0.6418 |
E13.5 wt_f1 | Indx11 | 41,338,700 | 26,193,747 | 0.6336 | 14,856,837 | 288,116 | 148,068 | 0.6372 |
E13.5 wt_f2 | Indx12 | 50,183,035 | 32,476,668 | 0.6472 | 17,347,409 | 358,958 | 152,887 | 0.6502 |
E13.5 KO_m1 | Indx13 | 34,342,309 | 22,409,676 | 0.6525 | 11,660,995 | 271,638 | 133,327 | 0.6564 |
E13.5 KO_f1 | Indx14 | 40,773,127 | 26,933,424 | 0.6606 | 13,505,587 | 334,116 | 135,519 | 0.6639 |
E13.5 KO_f2 | Indx15 | 44,644,927 | 29,356,424 | 0.6576 | 14,891,518 | 396,985 | 147,430 | 0.6609 |
Transfer and convert files
Transfer files
- Transfer files to genome-miner from meangenemachine
Transfer files to ruiliu@genome-miner:~/RNAseq/Hiseq111005/tophat: scp ruiliu@meangenemachine.dynamic.ucsd.edu:~/Hiseq111005/tophat_Indx15/tophat_out/accepted_hits.bam ./Indx15_accepted_hits.bam scp ruiliu@meangenemachine.dynamic.ucsd.edu:~/Hiseq111005/tophat_Indx14/tophat_out/accepted_hits.bam ./Indx14_accepted_hits.bam scp ruiliu@meangenemachine.dynamic.ucsd.edu:~/Hiseq111005/tophat_Indx13/tophat_out/accepted_hits.bam ./Indx13_accepted_hits.bam scp ruiliu@meangenemachine.dynamic.ucsd.edu:~/Hiseq111005/tophat_Indx12/tophat_out/accepted_hits.bam ./Indx12_accepted_hits.bam scp ruiliu@meangenemachine.dynamic.ucsd.edu:~/Hiseq111005/tophat_Indx11/tophat_out/accepted_hits.bam ./Indx11_accepted_hits.bam scp ruiliu@meangenemachine.dynamic.ucsd.edu:~/Hiseq111005/tophat_Indx10/tophat_out/accepted_hits.bam ./Indx10_accepted_hits.bam scp ruiliu@meangenemachine.dynamic.ucsd.edu:~/Hiseq111005/tophat_Indx5/tophat_out/accepted_hits.bam ./Indx5_accepted_hits.bam
- Transfer files from genome-miner to meangenemachine
ruiliu@meangenemachine-desktop:~/Hiseq111005/tophat_Indx9$ scp ruiliu@genome-miner:~/RNAseq/Hiseq111005/tophat/tophat_Indx9/* ./ scp ruiliu@genome-miner:~/RNAseq/Hiseq111005/tophat/tophat_Indx6/* ./
Convert files
- Convert bam to sam in genome-miner
samtools view -h Indx10_accepted_hits.bam > Indx10_accepted_hits.sam samtools view -h Indx11_accepted_hits.bam > Indx11_accepted_hits.sam nohup samtools view -h Indx12_accepted_hits.bam > Indx12_accepted_hits.sam & nohup samtools view -h Indx13_accepted_hits.bam > Indx13_accepted_hits.sam nohup samtools view -h Indx14_accepted_hits.bam > Indx14_accepted_hits.sam nohup samtools view -h Indx15_accepted_hits.bam > Indx15_accepted_hits.sam nohup samtools view -h Indx5_accepted_hits.bam > Indx5_accepted_hits.sam nohup samtools view -h Indx6_accepted_hits.bam > Indx6_accepted_hits.sam nohup samtools view -h Indx9_accepted_hits.bam > Indx9_accepted_hits.sam
- Convert bam to sam in meangenemachine
nohup samtools view -h ./tophat_Indx10/tophat_out/accepted_hits.bam > ./tophat_Indx10/tophat_out/accepted_hits.sam & nohup samtools view -h ./tophat_Indx11/tophat_out/accepted_hits.bam > ./tophat_Indx11/tophat_out/accepted_hits.sam & nohup samtools view -h ./tophat_Indx12/tophat_out/accepted_hits.bam > ./tophat_Indx12/tophat_out/accepted_hits.sam & nohup samtools view -h ./tophat_Indx13/tophat_out/accepted_hits.bam > ./tophat_Indx13/tophat_out/accepted_hits.sam & nohup samtools view -h ./tophat_Indx14/tophat_out/accepted_hits.bam > ./tophat_Indx14/tophat_out/accepted_hits.sam & nohup samtools view -h ./tophat_Indx15/tophat_out/accepted_hits.bam > ./tophat_Indx15/tophat_out/accepted_hits.sam & nohup samtools view -h ./tophat_Indx5/tophat_out/accepted_hits.bam > ./tophat_Indx5/tophat_out/accepted_hits.sam & nohup samtools view -h ./tophat_Indx6/tophat_out/accepted_hits.bam > ./tophat_Indx6/tophat_out/accepted_hits.sam & nohup samtools view -h ./tophat_Indx9/tophat_out/accepted_hits.bam > ./tophat_Indx9/tophat_out/accepted_hits.sam &
Cuffdiff
- Cuffdiff comparison in Ensemble (genome-miner):
- /GenomeDB/MmGenome/Mus_musculus.NCBIM37.64.gtf only differs from UCSC gene.gtf in Chr. ID (eg. 8 vs Chr8)
- Dr. Zhang ran a script to change to Chr.ID
cd cuffdiff_Ensemble cuffdiff /GenomeDB/MmGenome/Mus_musculus.NCBIM37.64.gtf -N -p 8 -o cuffDiff_wtE13.5f_vs_KOE13.5f ../topha/Indx6_accepted_hits.sam,../tophat/Indx11_accepted_hits.sam,../tophat/Indx12_accepted_hits.sam ../tophat/Indx14_accepted_hits.sam,../tophat/Indx15_accepted_hits.sam cuffdiff /GenomeDB/MmGenome/Mus_musculus.NCBIM37.64.gtf -N -p 8 -o cuffDiff_wtE13.5m_vs_KOE13.5m ../tophat/Indx5_accepted_hits.sam,../tophat/Indx9_accepted_hits.sam,../tophat/Indx10_accepted_hits.sam ../tophat/Indx13_accepted_hits.sam
- Cuffdiff comparison in UCSC (meangenemachine):
cd cuffdiff_UCSC cuffdiff ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf -N -p 8 -o cuffDiff_wtE13.5f_vs_KOE13.5f ../tophat_Indx6/tophat_out/accepted_hits.bam,../tophat_Indx11/tophat_out/accepted_hits.bam,../tophat_Indx12/tophat_out/accepted_hits.bam ../tophat_Indx14/tophat_out/accepted_hits.bam,../tophat_Indx15/tophat_out/accepted_hits.bam cuffdiff ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf -N -p 8 -o cuffDiff_wtE13.5m_vs_KOE13.5m ../tophat_Indx5/tophat_out/accepted_hits.bam,../tophat_Indx9/tophat_out/accepted_hits.bam,../tophat_Indx10/tophat_out/accepted_hits.bam ../tophat_Indx13/tophat_out/accepted_hits.bam
- Transfer files in meangenemachine to laptop
scp ruiliu@meangenemachine.dynamic.ucsd.edu:~/Hiseq111005/cuffdiff_UCSC/cuffDiff_wtE13.5f_vs_KOE13.5f/gene_exp.diff ./Desktop/wtE13.5f_vs_KOE13.5f_gene_exp.diff_UCSC scp ruiliu@meangenemachine.dynamic.ucsd.edu:~/Hiseq111005/cuffdiff_UCSC/cuffDiff_wtE13.5m_vs_KOE13.5m/gene_exp.diff ./Desktop/wtE13.5m_vs_KOE13.5m_gene_exp.diff_UCSC
- Transfer files in genome-miner to laptop
scp ruiliu@genome-miner:~/RNAseq/Hiseq111005/cuffdiff_Ensemble/cuffDiff_wtE13.5f_vs_KOE13.5f/gene_exp.diff ./Desktop/wtE13.5f_vs_KOE13.5f_gene_exp.diff_ensemble scp ruiliu@genome-miner:~/RNAseq/Hiseq111005/cuffdiff_Ensemble/cuffDiff_wtE13.5m_vs_KOE13.5m/gene_exp.diff ./Desktop/wtE13.5m_vs_KOE13.5m_gene_exp.diff_ensemble
Cufflinks
- genes.FPKM_tracking for each Indx is used to generate a matrix
- prepare a matrix for gene set analysis
cufflinks -g ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf -o Indx5_cufflinks ../tophat_Indx5/tophat_out/accepted_hits.sam nohup cufflinks -g ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf -o Indx6_cufflinks ../tophat_Indx6/tophat_out/accepted_hits.sam & nohup cufflinks -g ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf -o Indx9_cufflinks ../tophat_Indx9/tophat_out/accepted_hits.sam & nohup cufflinks -g ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf -o Indx10_cufflinks ../tophat_Indx10/tophat_out/accepted_hits.sam & nohup cufflinks -g ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf -o Indx11_cufflinks ../tophat_Indx11/tophat_out/accepted_hits.sam & nohup cufflinks -g ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf -o Indx12_cufflinks ../tophat_Indx12/tophat_out/accepted_hits.sam & nohup cufflinks -g ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf -o Indx13_cufflinks ../tophat_Indx13/tophat_out/accepted_hits.sam & nohup cufflinks -g ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf -o Indx14_cufflinks ../tophat_Indx14/tophat_out/accepted_hits.sam & nohup cufflinks -g ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf -o Indx15_cufflinks ../tophat_Indx15/tophat_out/accepted_hits.sam &
nohup cufflinks -g ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf -o Indx3_cufflinks ../tophat_Indx3/tophat_out/accepted_hits.sam & nohup cufflinks -g ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf -o Indx4_cufflinks ../tophat_Indx4/tophat_out/accepted_hits.sam & nohup cufflinks -g ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf -o Indx7_cufflinks ../tophat_Indx7/tophat_out/accepted_hits.sam & nohup cufflinks -g ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf -o Indx8_cufflinks ../tophat_Indx8/tophat_out/accepted_hits.sam &
- Transfer to genome-miner
nohup scp -r ruiliu@meangenemachine.dynamic.ucsd.edu:~/Hiseq111005/cufflinks_UCSC ./cufflinks_UCSC
Matrix
- Modify Dr. Zhang's script by replacing folder names (such as “tophat_Mm_Indx3” -> “Indx3_cufflinks)
- Under cufflinks_UCSC folder, nano and paste modified script in, save "Combine_geneTracking.pl" (automatically save as -rw-r--r-- in current directory)
- Executive: chomd 755 Combine_geneTracking.pl
- Run: ./Combine_geneTracking.pl > gene_expression_matrix.txt
Problem: I ran tophat w/o G correction, so that output of cufflinks (genes.FPKM_tracking) has no gene annotation. Have to repeat tophat with G correction
Tophat w/ G correction
Tophat mapping
- meangenemachine
nohup tophat -p 6 --solexa1.3-quals -o Indx15_tophat-G -G ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf ~/iGenome/Mus_musculus/UCSC/mm9/Sequence/BowtieIndex/genome ../Indx15.txt & nohup tophat -p 6 --solexa1.3-quals -o Indx14_tophat-G -G ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf ~/iGenome/Mus_musculus/UCSC/mm9/Sequence/BowtieIndex/genome ../Indx14.txt & nohup tophat -p 6 --solexa1.3-quals -o Indx13_tophat-G -G ~/iGenome/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf ~/iGenome/Mus_musculus/UCSC/mm9/Sequence/BowtieIndex/genome ../Indx13.txt &
- genome-miner
Need to download UCSC dataset from iGenome to genome-miner command: [1]