Rui:Data Analysis: Difference between revisions

From ZhangLabWiki
Jump to navigation Jump to search
>RuiLiu
>RuiLiu
 
(64 intermediate revisions by the same user not shown)
Line 1: Line 1:
==Examples==
==Examples==
KUN's pipeline:
http://genome-tech.ucsd.edu/LabNotes/index.php/RNAseq_pipeline [http://genome-tech.ucsd.edu/LabNotes/index.php/RNAseq_pipeline]
  Jeff's samples:
  Jeff's samples:
  Kun:LabNotes/Haplotyping/2010-9-17 [http://genome-tech.ucsd.edu/LabNotes/index.php/Kun:LabNotes/Haplotyping/2010-9-17]
  Kun:LabNotes/Haplotyping/2010-9-17 [http://genome-tech.ucsd.edu/LabNotes/index.php/Kun:LabNotes/Haplotyping/2010-9-17]
Line 9: Line 12:
  Kun:LabNotes/Haplotyping/2011-6-14 [http://genome-tech.ucsd.edu/LabNotes/index.php/Kun:LabNotes/Haplotyping/2011-6-14]
  Kun:LabNotes/Haplotyping/2011-6-14 [http://genome-tech.ucsd.edu/LabNotes/index.php/Kun:LabNotes/Haplotyping/2011-6-14]


==Haplotyping 6.21.11==
==Data analysis==
 
*[[Rui:Haplotyping 6.21.11|Haplotyping 6.21.11]]
===Data===
*[[Rui:RNAseq analysis from 7.18.11|RNAseq analysis from 7.18.11]]
ruiliu@genome-miner:~$ cd /home/kunzhang/haplotyping/Data/HL095/
*[[Rui:RNAseq analysis on HL098|RNAseq analysis on HL098]]
 
*[[Rui:RNAseq analysis on HL099|RNAseq analysis on HL099]]
ruiliu@genome-miner:/home/kunzhang/haplotyping/Data/HL095$ ls -l s_4_Indx1.*
*[[Rui:RNAseq analysis on HL101|RNAseq analysis on HL101]]
-rw-r--r-- 1 kunzhang kunzhang 114237942 2011-06-13 22:07 s_4_Indx1.bowtie.pileup
*[[Rui:RNAseq analysis on Hiseq111005|RNAseq_PGCsamples on Hiseq111005]]
-rw-r--r-- 1 kunzhang kunzhang  39977658 2011-06-13 21:28 s_4_Indx1.bowtie.sorted.bam
*[[Rui:Hap analysis on HL108|Hap analysis on HL108]]
-rw-r--r-- 1 kunzhang kunzhang  4442210 2011-06-13 22:05 s_4_Indx1.bowtie.sorted.unique.bam
*[[Rui:RNAseq analysis on HL109|RNAseq analysis on HL109]]
-rw-r--r-- 1 kunzhang kunzhang      278 2011-06-13 21:29 s_4_Indx1.bowtie.variants.txt
*[[Rui:DNAseq analysis on HL114|DNAseq analysis on HL114]]
-rw-r--r-- 1 kunzhang kunzhang      7103 2011-06-14 15:58 s_4_Indx1.hapCompare.txt
*[[Rui:DNAseq analysis on Hiseq120313|DNAseq analysis on Hiseq120313]]
-rw-r--r-- 1 kunzhang kunzhang      9399 2011-06-14 16:56 s_4_Indx1.idioGraph.txt
*[[Rui:DNAseq analysis on HL118|DNAseq analysis on HL118]]
-rw-r--r-- 1 kunzhang kunzhang    675115 2011-06-13 21:29 s_4_Indx1.log
*[[Rui:totoRNAseq on 121226_HL140|totoRNAseq on 121226_HL140]]
-rw-r--r-- 1 kunzhang kunzhang 273522562 2011-06-13 18:08 s_4_Indx1.txt
*[[Rui:totoRNAseq on 130120_HL141|totoRNAseq on 130120_HL141]]
 
*[[Rui:totoRNAseq on 130220_HL144|totoRNAseq on 130220_HL144]]
ruiliu@genome-miner:/home/kunzhang/haplotyping/Data/HL095$ less s_4_Indx1.bowtie.pileup
*[[Rui:totoRNAseq on BL-UHRR-IVT-May28_2013|totoRNAseq on BL-UHRR-IVT-May28_2013]]
1      557399  C      A      25      25      25      1      ^:A    f
*[[Rui:totoRNAseq on 131009_HL161|totoRNAseq on 131009_HL161]]
1      557400  A      A      25      0      25      1      .      f
1      557401  T      T      25      0      25      1      .      f
1      557402  C      C      25      0      25      1      .      f
1      557403  A      A      25      0      25      1      .      c
1      557404  C      C      25      0      25      1      .      g
1      557405  T      T      25      0      25      1      .      g
1      557406  A      A      25      0      25      1      .      g
1      557407  G      G      25      0      25      1      .      g
1      557408  A      A      25      0      25      1      .      e
1      557409  C      C      25      0      25      1      .      f
 
ruiliu@genome-miner:/home/kunzhang/haplotyping/Data/HL095$ more s_4_Indx11.log
/home/kunzhang/softwares/bowtie-latest/bowtie -k 1 -l 32 -n 2 -m 1 --best --quiet -p 4 /home/kunzhang/HsGenome/1KG.ref/HsB36m s_4_Indx
11.txt s_4_Indx11.bowtie.map.1
/home/kunzhang/Exome/Data/scripts/trimTerminalMismatches.pl < s_4_Indx11.bowtie.map.1 > s_4_Indx11.bowtie.map.2
/home/kunzhang/softwares/samtools-0.1.7_x86_64-linux/bowtie2sam.pl  s_4_Indx11.bowtie.map.2 > s_4_Indx11.bowtie.sam
/home/kunzhang/softwares/samtools-0.1.7_x86_64-linux/samtools import /home/kunzhang/HsGenome/1KG.ref/human_b36_male.fa.gz.fai s_4_Indx
11.bowtie.sam s_4_Indx11.bowtie.bam
[sam_header_read2] 114 sequences loaded.
/home/kunzhang/softwares/samtools-0.1.7_x86_64-linux/samtools sort s_4_Indx11.bowtie.bam  s_4_Indx11.bowtie.sorted
java -Xmx2g -jar /home/kunzhang/softwares/picard-tools-1.38/MarkDuplicates.jar ASSUME_SORTED=TRUE REMOVE_DUPLICATES=TRUE INPUT=s_4_In
dx11.bowtie.sorted.bam OUTPUT=s_4_Indx11.bowtie.sorted.unique.bam METRICS_FILE=dup.metric
[Tue Jun 14 15:23:03 PDT 2011] net.sf.picard.sam.MarkDuplicates INPUT=s_4_Indx11.bowtie.sorted.bam OUTPUT=s_4_Indx11.bowtie.sorted.uni
que.bam METRICS_FILE=dup.metric REMOVE_DUPLICATES=true ASSUME_SORTED=true    MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDL
ES_FOR_READ_ENDS_MAP=8000 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 TMP_DIR
=/tmp/kunzhang VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=fals
e CREATE_MD5_FILE=false
INFO 2011-06-14 15:23:03 MarkDuplicates Start of doWork freeMemory: 374846832; totalMemory: 377225216; maxMemory: 1908932608
INFO 2011-06-14 15:23:03 MarkDuplicates Reading input file and constructing read end information.
INFO 2011-06-14 15:23:03 MarkDuplicates Will retain up to 7575129 data points before spilling to disk.
INFO 2011-06-14 15:23:03 MarkDuplicates Assuming input is coordinate sorted.
INFO 2011-06-14 15:23:06 MarkDuplicates Read '''887559''' records. 0 pairs never matched.
INFO 2011-06-14 15:23:07 MarkDuplicates After buildSortedReadEndLists freeMemory: 733397640; totalMemory: 925368320; maxMemory
: 1908932608
INFO 2011-06-14 15:23:07 MarkDuplicates Will retain up to 59654144 duplicate indices before spilling to disk.
INFO 2011-06-14 15:23:07 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2011-06-14 15:23:07 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2011-06-14 15:23:09 MarkDuplicates Sorting list of duplicate records.
INFO 2011-06-14 15:23:09 MarkDuplicates After generateDuplicateIndexes freeMemory: 923987040; totalMemory: 1410072576; maxMemo
ry: 1908932608
INFO 2011-06-14 15:23:09 MarkDuplicates Marking '''837865''' records as duplicates.
INFO 2011-06-14 15:23:09 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2011-06-14 15:23:12 MarkDuplicates Before output close freeMemory: 920285040; totalMemory: 1408237568; maxMemory: 1908932
608
INFO 2011-06-14 15:23:12 MarkDuplicates After output close freeMemory: 876964080; totalMemory: 1364918272; maxMemory: 19089326
08
[Tue Jun 14 15:23:12 PDT 2011] net.sf.picard.sam.MarkDuplicates done.
Runtime.totalMemory()=1364918272
/home/kunzhang/softwares/samtools-0.1.7_x86_64-linux/samtools pileup -c -f /home/kunzhang/HsGenome/1KG.ref/human_b36_male.fa s_4_Indx1
1.bowtie.sorted.unique.bam > s_4_Indx11.bowtie.pileup
Use of uninitialized value in print at /home/kunzhang/Exome/Data/scripts/pileup2variants.pl line 145.
Use of uninitialized value in print at /home/kunzhang/Exome/Data/scripts/pileup2variants.pl line 145.
 
ruiliu@genome-miner:/home/kunzhang/haplotyping/Data/HL095$ /home/kunzhang/softwares/samtools-0.1.7_x86_64-linux/samtools view s_4_Indx11.bowtie.sorted.bam | wc -l
887559
 
ruiliu@genome-miner:/home/kunzhang/haplotyping/Data/HL095$ /home/kunzhang/softwares/samtools-0.1.7_x86_64-linux/samtools view s_4_Indx11.bowtie.sorted.unique.bam | wc -l
49694
 
===Reads===
{| {{table}}
| align="center" style="background:#f0f0f0;"|'''Barcode'''
| align="center" style="background:#f0f0f0;"|'''Reads'''
| align="center" style="background:#f0f0f0;"|'''Unique'''
| align="center" style="background:#f0f0f0;"|'''Percentage'''
|-
| Indx1||163556||15899||9.7
|-
| Indx2||196625||16718||8.5
|-
| Indx3||127140||10123||8.0
|-
| Indx4||189464||13511||7.1
|-
| Indx5||9536||4856||50.9
|-
| Indx6||220670||12779||5.8
|-
| Indx7||185404||12479||6.7
|-
| Indx8||209445||14185||6.8
|-
| Indx9||175999||12100||6.9
|-
| Indx10||212683||13919||6.5
|-
| Indx11||188707||14591||7.7
|-
| Indx12||326082||16300||5.0
|}


==RNAseq analysis key notes==
* iGenome packages [http://tophat.cbcb.umd.edu/igenomes.html]
# UCSC genome package: [http://genome-tech.ucsd.edu/LabNotes/index.php/Rui:Tophat_mapping_10.27.11]
# Ensemble genome package: [http://genome-tech.ucsd.edu/LabNotes/index.php/Rui:Tophat_11.21.11#Re-download_Ensemble_package_from_iGenome_.5B1.5D]


==RNAseq analysis 7.18.11==
* Comparison of RNAseq analysis programs: [http://genome-tech.ucsd.edu/LabNotes/index.php/Rui:RNAseq_analysis_on_Hiseq111005#Comparison_of_program]
# ID1 and ID2 are human samples; ID3-6 are mouse samples. De-multiplexing will give clearer idea on library mapping.
# Combine them into a single file, I can first check the sequencing quality with Galaxy.
# Athurva converted qseq files (55bp) for lane 1 (120) into a single fastq file [ruiliu@genome-miner:~/RNAseq/s_1_1_sequence.txt]
# s_1_1_sequence.txt: 4.4 GB, 50 million lines [wc -l s_1_1_sequence.txt]
# [head -n 1000000 s_1_1_sequence.txt] for the firstmillion lines [ruiliu@genome-miner:~/RNAseq/firstmillion.txt] uploaded to Galaxy (up limit is 2GB)
# Groomer --> fastq summary statistics (250000 fastq reads) --> compute quality statistics (good) --> draw quality score boxplot (40-35) --> Tophat (accepted hits /splice junction) --> flagstat (143296 in total, 0 QC failure, 0 duplicates, 143296 mapped (100.00%), 0 paired in sequencing, 0 read1) --> Mark Duplicate reads (empty??)
# To upload the original file (4GB), zip it to [tar -zcvf ~/new.tar.gz s_1_sequence.txt] new.tar.gz under home directly (permission issue), download to my laptop and install Cyberduck on Mac to upload file via FTP [main.g2.bx.psu.edu] with username [bsos@ucsd.edu] and password [zhanglab]

Latest revision as of 21:02, 31 October 2013

Examples[edit]

KUN's pipeline:
http://genome-tech.ucsd.edu/LabNotes/index.php/RNAseq_pipeline [1]
Jeff's samples:
Kun:LabNotes/Haplotyping/2010-9-17 [2]
Transcriptome:
Kun:LabNotes/SingleCellExpr/2011-6-13 [3]
Haplotyping:
Kun:LabNotes/Haplotyping/2011-6-14 [4]

Data analysis[edit]

RNAseq analysis key notes[edit]

  • iGenome packages [5]
  1. UCSC genome package: [6]
  2. Ensemble genome package: [7]
  • Comparison of RNAseq analysis programs: [8]