Daniel:Notebook/GenomeMiner/2013-9-19: Difference between revisions

From ZhangLabWiki
Jump to navigation Jump to search
>Djacobse
>Djacobse
 
(9 intermediate revisions by the same user not shown)
Line 2: Line 2:


[[Daniel:Notebook/GenomeMiner|Back to Calendar]]
[[Daniel:Notebook/GenomeMiner|Back to Calendar]]
==Update 09/19/2013==
The following is a table summarizing all of the data collected from mock sequencing reads so far.  Tables and graphs to follow.
{| class="wikitable" <hiddentext>generated with [[:de:Wikipedia:Helferlein/VBA-Macro for EXCEL tableconversion]] V1.8</hiddentext>
|- style="font-size:12pt"
| align="center" width="183" height="15"  valign="bottom" | &nbsp;
| align="center" width="60"  valign="bottom" | &nbsp;
| align="center" width="67"  valign="bottom" | &nbsp;
| align="center" width="58"  valign="bottom" | &nbsp;
| align="center" width="65"  valign="bottom" | &nbsp;
| align="center" width="65"  valign="bottom" | &nbsp;
| align="center" width="65"  valign="bottom" | &nbsp;
| align="center" width="71"  valign="bottom" | &nbsp;
| width="74" align="center" valign="bottom" | Measured
| align="center" width="65"  valign="bottom" | &nbsp;
| align="center" width="81"  valign="bottom" | &nbsp;
| width="65" align="center" valign="bottom" | Actual
| align="center" width="65"  valign="bottom" | &nbsp;
|- style="font-size:12pt;font-weight:bold" align="center" valign="bottom"
| height="30" | Name
| Date
| Base Quality
| 0 Aligned
| 1 Aligned
| >1 Aligned
| Total Aligned %
| Substitution Rate
| Insertion Rate
| Deletion Rate
| Substitution Rate
| Insertion Rate
| Deletion Rate
|- style="font-size:12pt"
|style="font-weight:bold" height="15"  valign="bottom" | Perfect Reads
| align="center" valign="bottom" | [[Daniel:Notebook/GenomeMiner/2013-9-11|9/11/13]]
| align="center" valign="bottom" | Random
| align="center" align="center" | 0
| align="center" align="center" | 99.79
| align="center" align="center" | 0.21
| align="center" align="center" | 100
| align="center" align="center" valign="bottom" | 0
| align="center" align="center" valign="bottom" | 0
| align="center" align="center" valign="bottom" | 0
| align="center" align="center" valign="bottom" | 0
| align="center" align="center" valign="bottom" | 0
| align="center" align="center" valign="bottom" | 0
|- style="font-size:12pt"
|style="font-weight:bold" height="15"  valign="bottom" | Perfect Reads, Perfect Quality
| align="center" valign="bottom" | [[Daniel:Notebook/GenomeMiner/2013-9-12|9/12/13]]
| align="center" valign="bottom" | Perfect
| align="center" align="center" | 0
| align="center" align="center" | 99.8
| align="center" align="center" | 0.2
| align="center" align="center" | 100
| align="center" align="center" valign="bottom" | 0
| align="center" align="center" valign="bottom" | 0
| align="center" align="center" valign="bottom" | 0
| align="center" align="center" valign="bottom" | 0
| align="center" align="center" valign="bottom" | 0
| align="center" align="center" valign="bottom" | 0
|- style="font-size:12pt"
|style="font-weight:bold" height="15"  valign="bottom" | 1% Substitutions
| align="center" valign="bottom" | [[Daniel:Notebook/GenomeMiner/2013-9-16|9/16/13]]
| align="center" valign="bottom" | Random
| align="center" align="center" valign="bottom" | 2.19
| align="center" align="center" valign="bottom" | 97.63
| align="center" align="center" valign="bottom" | 0.18
| align="center" align="center" valign="bottom" | 97.81
| align="center" align="center" | 0.885
| align="center" align="center" | 4.3eE-05
| align="center" align="center" | 2.09E-06
| align="center" align="center" | 1
| align="center" align="center" | 0
| align="center" align="center" | 0
|- style="font-size:12pt"
|style="font-weight:bold" height="15"  valign="bottom" | Mimic Data
| align="center" valign="bottom" | [[Daniel:Notebook/GenomeMiner/2013-9-16|9/16/13]]
| align="center" valign="bottom" | Random
| align="center" align="center" | 4.53
| align="center" align="center" | 95.3
| align="center" align="center" | 0.17
| align="center" align="center" | 95.47
| align="center" align="center" | 0.887
| align="center" align="center" | 0.195
| align="center" align="center" | 5.52E-04
| align="center" align="center" | 1.01
| align="center" align="center" | 0.26
| align="center" align="center" | 0.05
|- style="font-size:12pt"
|style="font-weight:bold" height="15"  valign="bottom" | High Deletions
| align="center" valign="bottom" | [[Daniel:Notebook/GenomeMiner/2013-9-17|9/17/13]]
| align="center" valign="bottom" | Random
| align="center" align="center" | 17.68
| align="center" align="center" | 82.18
| align="center" align="center" | 0.14
| align="center" align="center" | 82.32
| align="center" align="center" | 0.255
| align="center" align="center" | 0.252
| align="center" align="center" | 0.016
| align="center" align="center" | 0.05
| align="center" align="center" | 0.26
| align="center" align="center" | 1.01
|- style="font-size:12pt"
|style="font-weight:bold" height="15"  valign="bottom" | Mimic Errors Bases 30 to 50
| align="center" valign="bottom" | [[Daniel:Notebook/GenomeMiner/2013-9-17|9/17/13]]
| align="center" valign="bottom" | Random
| align="center" align="center" | 1.16
| align="center" align="center" | 98.65
| align="center" align="center" | 0.19
| align="center" align="center" | 98.84
| align="center" align="center" | 0.403
| align="center" align="center" | 0.129
| align="center" align="center" | 2.98E-04
| align="center" align="center" | 0.42
| align="center" align="center" | 0.15
| align="center" align="center" | 0.03
|- style="font-size:12pt"
|style="font-weight:bold" height="30"  valign="bottom" | Mimic Data Bases 30 to 50, Perfect Base Quality
| align="center" | [[Daniel:Notebook/GenomeMiner/2013-9-18|9/18/13]]
| align="center" | Perfect
| align="center" align="center" | 0.4
| align="center" align="center" | 99.41
| align="center" align="center" | 0.19
| align="center" align="center" | 99.6
| align="center" align="center" | 0.435
| align="center" align="center" | 0.091
| align="center" align="center" | 1.82E-04
| align="center" align="center" | 0.42
| align="center" align="center" | 0.11
| align="center" align="center" | 0.02
|- style="font-size:12pt"
|style="font-weight:bold" height="15"  valign="bottom" | Mimic Data (Ref Original Full)
| align="center" valign="bottom" | [[Daniel:Notebook/GenomeMiner/2013-9-18|9/18/13]]
| align="center" valign="bottom" | Random
| align="center" align="center" | 3.6
| align="center" align="center" | 96.08
| align="center" align="center" | 0.32
| align="center" align="center" | 96.4
| align="center" align="center" | 0.91
| align="center" align="center" | 0.188
| align="center" align="center" | 0.028
| align="center" align="center" | 1.01
| align="center" align="center" | 0.26
| align="center" align="center" | 0.05
|- style="font-size:12pt"
|style="font-weight:bold" height="15"  valign="bottom" | Mimic Data (Ref Revcomp Full)
| align="center" valign="bottom" | [[Daniel:Notebook/GenomeMiner/2013-9-18|9/18/13]]
| align="center" valign="bottom" | Random
| align="center" align="center" | 3.6
| align="center" align="center" | 96.08
| align="center" align="center" | 0.32
| align="center" align="center" | 96.4
| align="center" align="center" | 0.91
| align="center" align="center" | 0.188
| align="center" align="center" | 0.028
| align="center" align="center" | 1.01
| align="center" align="center" | 0.26
| align="center" align="center" | 0.05
|- style="font-size:12pt"
|style="font-weight:bold" height="15"  valign="bottom" | Mimic Data (Ref Original Read)
| align="center" valign="bottom" | [[Daniel:Notebook/GenomeMiner/2013-9-18|9/18/13]]
| align="center" valign="bottom" | Random
| align="center" align="center" | 4.48
| align="center" align="center" | 95.34
| align="center" align="center" | 0.17
| align="center" align="center" | 95.52
| align="center" align="center" | 0.887
| align="center" align="center" | 0.195
| align="center" align="center" | 5.48E-04
| align="center" align="center" | 1.01
| align="center" align="center" | 0.26
| align="center" align="center" | 0.05
|- style="font-size:12pt"
|style="font-weight:bold" height="15"  valign="bottom" | Single Error Per Read
| align="center" valign="bottom" | [[Daniel:Notebook/GenomeMiner/2013-9-19|9/19/13]]
| align="center" valign="bottom" | Random
| align="center" align="center" | 0.86
| align="center" align="center" | 98.97
| align="center" align="center" | 0.17
| align="center" align="center" | 99.14
| align="center" align="center" | 1.057
| align="center" align="center" | 0.241
| align="center" align="center" | 2.45E-04
| align="center" align="center" | 1.01
| align="center" align="center" | 0.26
| align="center" align="center" | 0.05
|- style="font-size:12pt"
|style="font-weight:bold" height="15"  valign="bottom" | Mimic Data, Bases 30 to 50
| align="center" valign="bottom" | [[Daniel:Notebook/GenomeMiner/2013-9-20|9/20/13]]
| align="center" valign="bottom" | Random
| align="center" align="center" | 0.27
| align="center" align="center" | 99.54
| align="center" align="center" | 0.19
| align="center" align="center" | 99.73
| align="center" align="center" | 0.431
| align="center" align="center" | 0.09
| align="center" align="center" | 1.63E-04
| align="center" align="center" | 0.42
| align="center" align="center" | 0.11
| align="center" align="center" | 0.02
|}


==Single Error per Read==
==Single Error per Read==
Line 15: Line 232:


'''MATLAB Error Count''':
'''MATLAB Error Count''':
  1493243 substitutions (1.01 pct reads)  
 
  386123 insertions (0.26 pct reads)  
  1494000 substitutions (1.01 pct)
  73712 deletions (0.05 pct reads)
384556 insertions (0.26 pct)
73592 deletions (0.05 pct)
 
===Alignment Results===
 
2959000 reads; of these:
  2959000 (100.00%) were unpaired; of these:
    25319 (0.86%) aligned 0 times
    2928519 (98.97%) aligned exactly 1 time
    5162 (0.17%) aligned >1 times
  99.14% overall alignment rate
 
===Error Counting Results===
 
Error Rate: 1.298%
Error Rate of Insertions: 0.241%
Error Rate of Deletions: 2.45e-04%
Error Rate of Substitutions: 1.057%
 
==Use Real Bases For Deletions==
 
Previous iterations have used the read length sequences as fasta files.  Therefore, deletions add on a random base at the end, since we didn't have data for what the base actually should be.  To verify this approach (maybe correct it), I have added a parameter in "gen_fastq_error_range.m" that can use the full length sequence.  For this approach, the file only prints bases 1:50 in the output file.  Since the reference sequence is longer than 50 bp, if a base is deleted the next base in the reference sequence is automatically added into the output.
 
===Workflow===
 
1. MockHL155_Master.m, Switch 5 (param reflength,full)
2. scp v4s1mockseq_fullref_errormimic.fq djacobse@132.239.135.41:/media/LTS_15T/DEJ_LTS/SeqStore/130628_HL155/mockseq/extdeletions/
3. hl155bash.sh
4. perl imp_count_mismatch.plx
 
'''MATLAB Error Counts'''
 
1495137 substitutions (1.01 pct)
384872 insertions (0.26 pct)  
  74058 deletions (0.05 pct)


===Alignment Results===
===Alignment Results===
Alignment was really low.  I found a bug in the code and I'm going to fix it.

Latest revision as of 21:51, 20 September 2013

Mock HL155 (Started 9/9/2013)[edit]

Back to Calendar

Update 09/19/2013[edit]

The following is a table summarizing all of the data collected from mock sequencing reads so far. Tables and graphs to follow.

                Measured     Actual  
Name Date Base Quality 0 Aligned 1 Aligned >1 Aligned Total Aligned % Substitution Rate Insertion Rate Deletion Rate Substitution Rate Insertion Rate Deletion Rate
Perfect Reads 9/11/13 Random 0 99.79 0.21 100 0 0 0 0 0 0
Perfect Reads, Perfect Quality 9/12/13 Perfect 0 99.8 0.2 100 0 0 0 0 0 0
1% Substitutions 9/16/13 Random 2.19 97.63 0.18 97.81 0.885 4.3eE-05 2.09E-06 1 0 0
Mimic Data 9/16/13 Random 4.53 95.3 0.17 95.47 0.887 0.195 5.52E-04 1.01 0.26 0.05
High Deletions 9/17/13 Random 17.68 82.18 0.14 82.32 0.255 0.252 0.016 0.05 0.26 1.01
Mimic Errors Bases 30 to 50 9/17/13 Random 1.16 98.65 0.19 98.84 0.403 0.129 2.98E-04 0.42 0.15 0.03
Mimic Data Bases 30 to 50, Perfect Base Quality 9/18/13 Perfect 0.4 99.41 0.19 99.6 0.435 0.091 1.82E-04 0.42 0.11 0.02
Mimic Data (Ref Original Full) 9/18/13 Random 3.6 96.08 0.32 96.4 0.91 0.188 0.028 1.01 0.26 0.05
Mimic Data (Ref Revcomp Full) 9/18/13 Random 3.6 96.08 0.32 96.4 0.91 0.188 0.028 1.01 0.26 0.05
Mimic Data (Ref Original Read) 9/18/13 Random 4.48 95.34 0.17 95.52 0.887 0.195 5.48E-04 1.01 0.26 0.05
Single Error Per Read 9/19/13 Random 0.86 98.97 0.17 99.14 1.057 0.241 2.45E-04 1.01 0.26 0.05
Mimic Data, Bases 30 to 50 9/20/13 Random 0.27 99.54 0.19 99.73 0.431 0.09 1.63E-04 0.42 0.11 0.02

Single Error per Read[edit]

Goal of this test is to see if bowtie's alignment results are dependent on the number of errors per read. So far the errors have been completely random, with each base potentially getting a substitution, insertion, or deletion, or even several errors. This new iteration rolls to see if an error will occur at all, and based on its roll also determines which of the three error types it will give.

Workflow[edit]

1. MockHL155_Master.m, Switch 6
2. scp v4s1mockseq_1errperread_mimic.fq djacobse@132.239.135.41:/media/LTS_15T/DEJ_LTS/SeqStore/130628_HL155/mockseq/error1per/
3. hl155bash.sh
4. perl imp_count_mismatch.plx (Matt's error counting script)

MATLAB Error Count:

1494000 substitutions (1.01 pct) 
384556 insertions (0.26 pct) 
73592 deletions (0.05 pct) 

Alignment Results[edit]

2959000 reads; of these:
 2959000 (100.00%) were unpaired; of these:
   25319 (0.86%) aligned 0 times
   2928519 (98.97%) aligned exactly 1 time
   5162 (0.17%) aligned >1 times
99.14% overall alignment rate

Error Counting Results[edit]

Error Rate: 1.298%
Error Rate of Insertions: 0.241%
Error Rate of Deletions: 2.45e-04%
Error Rate of Substitutions: 1.057%

Use Real Bases For Deletions[edit]

Previous iterations have used the read length sequences as fasta files. Therefore, deletions add on a random base at the end, since we didn't have data for what the base actually should be. To verify this approach (maybe correct it), I have added a parameter in "gen_fastq_error_range.m" that can use the full length sequence. For this approach, the file only prints bases 1:50 in the output file. Since the reference sequence is longer than 50 bp, if a base is deleted the next base in the reference sequence is automatically added into the output.

Workflow[edit]

1. MockHL155_Master.m, Switch 5 (param reflength,full)
2. scp v4s1mockseq_fullref_errormimic.fq djacobse@132.239.135.41:/media/LTS_15T/DEJ_LTS/SeqStore/130628_HL155/mockseq/extdeletions/
3. hl155bash.sh
4. perl imp_count_mismatch.plx

MATLAB Error Counts

1495137 substitutions (1.01 pct) 
384872 insertions (0.26 pct) 
74058 deletions (0.05 pct)

Alignment Results[edit]

Alignment was really low. I found a bug in the code and I'm going to fix it.