Benchmarks

Data sets for benchmarking variant callers

Manufacturing validation sets (2mar)

svclassify - builds a set of validation calls using a pedigree to enforce mendelian laws

• 2676 deletions, 68 insertions
• does it recognize complex variants?

gindel, speedseq, and manta all use some type of mendelian enforcement to select validation variants

Tumor/normal pairs

Barcelona/ICGC Real CLL/ML tumor/normal (gold set vcfs available, fastq mostly downloaded)

Craig/TGRI COLO829 tumor/normal cell line (all data waiting on agreement)

Horizon/Acrometrix/SeraCare engineered tumor/normal

WashU AML real tumor/normal (all data waiting on agreement)

DREAM synthetic tumor/normal (IS#1-4 fastq, 10xnormal/tumor fastq, IS#1-5 vcfs currently available)

Pedigree

NIST GiaB trios precisionFDA results (bams, consensus vcfs, fastq available)

Sanger Institute CEPH trio (bams, vcfs available)

Illumina Platinum Genome (bams, fastqs, and vcfs available)

Structure Variation validation data sets (2mar)

2017 pacbio/bionano alternate NA12878 assembly

• SVlib on the PacBio assemblies established a validation set of 4,482 deletions, 4,278 duplications, 488 inversions, 235,608 small deletions (≤ 50 bp) and 205,791 small insertions (≤ 50 bp)
• supplemented with 7,959 deletions that were identified by Pendleton using PacBio reads, Bio Nano assemblies, and the PBHoney detection tool.
  • 40 inversions, 80 PCR validated indels (200-5kbp), 94 large SVs (ins, del, tandemrepeat),
  • not sure why only the deletions were used from this data set
• Also used for validation by 2016 Svelter

2016 10XG/bionano/illumina hybrid NA12878 assembly

• used as a reference standard by valor
• more bionano variants referred to by valor

2015 Manta HCC1954 breast cancer cell line (original pub)

• original SV characterization has 244 structural variants
• manta uses CEPH 1463 (na12878) pedigree to generate their validation set - a variant should appear in 3+ samples without violating mendelian inheritance rules to be considered truth. manta finds 315 structural variants. Manta authors find 100 inversions, 87 translocations, 60 dups 10k+, 56 dels 10k+, 12 dels 1-10k, and compare Manta to Delly on them.
• Should be available in the data commons, but haven't found it yet. Truth calls are in Cosmic v70.
• characterized 1, 2,

HX1 Chinese genome

• vs GRCh38 genome assembly has 9,891 deletions and 10,284 insertions
• used by Canu assembler

2014 CHM1 haploid human genome

• lots of known structural variants (svelter)
• also used for validation by wham

"decode" icelandic

WashU AML (see Tumor/Normal Pairs section)

• BreaKmer - studies selected, specific genes with known e.g. translocations

NZYGMN (HX1?)

• human resequencing data set, can't find it. Does BGI have human reference assemblies we can use?
• used by breakseek, giving credit to Zhongsheng Sun from BGI

svclassify

• implements a mendelian validation strategy on trio data to get truth calls
• similar to validation strategies of gindel, speedseq, manta

2016 Illumina Platinum Genome

• lumpy (old version), gindel
• 
  

DREAM synthetic mutation calling challenges 3 & 4

• vardict

2014 TCGA lung carcinoma

• vardict - KRAS, EGFR, BRAF, PIK3CA and MET genes from 230 patients, compare to firehose and known mutants

2012 ICGC pediatric brain tumor

• delly

2011 1kGP

• 22,025 deletions and 6,000 additional SVs, including insertions and tandem duplications
• tigra, meerkat, lumpy, gindel, raptr-sv

2010 NA12878

• deletions and inversions validated by fosmid, deletions validated here as well
• ingap-sv, gasvpro, lumpy

2010 Kidd deletions

• 562 sequenced deletion breakpoints in 8 cell-line samples: long insert (~48 kb) fosmid end
• tigra

2010 Sanger CNV breakpoints

• multiply validated
• tigra

2009 LifeTech

2008 NA18507

• gasvpro, clever, softsearch, meerkat, indelminer
• softsearch adds simulated SVs to the known ones

2007 HuRef (Venter)

• gasvpro (simulated), clever (simulated)

2004 DbGV

• meerkat