2020-10-07 OCT

Terra (book)

Gus ...

PLCO

VCF centered test. What Box hosted VCF should we use to compare notes? Maybe a public VCF and a Box hosted VCF would do it.

What are VCF files?

https://samtools.github.io/hts-specs/VCFv4.3.pdf

Google Nucleus: library to read/write in common genomics formats

https://github.com/google/nucleus

Motivating example

https://colab.research.google.com/github/google/nucleus/blob/master/nucleus/examples/dna_sequencing_error_correction.ipynb

What are protocol buffers?

https://developers.google.com/protocol-buffers

Possible ETL workflow for Box to GCP? See figure below.

TFRecords (https://www.tensorflow.org/tutorials/load_data/tfrecord) -- suitable for big data and gives access to TF Data API (https://www.tensorflow.org/guide/data) to prevent GPU starvation. Compatible with Keras and Estimator APIs.

Creating BigQuery dataset programmatically (https://www.tensorflow.org/io/tutorials/bigquery) then import as data frames in any language of choice. Example with Pandas for Python (https://cloud.google.com/bigquery/docs/bigquery-storage-python-pandas).

VCF with Google Nucleus

Jeya: