Best Practices

Recommendations

Loam is a dialect of Scala, a general-purpose programming language. You can write anything you want (mostly; there some esoteric limits, but the basic point stands). Of course, just because one can do something does not mean that one should. To help with this, here are some recommendations from the LoamStream team.

(Note: a general Scala tutorial is beyond the scope of this document. If you'd like one, Twitter's Scala School is good, as is the book Programming in Scala. Scala for the Impatient and Atomic Scala also received favorable reviews, though we haven't read them. There are also many online Scala tutorials that we don't know about.)

Use config files for values that will change between runs.

Fundamentally, we encourage a separation between config files, containing just data, and Loam files, which declare stores and commands, and optionally declare pipeline topologies that depend on data in config files. Loam code can read from any type of file, but LoamStream includes built-in support for HOCON files. For example, given a config file like:

numFoos = 42

fooLabel = "some text"

xEnabled = false

foo {

  bar {

    baz = "xyz"

  labels = ["a", "b", "c"]

  counts = [42, 43, 44]

  bips = [

      bip = 99

      bop = "42"

},

      bip = 42

      bop = "hello"

it could be read in a Loam file like

val myConfigFile: Path = path("path/to/myDatasetConfig.conf")

val config: DataConfig = loadConfig(myConfigFile)

//Check for the presence of a value at a key

val fooIsDefined: Boolean = config.isDefined("foo")

//get values stored at keys:

val numFoos: Int = config.getInt("numFoos")

val label: String = config.getStr("fooLabel")

val isXEnabled: Boolean = config.getBool("xEnabled")

val foo: DataConfig = config.getObj("foo")

//nesting is possible:

val baz: String = config.getObj("foo").getObj("bar").getStr("baz")

//as are lists/arrays:

val labels: Seq[String] = foo.getStrList("labels")

val counts: Seq[Int] = foo.getIntList("counts")

val bips: Seq[DataConfig] = foo.getObjList("bips")

Be declarative in .loam files

Use .loam files to declare commands and stores. .loam files should be as declarative as possible; try to minimize logic in them, and make any logic expression-oriented instead of imperative.

Do this:

for(arrayName <- myConfig.getStrList("arrays")) {

  val input = store.at("in/$arrayName.vcf")

  val output = store.at("out/$arrayName.txt")

  cmd"foo --array $arrayName -i $input -o $output"

NOT this:

val dir = parentDir.listFiles.filter(_.getName.endsWith("foo"))

val arrayNameFile = parentDir.listFiles.filter(_.getName.endsWith("arrays")).head

val rawArrayNames = Files.readContents(arrayNameFile).split("\\s+")

val arrayNames = rawArrayNames.map(_.toUpperCase)

for(arrayName <- arrayNames) {

  val input = store.at("in/$arrayName.vcf")

  val output = store.at("out/$arrayName.txt")

  cmd"foo --array $arrayName -i $input -o $output"

Conditionals are fine, as is building the pipeline topology from config data:

/*

 * With config data like

* {

 *   fooEnabled = true

* }

 * wire up the pipeline one way if fooEnabled is true, and another way otherwise.

*/

val shouldFoo = myConfig.getBool("fooEnabled")

if(shouldFoo) {

  //N cmd"..."s

} else {

  //M slightly different cmd"..."s

or given config data like

blah {

  arrays = [

      name = "bar"

      fooLabels = ["x", "y", "z"]

},

...

      name = "baz"

      fooLabels = ["a", "b"]

},

for(arrayConfig <- myConfig.getObjList("arrays")) {

  val arrayName = arrayConfig.getStr("name")

  val fooLabels = arrayConfig.getStrList("fooLabels")

  for(fooLabel <- fooLabels) {

    cmd"some-command -a $arrayName -l $fooLabel"

Report abuse