Posts

Stop versioning libraries

posted Jan 16, 2018, 10:12 AM by Renato Athaydes   [ updated Jan 16, 2018, 10:18 AM ]

Versioning is one of the most basic concepts software developers have to deal with. But despite the apparent simplicity of the subject, we still haven't got it right!

Let me expand. We are used to a versioning scheme based on a set of numbers and a few qualifiers, so our versions look like 1.2.3 and 1.2.4-rc1. But have you ever stopped to think about what these versions really mean?

When you see a library that is on version 1.2.3, can you confidently claim that it is more mature than another library that is on version 1.0.0? Unfortunately, that's just impossible to tell. From real-world experience, I've come to believe that knowing a version number is almost completely useless. Version numbers are at the whim of library authors... they may not even follow semantic versioning, so version 1.2.0 might be completely different than version 1.1.0... or they might decide to skip a few major versions and go straight from 7 to 10!! But even if the library author follows semver religiously, which version of semver, given that semver itself is versioned?! They may decide to follow simver instead :D!

But seriously, when it comes to libraries (as opposed to apps/products, whose versions are normally meant for final users and marketing purposes, not dependency manager tools), I know you might be shocked by this and think that I have no idea what I am talking about, but try to give me a chance to explain it in the rest of this blog post... after all, one of the initial proponents of this idea was not me, but a really smart guy you may've heard of: Rich Hickey. Perhaps you'll start agreeing with this and join the cause, so in a not-so-distant future we might actually be able to build software confidently, even if we depend on hundreds of interdependent libraries!

This is my request to all library authors:

Don't ever make breaking changes in a library, regardless of the version change!
If breaking changes are unavoidable, change the library name instead.

That's because regardless of which language you use, there's no way to resolve versions correctly if different modules depend on different, incompatible versions of the same library.

I find it funny that some people believe that npm solves that. Or, if you're a little bit older, you might have thought that OSGi did it for Java 18 years ago. Well, they didn't! Because it's impossible.

OSGi will actually throw errors during startup if the two incompatible versions of a library are used by two different bundles and both can "see" each other (so, it actually solved a part of the problem by failing on startup, not runtime, but that doesn't help much and just got OSGi a bad rep), but in a node.js application, things might or might not work! Even though npm includes a module's dependencies into its own node_modules folder (so different consumers may use the correct version they require), it definitely won't work if objects from the two incompatible library versions happen to interact. This will cause difficult to debug runtime errors... that kind of error that only happens on a Saturday night with full moon.

To illustrate the problem, let's imagine we have a Javascript library, say lib1, that creates a tax summary given an income. This library was part of an effort to create tax reports, so the programmer thought it was a good idea to use a formatted String to describe the taxes... something like this:



function createTaxSummary(income) {
return {
"income": "Income = " + income,
"tax": "Tax = " + (income * 0.25),
}
}

exports.createTaxSummary = createTaxSummary;



Another library was used to actually generate a final report. Let's call it lib2. It uses lib1 to calculate the taxes, then returns a nice report:



const lib1 = require('../lib1')

function createReport(taxSummary) {
return "Tax Report\n----------\n" +
taxSummary.income.toUpperCase() + "\n" +
taxSummary.tax.toUpperCase()
}

exports.createReport = createReport;



The programmer made the line that contains the tax amount uppercase because that's the tradition in accounting circles.

The application code developed by another company should just call lib2 to generate the report and print it out, but lib2 did not have a way to add "extra" taxes that can be charged in some states... because that was only noticed when the release was already several weeks late, this feature was hacked into the application code by directly using lib1:



const
lib1 = require('../lib1')
const lib2 = require('../lib2')

const income = 100000;
const extras = 2500;

const summary = lib1.createTaxSummary(income);
var report = lib2.createReport(summary);
const extraTaxes = lib1.createTaxSummary(extras).tax;
report += "\nExtra taxes:\n" + extraTaxes.toUpperCase();

console.log(report);



The report now looked perfect!


Tax Report
----------
INCOME = 100000
TAX = 25000
Extra taxes:
TAX = 625



A few years later (the code was working, so no one had the time to go back "fix" the hack), someone noticed that the createTaxReport function was calculating the taxes directly. That doesn't make any sense! So the programmer changed lib1 so that it would provide a function just to calculate taxes, as well as fixing the tax summary object to contain only numbers, separating its logic from report logic... semantic versioning was being followed, so the version was bumped from 1.23 to 2.0. After all, this was a breaking change:



function calculateTax(income) {
return income * 0.25;
}

function createTaxSummary(income) {
return {
"income": income,
"tax": calculateTax(income),
}
}

exports.createTaxSummary = createTaxSummary;
exports.calculateTax = calculateTax;



The application developer, worried that the application was depending on an old versions of some libraries, decided to upgrade all versions. The change log on lib2 was pretty small, even though it was a major version change. It looked like the tax property was changed to a number, and that was pretty much it.

That's a one-line change.



const
lib1 = require('../lib1')
const lib2 = require('../lib2')

const income = 100000;
const extras = 2500;

const summary = lib1.createTaxSummary(income);
var report = lib2.createReport(summary);
const extraTaxes = lib1.createTaxSummary(extras).tax;
report += "\nExtra taxes:\nTAX = " + extraTaxes;

console.log(report);



Running it didn't work, though!


/lib2/index.js:6
      taxSummary.income.toUpperCase() + "\n" +
                        ^

TypeError: taxSummary.income.toUpperCase is not a function
    at Object.createReport (/lib2/index.js:6:25)
    at Object.<anonymous> (/app/index.js:8:19)
    at Module._compile (module.js:660:30)
    at Object.Module._extensions..js (module.js:671:10)
    at Module.load (module.js:573:32)
    at tryModuleLoad (module.js:513:12)
    at Function.Module._load (module.js:505:3)
    at Function.Module.runMain (module.js:701:10)
    at startup (bootstrap_node.js:193:16)
    at bootstrap_node.js:617:3


The author of lib2 is no longer maintaining it, so lib2 hadn't been updated with the changes. The only way to fix the error was to revert to the old version.

But what if there was a critical security bug in the old version? What if other libraries that were being used also expected the old summary format?

In this simple case (I admit it is contrived and looks artificial... sorry, that's the best simple example I could come up with - but can you be sure that something like this won't happen to you??) you might just have to fork the project(s) and patch it... but that could cost a lot of time. But even if this scenario looks unlikely, with a typical node_modules folder containing nearly 100MB of libraries, I think it should be almost unavoidable that something like this will happen at some point.

Just by using React/Redux, you get 40MB worth of dependencies:


➜  npm install --save react-redux
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN react-redux@5.0.6 requires a peer of react@^0.14.0 || ^15.0.0-0 || ^16.0.0-0 but none is installed. You must install peer dependencies yourself.
npm WARN react-redux@5.0.6 requires a peer of redux@^2.0.0 || ^3.0.0 but none is installed. You must install peer dependencies yourself.

+ react-redux@5.0.6
added 21 packages in 2.573s
➜  npm install --save react
npm WARN react-redux@5.0.6 requires a peer of redux@^2.0.0 || ^3.0.0 but none is installed. You must install peer dependencies yourself.

+ react@16.2.0
added 1 package in 0.419s
➜  npm install --save redux

+ redux@3.7.2
added 2 packages in 0.443s
➜  du -sh node_modules
40M     node_modules



Here's the full dependency tree:


➜  npm ls
lib3@1.0.0 /lib3
├─┬ react@16.2.0
│ ├─┬ fbjs@0.8.16
│ │ ├── core-js@1.2.7
│ │ ├─┬ isomorphic-fetch@2.2.1
│ │ │ ├─┬ node-fetch@1.7.3
│ │ │ │ ├─┬ encoding@0.1.12
│ │ │ │ │ └── iconv-lite@0.4.19
│ │ │ │ └── is-stream@1.1.0
│ │ │ └── whatwg-fetch@2.0.3
│ │ ├── loose-envify@1.3.1 deduped
│ │ ├── object-assign@4.1.1 deduped
│ │ ├─┬ promise@7.3.1
│ │ │ └── asap@2.0.6
│ │ ├── setimmediate@1.0.5
│ │ └── ua-parser-js@0.7.17
│ ├─┬ loose-envify@1.3.1
│ │ └── js-tokens@3.0.2
│ ├── object-assign@4.1.1
│ └─┬ prop-types@15.6.0
│   ├── fbjs@0.8.16 deduped
│   ├── loose-envify@1.3.1 deduped
│   └── object-assign@4.1.1 deduped
├─┬ react-redux@5.0.6
│ ├── hoist-non-react-statics@2.3.1
│ ├─┬ invariant@2.2.2
│ │ └── loose-envify@1.3.1 deduped
│ ├── lodash@4.17.4
│ ├── lodash-es@4.17.4
│ ├── loose-envify@1.3.1 deduped
│ └── prop-types@15.6.0 deduped
└─┬ redux@3.7.2
  ├── lodash@4.17.4 deduped
  ├── lodash-es@4.17.4 deduped
  ├── loose-envify@1.3.1 deduped
  └── symbol-observable@1.1.0


If you think the JavaScript world is insane, so that kind of thing is to be expected, let's see how the Java world compares.

A typical Java web backend will use some kind of framework like Dropwizard. If you want to see how many libraries you'll get just by making that choice, create a build.gradle file with the following contents:



repositories {
jcenter()
}

apply plugin: 'java'

dependencies {
compile "io.dropwizard:dropwizard-core:1.3.0-rc3"
}

task copyLibs(type: Copy) {
into "libs"
from configurations.runtime
}



Now, run the copyLibs task with Gradle and check how heavy the libs folder is:


➜  gradle copyLibs
Starting a Gradle Daemon (subsequent builds will be faster)
Download https://jcenter.bintray.com/io/dropwizard/dropwizard-core/1.3.0-rc3/dropwizard-core-1.3.0-rc3.pom
Download https://jcenter.bintray.com/io/dropwizard/dropwizard-parent/1.3.0-rc3/dropwizard-parent-1.3.0-rc3
.pom

......... hundreds of lines like that .........


BUILD SUCCESSFUL in 27s
1 actionable task: 1 executed
➜  du -sh libs
17M     libs


Well, at least that's better than the Node example, but 17MB of jars is still quite a lot!! Let's see what dependencies are responsible for that:


➜  gradle dep --configuration runtime

> Task :dependencies

------------------------------------------------------------
Root project
------------------------------------------------------------

runtime - Runtime dependencies for source set 'main' (deprecated, use 'runtimeOnly ' instead).
\--- io.dropwizard:dropwizard-core:1.3.0-rc3
     +--- io.dropwizard:dropwizard-util:1.3.0-rc3
     |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
     |    +--- com.google.guava:guava:23.5-jre
     |    |    +--- com.google.code.findbugs:jsr305:1.3.9 -> 3.0.2
     |    |    +--- org.checkerframework:checker-qual:2.0.0
     |    |    +--- com.google.errorprone:error_prone_annotations:2.0.18
     |    |    +--- com.google.j2objc:j2objc-annotations:1.1
     |    |    \--- org.codehaus.mojo:animal-sniffer-annotations:1.14
     |    +--- com.google.code.findbugs:jsr305:3.0.2
     |    \--- joda-time:joda-time:2.9.9
     +--- io.dropwizard:dropwizard-jackson:1.3.0-rc3
     |    +--- com.google.guava:guava:23.5-jre (*)
     |    +--- io.dropwizard:dropwizard-util:1.3.0-rc3 (*)
     |    +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
     |    +--- com.fasterxml.jackson.core:jackson-databind:2.9.3
     |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
     |    |    \--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    +--- com.fasterxml.jackson.datatype:jackson-datatype-guava:2.9.3
     |    |    +--- com.google.guava:guava:18.0 -> 23.5-jre (*)
     |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
     |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.9.3
     |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
     |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
     |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.9.3
     |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
     |    +--- com.fasterxml.jackson.module:jackson-module-parameter-names:2.9.3
     |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
     |    +--- com.fasterxml.jackson.module:jackson-module-afterburner:2.9.3
     |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
     |    +--- com.fasterxml.jackson.datatype:jackson-datatype-joda:2.9.3
     |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
     |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    |    +--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
     |    |    \--- joda-time:joda-time:2.7 -> 2.9.9
     |    \--- org.slf4j:slf4j-api:1.7.25
     +--- io.dropwizard:dropwizard-validation:1.3.0-rc3
     |    +--- io.dropwizard:dropwizard-util:1.3.0-rc3 (*)
     |    +--- org.hibernate:hibernate-validator:5.4.2.Final
     |    |    +--- javax.validation:validation-api:1.1.0.Final
     |    |    +--- org.jboss.logging:jboss-logging:3.3.0.Final
     |    |    \--- com.fasterxml:classmate:1.3.1
     |    +--- org.glassfish:javax.el:3.0.0
     |    +--- org.javassist:javassist:3.22.0-GA
     |    \--- org.slf4j:slf4j-api:1.7.25
     +--- io.dropwizard:dropwizard-configuration:1.3.0-rc3
     |    +--- io.dropwizard:dropwizard-jackson:1.3.0-rc3 (*)
     |    +--- io.dropwizard:dropwizard-validation:1.3.0-rc3 (*)
     |    +--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.9.3
     |    |    +--- org.yaml:snakeyaml:1.18
     |    |    \--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    +--- org.apache.commons:commons-lang3:3.6
     |    \--- org.apache.commons:commons-text:1.1
     |         \--- org.apache.commons:commons-lang3:3.5 -> 3.6
     +--- io.dropwizard:dropwizard-logging:1.3.0-rc3
     |    +--- io.dropwizard:dropwizard-jackson:1.3.0-rc3 (*)
     |    +--- io.dropwizard:dropwizard-validation:1.3.0-rc3 (*)
     |    +--- io.dropwizard.metrics:metrics-logback:4.0.2
     |    |    \--- io.dropwizard.metrics:metrics-core:4.0.2
     |    +--- org.slf4j:slf4j-api:1.7.25
     |    +--- org.slf4j:jul-to-slf4j:1.7.25
     |    |    \--- org.slf4j:slf4j-api:1.7.25
     |    +--- ch.qos.logback:logback-core:1.2.3
     |    +--- ch.qos.logback:logback-classic:1.2.3
     |    |    \--- ch.qos.logback:logback-core:1.2.3
     |    +--- org.slf4j:log4j-over-slf4j:1.7.25
     |    |    \--- org.slf4j:slf4j-api:1.7.25
     |    +--- org.slf4j:jcl-over-slf4j:1.7.25
     |    |    \--- org.slf4j:slf4j-api:1.7.25
     |    \--- org.eclipse.jetty:jetty-util:9.4.8.v20171121
     +--- io.dropwizard:dropwizard-metrics:1.3.0-rc3
     |    +--- io.dropwizard:dropwizard-lifecycle:1.3.0-rc3
     |    |    +--- org.slf4j:slf4j-api:1.7.25
     |    |    +--- com.google.guava:guava:23.5-jre (*)
     |    |    +--- org.eclipse.jetty:jetty-server:9.4.8.v20171121
     |    |    |    +--- javax.servlet:javax.servlet-api:3.1.0
     |    |    |    +--- org.eclipse.jetty:jetty-http:9.4.8.v20171121
     |    |    |    |    +--- org.eclipse.jetty:jetty-util:9.4.8.v20171121
     |    |    |    |    \--- org.eclipse.jetty:jetty-io:9.4.8.v20171121
     |    |    |    |         \--- org.eclipse.jetty:jetty-util:9.4.8.v20171121
     |    |    |    \--- org.eclipse.jetty:jetty-io:9.4.8.v20171121 (*)
     |    |    \--- io.dropwizard:dropwizard-util:1.3.0-rc3 (*)
     |    +--- io.dropwizard:dropwizard-jackson:1.3.0-rc3 (*)
     |    +--- io.dropwizard:dropwizard-validation:1.3.0-rc3 (*)
     |    +--- io.dropwizard.metrics:metrics-core:4.0.2
     |    \--- org.slf4j:slf4j-api:1.7.25
     +--- io.dropwizard:dropwizard-jersey:1.3.0-rc3
     |    +--- io.dropwizard:dropwizard-jackson:1.3.0-rc3 (*)
     |    +--- io.dropwizard:dropwizard-validation:1.3.0-rc3 (*)
     |    +--- io.dropwizard:dropwizard-logging:1.3.0-rc3 (*)
     |    +--- org.glassfish.jersey.core:jersey-server:2.25.1
     |    |    +--- org.glassfish.jersey.core:jersey-common:2.25.1
     |    |    |    +--- javax.ws.rs:javax.ws.rs-api:2.0.1
     |    |    |    +--- javax.annotation:javax.annotation-api:1.2
     |    |    |    +--- org.glassfish.jersey.bundles.repackaged:jersey-guava:2.25.1
     |    |    |    +--- org.glassfish.hk2:hk2-api:2.5.0-b32
     |    |    |    |    +--- javax.inject:javax.inject:1
     |    |    |    |    +--- org.glassfish.hk2:hk2-utils:2.5.0-b32
     |    |    |    |    |    \--- javax.inject:javax.inject:1
     |    |    |    |    \--- org.glassfish.hk2.external:aopalliance-repackaged:2.5.0-b32
     |    |    |    +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
     |    |    |    +--- org.glassfish.hk2:hk2-locator:2.5.0-b32
     |    |    |    |    +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
     |    |    |    |    +--- org.glassfish.hk2.external:aopalliance-repackaged:2.5.0-b32
     |    |    |    |    +--- org.glassfish.hk2:hk2-api:2.5.0-b32 (*)
     |    |    |    |    +--- org.glassfish.hk2:hk2-utils:2.5.0-b32 (*)
     |    |    |    |    \--- org.javassist:javassist:3.20.0-GA -> 3.22.0-GA
     |    |    |    \--- org.glassfish.hk2:osgi-resource-locator:1.0.1
     |    |    +--- org.glassfish.jersey.core:jersey-client:2.25.1
     |    |    |    +--- javax.ws.rs:javax.ws.rs-api:2.0.1
     |    |    |    +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
     |    |    |    +--- org.glassfish.hk2:hk2-api:2.5.0-b32 (*)
     |    |    |    +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
     |    |    |    \--- org.glassfish.hk2:hk2-locator:2.5.0-b32 (*)
     |    |    +--- javax.ws.rs:javax.ws.rs-api:2.0.1
     |    |    +--- org.glassfish.jersey.media:jersey-media-jaxb:2.25.1
     |    |    |    +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
     |    |    |    +--- org.glassfish.hk2:hk2-api:2.5.0-b32 (*)
     |    |    |    +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
     |    |    |    +--- org.glassfish.hk2:hk2-locator:2.5.0-b32 (*)
     |    |    |    \--- org.glassfish.hk2:osgi-resource-locator:1.0.1
     |    |    +--- javax.annotation:javax.annotation-api:1.2
     |    |    +--- org.glassfish.hk2:hk2-api:2.5.0-b32 (*)
     |    |    +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
     |    |    +--- org.glassfish.hk2:hk2-locator:2.5.0-b32 (*)
     |    |    \--- javax.validation:validation-api:1.1.0.Final
     |    +--- org.glassfish.jersey.ext:jersey-metainf-services:2.25.1
     |    |    +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
     |    |    \--- javax.ws.rs:javax.ws.rs-api:2.0.1
     |    +--- org.glassfish.jersey.ext:jersey-bean-validation:2.25.1
     |    |    +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
     |    |    +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
     |    |    +--- org.glassfish.jersey.core:jersey-server:2.25.1 (*)
     |    |    +--- javax.validation:validation-api:1.1.0.Final
     |    |    +--- org.hibernate:hibernate-validator:5.1.3.Final -> 5.4.2.Final (*)
     |    |    \--- javax.ws.rs:javax.ws.rs-api:2.0.1
     |    +--- io.dropwizard.metrics:metrics-jersey2:4.0.2
     |    |    +--- io.dropwizard.metrics:metrics-core:4.0.2
     |    |    \--- io.dropwizard.metrics:metrics-annotation:4.0.2
     |    |         \--- org.slf4j:slf4j-api:1.7.25
     |    +--- com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:2.9.3
     |    |    +--- com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:2.9.3
     |    |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
     |    |    \--- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:2.9.3
     |    |         +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
     |    |         +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    |         \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
     |    +--- org.glassfish.jersey.containers:jersey-container-servlet:2.25.1
     |    |    +--- org.glassfish.jersey.containers:jersey-container-servlet-core:2.25.1
     |    |    |    +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
     |    |    |    +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
     |    |    |    +--- org.glassfish.jersey.core:jersey-server:2.25.1 (*)
     |    |    |    \--- javax.ws.rs:javax.ws.rs-api:2.0.1
     |    |    +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
     |    |    +--- org.glassfish.jersey.core:jersey-server:2.25.1 (*)
     |    |    \--- javax.ws.rs:javax.ws.rs-api:2.0.1
     |    +--- org.eclipse.jetty:jetty-server:9.4.8.v20171121 (*)
     |    +--- org.eclipse.jetty:jetty-webapp:9.4.8.v20171121
     |    |    +--- org.eclipse.jetty:jetty-xml:9.4.8.v20171121
     |    |    |    \--- org.eclipse.jetty:jetty-util:9.4.8.v20171121
     |    |    \--- org.eclipse.jetty:jetty-servlet:9.4.8.v20171121
     |    |         \--- org.eclipse.jetty:jetty-security:9.4.8.v20171121
     |    |              \--- org.eclipse.jetty:jetty-server:9.4.8.v20171121 (*)
     |    +--- org.eclipse.jetty:jetty-continuation:9.4.8.v20171121
     |    \--- org.apache.commons:commons-lang3:3.6
     +--- io.dropwizard:dropwizard-servlets:1.3.0-rc3
     |    +--- org.slf4j:slf4j-api:1.7.25
     |    +--- io.dropwizard:dropwizard-util:1.3.0-rc3 (*)
     |    +--- io.dropwizard.metrics:metrics-annotation:4.0.2 (*)
     |    +--- io.dropwizard.metrics:metrics-core:4.0.2
     |    \--- ch.qos.logback:logback-classic:1.2.3 (*)
     +--- io.dropwizard:dropwizard-jetty:1.3.0-rc3
     |    +--- io.dropwizard:dropwizard-logging:1.3.0-rc3 (*)
     |    +--- io.dropwizard.metrics:metrics-jetty9:4.0.2
     |    |    \--- io.dropwizard.metrics:metrics-core:4.0.2
     |    +--- org.eclipse.jetty:jetty-server:9.4.8.v20171121 (*)
     |    +--- org.eclipse.jetty:jetty-servlet:9.4.8.v20171121 (*)
     |    +--- org.eclipse.jetty:jetty-servlets:9.4.8.v20171121
     |    |    +--- org.eclipse.jetty:jetty-continuation:9.4.8.v20171121
     |    |    +--- org.eclipse.jetty:jetty-http:9.4.8.v20171121 (*)
     |    |    +--- org.eclipse.jetty:jetty-util:9.4.8.v20171121
     |    |    \--- org.eclipse.jetty:jetty-io:9.4.8.v20171121 (*)
     |    \--- org.eclipse.jetty:jetty-http:9.4.8.v20171121 (*)
     +--- io.dropwizard:dropwizard-lifecycle:1.3.0-rc3 (*)
     +--- io.dropwizard.metrics:metrics-core:4.0.2
     +--- io.dropwizard.metrics:metrics-jvm:4.0.2
     |    \--- io.dropwizard.metrics:metrics-core:4.0.2
     +--- io.dropwizard.metrics:metrics-jmx:4.0.2
     |    \--- io.dropwizard.metrics:metrics-core:4.0.2
     +--- io.dropwizard.metrics:metrics-servlets:4.0.2
     |    +--- io.dropwizard.metrics:metrics-core:4.0.2
     |    +--- io.dropwizard.metrics:metrics-healthchecks:4.0.2
     |    +--- io.dropwizard.metrics:metrics-json:4.0.2
     |    |    \--- io.dropwizard.metrics:metrics-core:4.0.2
     |    +--- io.dropwizard.metrics:metrics-jvm:4.0.2 (*)
     |    \--- com.papertrail:profiler:1.0.2
     |         \--- joda-time:joda-time:2.9.1 -> 2.9.9
     +--- io.dropwizard.metrics:metrics-healthchecks:4.0.2
     +--- io.dropwizard:dropwizard-request-logging:1.3.0-rc3
     |    +--- io.dropwizard:dropwizard-jetty:1.3.0-rc3 (*)
     |    +--- io.dropwizard:dropwizard-logging:1.3.0-rc3 (*)
     |    \--- ch.qos.logback:logback-access:1.2.3
     |         \--- ch.qos.logback:logback-core:1.2.3
     +--- net.sourceforge.argparse4j:argparse4j:0.7.0
     \--- org.eclipse.jetty.toolchain.setuid:jetty-setuid-java:1.0.3

(*) - dependencies omitted (listed previously)


BUILD SUCCESSFUL in 0s
1 actionable task: 1 executed


Wholly crap!!! That's a lot of jars!!! 90, to be accurate.

Well, surely Java and JavaScript must be some of the worst offenders... but given it is just rational to try to avoid implementing functionality that already is implemented by libraries (which is nearly everything for the most popular languages), this proliferation of dependencies seems unavoidable.

For curiosity, I compared Java/JavaScript with Go and one of its heavier web frameworks, Beego, and was shocked to find out that this framework seems to have no external dependencies (even though it implements ORM, MVC, i18n and logging in the core module), weighs in at only 3,7MB, and a hello world, statically compiled web server (which, unlike Java and JS, does not require a runtime) compiles to around 11MB.

Not too bad! But that brings us to the curious case of Go dependency management (which may help explain how a large framework can be implemented without using libraries, even considering Go's stance to have a complete, large standard library).

You see, Go is very different from other languages in that it does not have a package repository! Until very recently, it did not even have a standard package manager.

Want to use a dependency in Go? Just run go get to download it from GitHub and start using it... For example, to use Beego, you first run this:


go get github.com/astaxie/beego


This downloads the source adds it to the GOPATH (where it can be found by the Go compiler). Now you can just import it in the source code with import github.com/astaxie/beego!

go get supports downloading from GitHub and a few other sources (e.g. BitBucket, LaunchPad), but it doesn't do any kind of version checks! It just gets the source currently in the default branch (it might look at tags with a Go version as well) and calls it a day.

This might sound like complete chaos! Every time you build your project in a new machine, you might get different code in your dependencies. The solution the Google Go team suggested was to not break backwards compatibility - because then, it doesn't matter when you build it, it will still work if it worked last time!

I was first shocked when I found out about that! But then I reasoned that Google engineers probably know what they are doing, even if it looks like they just came up with the most terrible solution to the hard problem of package management.

Have you ever tried to build software that is shared between hundreds of developers, maybe hundreds of teams? Keeping dependencies up-to-date in all projects so that everything works and remains free of security vulnerabilities is difficult. Some libraries may have no maintainer for a while, causing old versions of its transitive dependencies to spread like a fire.

But that's exactly the situation nearly everyone is at now. We all use hundreds of dependencies in our projects, some of which may not have been updated in a decade. We just can't keep up, this is a losing fight.

We must give up on trying to be heroic and accept the facts: updating all transitive dependencies of every library and project till the end of time is just unfeasible. We must find another way.

Here's how we can solve the problem:

  • No breaking changes at the binary/source level are allowed once a library is declared stable.
  • If a function is not ideal, write a new function that gets closer to perfection, but keep the old one with a "DEPRECATED" warning.
  • Don't ever remove anything. We all make bad decisions, but removing a function from a library will not solve anything.
  • When the burden of maintaining the legacy code becomes to heavy to carry, start a new library, with a new name, to obsolete the old one.

In the Java world, many of the most common libraries have been doing that for years, and it works great!

Check this gist that shows the evolution of the HTTP Components Client library. How hard can it be to design a HTTP client, right? Every version changed basically everything! But because they have been changing the package names from version to version, it is possible to run an application that depends on all 3 versions at the same time (which amazingly, can easily happen given how widespread this HTTP Commons is). Objects coming from one version will not have the same type than the equivalent Object coming from another, so any mismatches are caught by the type system auomatically (JS and other dynamic languages have the "advantage" that it will only blow up at runtime and you'll probably only ever notice the problem when it happens, hopefully during tests).

Same thing with Log4j, which changed its namespace (package) in version 2, effectively becoming a new library.

Unfortunately, we're humans so I acknowledge we cannot expect everyone to follow these rules... after all, even the Go community seems to have finally settled with a dependency manager (go dep). It works a little bit like npm and Cargo, pinning dependencies following version constraints present in the build file. But that solution still suffers from the problems I have mentioned above. Just because your version range should work according to semantic versioning, it doesn't mean it will. And as Gavin King, lead developer of Ceylon, said: version range compatibility is a lie. Remember, we can't trust people!

In the long term, the only viable solution is to automate the process of checking for backwards compatibility and stopping breaking changes from happening, so we don't need to trust people to achieve that.

In the Java world, there's already a tool, japicmp, that does that. Elm, a strict functional programming language for the web, aims to support automated version change checks at the language level! With that in place, you simply can't break backwards compatibility and cause havoc for all poor souls who thought it would be a good idea to rely on your software in their applications.

If you try hard, you actually can break compatibility by changing semantics... but that's just plain evil! If you do that, you should feel bad.

What about repeatable builds, you may ask. Well, perhaps what you're really asking for is archived builds, not the possibility to download all exact versions of all dependencies in every build?

To conclude, I acknowledge that we need to adapt to a world where versioning is a reality. I suggest just letting an automated tool bump our minor/patch versions every time we release (maybe every day, or several times a day). If we never bump our major version number and have automated checks in place to keep us honest, everyone can depend on version [1.0, 2.0) (or 1.* in semver language) forever without fear of being screwed, automatically receiving all fixes and improvements library authors add over time... even old examples in Stack Overflow will still work (if the example code becomes obsolete, the worst that will happen is that the developer using it will see a deprecation warning, but it will still work)! I am sure that if we do that, building software in 20 years will be a lot more pleasant and reliable than the current chaotic situation.



Impressions of Go: a wonderfully simple, weird, native language

posted Dec 28, 2017, 11:09 AM by Renato Athaydes   [ updated Jan 9, 2018, 2:13 PM ]

At my current job, I have joined a team that, among other things, is responsible for creating tools for developers. I'm a long-time JVM (Java/Kotlin/Groovy) developer, but I have to admit that for some kinds of tools we need to create, specially CLIs (command-line interfaces), the JVM (regardless of the language) is not the best choice. That's mostly because of the startup time of the JVM. It has gotten a lot better in recent JVM versions, but there's still a perceptible, even if under-a-second, lag to start even the simplest applications.

For some kinds of tool we might need to develop, a cross-platform, native language can be defintely a better choice.

I've been playing with some native languages like Haskell, Rust and Pony to see if I could find a language that fits that purpose, but unfortunately none of them made me feel satisfied.

Haskell and Rust, while being incredibly powerful tools, are very complex and I found myself being unable to become productive with them within a short time frame (several weeks).
I managed to write a couple of small projects in Rust (JGrab and the unfinished Vinegar), but it felt really constraining to me the whole time, without sign of getting much better with time.

Pony is another really interesting language, with a Python-like syntax, an Erlang-inspired actor-based runtime, a powerful type-system that can encode capabilities which make lock-free concurrency verifiably safe in a way similar to Rust, though using completely different means... but I just don't have the patience to write serious code using it due to the lack of tooling. It's extremely taxing for me to have to keep a browser open to check what method I can call, read basic documentation, read the standard lib code etc. (I've been spoiled by the wonderful tooling one gets to use with Java/Kotlin in IntelliJ).

That's one of the reasons I decided to finally try Go, the language developed by Google that has been a hit in the devops space, with arguably the most respected tools in this space written in Go: Kubernetes, Docker, rkt, several AWS utilities.

Go has been a hit with databases (CockRoachDB, InfluxDB) as well, and many other interesting, groundbreaking projects use Go! Just to give a few more examples of projects mostly written in Go that I felt excited about:

* IPFS - a huge effort towards a decentralized, permanent web.
* Keybase - end-to-end encryption for everyone! Supports chat, a distributed file system (public content can be viewed online) and Git.
* Hugo - static website generator that's becoming hugely popular (I know, I should be using it for this blog... hoping to switch to it soon).

That shows that a lot of cool stuff is being built with Go, which in my opinion is a great reason to give it a try.

Another reason for me, personally, is that Go is actively being used in another of our offices and there's been some talk about trying it in our office as well.

And that's how I got to write a mildly serious project in Go (the project is not ready yet, so I will not publish the link just yet... will update it as soon as I have something useful). UPDATE: the project is now useable and it's a CLI password manager called go-hash.

First impressions


At first look, Go code looks quite familiar, with a C-like syntax. The first not-so--obvious thing I noticed is how variables are declared:



i
:= 0



This declares a variable i with value 0. The type is inferred to be int.

You could also do this, equivalently:



var i int = 0



Or even just this:



var i int



Here, there's two things to understand... first, the type of the variable is declared after the variable name, so here i is the name of the variable, while int is its type.
Second, there's no need to explicitly provide an initial value!

That's because in Go, all types have a zero-value, so if you do not initialize a variable, it just takes that zero-value. For int, the value is, obviously enough, 0.

The next thing I noticed is that in many function declarations, the type of a variable seemed to be omitted:



func
HasPrefix(s, prefix string) bool { /* implementation */ }

 

That means that the variable type is the same as the one following it... so in this example, both s and prefix are of type string. bool is the return type of this function.

You might have noticed that most types start with a lower-case letter, and this function starts with a capital-letter. However, that's not really the case. In reality, any exported symbol in Go needs to start with a capital-letter, otherwise it is private to the scope where it is declared.
An exception to this rule seems to be primitive types, which always start with a lower-case letter, even though they can be used anywhere.

This, for me, is a strange choice because it makes it easy for types and variables names to conflict. I'm used to declare variables (for good or bad) with the same name as their types (but starting with lower-case), which in Go is impossible unless you export your type or the variable. The choice in Go, as the above function (taken from the strings package from the standard lib) shows, seems to be to use one-letter variables instead, which I am not a big fan of.

Anyway, the next thing to notice is that Go does not have classes. Instead, it offers a mix of interfaces and structs, similar to Rust (which has traits instead of interfaces).

A struct, as one might expect from other languages, is just a group of values:



type MyStruct struct {
     enabled bool
     name string
     values []int
}



It is interesting to notice that you can create an instance of a struct without providing values for all (or any) of its fields. The fields are just initialized to their zero-value if no value is provided:



m := MyStruct{
   enabled: true,
 }
 fmt.Printf("%+v\n", m)



Running this code prints this:


{enabled:true name: values:[]}


As you can see, the zero-value of a string is the empty string, and of an array (technically, a slice, actually), the empty array.

Interestingly, nil exists in Go but only pointers can be nil (as the zero-value of a pointer has to be something!) which seems to help limit the damage quite a lot.


The weird bits



The above code brings us to the first weird thing about Go (at least for me): when declaring a struct literal, you must write a trailing comma after each field value, including the last one.

That is, presumably, to be diff-friendly (so adding a new value won't cause the line above it to also change). But it's the first time I see this approach being used... I've seen other languages address this by formatting code differently, for example, Haskell and Elm prefer to just place the comma as the first character of a new line... but anyway, even though this is surprising and a bit weird, there seems to be a justification for it.

If we wanted to initialize the slice, we would need to learn about slice literals and the make built-in.

make is a built-in which cannot be written in Go itself. It can be used to create a slice, map or channel.

So, in order to provide an initial value for the values field of MyStruct, you could either give it a literal value, or use make to just allocate memory for it (in this case, with size 0, capacity 10):



    // literal slice
    m := MyStruct{
        values: []int{1, 2, 3},
    }

    // allocate array for slice with make
    n := MyStruct{
        values: make([]int, 0, 10),
    }



Creating a map with make looks like this (with a map literal also shown):



    // create map using make
    myMap := make(map[string]int)

    myMap["one"] = 1
    myMap["two"] = 2
    myMap["three"] = 3

    // create map literal
    mapLiteral := map[string]int{
        "one": 1,
        "two": 2,
        "three": 3,
    }



Looking at the map declaration above, you might think that Go supports generics. If you've ever read a discussion about Go anywhere, you would know that that's not the case! The Go creators famously refuse to add generics to the language, much to the despair of some people in every discussion around the topic.

The reason maps can be declared in Go is that they are a Go primitive, just like arrays/slices. It's not possible to create your own type that behaves similarly.

That's another weird thing about Go: the amount of constructs available to the user is much smaller than what the language itself makes use of.

So, while Go has no generics, it still has a generic-like thing: map. While there's no overloading in Go, built-ins can still behave as if there were, as in this example with the delete built-in, which removes entries from maps:



    myMap := map[string]int{
        "one": 1,
        "two": 2,
    }

    delete(myMap, "one")

    otherMap := map[int]string{
        1: "one",
        2: "two",
    }

    delete(otherMap, 1)



Declaring a function like delete in user code is just not possible. Notice, though, that unlike in most languages with generics, delete is type-safe: trying to delete a key with the wrong time is a compiler error.

To follow up on the delete built-in, it was very interesting to discover how to remove an item from a slice. Knowing about delete, I supposed that there would be something similar for slices. But there isn't! How to delete an item from a slice?

I bet you wouldn't guess it. This is how:



    slice := []string{"a", "b", "c"}
    i := 1

    // remove item at index i
    slice = append(slice[:i], slice[i+1:]...)



This idiom is used all over the place for this purpose because there's just no way to declare a function that does it for slices of any type (that would require generics) and the Go creators decided a built-in like delete was not necessary.

But at least they provided Go programmers with the append built-in, so they can just implement that themselves.

You might have notice the ... in the code above. We'll see what that means in a second, but before we move on, there's a few more weird thins I think I should mention!

Another feature that nearly every other language has but is missing in Go is enums.

To be fair, it's not completely missing, as there is a construct called iota which is supposed to compensate for that. I'll leave the details out (check the link for details), but here's, more or less, how you handle enumerated values in Go:



type Weekday int

const (
     Sunday = iota
     Monday
     Tuesday
    Wednesday
     Thursday
     Friday
     Saturday
)



Unfortunately, although this allows you to have something that resembles proper enums, this is not type-safe in the sense that you could pass -10 or whatever to any function that accepts a variable of type Weekday.

To finalize a section on the weird bits of Go, I couldn't do better than show how time format works in Go.

In Java and every other language I know of, time formatting is done via a String template with codes like H for hour and m for month, so you end up with templates like yyyy-MM-dd HH:mm:ss, which resolve to date-times like 2017-12-28 17:21:00.

Not so in Go! Because the traditional template may be confusing to newbies (for example, is mm month or minutes?), Go came up with a novel system.

Here's how to print the current time using the same format mentioned above with Go:



    println(time.Now().Format("2006-01-02 15:04:05"))



To understand why, I will just quote the Go docs:

These are predefined layouts for use in Time.Format and Time.Parse. The reference time used in the layouts is the specific time:

Mon Jan 2 15:04:05 MST 2006

which is Unix time 1136239445. Since MST is GMT-0700, the reference time can be thought of as

01/02 03:04:05PM '06 -0700

The cool bits


The append built-in is interesting because it shows how to use variadic parameters in Go. In the case we saw previously, to spread the slice into a variadic parameter.
Luckily, it is actually possible to declare a function that takes a variadic parameter:



func myAppend(s string, ss ...string) string {
     b := bytes.NewBufferString(s)
     for _, item := range ss {
         b.WriteString(item)
     }
     return b.String()
}



This function can then be used in any of the following ways:



    strings := []string{"dear ", "world"}

    println(myAppend("hello dear world"))
    println(myAppend("hello ", "dear ", "world"))
    println(myAppend("hello ", strings...))



Which prints hello dear world 3 times.

As mentioned briefly before, unlikely Java and most other object-oriented languages, Go does not have classes. But it still allows us to define methods for structs, maintaining the benefits of strong encapsulation!


By adding methods to structs, we can also make the struct implement an interface to achieve polymorphism -
a struct implements an interface implicitly, i.e. it can be used where an interface is required as long as it implements all methods of the interface.



For example, consider the MyStruct defined earlier. Here's its definition again:



type MyStruct struct {
     enabled bool
     name string
     values []int
}



We could define a method called Sum that sums over all values of a MyStruct instance with the following code:



func (m *MyStruct) Sum() (result int) {
     for _, item := range m.values {
         result += item
     }
     return
}



Here, I intentionally introduce quite a few features not mentioned earlier. For example, notice the return value is named (called result in this case), so we don't need to declare it in the function body or in the return statement at the end.

Also notice the for loop. Go supports simple indexed loops like most other imperative languages as well:



    for i := 0; i < 10; i++ {
        println(i)
    }



But the for loop in the previous example looks quite different. That's because that's a for-each loop.
It works like this: the range keyword iterates over data structures, returning two values. In the case of slices, an index and the current item.
In the Sum function, we throw away the index (because it is not needed in this case) by declaring it with _, using only the current item.

You cannot declare variables and not use them in Go, doing that causes a compiler error! That's why we need to say we don't care about the index by naming it _. I noticed that this feature saved me from writing quite a few bugs!

One more detail in the Sum function: notice that the parameter type is prefixed with a *. That indicates the variable is pointer to something of type MyStruct, very similar to C. And as in C, to deference a pointer, you would normally use the & character. But that's only required to call functions in Go, methods can be called without that!

Hence, this is how you can call the Sum method on an existing instance of a struct:


   
n
:= MyStruct{
        values: []int{1, 2, 3, 4, 5},
    }

    fmt.Printf("Sum of n: %d\n", n.Sum())



If Sum were a function, you would need the & symbol:



    fmt.Printf("Sum of n: %d\n", Sum(&n))



It is preferable to use pointers when passing structs around because otherwise Go would make a copy of the struct when passing it to the function or method (unless that's exactly what you need, even if it might be less efficient).

Before we move on, remember how pointers can be nil? So every time we handle a pointer, we need to be careful to handle nil, so our Sum function should really be something like this instead:



func (m *MyStruct) Sum() (result int) {
     if m == nil {
         return
     }
     for _, item := range m.values {
         result += item
     }
     return
}



Now, moving on... Go has a feature to defer the execution of a method invocation until the end of the function where it is invoked, regardless of whether the function panics (for Java developers, similar to throwing an Error).

This feature is commonly used to release resources in a safe manner. For example, suppose you need to write to a file. The common idiom is to defer a call to its Close method immediately after assigning it (and checking for an error):


    
    file, err := os.Create(filePath)
    if err != nil {
        return err
    }
    defer file.Close()



There's a few things you must understand to be able to use defer correctly, but the Go Tour explains that much better than I could, so just have a look at the Tour for the details (the Tour is a really great way to learn Go, by the way!).

This code snippet shows another cool thing about Go: its error handling.

Most functions that have a good likelihood of failing, like opening a file or reading from a socket, return both a value and an error. If an error occurs, then the error result will not be nil, otherwise you can safely use the success return value.

To propagate the error, just return it from your own function. To give up on error, call panic(err), which is meant to crash the program (though there are ways to recover upstream), similar to Exceptions in languages that have them.

The possibility to return multiple values from a function is really useful, and not only to return the tuple (success, error).

For example, this is a function I wrote in the project I am working on, currently:



func removeEntryFrom(entries *[]LoginInfo, name string) ([]LoginInfo, bool) {
     if i, found := findEntryIndex(entries, name); found {
         return append((*entries)[:i], (*entries)[i+1:]...), true
     }
     return *entries, false
}



It returns the new slice with the target entry removed if it is found, along with a boolean that tells the caller whether the entry was removed.

There's quite a few other cool things about Go, but to not make this blog post even longer, I should stop here and just mention goroutines and channels, which are the primitives used to achieve concurrency in Go.

Even though I haven't had the chance to use them yet, they seem quite approachable and are definitely one of the strongest points of Go.

Again, I will defer (sorry the pun) to the Go Tour (click on the link) for a short and easy explanation of how they work.



Conclusion


Go does justice to its reputation: it is a really simple language and runs quite fast.

On the one hand, the haters seem to be right about some things. Go really misses some very basic features that most modern languages have adopted: generics, enums with powerful tricks like in Kotlin and Rust, functional programming features (no sign of map/filter/reduce in Go - no doubt due to the lack of generics), and even a built-in package manager (though I really like the simplicity of go get, and go dep seems to be finally unifying the tooling around Go dependency management).

But on the other hand, the weaknesses of Go may play to its advantage: I read a lot of Go code while working on my first project with the language, and I could understand basically all of it without trouble! The simplicity of the language is essential for that to be possible.

The Go compiler is extremely fast. My project was compiling so fast sometimes I was not sure it actually did anything :) Only after the project grew to a couple of thousand lines of code, did I notice a small, sub-second delay when compiling!

There are excellent IDEs for Go: the free VSCode plugin is very good even for someone accustomed to the power of IntelliJ.

Jetbrains has a full-blown IDE for Go, Goland but it is not free... but I found out that because I already have a IntelliJ Ultimate license from work, I could just use the Go Plugin in IntelliJ Ultimate without extra cost (unfortunately, it seems that a license is required to install this plugin... I was not able to install it at home without the license - hence why I've been using mostly VSCode). The IntelliJ plugin is definitely better than VSCode, with lots of small conveniences missing from VSCode (automatic imports, refactoring, quick-fixes etc), but I was actually surprised at how close VSCode is to the IntelliJ experience!

All in all, I can definitely understand why so many successful products are written in Go: it is a very productive language, and even though it is native, it is still very easy to compile to multiple targets.

The simplicity of the language does annoy me at times when writing yet another old-fashioned for-loop, but that simplicity seems to be its greatest feature, after all.





Announcing RawHTTP - a JVM library for handling raw HTTP

posted Dec 10, 2017, 1:24 PM by Renato Athaydes   [ updated Dec 10, 2017, 2:39 PM ]

HTTP is the language of the internet. It is also the most commonly used communication protocol for REST APIs, which have become the backbone of most distributed applications.
That's probably not news to anyone. But something that a lot of people may not realize is that HTTP (up to version 1.1, more on that later) is a textual protocol you can quite easily write by hand!

For example, here is a GET request directed at the jsontest.com REST API that will return the headers we send out as a JSON object:


GET / HTTP/1.1
Host: headers.jsontest.com
Accept: application/json


Running this request results in the following response (shown exactly as it is returned by the server in bytes-form):


HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json; charset=ISO-8859-1
X-Cloud-Trace-Context: e7be085086214dffdeb6350ad37672ed
Date: Sun, 10 Dec 2017 18:33:53 GMT
Server: Google Frontend
Content-Length: 155

{
   "X-Cloud-Trace-Context": "e7be085086214dffdeb6350ad37672ed/9779089525915638501",
   "Host": "headers.jsontest.com",
   "Accept": "application/json"
}


As you can see, both the request and response are easy to read and write... one could argue you could write them easily by hand. And that's exactly why I decided to create RawHTTP.

RawHTTP is a library written in pure Java, with no dependencies, which lets you create requests and responses from their textual representation, letting you then turn them into bytes which can be sent to either a server in the case of a request, or a HTTP client in the case of a response!

As a side note: do you know the difference between a HTTP request and a HTTP response?

Well, only the first line (called start-line by the HTTP RFC). The rest of the HTTP message (either request or response) is exactly the same. Sure, requests and responses normally include different headers, but the message format is exactly the same:


start-line\r\n
header-name: header-value\r\n
...
\r\n
<optional body>

There is some complication in the framing of the optional body, but essentially that's all there is to a HTTP message.

Pretty neat.

So, back to RawHTTP! Given that HTTP messages are so easy to write, you might ask yourself why even bother creating a library to parse it.

That's a fair question. And the answer is that even though it is in fact very easy to write, it is also very easy to make mistakes that make your HTTP message completely invalid, which may cause a connection to hang (if you, for example, get the Content-Length header value wrong!) or to just be rejected right away by the recipient (if you forget to use \r\n as line-separators and carelessly use just \n).

If you can make sure you never make these mistakes (and many others, of course), then you can definitely just do the following, in Kotlin, for example:


All examples in this blog post are written in Kotlin, but should look similar in Java or any other JVM language.



val
request = "GET / HTTP/1.1\r\n" +
"Host: ip.jsontest.com\r\n" +
"Accept: application/json\r\n" +
"\r\n"

val response = Socket(InetAddress.getByName("ip.jsontest.com"), 80).use {
it.getOutputStream().write(request.toByteArray())
// TODO read the response
}


This works perfectly, but reading the response might be a little tricky because you need to be able to read the headers first to be able to determine how to read the body (if any), and as mentioned earlier, reading the body can be tricky if the sender decides to send it out in chunks for efficiency.

There are more reasons why you probably don't want to do it all by yourself, but let it suffice to say that deciding when you should attempt to read a HTTP message's body, which headers, status code, methods, impact on that decision, is not trivial to decide.

RawHTTP helps with all of these problems.

Here's how you would send a request AND read the response correctly, using RawHTTP:



val http = RawHttp()

val request = http.parseRequest("""
GET / HTTP/1.1
Host: headers.jsontest.com
Accept: application/json
""".trimIndent())

val response = Socket(InetAddress.getByName("date.jsontest.com"), 80).use {
request.writeTo(it.getOutputStream())
http.parseResponse(it.getInputStream()).eagerly()
}



Notice that RawHTTP keeps your HTTP message pretty raw, but it does fix a few things for you to make sure it is a little harder to make invalid requests and responses.

For one thing, you can use simple new-lines, no need to insist on using \r\n (though it works as well).
RawHTTP also inserts a final new-line for you automatically if needed.


You might have noticed the eagerly() method being called before we return the response? That's because, by default, HTTP message's bodies are not read unless you call asStream() and read the InputStream yourself, or you just call eagerly(), which does that for you. So, unless you only want to download the body of a message conditionally after processing the message's headers or you need custom behaviour, always call eagerly() after parsing a request/response to avoid surprises.


Other things RawHTTP does is make sure that you have the minimum set of headers necessary for your HTTP message to be generally valid, and set a HTTP version in the start-line if none is specified.

In the case of requests, you must make sure to include the Host header (at least in HTTP/1.1).

For the sake of convenience, you can simply specify your request as follows:


GET http://example.com/path/to/resource


RawHTTP will insert a Host header with value example.com and set the HTTP version to HTTP/1.1, so the end result should be this:


GET /path/to/resource HTTP/1.1
Host: example.com


If you do not want RawHTTP to fix the Host header for you (or fix new-lines), you can actually configure it to not do that! Just give an instance of com.athaydes.rawhttp.core.RawHttpOptions to the RawHttp constructor configured the way you want and you're done.

If your request needs to include a body, you can simply write the body after an empty line (don't forget to add the Content-Length header so the recipient of the message knows how to read the body - keep reading to see how RawHTTP can set a body from a String or File so you don't need to include it in the raw message):



val
response = RawHttp().parseResponse("""
HTTP/1.1 200 OK
Server: RawHTTP
Content-Length: 12

Hello World!
""".trimIndent())



Notice that just like you can stream a request to a server, you can stream a response to a client, just use its writeTo(OutputStream) method! That means that using RawHTTP, you can quite simply create working HTTP clients and servers!

But please beware that RawHTTP is still VERY low level. It concerns itself only with the format of the HTTP messages as defined by the HTTP/1.1 RFC (RFC-7230). It currently does not validate headers' names and values, for example, or support any kind of common HTTP extension features, such as cookies, caching etc. So it may not be a replacement to HttpClient or Jetty :)

RawHTTP also allows some HTTP/1.0 behaviour, such as not setting the Host header if the message has HTTP/1.0 version, and downloading the body until the connection is closed in case of a missing Content-Length and Transfer-Encoding headers, but it generally adheres to HTTP/1.1. In the future, I plan to add support for setting the message's version to HTTP/2.0 and then process it as such, but unfortunately HTTP/2.0 messages are binary rather than textual, so they will have to be converted from the HTTP/1.1 format rather than written raw.

If you just need a very lightweight way to send requests/responses, though, specially if you just want to test your client or server, RawHTTP may be ideal for your use-case.


One last thing about sending HTTP messages with bodies: RawHTTP messages have a replaceBody method that lets you, you guessed it, replace a HTTP message's body with the contents of a String (via StringBody) or a file (via FileBody).

When you use replaceBody, RawHTTP changes the Content-Type and Content-Length headers automatically.

All RawHTTP objects are completely immutable, so methods like replaceBody do not modify the instance they are called on, but return a new, modified version of the object instead.


For example, to use a file as the body of a HTTP response:



val response = RawHttp().parseResponse("""
HTTP/1.1 200 OK
Server: RawHTTP
""".trimIndent()
).replaceBody(FileBody(File("example.json"), "application/json", true))

println(response.eagerly())


Which, in the case of my example.json file, prints the following:


HTTP/1.1 200 OK
Server: RawHTTP
Content-Type: application/json
Content-Length: 47

{
  "hello": "Hi",
  "location": "Stockholm"
}



To finish off, notice that RawHTTP ships with a very basic HTTP client that makes it just a little more convenient to send requests and receive responses. Using the included client, called TcpRawHttpClient, your code would look like this:



val request = RawHttp().parseRequest("""
GET / HTTP/1.1
Host: headers.jsontest.com
Accept: application/json
""".trimIndent())

val response = TcpRawHttpClient().send(request).eagerly()



I am planning to write a very simple, asynchronous server implementation as well so that the library ships with both a functional client and server, but I haven't had the time to finish that yet... I also want to make sure the server implementation does a little bit more validation on the messages it accepts to avoid security issues, but in the spirit of the library's goals (namely, to provide the bare minimum HTTP implementation) it will still not take care of things like caching, CORS headers or anything on those lines... I hope this will be useful anyway as a test library or perhaps as the backbone of full-featured HTTP clients and servers in the future.

To use RawHTTP, just include it as a dependency of your project, it's on Maven Central and JCenter, see the RawHTTP Bintray page for details.

All source code shown in this blog post is available on GitHub.

Say no to Electron! Using JavaFX to write a fast, responsive desktop application.

posted Oct 3, 2017, 6:11 AM by Renato Athaydes   [ updated Oct 21, 2017, 7:42 AM ]

        This article has been translated to Russian!

Lately, there has been a lot of discussions (see here, here and here for a few examples - and today this one) in programming forums about Electron and its impact on the desktop app world.

If you don't know Electron, it's basically a web browser (Chromium) that hosts only your web application... as if it were a desktop application (no, it's not a joke)... that lets you use the web stack to develop cross-platform desktop applications.

Most new, hipster desktop apps these days are  built on Electron, including Slack, VS Code, Atom and GitHub DesktopThis is an extraordinary development.

We've been writing desktop apps for decades. The web, on the other hand, only really got started less than 20 years ago, and most of that time it was only used for serving documents and animated gifs, not creating full-fledged applications, or even simple ones!

To think that the web stack would be used to create desktop applications 10 years ago would be unthinkable. But here we are, in 2017, and a lot of intelligent people think that Electron is a great idea!

This is not as much a result of the superiority of the web stack for building applications (far from that, I don't think anyone disagrees that the web is a mess), as a failure of the current desktop UI frameworks. If people are preferring to ship a full web browser with their apps just so they can use great tools such as JavaScript (sarcasm) to build them, something must have gone terribly wrong.

So, what are these terrible alternatives that are losing out badly to the web stack?

I decided to have a look and build myself a real application with one of them.

Electron alternatives


If you don't mind having different teams developing for each popular OS, the options seem to be, currently, AppKit on MacOS, WPF on Windows (I am not a specialist in platform-specific development, so please let me know if other options are more popular nowadays).

However, the real competitors to Electron are the multi-platform frameworks. I believe the most common multi-platform frameworks today are GTK+Qt and JavaFX.

Gtk+


GTK+ is written in C but has bindings to many other languages as well. It was used to develop the beautiful GNOME-3 platform.

Qt


Qt seems to be the most often suggested alternative to Electron in the discussions I've seen... It's a C++ library but also has bindings in other languages (though it seems that none of them are supported commercially and it's hard to say how complete they are). It looks like a popular choice for embedded systems.

More alternatives!

UPDATE (2017-Oct-21): Thanks to comments from people on Reddit, I decided to add a link to a few more alternatives here:


Embeddable HTML/CSS/script engine for modern UI development.


Open source Python library for rapid development of applications
that make use of innovative user interfaces, such as multi-touch apps.


NW.js (previously known as node-webkit) lets you call all Node.js modules directly from DOM and enables a new way of writing applications with all Web technologies.


Lazarus is a Delphi compatible cross-platform IDE for Rapid Application Development. It has variety of components ready for use and a graphical form designer to easily create complex graphical user interfaces.


JavaFX


In this post, however, I will focus on development of a desktop application using JavaFX because I believe JavaFX and the JVM are great for developing desktop applications.

Whatever you think about the JVM, no other platform (except perhaps Electron itself!) is as easy to use for development on multiple platforms. Once you've created your jar, in any platform, you can distribute it to any user on any OS and it will just work.

With the large variety of languages that currently run on the JVM, language shouldn't be an issue: there will definitely be one that you like (even including JavaScript if you just can't help it), and you can use JavaFX with any JVM language without big issues. In this blog post, besides Java, I will show a little bit of Kotlin code as well.

The UI itself can be built with just code (in which case you have wonderful IDE support from IntelliJ, Eclipse or NetBeans, all excellent, free IDEs that probably beat any of the competitors and, by the way, probably the best examples of Java desktop applications), or using a UI visual builder: SceneBuilder (which can be integrated into IntelliJ) or NetBeans' Visual Debugger.


JavaFX history

JavaFX is not a new technology. It was first released in December 2008 and looked quite different from what it looks like today. The idea was to create a modern UI framework to replace the ageing Swing Framework which had been the official framework in the JVM since the late 90's.

Oracle almost messed up its future from the beginning by creating a special, declarative language that was supposed to be used to create application UIs. That didn't go down well with Java developers and it almost killed JavaFX.

Noticing that, Oracle decided to release JavaFX 2 in 2011 without the special language, but using instead FXML as an option to pure Java code (as we'll see below).

Around 2012, JavaFX gained some traction and Oracle spent a substantial effort trying to improve it and make it become more popular. With version 2.2, JavaFX became a fairly complete platform, but it still was not included in the standard Java runtime (though it was always shipped with the JDK).

Only with JavaFX 8 (the version change was just to match Java 8) did it become part of the standard Java runtime.

Today, JavaFX may not be a very large player in the UI world but it still has a good number of real-world applications, a quite a large number of related libraries, and has even been ported to mobile.


Creating a JavaFX application


For my application, a log viewer I called, un-inspiringly, LogFX, I chose to just use Java (because it's mostly pretty low level code and I wanted to focus on speed and small package size) and IntelliJ for an IDE. I almost went with Kotlin, but IntelliJ's Java support is just so great that having to write Java (or let IntelliJ do it for me would be a better description) was not an issue large enough to justify adding an extra 0.9MB to my packaged application.

I chose not to use FXML (the XML view-descriptor language) or a UI visual builder as the UI was really simple.

So, let's get started by looking at some code!

Java Hello World


A JavaFX application is just a class that extends javafx.application.Application and shows a JavaFX Stage.

Here's a JavaFX Hello World:


public class JavaFxExample extends Application {

@Override
public void start(Stage primaryStage) throws Exception {
Text helloWorld = new Text("Hello world");
StackPane root = new StackPane(helloWorld);
primaryStage.setScene(new Scene(root, 300, 120));
primaryStage.centerOnScreen();
primaryStage.show();
}

public static void main(String[] args) {
launch(JavaFxExample.class, args);
}
}
 src/main/java/main/JavaFxExample.java


On Mac, this will show something like this:


JavaFX Hello World


FXML+Java Hello World


If you have a hard time writing code for UIs and prefer using a markup language, here's the equivalent code + FXML:


 <?xml version="1.0" encoding="UTF-8"?>

<?import javafx.scene.layout.StackPane?>
<?import javafx.scene.Scene?>
<?import javafx.scene.text.Text?>
<Scene xmlns="http://javafx.com/javafx"
width="300.0" height="120.0">
<StackPane>
<Text>Hello world</Text>
</StackPane>
</Scene>
 src/main/resources/main/Example.fxml


public class JavaFxExample extends Application {

@Override
public void start(Stage primaryStage) throws Exception {
Scene scene = FXMLLoader.load(getClass().getResource("Example.fxml"));
primaryStage.setScene(scene);
primaryStage.centerOnScreen();
primaryStage.show();
}

public static void main(String[] args) {
launch(JavaFxExample.class, args);
}
}
 src/main/java/main/JavaFxExample.java


Notice that IntelliJ has support for FXML and will link its contents to the relevant Java code and vice-versa, highlight errors, auto-complete, manage imports, show documentation inline and so on, which is pretty cool... but as I said before, I decided to not use FXML as the UI I had in mind was very simple and quite dynamic.... so I will not show any more FXML. Have a look at the FXML tutorial if you're interested in it.

Kotlin+TornadoFX Hello World


Before we move on, let's see what this would look like with a modern language like Kotlin and its own library for writing JavaFX applications, TornadoFX:


class Main : App() {
override val primaryView = HelloWorld::class
}

class HelloWorld : View() {
override val root = stackpane {
prefWidth = 300.0
prefHeight = 120.0
text("Hello world")
}
}
src/main/kotlin/main/app.kt


Using Kotlin and JavaFX may be attractive to many teams, specially if you like type-safety (TornadoFX has a nice feature called type-safe stylesheets) and if adding up an extra 5MB to your application is not a big concern.

Even if the overhead is too much and you want to avoid including a framework (and having to deal with all its complexity and quirks), Kotlin works just fine with pure JavaFX as well.


Styling and theming JavaFX user interfaces



Now that we've seen how to get started with JavaFX, let's see how JavaFX applications can be styled.

Just as there are different layout approaches, there are different ways to style things in JavaFX.

Let's say we want to make the background dark and the text white as shown below:


Styled JavaFX Hello World


Programmatic and inline styles


One (painful, though type-safe) way is to do it programmatically:


root.setBackground(new Background(new BackgroundFill(
Color.DARKSLATEGRAY, CornerRadii.EMPTY, Insets.EMPTY)));

helloWorld.setStroke(Color.WHITE);


An easier way to do it programmatically is by setting the styles as in CSS:


root.setStyle("-fx-background-color: darkslategray");
helloWorld.setStyle("-fx-stroke: white");


Notice that IntelliJ, again, provides auto-complete support for the String values above.

If you're using FXML:


 <StackPane style="-fx-background-color: darkslategray">
<Text style="-fx-stroke: white">Hello world</Text>
</StackPane>

Same deal...

Using separate stylesheets


If you prefer to stay close to the web world and set styles using a separate stylesheet, JavaFX also allows that! That's the approach I decided to use because that allows me to style everything in a central place and even let users provide a custom stylesheet to change all styles to their own preferences.

To do that, first create a stylesheet:


.root {
-fx-base: darkslategray;
}

Text {
-fx-stroke: white;
}

src/main/resources/css/hello.css


Now, add the stylesheet to the Scene:


primaryStage.getScene().getStylesheets().add("css/hello.css");


And that's it.

Notice that the stylesheet sets not only the StackPane's background color to darkslategray, it changes the base color of the theme.

That means that all controls and "background" items will take a color that's based on this color. That's a pretty neat feature as you can set colors based on the base color, ensuring that if you ever change the base color, most things will look good automatically.

For example, a better stylesheet for our case would not set the text stroke to white, but to an "opposite" color to the theme's base color, so that the text is always readable:


-fx-stroke: ladder(-fx-base, white 49%, black 50%);


JavaFX stylesheets are pretty smart, check the CSS Reference Guide for more information.

Here's an example of a simple application where we replace the Text with a Button, with the default styles on the left, and with the stylesheet we just used above on the right:


JavaFX Button with and without styling
On the left: default JavaFX styles. On the right: using the custom stylesheet introduced above.


In my application, I wanted to have a dark theme by default, but also allow users to provide their own stylesheets so they could use whatever theme they like best.

This is what LogFX ended up looking like with the default theme:


LogFX default theme


Notice that I used FontAwesome's icons in the buttons. It was fairly simple to get the buttons styled with css. Just make sure to install the font first as early as possible with this instruction:


Font.loadFont( LogFX.class.getResource( "/fonts/fontawesome-webfont.ttf" ).toExternalForm(), 12 );


With a custom stylesheet, the look-and-feel can be completely changed. For example, here's an overly green theme in Linux Mint:



LogFX styled green in Linux


Although the good taste of doing this may be questionable, it shows that JavaFX allows for powerful styling, you can achieve pretty much anything your imagination comes up with.

To finalise, I would like to mention the cool effects JavaFX provides... I wanted to make a start screen that looked good with just a nicely formatted text.

JavaFX makes that easy. This is what I came up with (I based this on the GroovyFX example):


LogFX logo


And here's the stylesheet I used to achieve that:


Text {
-fx-fill: white;
}

#logfx-text-log {
-fx-font-family: sans-serif;
-fx-font-weight: 700;
-fx-font-size: 70;
-fx-fill: linear-gradient(to top, cyan, dodgerblue);
}

#logfx-text-fx {
-fx-font-family: sans-serif;
-fx-font-weight: 700;
-fx-font-size: 86;
-fx-fill: linear-gradient(to top, cyan, dodgerblue);
-fx-effect: dropshadow(gaussian, dodgerblue, 15, 0.25, 5, 5);
}


Very nice effects are available. See the tutorial for more details.

In the next sections I will discuss how to change your views, hot reload code and refresh stylesheets all at runtime.


Designing, debugging and hot code reloading


To write user interfaces without being able to instantly see what your changes do is next to impossible. For that reason, hot code reloading or some sort of UI builder is essential in any UI framework.

JavaFX (and the JVM itself) has a few solutions to this problem.

SceneBuilder


The first one is the SceneBuilder, a visual UI builder that lets you create FXML by drag-and-dropping UI components.


SceneBuilder screenshot

It can be integrated into all Java IDEs, making it easy to create new views.

I've used SceneBuilder before to create forms and similar complex views, but I usually just use it to sketch something quickly, then start editing the code by hand to polish it off.

If you do that and later re-open the view in SceneBuilder, it still works fine, so you can alternate how you change the view between manually coding it and designing it with SceneBuilder.

ScenicView


Once you've got the basic design ready, you can use ScenicView to see and edit your scene-graph while the application is running.

You can think of it as the equivalent of the browser's developer view.


ScenicView screenshot


To start it with your application, just download the jar and pass the option -javaagent:/path-to/scenicView.jar to the JVM.

ScenicView lets you change and remove Nodes, track events, and read Javadocs for the selected nodes.


JVM hot code reload


If you want to change the application code that is not directly related to the UI, you can use the Java debugger to hotswap code as your application is running. Basic support for code reloading exists in the Oracle's JVM, HotSpot, and I believe also in the OpenJDK JVM.

However, the built-in code reloading is pretty basic: you can only change the implementation of existing methods.

But there is a HotSpot VM extension called DCEVM (Dynamic Code Evolution VM) that allows you to do a lot more: add/remove methods and fields, add/remove classes, change the value of final variables and more. I wrote about this and other ways to reload code in a running JVM in another post.

I used this to develop LogFX and it worked great. Views that you don't close and re-open are not automatically changed when you reload code, but that's not a problem if you change something that goes into a Stage that can be closed and re-opened... besides, if you want to change the UI component only, you can use the ScenicView or just go back to ScenicBuilder and re-design it the way you want.  

To try DCEVM, you just need to install it and make sure you get the correct version for the version of the JVM you're using. After that, you can run your application with a debugger attached to it and every time you recompile it in the IDE, the code will be automatically reloaded into the running application.

In IntelliJ, you will see something like this when you change a class and re-compile (Cmd+F9 on Mac):


IntelliJ code reload



Refreshing stylesheets



JavaFX does not automatically refresh stylesheets. In LogFX, I wanted to make it possible to change the stylesheets and see immediately the effect of doing so in the application.

As LogFX is a log viewer, it has a pretty advanced FileChangeWatcher that I could use to watch stylesheets and reload them quite easily.

But this only works if the stylesheets come from a file, not from within the application jar itself.

As I already allowed users to specify a custom spreadsheet file to use, that was not a problem for me.

I used this feature during development, and it was really awesome. If you need this feature, you can implement your own file watcher or copy mine (it's open source after all).

To specify a stylesheet as a file (as opposed to a jar resource), you must use different syntax in Unix/Mac and Windows, unfortunately! I used this method to fix that problem:


private static String toAbsoluteFileUri( File file ) {
String absolutePath = file.getAbsolutePath();
if ( File.separatorChar == '\\' ) {
// windows stuff
return "file:///" + absolutePath.replace( "\\", "/" );
} else {
return "file:" + absolutePath;
}
}


This works in Mac, Windows and Linux Mint. But this was one of the only two problems I had related to differences in OS's (the other one was the icon in the system tray on Mac does not work, but there is an ugly workaround for that). JavaFX abstracts that away pretty well, most of the time!

Finally, when you detect a change in a stylesheet file, you can refresh it simply by removing it, then immediately adding it back:


Runnable resetStylesheet = () -> Platform.runLater( () -> {
scene.getStylesheets().clear();
scene.getStylesheets().add( stylesheet );
} );


This works pretty well. But if you don't want to build it yourself, ScenicView can also watch stylesheet in external files (but not inside jars), and TornadoFX also supports that, so you have a few choices there.


Conclusion


Writing an application in JavaFX was a pretty nice experience. I had a brief experience writing a JavaFX application professionally several years ago (when JavaFX was still in its early days, which is not the case anymore), so I certainly had a good head start... but I've also worked as a web developer and I find it hard to believe that anyone would prefer the webstack to working with a sane environment like the JVM.

The application I wrote, LogFX, works very well in my opinion, and it achieves the goals of being very fast, responsive and at the same time look good on all operating systems without changes... please try it yourself and let me know what you think:


curl -sSfL https://jcenter.bintray.com/com/athaydes/logfx/logfx/0.7.0/logfx-0.7.0-all.jar -o logfx.jar


Even though it is a fully functional app, the jar weighs in at just 303 KB. That's 0.3MB, and it includes a couple of pictures, the fontawesome TTF file and some HTML and CSS besides the Java class files!!

Of course, that does not include the JVM itself, but the JVM is not part of the application and can be shared between many apps! With Java 9, you can create native executables with just the JVM parts you need, though, so if your users would not be satisfied with just the jar, you can package it as a native app, as I showed in a previous blog post (a small native JVM app seems to take up around 35MB, or 21MB after stripping it).

To run LogFX you need around 50MB of RAM (not for LogFX itself, nearly all of that is used up by JavaFX itself). You can verify that by starting it with this command:


java -Xmx50m -jar logfx.jar


That's a world of difference from an Electron app, which typically needs 200MB just to open.

JavaFX is not perfect and there's many areas where some improvements are still needed. Distribution and auto-update is one of them. The current solution, JNLP and Java WebStart, seems to be poorly implemented, though there are community alternatives such as Getdown and FxLauncher, or if you want a proper native installer, the commercial solution is Install4J (notice that Install4J has free licenses for open source projects).

There is a lot of things I did not have time to mention in this already rather long blog post about JavaFX that I think you should check out if you're interested:


 

Spock-Reports - great test reports that you actually want to read

posted Sep 26, 2017, 11:34 AM by Renato Athaydes   [ updated Sep 26, 2017, 11:52 AM ]

Spock-Reports introduced in version 1.3.0 the possibility to embed source code in reports, a feature we call vivid reports.

This may sound like a crazy idea if you are not used to reading and writing Spock specifications! But trust me, the reports created when vivid report is enabled look awesome!

Spock specifications (or tests, if you prefer) tend to be highly readable. Many people have said that the code is normally clear enough that they'd prefer to skip the description block you would see in traditional Spock specifications and just show the code in reports. That's how vivid reports were born.

Just as importantly, Spock error messages (thanks mostly to Groovy power-assertions) are as good as they get.


Simple Spock Report
  A Spock Report for a failed test


When combined, awesome error messages and embedding the source code make these reports extremely useful (unlike reports from most other testing frameworks)!

For example, take this little test that is used to test spock-reports itself:


@Unroll
def "An @Unrolled spec with x=#x and y=#y"() {
setup:
"nothing"
expect:
x == 0
and:
"An error if y is 5"
if ( y == 5 ) {
throw new RuntimeException( 'y is 5' )
}

where:
x | y
0 | 1
2 | 3
0 | 5
}


It is obviously going to fail in the second (because x != 0) and third (because y == 5) iterations.

Here's what the report for the failed iterations look like:


Notice that when a test is Unrolled, spock-reports shows each iteration as a separate feature, unlike on the first report shown at the top of this blog post, where the test is example-based but not unrolled.


Older Vivid Report
 Example of a failed vivid report


This is what the reports look like in version 1.3.0 and 1.3.1.

In version 1.3.2, which is just about to come out, we take things even further and highlight the line of code that caused the test to fail!

Below, you can the same section shown in the report above, but with the line that caused the test to fail highlighted:



New Vivid Report
 Vivid report with line of code that caused the test to fail highlighted


This is really nice. Notice that even the line number is shown as a comment at the end of the guilty line.


Real-world project example


To give a more realistic example, let's look at a Specification for another side project of mine, LogFX. As a log viewer, LogFX needs to watch files for changes, and the FileChangeWatcherSpec tests the code that does it.

Below, we see what the default spock-report for the first feature test looks like:


Realistic standard spock-report
Real-world project report snippet example


This is good and nice for non-coders... but from experience, most actual consumers of this information are technical people, even if not only programmers... most of them would appreciate the further details provided by source code embedded in the report, as shown below (top summary omitted for brevity):



Real-world project report with source code
Real-world project report snippet with source code example


A lot more information is conveyed in this report, but it still looks like a report (as opposed to just printed code, for example)... and in my modest opinion, this is much more useful.

It allows teams to see what's tested, discuss not only the test plan but the implementation... as a side-effect, it might remind the developers writing the test to try to make the tests just a bit more readable as well, as this increases the visibility of the test code. Finally, when the test fails, you get right to the heart of the information you need:



Failure Real-world report with source code
Failed test real-world project report with source code


Even the stack-trace could be displayed if desired by setting the com.athaydes.spockframework.report.internal.HtmlReportCreator.printThrowableStackTrace property to true.

This failed test report is quite useful! The cause of the test failure is much easier to find if you can see exactly how the test is implemented. The fact that Groovy code can be quite compact and easy to read helps a lot... knowing exactly where the error happened without digging through build logs and loads of system output is really valuable as well.


Enabling vivid Spock reports in your tests


First of all, assuming you already use Spock, to start generating reports with spock-reports, just add a test dependency on spock-reports as shown below.

If you're not using Spock yet, head to the Spock documentation and do it now! You won't regret it.


Exclude transitive dependencies to avoid affecting your version of Spock and Groovy.
Notice that spock-reports 1.3.2 requires Spock version 1.1 at least.

Maven

<dependency>
    <groupId>com.athaydes</groupId>
    <artifactId>spock-reports</artifactId>
    <version>1.3.2</version>
    <scope>test</scope>
    <exclusions>
        <exclusion>
            <groupId>*</groupId>
            <artifactId>*</artifactId>
        </exclusion>
    </exclusions>
</dependency>


Gradle

testCompile 'com.athaydes:spock-reports:1.3.2', { transitive = false } 


After you run your tests now, you should find your reports under ${project.buildDir}/spock-reports/. The main report is at index.html.

As vivid reports are not enabled by default, to enable it, set the com.athaydes.spockframework.report.showCodeBlocks system property to true.

In Gradle, that's done by adding these lines to your build file:


test {
    systemProperty 'com.athaydes.spockframework.report.showCodeBlocks', true
}


You can also do that via a config file if you prefer, which allows you to configure just about anything about the reports. See the spock-reports README page for more information.




Faster command line tools with Kotlin

posted Aug 6, 2017, 5:35 AM by Renato Athaydes   [ updated Aug 6, 2017, 5:37 AM ]

Inspired by Faster command line tools with Haskell 
    which itself was inspired by Faster command line tools with Go
        (which inspired Faster command line tools with Nim)
        which itself was inspired by Faster Command Line Tools in D.

In this article, I will show how Kotlin (when run on the JVM, in this case) can be used to create fast command-line tools that run at similar speed to the fastest native languages.

Even though that is the main goal, I will also go beyond the basics to show how programs like this can be optimised to a high degree with the help of basic JVM tools and some intuition.

To recap from the original D blog post:

The original problem statement:

It’s a common programming task: Take a data file with fields separated by a delimiter (comma, tab, etc), and run a mathematical calculation involving several of the fields. Often these programs are one-time use scripts, other times they have longer shelf life. Speed is of course appreciated when the program is used more than a few times on large files.

The specific exercise we’ll explore starts with files having keys in one field, integer values in another. … Fields are delimited by a TAB, and there may be any number of fields on a line. The file name and field numbers of the key and value are passed as command line arguments.



From previous benchmarks I did on Kotlin, I know Kotlin is a pretty fast language... at least it is in the same league as Java and Go, which is a great thing!

Note: when I talk about Kotlin performance, I am talking about running Kotlin with the Oracle JVM, HotSpot. In particular, I've used Java version 8u141, Java HotSpot(TM) 64-Bit Server VM (build 25.141-b15, mixed mode).

Kotlin version 1.1.3-2 compiled for Java 8 (kotlin-stdlib-jre8)

However, the JVM is not known for good performance for small scripts like the one used in this experiment due to the slow startup time and the fact that it may not even have time to warm up properly (i.e. the JIT compiler does not have time to compile efficient machine code, so the JVM ends up just interpreting bytecode most of the time).

Anyway, let's get right into it and see how Kotlin and the JVM perform!


V1 - Initial Version


The first version of the code looks almost exactly the same as the original D code (allowing for syntax differences):


val delim = '\t'

private fun run(file: File, keyIndex: Int, valueIndex: Int) {
val maxFieldIndex = maxOf(keyIndex, valueIndex)
val sumByKey = mutableMapOf<String, Int>()

file.forEachLine { line ->
val fields = line.split(delim)

if (maxFieldIndex < fields.size) {
sumByKey.merge(fields[keyIndex], fields[valueIndex].toInt(), Int::plus)
}
}

if (sumByKey.isEmpty()) {
println("No entries")
} else {
val (key, maxEntry) = sumByKey.maxBy { it.value }!!
println("max_key: $key, sum: $maxEntry")
}
}



The boilerplate of parsing arguments and handling errors is omitted for brevity, but you can see it here.

To run the program, we type:

java -cp "build/libs/*" Max_column_sum_by_keyKt ngrams.tsv 1 2


Here's the result:


max_key: 2006, sum: 22569013
Total time: 3.37 sec


Note: all times reported in this article are self-reported by the Kotlin code, similar to what was done in the Go and Haskell articles.
Using the shell's time command to measure the time adds at most 0.1 seconds to all reported times. 
In any case, comparing the times obtained by different languages using different hardware
should be seen only as a means to get an idea of whether speeds are around the same order of magnitude.


Given the first D version ran in 3.2 seconds (and Go, in 3.49s), Kotlin is not looking too bad!

However, we can definitely improve on this!

Let's see what the poor man's profiler on the JVM, jvisualvm (with the Startup Profiler Plugin), has to say about our program so far:


profiler results 1


First of all, Kotlin seems to be introducing a lot of runtime null checks! We probably don't need that, so it would be nice to find out where that's being called from!


It is possible to disable null checks at runtime, but I won't do that as the options to do so are currently unofficial.


Using the Visual VM Sampler to inspect the call stack, the cause becomes obvious:


call stack



V2 - Java String::split


Kotlin's extension function CharSequence::split, which we use to split each line, seems to be guilty. Even though this function is much more powerful than Java's, all we need here is the simpler Java version, so let's replace the line:


val fields = line.split(delim)


With this:


@Suppress("PLATFORM_CLASS_MAPPED_TO_KOTLIN")
val fields = (line as java.lang.String).split(delim)


Because Java's split takes a String rather than a char, the type of delim had to be changed as well. See the full code here.

Result:


max_key: 2006, sum: 22569013
Total time: 2.73 sec


Nice! That's over 0.6 seconds off the first version with a one-line change!

And this dramatically changes the profiling characteristics of our application:


v2 profiling


The bottleneck now seems to be related to simply iterating over the file's lines.

Other than doing something like the Go guy did and iterate over the file bytes directly rather than the line Strings to solve that, we could try a few simpler things first that might help...

V3 - Wrap Integers into IntBox


Let's see if we can avoid any Integer boxing by implementing a int wrapper, thus avoiding lots of boxing/unboxing values to/from the Map with the results.

Here's the new code with just some simple modifications based on the introduction of the IntBox data class:


data class IntBox(var int: Int = 0) {
operator fun plus(other: Int) {
int += other
}
}

const val delim = "\t"

private fun run(file: File, keyIndex: Int, valueIndex: Int) {
val maxFieldIndex = maxOf(keyIndex, valueIndex)
val sumByKey = mutableMapOf<String, IntBox>()

file.forEachLine { line ->
@Suppress("PLATFORM_CLASS_MAPPED_TO_KOTLIN")
val fields = (line as java.lang.String).split(delim)

if (maxFieldIndex < fields.size) {
sumByKey.getOrPut(fields[keyIndex]) { IntBox() } + fields[valueIndex].toInt()
}
}

if (sumByKey.isEmpty()) {
println("No entries")
} else {
val (key, maxEntry) = sumByKey.maxBy { it.value.int }!!
println("max_key: $key, sum: ${maxEntry.int}")
}
}



As the box allows the code to always perform operations on primitive int's (new values are not put into the Map, the current box's value is just changed), this could run faster, in theory.

The result:


max_key: 2006, sum: 22569013
Total time: 2.52 sec


Hm.. not a huge difference, just a couple of tenths of a second (but faster). Running this a few times, the time does not vary significantly...

One thing we can do now to get to the bottom of this, is to include java.* classes in the profiler configuration (by default, they are excluded) and see exactly where time is being spent:


Profiler data with Java classes


Unfortunately, looks like now we're down to the lowest level Java methods taking up most of the time: copying array ranges (called by the split method), the String constructor itself (there's a lot of Strings being created!), String::indexOf (also from split)...

V4 - Faster Map


I had thought of trying to use a faster Map implementation, but the profiler's information does not seem to back up the assumption that doing that would help much!

But I am stubborn, and decided to try anyway... first, by explicitly instantiating a java.util.HashMap instead of using Kotlin's mutableMapOf function (which gives a LinkedHashMap).

Then, by using a heavyweight library called fastutil, which contains lots of fast implementations of primitive collections for Java (which we can use freely from Kotlin, of course)... I decided to try Object2IntOpenHashMap. It even has an appropriate method with signature

    int addTo(final K k, final int incr)

which I thought would give us at least a modest boost in speed (notice that this method allowed me to get rid of IntBox).

But, as usual with performance guesses, I was wrong!

Here's the result using HashMap:


max_key: 2006, sum: 22569013
Total time: 2.39 sec


And using Object2IntOpenHashMap:


max_key: 2006, sum: 22569013
Total time: 2.58 sec


Well, it turns out fastutil was not too fast on this problem (the library seems to optimise for large collections, not only speed)... but at least we're gonna take the tiny improvement of using HashMap, which brings the total time down another 100ms from the previous result to a respectable 2.39 seconds for a non-native, VM-based language (the only one in the articles I've read so far besides the mention of Python in the D article... it's worth mentioning that Kotlin Native is being developed, so Kotlin may not always rely on the JVM for best performance).

And the code so far remains extremely clean.

But we're still well behind D and Nim, and not quite better than Haskell yet.

V5 - Process bytes, not Strings


To make things even faster, it seems we'll need to bite the bullet and get rid of all those byte-to-String + String::split operations. They are not strictly necessary anyway, as most of the work can be done on the byte level, as shown in the Go article.

But how less readable and maintainable will the code become?!

Judge it by yourself:


const val delim = '\t'

private fun run(file: File, keyIndex: Int, valueIndex: Int) {
val maxFieldIndex = maxOf(keyIndex, valueIndex)
val sumByKey = HashMap<String, IntBox>()

val inputStream = file.inputStream()

val buffer = ByteArray(1024 * 1_000)
var fieldIndex = 0
val currentKey = StringBuilder(12)
val currentVal = StringBuilder(12)

fun startLine() {
if (currentVal.isNotEmpty()) {
sumByKey.getOrPut(currentKey.toString()) { IntBox() } + currentVal.toString().toInt()
}

fieldIndex = 0
currentKey.setLength(0)
currentVal.setLength(0)
}

inputStream.use {
while (true) {
val bytesCount = inputStream.read(buffer)

if (bytesCount < 0) {
break
}

(0 until bytesCount).forEach { i ->
val char = buffer[i].toChar()

if (fieldIndex <= maxFieldIndex) {
when (char) {
delim -> {
fieldIndex++
}
'\n' -> {
startLine()
}
else -> {
if (fieldIndex == keyIndex) {
currentKey.append(char)
} else if (fieldIndex == valueIndex) {
currentVal.append(char)
}
}
}
} else if (char == '\n') {
startLine()
}
}
}
}

if (sumByKey.isEmpty()) {
println("No entries")
} else {
val (key, maxEntry) = sumByKey.maxBy { it.value.int }!!
println("max_key: $key, sum: ${maxEntry.int}")
}
}


Result:


max_key: 2006, sum: 22569013
Total time: 1.39 sec


Really good speed up!

V6 - ByteArray conversion straight into Int


But notice that we're still wastefully converting every value into a String before parsing it as an int, something that can be avoided without much effort...

The code doesn't change too much from the last one shown above, so you can just click here to see the diff if you're interested!

Here's just the ByteArray-to-Int function used:


private fun ByteArray.toIntUpTo(maxIndex: Int): Int {
var result = 0
var multiplier = 1
((maxIndex - 1) downTo 0).forEach { index ->
val digit = this[index].toInt() - 48
result += digit * multiplier
multiplier *= 10
}
return result
}


And here's the result after doing that:


max_key: 2006, sum: 22569013
Total time: 1.18 sec


This is pretty close to the best D version!

But we still have one card up our sleeves with Kotlin!

V7 - Parallelise


Given that the current fastest implementation we have can be easily parallelised, there's no reason to not try doing that.

All we need is a Java's RandomAccessFile to be able to read different parts of the file concurrently, and some setup code to break the file up into partitions.

Here's the setup code:


private fun run(file: File, keyIndex: Int, valueIndex: Int) {
val maxFieldIndex = maxOf(keyIndex, valueIndex)

val partition1 = RandomAccessFile(file, "r")
val partition2 = RandomAccessFile(file, "r")

val fileLength = file.length()
var secondPartitionStartIndex = fileLength / 2L

partition2.seek(secondPartitionStartIndex)

while (true) {
val byte = partition2.read()
secondPartitionStartIndex++
if (byte < 0 || byte.toChar() == '\n') {
break
}
}

val firstPartitionResult = CompletableFuture.supplyAsync {
maxOfPartition(partition1, keyIndex, valueIndex, maxFieldIndex, secondPartitionStartIndex.toInt())
}

val secondPartitionResult = CompletableFuture.supplyAsync {
maxOfPartition(partition2, keyIndex, valueIndex, maxFieldIndex, fileLength.toInt())
}

val sumByKey = firstPartitionResult.join().toMutableMap()
.mergeWith(secondPartitionResult.join())

val maxEntry = sumByKey.maxBy { it.value.int }

if (maxEntry == null) {
println("No entries")
} else {
val (key, value) = maxEntry
println("max_key: $key, sum: ${value.int}")
}
}



Most of the code that was in the run function in the previous examples was moved to a new function called maxOfPartition, which only scans a partition of the file and returns the sumByKey Map only for its partition. After computing the maps for each partition in parallel, we merge them together to get the final result.

Why not use Kotlin coroutines?

Why did I not use Kotlin coroutines for this?

Kotlin coroutines are still experimental, but they are expected to become one of the bigger and coolest features of the language.

However, when I tried to write this code using coroutines, I ran into a compiler bug which made it impossible to use them, unfortunately.

If and when JetBrains addresses this bug, I might update this blog post with the code using coroutines, but for now, CompletableFuture it is.



Here is the result when reading two separate partitions of the file in parallel:


max_key: 2006, sum: 22569013
Total time: 0.71 sec


Even though I got some variance between different runs, the above result seems to be around the average for this code (slower runs ended in around 0.9 seconds, but the fastest runs were down at 0.67 seconds, with most runs close to the latter than the former).

So, even though we had to employ some pretty advanced stuff compared to the simple v1, we were able to increase the speed from over 3 seconds to well under 1 second. And, remember, we're processing a 192.4MB file.

So, I hope this shows that writing Kotlin command-line application that run fast is possible... and yes, you can get speeds similar to Go and D.


Bonus solution

Bonus solution: MappedByteBuffer

As a bonus solution, I decided to map the file bytes into a MappedByteBuffer, which makes reading files extremely fast!

I just replaced RandomAccessFile with MappedByteBuffer, making the necessary changes to compile, but otherwise keeping the last version of the code unchanged (i.e. two buffers running in parallel).

This is not a fair solution as it maps the whole file into memory rather than consuming small chunks of it at a time like the previous solutions did (which allows reading files of any size, not just whatever fits into memory).

In any case, if you don't need to worry about files that won't fit into memory, this would be the fastest thing to do.

Here's the result:

max_key: 2006, sum: 22569013
Total time: 0.58 sec


And that's our absolute best time!



Please leave your comments on Reddit.


Summary of results:

V1 - 3.37s

V2 - 2.73s (java.lang.String::split)

V3 - 2.52s (IntBox)

V4 - 2.39s (java.util.HashMap)

V5 - 1.39s (process bytes, not Strings)

V6 - 1.18s (ByteArray -> Int)

V7 - 0.71s (parallelise)

(Bonus version) - 0.58s (MappedByteBuffer)


The (gzipped) file used as an input can be downloaded from here:


Repository containing all the code shown in this blog post:






Kotlin's hidden costs - Benchmarks

posted Jun 25, 2017, 6:56 AM by Renato Athaydes   [ updated Oct 21, 2017, 10:27 AM ]

This article has been translated into Japanese!


A series of blog posts called Exploring Kotlin’s hidden costs , written by @BladeCoder, demonstrated how certain Kotlin constructs have a hidden cost.


The actual hidden cost is normally due to the instantiation of an invisible Object or the boxing/unboxing of primitive values. These costs are specially hard to see for a developer who doesn't understand how the Kotlin compiler translates such constructs to JVM bytecode.

However, just talking about hidden costs without putting some numbers on said costs makes one wonder how much they should actually worry about them. Should these costs be taken into consideration everywhere in the codebase, meaning that some Kotlin constructs should just be forbidden outright? Or should these costs only be taken into consideration in the tightest inner loops?

Even more provocatively, do these so-called costs actually result in performance penalties (given how the JVM actively optimises code at runtime, compiling it to efficient machine code based on actual usage, the answer to this question may not be as clear as it seems)?

Without putting numbers to the hidden costs, that's impossible to answer.

For that reason, I decided to write JMH benchmarks to try to quantify the actual costs of each Kotlin construct mentioned in all the 3 parts of that blog post series published so far.

Methodology and system


Some of the Kotlin constructs' costs mentioned in the blog posts can be directly compared to the equivalent Java construct. For example, the cost of Kotlin lambdas can be directly compared to the cost of Java lambdas. However, many Kotlin constructs don't have a Java equivalent, in which case instead of comparing the Kotlin construct with the equivalent Java version of it, I compare them with the author's suggestions to improve on the costly constructs.

The code is on GitHub, so anyone can run it in their own system to see if the numbers match (it would be very interesting to collect some results in the Reddit's comments) in different systems.

If you do run them, beware the full benchmark takes a few hours to run.

All results were collected using the following system:

Macbook Pro (2,5 GHz Intel Core i7, 16GB of RAM)

Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

The Kotlin version (1.1.3) and JMH (0.5.6) are the latest as of writing (see the pom.xml).

Update: Android ART benchmarks are available in a WillowTreeApps.com blog post!


Part 1

https://medium.com/@BladeCoder/exploring-kotlins-hidden-costs-part-1-fbb9935d9b62


Higher-order functions and Lambda expressions


In this first example, it was not very clear what cost the author was talking about. It appears that he is just referring to the cost of using Kotlin lambdas, given that Kotlin, by default, targets the Java 6 VM which doesn't have lambdas.

However, the advice given later in the post is specific to capturing lambdas, not just any lambdas, and specifically, not the one given in the example:


fun transaction(db: Database, body: (Database) -> Int): Int {
db.beginTransaction()
try {
val result = body(db)
db.setTransactionSuccessful()
return result
} finally {
db.endTransaction()
}
}


Which is used with the following syntax:

    
val deletedRows = transaction(db) {
it.delete("Customers", null, null)
}

The hidden cost here appears to be only the fact that a Function Object might be created when the transaction function is invoked above. But as the author himself notices, that's not the case for this particular function because it is not a capturing lambda, so a singleton Function instance is created and used on every invocation.

The only remaining cost mentioned in the post is not a runtime cost: the extra 3 or 4 methods created by the Function class generated by the Kotlin compiler.

Anyway, I decided to check whether there's a real runtime cost related to Kotlin lambdas when compared to Java 8's lambdas because I was left with the impression that such cost should exist from the description of the problem (the cost of using a capturing lambda, which the actor advises against in the advice given in this part of the post, will be benchmarked in Part 2, so read on).

The equivalent function was implemented by me using Java 8:


public static int transaction( Database db, ToIntFunction<Database> body ) {
db.beginTransaction();
try {
int result = body.applyAsInt( db );
db.setTransactionSuccessful();
return result;
} finally {
db.endTransaction();
}
}


The syntax for calling this function in Java 8 is just slightly different from Kotlin:


 int deletedRows = transaction( db, ( database ) ->
database.delete( "Customer", null, null ) );

And here is the results of the benchmark comparing the Kotlin version with the Java 8 version:

RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part1.KotlinBenchmarkPart1.javaLambda                                          thrpt       200  1024302.409     1851.789   ops/ms

c.a.k.part1.KotlinBenchmarkPart1.kotlinLambda                                        thrpt       200  1362991.121     2824.862   ops/ms






In the above chart, higher is better (more ops/ms).

Notice that this example should also show the overhead of using a Kotlin lambda which returns an Integer VS the specialized version (ToIntFunction) used in the Java 8 example.

However, the Kotlin lambda seems to be significantly faster than the Java lambda. So, the cost here seems to be negative, as Kotlin actually ran around 30% faster than Java! The mean error is a little bit larger for Kotlin than for Java, but that' not even visible in the above chart, which does attempt to show the error bar (but it's just too small to be seen).

Well, in any case, the solution offered by the author to make the Kotlin lambda less costly (even though it doesn't seem like you should worry about that) is to inline the function, which can be done by simply declaring the transaction function with the inline keyword.

When that's done, we get the following result:

RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part1.KotlinBenchmarkPart1.kotlinInlinedFunction                               thrpt       200  1344885.445     2632.587   ops/ms





As you can see, using an inline function in this example did not significantly improve the performance of the Kotlin lambda at all. If anything, it got just a little bit worse.

I have absolutely no idea why these results are the complete opposite of what the author and I, admittedly, expected. Looking at the benchmark code, I can't see anything that might be obviously wrong, so I am carefully confident that these figures are real.


UPDATE: See this discussion for possible reasons for the surprise results.


Companion Objects


As the author shows, companion objects seem to imply an overhead due to the generated synthetic getters and setters for access to class properties. In the worst case scenario, the first getter might need to even call a second getter, an instance method of the companion object, in order to get a simple constant value.

I decided to combine the companion object examples shown in the blog post to try to measure the cost of the worst case scenario using the following Kotlin class:


class MyClass private constructor() {

companion object {
private val TAG = "TAG"

fun newInstance() = MyClass()
}

fun helloWorld() = TAG
}

Here's the benchmarked Kotlin function:


fun runCompanionObjectCallToPrivateConstructor(): String {
val myClass = MyClass.newInstance()
return myClass.helloWorld()
}


The cost of the above Kotlin code is being compared against the equivalent, straight-forward Java implementation using just a simple static final String within the class itself:


class MyJavaClass {

private static final String TAG = "TAG";

private MyJavaClass() {
}

public static String helloWorld() {
return TAG;
}

public static MyJavaClass newInstance() {
return new MyJavaClass();
}
}

The Java method used was this one:


public static String runPrivateConstructorFromStaticMethod() {
MyJavaClass myJavaClass = newInstance();
return myJavaClass.helloWorld();
}

RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part1.KotlinBenchmarkPart1.javaPrivateConstructorCallFromStaticMethod          thrpt       200   398709.154      800.190   ops/ms

c.a.k.part1.KotlinBenchmarkPart1.kotlinPrivateConstructorCallFromCompanionObject     thrpt       200   404746.375      621.591   ops/ms





Again, Kotlin seems to have a better performance than Java, if only by a tiny margin this time.

Part 2


Local Functions


In this part of the blog post, the author theorizes about the hidden costs of Kotlin's local functions. The only cost, it seems, is the creation of a Function object for capturing functions, but not for functions that do not capture anything from its context.

To test that, we start with a Java local function, or lambda, that avoids boxing but captures one variable from the context:


public static int someMath( int a ) {
IntUnaryOperator sumSquare = ( int b ) -> ( a + b ) * ( a + b );

return sumSquare.applyAsInt( 1 ) + sumSquare.applyAsInt( 2 );
}

The exact same function is implemented in Kotlin as an example given in the blog post:


fun someMath(a: Int): Int {
fun sumSquare(b: Int) = (a + b) * (a + b)

return sumSquare(1) + sumSquare(2)
}


A second Kotlin version which avoids capturing anything from its context is also tried:


fun someMath2(a: Int): Int {
fun sumSquare(a: Int, b: Int) = (a + b) * (a + b)

return sumSquare(a, 1) + sumSquare(a, 2)
}


RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part2.KotlinBenchmarkPart2.javaLocalFunction                                   thrpt       200   897015.956     1951.104   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.kotlinLocalFunctionCapturingLocalVariable           thrpt       200   909087.356     1690.368   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.kotlinLocalFunctionWithoutCapturingLocalVariable    thrpt       200   908852.870     1822.557   ops/ms




By now, it should be no surprise! Kotlin again seems to beat Java by a tiny margin. And that means, once again, if there's any cost to Kotlin, that cost is negative!


Null safety


The Kotlin compiler adds a null-check on each public function's non-null parameter. But how much does that cost?

Let's see.

Here's the tested Kotlin function:


fun sayHello(who: String, blackHole: BlackHole) = blackHole.consume("Hello $who")


Blackhole is a JMH class that can be used to consume values during benchmarks, ensuring that the compiler cannot just avoid computing the value, making the benchmark meaningless.


And the Java baseline:


public static void sayHello( String who, BlackHole blackHole ) {
blackHole.consume( "Hello " + who );
}


RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part2.KotlinBenchmarkPart2.javaSayHello                                        thrpt       200    73353.725      155.551   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.kotlinSayHello                                      thrpt       200    75637.556      162.963   ops/ms






Again, the cost of using Kotlin is negative, or in other words, using Kotlin seems to help with performance over using Java, going against our predictions based on differences in the bytecode.

Note: I skipped benchmarking the Nullable Primitive Types Part because that is not, in my opinion, a hidden cost of Kotlin as compared to Java, as in Java nullable primitives would also incur the exact same boxing costs as in Kotlin.

Varargs


The cost of using varargs for method parameters, as the author points out, occurs only when you need to use the spread operator to use an existing array as a method argument, something that is not necessary in Java.

So, to test the overhead, we compare the following Java method call:


public static void runPrintDouble( BlackHole blackHole, int[] values ) {
printDouble( blackHole, values );
}

public static void printDouble( BlackHole blackHole, int... values ) {
for (int value : values) {
blackHole.consume( value );
}
}


With the equivalent Kotlin implementation:


fun runPrintDouble(blackHole: BlackHole, values: IntArray) {
printDouble(blackHole, *values)
}

fun printDouble(blackHole: BlackHole, vararg values: Int) {
for (value in values) {
blackHole.consume(value)
}
}


RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part2.KotlinBenchmarkPart2.javaIntVarargs                                      thrpt       200   173265.270      260.837   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.kotlinIntVarargs                                    thrpt       200    83621.509      990.854   ops/ms





Finally, a Kotlin hidden cost which you should definitely avoid! Using Kotlin's spread operator, which causes a full copy of the array to be created before calling a method, has a very high performance penalty (and that might increase with the size of the array). In our case, the Java version ran 200% faster than the seemingly equivalent Kotlin version.


Note: The Passing a Mix of Arrays and Arguments Part is also skipped because there's just no equivalent in Java to compare with.

Part 3


Delegated Properties

To measure the actual cost of using delegated properties in Kotlin, I decided to use the most efficient possible equivalent implementation in Java as a baseline. This may not be totally fair to Kotlin as the two things are not the same, and delegated properties enable using patterns that are just not possible in Java.

However, with this in mind, I think it is useful to know just what kind of cost this really entails, even if compared to a Java version that is specifically written for this one case.

The Java baseline uses the following, trivial classes:


class DelegatePropertyTest {

public static String stringValue = "hello";

public static String someOperation() {
return stringValue;
}

}

class Example2 {
public String p;

public void initialize() {
p = DelegatePropertyTest.someOperation();
}
}


As you can see, the caller must remember to call initialize on Example2 in order to initialize the p property!


public static void runStringDelegateExample( BlackHole blackHole ) {
Example2 example2 = new Example2();
example2.initialize();
blackHole.consume( example2.p );
blackHole.consume( example2.p );
}


The Kotlin code uses a delegate class for initializing the p property:


class StringDelegate {
private var cache: String? = null

operator fun getValue(thisRef: Any?, property: KProperty<*>): String {
var result = cache
if (result == null) {
result = someOperation()
cache = result
}
return result!!
}

operator fun setValue(thisRef: Any?, property: KProperty<*>, value: String) {
cache = value
}
}



class Example {
var p: String by StringDelegate()
}


And the Kotlin test function does roughly the same thing as the Java test function, except it doesn't need to explicitly initialize the Example class property:


fun runStringDelegateExample(blackHole: BlackHole) {
val example = Example()
blackHole.consume(example.p)
blackHole.consume(example.p)
}


RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part3.KotlinBenchmarkPart3.javaSimplyInitializedProperty                       thrpt       200   274394.088      554.171   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinDelegateProperty                              thrpt       200   255899.824      910.112   ops/ms





Here, we see that there's a small cost, in the order of 10%, associated with using Kotlin delegated properties when compared to manual Java property initialization.

Note: We skip Generic Delegates because, again, the cost they might incur is related to boxing/unboxing primitive types, not the feature itself.
We also skip the Lazy Delegate Part as that's not a hidden cost, just an informational section regarding how to correctly specify lazy delegates synchronization properties.


Ranges (indirect reference)


To find out the cost of using ranges, as Java does not currently have the equivalent concept at all, we compare the performance of the suggested solutions to the posed performance problems in most examples.

To start with, we compare the cost of using a range with at least on indirection, with just using a range directly where it's needed.

The code that uses indirection involves getting the range from a calling a getter:


private val myRange get() = 1..10

fun isInOneToTenWithIndirectRange(i: Int) = i in myRange


As opposed to using a range directly:


fun isInOneToTenWithLocalRange(i: Int) = i in 1..10


RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part3.KotlinBenchmarkPart3.kotlinIndirectRange                                 thrpt       200  1214464.562     2071.128   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinLocallyDeclaredRange                          thrpt       200  1214883.411     1797.921   ops/ms





Even though there seems to be a tiny cost associated with using indirect references to ranges, the cost is not significant.


Ranges (non-primitive types)


Another range cost found by the author is that, when Ranges are used with non-primitive types, a new ClosedRange instance is created even for locally declared ranges, as in this example:


fun isBetweenNamesWithLocalRange(name: String): Boolean {
return name in "Alfred".."Alicia"
}


Hence, the above should be more expensive than this:


private val NAMES = "Alfred".."Alicia"

fun isBetweenNamesWithConstantRange(name: String): Boolean {
return name in NAMES
}


RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part3.KotlinBenchmarkPart3.kotlinStringRangeInclusionWithLocalRange            thrpt       200   211468.439      483.879   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinStringRangeInclusionWithConstantRange         thrpt       200   218073.886      412.408   ops/ms





It turns out that, yes, it is better to use constant, non-primitive ranges, than locally declared ones, if you need the absolute best performance possible.

The cost of using local ranges instead of a constant range is around 3%, so this one is not something to worry too much about, though.

Ranges (iteration)


One more potential issue with ranges arises when we iterate over them.

Iterating over a primitive range should have zero overhead:


fun rangeForEachLoop(blackHole: BlackHole) {
for (it in 1..10) {
blackHole.consume(it)
}
}


However, iterating using the forEach method should have an overhead, according to the blog post:


fun rangeForEachMethod(blackHole: BlackHole) {
(1..10).forEach {
blackHole.consume(it)
}
}


As should iterating over a range created using a step:


fun rangeForEachLoopWithStep1(blackHole: BlackHole) {
for (it in 1..10 step 1) {
blackHole.consume(it)
}
}


RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part3.KotlinBenchmarkPart3.kotlinRangeForEachFunction                          thrpt       200   108382.188      561.632   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinRangeForEachLoop                              thrpt       200   331558.172      494.281   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinRangeForEachLoopWithStep1                     thrpt       200   331250.339      545.200   ops/ms




The above graph shows that using the forEach function for a Range, as the author predicted, should absolutely be avoided. It performs 300% slower than a simple for-loop!

On the other hand, using an explicit step does not seem to impact the performance of a range for-loop, contradicting the advice in the blog post.

Iterations: Collection indices


Finally, let's measure the cost of using indices on custom classes that are not optimised by the compiler.

For this example, we create a mock version of SparseArray:


class SparseArray<out T>(val collection: List<T>) {
fun size() = collection.size
fun valueAt(index: Int) = collection[index]
}


As suggested by the author, we extend it with a custom indices property:


inline val SparseArray<*>.indices: IntRange
get() = 0..size() - 1


Now, we iterate over the indices:


fun printValuesUsingIndices(map: SparseArray<String>, blackHole: BlackHole) {
for (i in map.indices) {
blackHole.consume(map.valueAt(i))
}
}


The better solution, according to the author, is to use lastIndex instead:


inline val SparseArray<*>.lastIndex: Int
get() = size() - 1



fun printValuesUsingLastIndexRange(map: SparseArray<String>, blackHole: BlackHole) {
for (i in 0..map.lastIndex) {
blackHole.consume(map.valueAt(i))
}
}


RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part3.KotlinBenchmarkPart3.kotlinCustomIndicesIteration                        thrpt       200    79096.631      134.813   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinIterationUsingLastIndexRange                  thrpt       200    80811.554      122.462   ops/ms





Even though it might be slightly safer to use a range from 0 to lastIndex to iterate over a custom collection, the impact of using indices is really small, it seem to only run around 2% slower.

Conclusion


So, which features should you use without concern, and which ones should you avoid?!

The ones with a green tick mark below are the features that can, according to this benchmark, be used without concern (cost below 5%).

The red ones should be avoided if possible, unless performance is just a secondary concern.

In any case, I hope this analysis has demonstrated how, when it comes to performance, the only thing you can be sure about is that without measuring, you know nothing.




pass      Higher-order functions and Lambda expressions









No evidence was found to suggest that Kotlin lambdas and higher-order functions should be avoided. To the contrary, they seem to run faster than Java 8 lambdas.


pass      Companion Objects









Companion Objects have no significant performance costs that could be measured, so there's no reason to avoid them from a performance perspective.


pass      Local Functions








Local Functions, capturing or not, do not seem to affect the performance of Kotlin code.


pass
      Null safety









Kotlin null-safety checks appear to have a negligible performance impact that can be safely ignored.


fail       Varargs + Spread Operator

Kotlin varargs, when used with the spread operator, have a high performance cost due to an extra, unnecessary array copy. Avoid it if performance is a concern.


fail       Delegate Properties

Avoid delegated properties on performance-critical code. Even though the overhead is quite small, around 10%, this may be unacceptable in certain circumstances.


pass
      Indirect access to Range










No performance impact was observed from accessing Ranges indirectly.


pass
      Ranges (local, non-primitive types)










The cost of using local Ranges of non-primitive types is almost insignificant (measured to be around 3%), so only avoid in the most extreme circumstances where performance is the main concern.



fail
       Ranges (forEach function)

Absolutely avoid calling forEach on Ranges. The cost is extremely high at around 300%. Hopefully, the Kotlin team will be able to address this problem in time, but for now, it's a bad idea to use it.



pass
      Ranges (iteration with explicit step)










Using an explicit step does not seem to impact on the speed of an iteration over Ranges.


pass
       Use of indices on custom collection










Using the indices property on a custom Collection did not present a very significant cost over using a range to lastIndex. There is still a cost of around 2%, so using lastIndex may still be good advice for performance-critical applications.


All Results


Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part1.KotlinBenchmarkPart1.empty                                               thrpt       200  3540527.759    23025.839   ops/ms

c.a.k.part1.KotlinBenchmarkPart1.javaLambda                                          thrpt       200  1024302.409     1851.789   ops/ms

c.a.k.part1.KotlinBenchmarkPart1.javaPrivateConstructorCallFromStaticMethod          thrpt       200   398709.154      800.190   ops/ms

c.a.k.part1.KotlinBenchmarkPart1.kotlinInlinedFunction                               thrpt       200  1344885.445     2632.587   ops/ms

c.a.k.part1.KotlinBenchmarkPart1.kotlinLambda                                        thrpt       200  1362991.121     2824.862   ops/ms

c.a.k.part1.KotlinBenchmarkPart1.kotlinPrivateConstructorCallFromCompanionObject     thrpt       200   404746.375      621.591   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.javaIntVarargs                                      thrpt       200   173265.270      260.837   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.javaLocalFunction                                   thrpt       200   897015.956     1951.104   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.javaSayHello                                        thrpt       200    73353.725      155.551   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.kotlinIntVarargs                                    thrpt       200    83621.509      990.854   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.kotlinLocalFunctionCapturingLocalVariable           thrpt       200   909087.356     1690.368   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.kotlinLocalFunctionWithoutCapturingLocalVariable    thrpt       200   908852.870     1822.557   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.kotlinSayHello                                      thrpt       200    75637.556      162.963   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.javaSimplyInitializedProperty                       thrpt       200   274394.088      554.171   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinCustomIndicesIteration                        thrpt       200    79096.631      134.813   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinDelegateProperty                              thrpt       200   255899.824      910.112   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinIndirectRange                                 thrpt       200  1214464.562     2071.128   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinIterationUsingLastIndexRange                  thrpt       200    80811.554      122.462   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinLocallyDeclaredRange                          thrpt       200  1214883.411     1797.921   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinRangeForEachFunction                          thrpt       200   108382.188      561.632   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinRangeForEachLoop                              thrpt       200   331558.172      494.281   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinRangeForEachLoopWithStep1                     thrpt       200   331250.339      545.200   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.javaStringComparisons                               thrpt       200   211488.726      450.531   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinStringRangeInclusionWithConstantRange         thrpt       200   218073.886      412.408   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinStringRangeInclusionWithLocalRange            thrpt       200   211468.439      483.879   ops/ms



JGrab - run Java code fast, from source, with a little Rust help

posted May 21, 2017, 4:19 AM by Renato Athaydes   [ updated May 21, 2017, 4:24 AM ]

One of the main problems Java has, in my opinion, is how hard it is to run a simple Java class without the help of a complex build system.

If you don't need any external dependencies, it may be easy enough to run javac Hello.java, followed by java Hello (but even this can get tiring quickly). But if you need dependencies, then it becomes completely impractical to download the dependencies by hand, and then type the full classpath every time you need to run the code.

Not to mention the performance of code run in this way is pretty bad (due to the JVM startup and warmup times), even with the great improvements seen in the latest Java versions.

For this reason, even veteran Java developers often choose to write tiny applications in JavaScript or Python, for example, or even bash on a bad day. These scripting languages let you write a file and run it, with decent performance, without fuss... and installing dependencies with npm (for NodeJS) or pip (for Python) is just a simple command away.

But using these languages comes with its own problems if you use Java as your main language: unfamiliar syntax and libraries, often lower quality libraries than what we're used in the Java world, low performance for longer-running tasks, often complex ecosystems once you start trying to use more advanced things (especially in the JS world), lack of static type checking causing trivial errors to only be caught at runtime, sometimes.

Each reason may be weak on its own, but put together they really add up and, at least for me, make it hard to justify using these languages for anything but the most trivial utilities.

But the Java world offers basically no alternatives (except perhaps Groovy, which solves most, but not all of these problems).

That's why I created JGrab!

JGrab lets you run your Java source file quickly and with 0 setup. If you need to use dependencies, just add them to the Java file itself and JGrab will download and cache it for the next runs.

As explained in the JGrab README page, you can download and install JGrab on most OSs with the following command:

curl https://raw.githubusercontent.com/renatoathaydes/jgrab/master/releases/install.sh -sSf | sh
If you're on Unix-like systems (or Windows using a bash system), create a link to the JGrab executable to make it easy to run it from anywhere:

sudo ln -s $HOME/.jgrab/jgrab-client /usr/local/bin/jgrab
And that's it! You're all set up.

Using JGrab


To try it out, you can run a simple Java expression:

jgrab -e 2 + 2

The first time you run JGrab, it will automatically start the JGrab Daemon, which will be used in to compile/run code in subsequent runs.


Or a statement (statements are things that do not return any value, as opposed to expressions) by terminating it with a semi-colon:

jgrab -e 'System.out.println("Hello world!");'
To run a Java class (must either be Runnable or have a traditional main method), just pass its path to JGrab.

For example, create the following file in the local directory:

Hello.java

public class Hello implements Runnable {
public void run() {
System.out.println( "Hello JGrab" );
}
}


Then, run it with:

jgrab Hello.java
You can also pipe source code directly to JGrab:

cat Hello.java | jgrab
If you want to use some dependency, declare it as a #jgrab groupdId:artifactId:[version] comment directive anywhere in the source.

For example, to use Guava:

UsesGuava.java

 // #jgrab com.google.guava:guava:19.0

import com.google.common.collect.ImmutableMap;
import java.util.Map;

public class UsesGuava {

public static void main(String[] args) {

Map<String
, Integer> items = ImmutableMap.of("coin", 3, "glass", 4, "pencil", 1);

items.entrySet()
.stream()
.forEach(System.
out::println);
}
}

You can run the file with JGrab, as usual:

jgrab UsesGuava.java

Notice that JGrab will download all transitive dependencies the first time you run this file...
subsequent runs will use the cached jars, so will run much faster!


JGrab Performance


To get an idea JGrab's performance compared to running a compiled Java class using the java command, here's the results I got on my machine using Java 8 update 131, as measured by using the shell time command:

JGrab command:

jgrab UsesGuava.java

Javac + Java command:

javac -cp ~/.ivy2/cache/com.google.guava/guava/bundles/guava-19.0.jar UsesGuava.java
java
-cp ~/.ivy2/cache/com.google.guava/guava/bundles/guava-19.0.jar:. UsesGuava

 Run JGrab (secs)  Java (secs) 
 1 0.26  0.15 
 2  0.13  0.16
 3  0.10  0.15
 4  0.11  0.15
 5  0.10  0.15


So, performance-wise, JGrab does not suffer much from having to compile the source before running it, if at all (of course, this will depend on how long the Java compiler takes to generate the bytecode, but from experience, the compiler is really fast even for bigger files). And the simplicity of using JGrab VS javac+java is pretty obvious!

I hope to improve JGrab performance further in the future by re-using the ClassLoader used to execute each source (so that not all classes need to be loaded on each run).

How does JGrab work?


JGrab uses a native client written in Rust to send code to execute to the JGrab Daemon, written in Java.

I should mention that to build the Rust binaries for all different operating systems, I used the japaric/trust template, which makes it easy to leverage GitHub, Travis CI and AppVeyor to build, test and distribute the executables. The install script was copied and adapted from the rustup project (the script detects which OS, architecture, bitness a system has, then automatically selects the appropriate binary to download).

The Java daemon is hosted on JCenter and uses Apache Ivy to resolve dependencies, so it works really well with Maven repositories.

An in-memory Java compiler I wrote for the OSGiaaS project is used to actually compile the Java source. This compiler was originally written for the OSGiaaS Java REPL and is very fast and reliable because it never writes anything to disk and leverages the JavaCompiler mechanism (which means you must have tools.jar, distributed with the Java SDK, in the classpath). This allows JGrab to support both Java 8 and Java 9 seamlessly (all that matters is which Java version you use to run JGrab).

The JGrab client sends code to the JGrab daemon via a socket on 127.0.0.1:5002. It is not meant for remote code execution, though it would probably work if you exposed this port on the network (which I do not recommend at all!). By design, JGrab only handles a single program at a time, so multi-user setups are not supported.

Conclusion


JGrab offers a new alternative to running Java code simply and easily from the command-line. It is part of my long-term crusade against bash programming!

I had already created an alternative to using bash, awk and company on the shell that allows invoking code from a shell in almost any JVM language (including Java, Groovy, Scala, Clojure and Frege)... with JGrab, I hope to make it easy to replace the cryptic languages of times past with proper Java utilities which are debuggable and readable.

JGrab has also been a great exercise to get my feet wet with Rust, and though I found that learning it was quite hard, it was fun to learn really interesting low-level language concepts as well as a quite advanced type-system. I am happy with the results!


A practical guide to Java 9 - compile, jar, run

posted Apr 24, 2017, 2:42 PM by Renato Athaydes   [ updated Apr 28, 2017, 1:33 PM ]

The java and javac commands are rarely used by Java programmers... build tools like Maven and Gradle make that mostly unnecessary.

However, as of writing (April 2017), both Maven and Gradle still don't provide full support for Java 9, so if you want to start using Java 9 now, or if you just want to know how it will all work behind the scenes when you finally start using it with your favourite tool, it is a good thing to learn how to call java, javac and jar to manage your Java code.

The purpose of this guide is to show how these command have changed from the previous Java versions (and they did change substantially in Java 9) and how you can already use them to manage your applications. It will also discuss the new tools, jdeps and jlink.

It is assumed that you have at least a little bit of familiarity with previous versions of the java/javac/jar command and with the Java 9 module system.

If you need to learn the basics of the new module system, check The State of the Module System, which gives a pretty good overview of it.



Getting Java 9


Before getting started with the Java 9 commands, you need to get Java 9!

You can download it from the Oracle website, but I'd recommend using SdkMAN! instead because besides making it trivial to upgrade Java (and many other packages), it lets you switch between Java versions with ease (so you can continue using your current tools without trouble).

You can install SdkMAN! with this command (inspect the bash script here):

curl -s "https://get.sdkman.io" | bash

Then, install Java 9:

Check what's the latest available version with  sdk list java

sdk install java 9ea163

Now, as long as you have other Java versions installed, you can switch between them with  sdk use java <version>.

Compiling/Running the old way


With Java 9 installed, the first thing to do is to write some code to try our tools!

If we don't use a module descriptor, everything looks pretty much the same as before!

Take this simple Java class:

src/app/Main.java

package app;

public class Main {
public static void main( String[] args ) {
System.out.println( "Hello Java 9" );
}
}


Now, as we're not really using any Java 9 feature yet, we can pretty much compile it as we always did:

javac -d out src/app/Main.java

This will create a class file at out/app/Main.class. To run that, you do it, again, just like in all previous versions:

java -cp out app.Main

Which prints the expected message, Hello Java 9.

By the way, the "hello world" program takes around 0.1 seconds to run with Java 9 on my machine (it's a few years old Macbook Pro).
Not bad for a language that needs to boot up a whole VM to run.

Now, let's create a Greeting library, also without any Java 9 feature yet, to see how that works.

Create the following file:

greeting/src/lib/Greeting.java

package lib;

public class Greeting {
public String hello() {
return "Hi there!";
}
}


Change the app.Main class to use this mini library:

src/app/Main.java

package app;

import lib.Greeting;

public class Main {
public static void main( String[] args ) {
System.out.println( new Greeting().hello() );
}
}


Compile the Greeting lib:

javac -d greeting/out greeting/src/lib/Greeting.java

We want to show how to use legacy, non-modular, pre-Java 9 libraries, so let's jar this library without a Java 9 module descriptor or anything (you could do this with the Java 8 compiler, or even some earlier version), just to make sure that this still works:

mkdir libs
jar cf libs/lib.jar -C greeting/out .

This will create the libs/lib.jar file containing just the lib.Greeting class (and an auto-generated manifest file).

You can inspect the jar with the tf option:

jar tf libs/lib.jar

Which should print:

META-INF/
META-INF/MANIFEST.MF
lib/
lib/Greeting.class


Now to compile app.Main, we need to tell the compiler where to find the lib.Greeting class.

We can do that using the good old cp (classpath) option for now:

javac -d out -cp libs/lib.jar src/app/Main.java

Same thing to run the program:

java -cp out:libs/lib.jar app.Main

In MS Windows, remember to replace the : separator in the command above with ;


We could package the app into a jar as well, of course:

jar cf libs/app.jar -C out .

And then, run it with:

java -cp 'libs/*' app.Main

This is a good time to see what our current project directory structure looks like so far:

.
├── greeting
│ ├── out
│ │   └── lib
│ │       └── Greeting.class
│ └── src
│     └── lib
│         └── Greeting.java
├── libs
│   ├── app.jar
│   └── lib.jar
├── out
│   └── app
│       └── Main.class
└── src
    └── app
        └── Main.java


Modularizing the application


So far, nothing new (which is a great thing, as most of what you know is still valid).

But now, let's finally start modularizing our application by creating a module descriptor (always called module-info.java and placed under the root source directory):

src/module-info.java

module com.app {
}

The command to compile a Java 9 module is a little bit different from what we saw previously. Trying to use the same command as before and just adding the module file to the list of files to compile causes an error:

javac -d out -cp 'libs/lib.jar' src/module-info.java src/app/Main.java # doesn't work
src/app/Main.java:3: error: package lib does not exist

import lib.Greeting;

          ^

src/app/Main.java:7: error: cannot find symbol

    System.out.println( new Greeting().hello() );

                            ^

symbol: class Greeting

location: class Main

2 errors


So, what's going on? By just adding the module descriptor, our code no longer compiles. To understand why, we need to understand unnamed modules.


The unnamed module

Any class that is not loaded from a named module is automatically made part of the unnamed module. In our case above, before creating a module descriptor, our code was not part of any module, hence it was associated with the unnamed module. The unnamed module is a compatibility mechanism, basically, that allows code that has not been migrated to Java 9 and modularized yet to be used by Java 9 applications. For this reason, code belonging to the unnamed module have rules akin to Java 8 and earlier: it can see all packages exported by all other modules, and all packages of the unnamed module are exported (ie. visible from other modules).

Once a module descriptor is added to a module, its code is no longer part of the unnamed module and therefore cannot see code from other modules unless it imports them! In the case above, the com.app module does not explicitly require any module, so the greeting library module is not visible to it. It can only "see" the java.base module's packages (which are a mandatory, implicit requirement of all modules).


Java 9 modules, except for the elusive unnamed module discussed above, must declare which other modules they require. In the case of the com.app module, the only requirement is the greeting library. But, as you might have guessed, the greeting library (as well as any of the libraries you use that do not support Java 9 yet) is not a Java 9 module, so how can we declare a requirement on it?

Well, in cases like this, you need to know the name of the library's jar file. That's right, if you have a dependency on a library that has not been converted to use Java 9 modules yet, you need to know what the jar file for that library is called, because Java 9 will convert the file name to a valid Java 9 module name.

This is called an automatic module.

Similarly to unnamed modules, automatic modules can read from all other modules, and all of its packages are exported. But unlike the unnamed modules, it is possible to refer to automatic modules by name from explicit modules.

To figure out the automatic module's name, the compiler converts non-alphabetic characters .'s, so something like the slf4j-api-1.7.25.jar file would become the module name sl4j.api.

We gave the greetings' library jar a pretty bad name: lib.jar. To avoid requiring the lib library, let's rename the jar file to greetings-1.0.jar:

mv libs/lib.jar libs/greetings-1.0.jar

That's a much more standard file name, and now we can tell Java to turn this into an automatic module with a somewhat acceptable name: greetings. And we can require it from the com.app module:

src/module-info.java

module com.app {
requires greetings;
}


Modules are not added to the classpath, like regular jar files... they use the new --module-path  (or -p) option. So, we can now compile our modules with the following command:

javac -d out -p 'libs/greetings-1.0.jar' src/module-info.java src/app/Main.java

Now, to run the app.Main class with the java command, we can use the new --module (-m) option, which takes either a module name (in which case the Manifest file of the module must declare the Main-Class attribute) or the pattern module-name/main-class:

java -p 'libs/greetings-1.0.jar:out' -m com.app/app.Main

And, again, this prints the greeting, Hi there!.

To create and use the app.jar as a runnable jar, instead of the out dir, and using shorter options for the java command to run that jar:

jar --create -f libs/app.jar -e app.Main -C out . 
java -p libs -m com.app

Pretty good! The next step is to modularize the library(ies) used by our application as well.


Modularizing libraries


To modularize a library, you can't do better than to start out by using jdeps, a static analysis tool that is part of the Java SE distribution.

For example, to see the dependencies of our tiny Greetings library:

jdeps -s libs/greetings-1.0.jar
greetings-1.0.jar -> java.base


As expected, the greetings library only depends on the java.base module.

We know that com.app depends on greetings. Let's try to get jdeps to tell us that by removing the module-info.class file from the app.jar file first, then running jdeps on it:

zip -d libs/app.jar module-info.class
deleting: module-info.class

jdeps -s -cp libs/greetings-1.0.jar libs/app.jar
app.jar -> libs/greetings-1.0.jar
app.jar -> java.base


Perfect! But it gets better... we can even ask jdeps to auto-generate a module descriptor for a set of jars. Just tell it where to save the generated files, for example, in generated-mods, and where the jars are:

jdeps --generate-module-info generated-mods libs/greetings-1.0.jar libs/app.jar

This creates two files, generated-mods/app/module-info.java and generated-mods/greetings/module-info.java, with the following contents:

generated-mods/app/module-info.java

module app {
requires greetings;
exports app;
}


generated-mods/greetings/module-info.java

module greetings {
exports lib;
}


Very cool! We can now just add the auto-generated module descriptor for the greetings library in the library's source code, re-package it and have a fully modular hello world application.

javac -d greeting/out $(find greeting/src -name *.java)
jar cf libs/greetings-1.0.jar -C greeting/out .

Now we have both the application and the library we were using fully modularized! After removing the generated and binary files, our application structure now looks like this:

├── greeting
│   └── src
│       ├── lib
│       │ └── Greeting.java
│       └── module-info.java
├── libs
│   ├── app.jar
│   └── greetings-1.0.jar
└── src
    ├── app
    │   └── Main.java
    └── module-info.java



One final improvement you may want to make is to rename the src folders with the module's full names (or add a new level under src with the module's full names), as that's the new recommendation for Java 9.


Notice that to be able to get good data from jdeps, you must provide the locations of the jars of all transitive dependencies that are used in the application, so that it can create a full module graph.

An easy way to get the list of jars a library requires is to use this simple Gradle script shown below. It will print the location of the local jars for all dependencies of the libraries you add in the dependencies section, downloading them if necessary (this example will show the jars for the JavaSlang library, recently renamed VAVR):

build.gradle

apply plugin: 'java'

repositories {
mavenLocal()
mavenCentral()
}

dependencies {
compile 'io.javaslang:javaslang:2.0.6'
}

task listDeps {
println configurations.compile*.absolutePath.join(' ')
}


If you don't have Gradle, you can use SDKMAN! (or your favourite package manager) to install it:

sdk install gradle

To get the list of dependencies, run:

gradle listDeps

Then, give the output to jdeps for analysis and auto-generation of module metadata.

This is the file jdeps outputs for javaslang.match:


module javaslang.match {
requires transitive java.compiler;
exports javaslang.match;
exports javaslang.match.annotation;
exports javaslang.match.generator;
exports javaslang.match.model;
provides javax.annotation.processing.Processor with
javaslang.match.PatternsProcessor;
}

As easy as it gets, so hopefully most Java libraries will be fully modularized within a short amount of time.


Creating a native runtime image


With jlink, Java applications can be distributed as native runtime images that do not require the JVM to be installed in the target system to run.

The following command creates an image for our com.app module without optimisations/compression or stripping debug information:

jlink -p $JAVA_HOME/jmods:libs --add-modules com.app \
  --launcher start-app=com.app \
  --output dist

A smaller distribution can be obtained by using some of jlink options like --strip-debug and --compress:

jlink -p $JAVA_HOME/jmods:libs --add-modules com.app \
  --launcher start-app=com.app \
  --strip-debug --compress=2 \
  --output small-dist

On my machine, the size of the distributions, according to du -sh, are:

du -sh small-dist dist
21M small-dist
35M dist


To launch the application, use the provided launcher in the bin directory:

dist/bin/start-app

This time, without the help of the local java command, you should see the Hi there! message printed out again from our com.app module, with the help of the greeting library!

And with that, we come to the end of this Java 9 guide. There is a lot more about Java 9 and its module system that has not been covered in this guide, but I hope that this guide will show you most of what you need to know to quickly get started with Java 9. Have fun!


Writing JVM bytecode by hand using Groovy

posted Jan 30, 2017, 2:18 PM by Renato Athaydes   [ updated Jan 30, 2017, 2:24 PM ]

JVM bytecode looks similar to Assembly code. It maps to it closely enough that translating it into machine code shouldn't be too much of a challenge.

Writing it can be quite challenging, but it can be quite useful to understand the inner workings of the JVM.

But before we see how to use the Groovy compiler to let us write bytecode by hand (and create pure JVM bytecode that does not require Groovy at runtime!), let's have a quick crash-course on JVM bytecode.

A short introduction to JVM bytecode


Instructions operate on a frame consisting of an operand stack and an array of local variables (amongst other things).

For example, to load the value of an object reference stored in the local variable at index 1 onto the stack:

aload_1

Instance methods always get a reference to the this Object in the local variable at index 0. Method arguments are passed in as the next elements in the local variables array.

So, to invoke the Object#toString() method, for example, within an instance method, one can do:

aload_0
invokevirtual java/lang/Object.toString()Ljava/lang/String;

This will leave the String returned by toString() onto the stack, where it can be used by another JVM instruction.

The above code example shows how JVM methods and types are represented in bytecode. Methods follow the pattern:

<package>/<className>.<method>([argTypes])<returnType>.

Reference types are represented by:

L<package>/<typeName>;

Primitive types, by a single letter:


 Type  JVM representation
 boolean  Z
 byte  B
 char  C
 double  D
 float  F
 int  I
 long  J
 short  S
 void  V


With this knowledge, we can understand much easier the output of javap, the Java class file disassembler.

As an example, take the following, simple Java class:


public class Hello {

public static void main(String[] args) {
System.out.println("Hello world");
}
}


If you compile and run javap on the resulting class file:

javac Hello.java
javap -c Hello


The following output is shown:

public class Hello {
public Hello();
Code:
0: aload_0
1: invokespecial #1 // Method java/lang/Object."<init>":()V
4: return

public static void main(java.lang.String[]);

Code:
0: getstatic #2 // Field java/lang/System.out:Ljava/io/PrintStream;
3: ldc #3 // String Hello world
5: invokevirtual #4 // Method java/io/PrintStream.println:(Ljava/lang/String;)V
8: return
}


As you can see, it's pretty simple. A default constructor is added automatically to all Java classes.

Lets' go through each instruction in the main method:

getstatic java/lang/System.out:Ljava/io/PrintStream;

This loads the static field System.out onto the stack.

ldc #3

Loads the "Hello world" String onto the stack. #3 refers to the position of the String in the constant pool

invokevirtual java/io/PrintStream.println:(Ljava/lang/String;)V

Invokes the println method using the variables in the stack (System.out, "Hello world").

return

The return instruction is used to return the flow control to the caller without returning any value.


To find out what an instruction expects to be in the stack when it is called, and what it leaves behind, you must consult the JVM documentation.

The invokevirtual instruction, for example, uses the stack as follows:


Operand Stack

..., objectref, [arg1, [arg2 ...]] →

...


This means it expects an object reference (on which to call the provided method), plus the method arguments to be on the top of the stack... and after it returns, it leaves whatever was on the stack below the object reference.

Writing JVM bytecode with Groovy


The ability to use local variables makes it much easier to write bytecode than if we had only the stack at our disposal (see Forth for an actual language where that's exactly what you must do!).

But that begs the question:

How much difference does it make to use local variables as opposed to the stack?

To find out, the only way is to try it!

To write bytecode by hand is not very easy, though... There are some tools such as ByteBuddy and the lower-level ASM (on which ByteBuddy is based) that let you generate JVM class files, but even they are too high-level for what I wanted... they don't really let you write bytecode exactly as javap outputs it. For that, the only tool I could find was Jasmin, which has been abandoned, unfortunately, since around 2010... so it is limited to creating Java 6 bytecode.

The good news is that JVM bytecode evolves even slower than Java, the language! So this limitation is not nearly as bad as it sounds.

To make it easier to use Jasmin, I created Grasmin, a Groovy AST (Abstract Syntax Tree) Transformation which reads the first String in the body of a method as JVM bytecode (in the Jasmin syntax, which closely mirrors javap's output).

The advantage of using Grasmin over Jasmin is that the class definition itself can be written in Groovy. This means you can use the class in your other code as if it were just a normal class, even though the implementation will be written in bytecode directly!

This is what a hello-world Grasmin class looks like:


@CompileStatic
@JasminCode
class Test1 {

void exampleJasminCode() {
"""
.limit stack 2
getstatic java/lang/System/out Ljava/io/PrintStream;
ldc "Hello World!"
invokevirtual java/io/PrintStream/println(Ljava/lang/String;)V
return
"""
}

}

That's it! Notice that users of this class don't need to know anything about how it is implemented. It could return values and take arguments. But the implementation of this class contains the exact bytecode typed above.

To prove it, we can attempt a more interesting example that we can use to answer the question posed in the beginning of this section: How much difference does it make to use local variables as opposed to the stack?

Here's a simple Java function that converts an array of integers to an array of Strings:


public class JavaArrays {

public String[] stringify( int[] ints ) {
String[] result = new String[ ints.length ];
for (int i = 0; i < ints.length; i++) {
result[ i ] = Integer.toString( ints[ i ] );
}
return result;
}
}


The implementation is as simple as it gets, in good old-fashioned Java style.

According to javap, the bytecode of this Java class looks like this:



public class grasmin.test_target.JavaArrays {
  public grasmin.test_target.JavaArrays();
    Code:
       0: aload_0
       1: invokespecial #1                  // Method java/lang/Object."<init>":()V
       4: return

  public java.lang.String[] stringify(int[]);
    Code:
       0: aload_1
       1: arraylength
       2: anewarray     #2                  // class java/lang/String
       5: astore_2
       6: iconst_0
       7: istore_3
       8: iload_3
       9: aload_1
      10: arraylength
      11: if_icmpge     29
      14: aload_2
      15: iload_3
      16: aload_1
      17: iload_3
      18: iaload
      19: invokestatic  #3                  // Method java/lang/Integer.toString:(I)Ljava/lang/String;
      22: aastore
      23: iinc          3, 1
      26: goto          8
      29: aload_2
      30: areturn
}


This code uses if_icmpge as the loop condition. It compares the 2 ints at the top of stack, ie. value1 ≥ value2.

My first attempt at implementing this in bytecode was similar to the above, but instead of using id_icmpge, I decided to use ifeq, which compares the int on the top of the stack to zero, because that seemed to be potentially more efficient. To do this, of course, I would have to start the loop from the last index rather than the first.

This is what I ended up with (with lots of comments, as that's the only way I can understand what I am writing!):


@CompileStatic
@JasminCode
class ArraysCode {

String[] stringify( int[] ints ) {
"""\
.limit locals 4
.limit stack 3
aload_1
arraylength ; put the length of the array on the stack
istore_2 ; store the array length on local variable 2
iload_2 ; read the array length
anewarray java/lang/String ; create a String array with the same length as the input array
astore_3 ; store the result in the local variable 3
Loop:
iload_2
ifeq End ; check if the remaining length is 0, go to End if so
iinc 2 -1 ; decrement index
aload_1 ; load the input array
iload_2 ; load current index
iaload ; read current value
invokestatic java/lang/Integer.toString(I)Ljava/lang/String;
aload_3 ; load the result array
swap ; swap so the String value is on top of the stack
iload_2 ; load current index
swap ; swap so the stack has arrayRef, index, value
aastore ; put the result String into the result array
goto Loop ; loop
End:
aload_3
areturn
"""
}

}

If you're wondering about why I used goto... well, because the only way the computer actually knows how to loop, is by using jump instructions such as goto, even if in higher-level languages we can hide those.


Unfortunately, this hand-written bytecode implementation had the exact same performance as the Java code.

I wondered if using one less local variable to store the current index (by keeping it always on the stack instead) would make this code run faster.

So, here is my second attempt (which took hours to write! Bytecode is not easy to write, especially without using local variables too much):



@CompileStatic
@JasminCode( outputDebugFile = 'build/arrays-code.j' )
class ArraysCode {

String[] stringify( int[] ints ) {
"""\
.limit locals 3
.limit stack 5
aload_1
arraylength ; put the length of the array on the stack
dup
dup ; Stack: length, length, length |TOP
anewarray java/lang/String ; create a String array with the same length as the input array
astore_2 ; store the result in the local variable 2
; Stack: length, length |TOP
Loop:
ifeq End ; check if the remaining length is 0, go to End if so
iconst_1 ; Stack: length, 1 |TOP
isub ; decrement index
dup ; Stack: index, index |TOP
aload_2 ; load the result array
swap ; Stack: index, resultArray, index |TOP
dup ; Stack: index, resultArray, index, index |TOP
aload_1 ; load the input array
swap ; Stack: index, resultArray, index, inputArray, index |TOP
iaload ; read current value
; Stack: index, resultArray, index, intValue |TOP
invokestatic java/lang/Integer.toString(I)Ljava/lang/String;
; Stack: index, resultArray, index, stringValue |TOP
aastore ; put the result String into the result array
dup ; Stack: index, index |TOP
goto Loop ; loop
End:
aload_2
areturn
"""
}

}

This implementation is a little bit shorter, and as you may notice, uses one less local variable than the previous ones. I expected this to run faster.

To my surprise, it actually runs consistently slower. Around 50% slower, to be more accurate. I have no idea why, and would love to hear if anyone has any idea!

Is some bytecode instruction I used too slow? Maybe decreasing the index with isub instead of iinc is the problem?

No idea... but this seems to show that, answering the initial question:

Trying to save local variables by using the stack does NOT seem to be a good approach!

Well, in any case, it does not really matter... the important is that writing bytecode by hand is so much fun, if you ask me... in a weird, painful yet rewarding kind-of-way.

By the way, here's the output of javap for the above implementation of ArraysCode:


public class grasmin.test_target.ArraysCode {
  public java.lang.String[] stringify(int[]);
    Code:
       0: aload_1
       1: arraylength
       2: dup
       3: dup
       4: anewarray     #7                  // class java/lang/String
       7: astore_2
       8: ifeq          28
      11: iconst_1
      12: isub
      13: dup
      14: aload_2
      15: swap
      16: dup
      17: aload_1
      18: swap
      19: iaload
      20: invokestatic  #15                 // Method java/lang/Integer.toString:(I)Ljava/lang/String;
      23: aastore
      24: dup
      25: goto          8
      28: aload_2
      29: areturn
}


As you can see, it's identical to the source.

You can compile and test yourself. Here's the first implementation, and here, this last one!

Thanks for reading.

1-10 of 32

Comments