Stop versioning libraries
Post date: Jan 16, 2018 6:12:51 PM
Versioning is one of the most basic concepts software developers have to deal with. But despite the apparent simplicity of the subject, we still haven't got it right!
Let me expand. We are used to a versioning scheme based on a set of numbers and a few qualifiers, so our versions look like 1.2.3 and 1.2.4-rc1. But have you ever stopped to think about what these versions really mean?
When you see a library that is on version 1.2.3, can you confidently claim that it is more mature than another library that is on version 1.0.0? Unfortunately, that's just impossible to tell. From real-world experience, I've come to believe that knowing a version number is almost completely useless. Version numbers are at the whim of library authors... they may not even follow semantic versioning, so version 1.2.0 might be completely different than version 1.1.0... or they might decide to skip a few major versions and go straight from 7 to 10!! But even if the library author follows semver religiously, which version of semver, given that semver itself is versioned?! They may decide to follow simver instead :D!
But seriously, when it comes to libraries (as opposed to apps/products, whose versions are normally meant for final users and marketing purposes, not dependency manager tools), I know you might be shocked by this and think that I have no idea what I am talking about, but try to give me a chance to explain it in the rest of this blog post... after all, one of the initial proponents of this idea was not me, but a really smart guy you may've heard of: Rich Hickey. Perhaps you'll start agreeing with this and join the cause, so in a not-so-distant future we might actually be able to build software confidently, even if we depend on hundreds of interdependent libraries!
This is my request to all library authors:
Don't ever make breaking changes in a library, regardless of the version change!
If breaking changes are unavoidable, change the library name instead.
That's because regardless of which language you use, there's no way to resolve versions correctly if different modules depend on different, incompatible versions of the same library.
I find it funny that some people believe that npm solves that. Or, if you're a little bit older, you might have thought that OSGi did it for Java 18 years ago. Well, they didn't! Because it's impossible.
OSGi will actually throw errors during startup if the two incompatible versions of a library are used by two different bundles and both can "see" each other (so, it actually solved a part of the problem by failing on startup, not runtime, but that doesn't help much and just got OSGi a bad rep), but in a node.js application, things might or might not work! Even though npm includes a module's dependencies into its own node_modules folder (so different consumers may use the correct version they require), it definitely won't work if objects from the two incompatible library versions happen to interact. This will cause difficult to debug runtime errors... that kind of error that only happens on a Saturday night with full moon.
To illustrate the problem, let's imagine we have a Javascript library, say lib1, that creates a tax summary given an income. This library was part of an effort to create tax reports, so the programmer thought it was a good idea to use a formatted String to describe the taxes... something like this:
function createTaxSummary(income) {
return {
"income": "Income = " + income,
"tax": "Tax = " + (income * 0.25),
}
}
exports.createTaxSummary = createTaxSummary;
Another library was used to actually generate a final report. Let's call it lib2. It uses lib1 to calculate the taxes, then returns a nice report:
const lib1 = require('../lib1')
function createReport(taxSummary) {
return "Tax Report\n----------\n" +
taxSummary.income.toUpperCase() + "\n" +
taxSummary.tax.toUpperCase()
}
exports.createReport = createReport;
The programmer made the line that contains the tax amount uppercase because that's the tradition in accounting circles.
The application code developed by another company should just call lib2 to generate the report and print it out, but lib2 did not have a way to add "extra" taxes that can be charged in some states... because that was only noticed when the release was already several weeks late, this feature was hacked into the application code by directly using lib1:
const lib1 = require('../lib1')
const lib2 = require('../lib2')
const income = 100000;
const extras = 2500;
const summary = lib1.createTaxSummary(income);
var report = lib2.createReport(summary);
const extraTaxes = lib1.createTaxSummary(extras).tax;
report += "\nExtra taxes:\n" + extraTaxes.toUpperCase();
console.log(report);
The report now looked perfect!
Tax Report
----------
INCOME = 100000
TAX = 25000
Extra taxes:
TAX = 625
A few years later (the code was working, so no one had the time to go back "fix" the hack), someone noticed that the createTaxReport function was calculating the taxes directly. That doesn't make any sense! So the programmer changed lib1 so that it would provide a function just to calculate taxes, as well as fixing the tax summary object to contain only numbers, separating its logic from report logic... semantic versioning was being followed, so the version was bumped from 1.23 to 2.0. After all, this was a breaking change:
function calculateTax(income) {
return income * 0.25;
}
function createTaxSummary(income) {
return {
"income": income,
"tax": calculateTax(income),
}
}
exports.createTaxSummary = createTaxSummary;
exports.calculateTax = calculateTax;
The application developer, worried that the application was depending on an old versions of some libraries, decided to upgrade all versions. The change log on lib2 was pretty small, even though it was a major version change. It looked like the tax property was changed to a number, and that was pretty much it.
That's a one-line change.
const lib1 = require('../lib1')
const lib2 = require('../lib2')
const income = 100000;
const extras = 2500;
const summary = lib1.createTaxSummary(income);
var report = lib2.createReport(summary);
const extraTaxes = lib1.createTaxSummary(extras).tax;
report += "\nExtra taxes:\nTAX = " + extraTaxes;
console.log(report);
Running it didn't work, though!
/lib2/index.js:6
taxSummary.income.toUpperCase() + "\n" +
^
TypeError: taxSummary.income.toUpperCase is not a function
at Object.createReport (/lib2/index.js:6:25)
at Object.<anonymous> (/app/index.js:8:19)
at Module._compile (module.js:660:30)
at Object.Module._extensions..js (module.js:671:10)
at Module.load (module.js:573:32)
at tryModuleLoad (module.js:513:12)
at Function.Module._load (module.js:505:3)
at Function.Module.runMain (module.js:701:10)
at startup (bootstrap_node.js:193:16)
at bootstrap_node.js:617:3
The author of lib2 is no longer maintaining it, so lib2 hadn't been updated with the changes. The only way to fix the error was to revert to the old version.
But what if there was a critical security bug in the old version? What if other libraries that were being used also expected the old summary format?
In this simple case (I admit it is contrived and looks artificial... sorry, that's the best simple example I could come up with - but can you be sure that something like this won't happen to you??) you might just have to fork the project(s) and patch it... but that could cost a lot of time. But even if this scenario looks unlikely, with a typical node_modules folder containing nearly 100MB of libraries, I think it should be almost unavoidable that something like this will happen at some point.
Just by using React/Redux, you get 40MB worth of dependencies:
➜ npm install --save react-redux
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN react-redux@5.0.6 requires a peer of react@^0.14.0 || ^15.0.0-0 || ^16.0.0-0 but none is installed. You must install peer dependencies yourself.
npm WARN react-redux@5.0.6 requires a peer of redux@^2.0.0 || ^3.0.0 but none is installed. You must install peer dependencies yourself.
+ react-redux@5.0.6
added 21 packages in 2.573s
➜ npm install --save react
npm WARN react-redux@5.0.6 requires a peer of redux@^2.0.0 || ^3.0.0 but none is installed. You must install peer dependencies yourself.
+ react@16.2.0
added 1 package in 0.419s
➜ npm install --save redux
+ redux@3.7.2
added 2 packages in 0.443s
➜ du -sh node_modules
40M node_modules
Here's the full dependency tree:
➜ npm ls
lib3@1.0.0 /lib3
├─┬ react@16.2.0
│ ├─┬ fbjs@0.8.16
│ │ ├── core-js@1.2.7
│ │ ├─┬ isomorphic-fetch@2.2.1
│ │ │ ├─┬ node-fetch@1.7.3
│ │ │ │ ├─┬ encoding@0.1.12
│ │ │ │ │ └── iconv-lite@0.4.19
│ │ │ │ └── is-stream@1.1.0
│ │ │ └── whatwg-fetch@2.0.3
│ │ ├── loose-envify@1.3.1 deduped
│ │ ├── object-assign@4.1.1 deduped
│ │ ├─┬ promise@7.3.1
│ │ │ └── asap@2.0.6
│ │ ├── setimmediate@1.0.5
│ │ └── ua-parser-js@0.7.17
│ ├─┬ loose-envify@1.3.1
│ │ └── js-tokens@3.0.2
│ ├── object-assign@4.1.1
│ └─┬ prop-types@15.6.0
│ ├── fbjs@0.8.16 deduped
│ ├── loose-envify@1.3.1 deduped
│ └── object-assign@4.1.1 deduped
├─┬ react-redux@5.0.6
│ ├── hoist-non-react-statics@2.3.1
│ ├─┬ invariant@2.2.2
│ │ └── loose-envify@1.3.1 deduped
│ ├── lodash@4.17.4
│ ├── lodash-es@4.17.4
│ ├── loose-envify@1.3.1 deduped
│ └── prop-types@15.6.0 deduped
└─┬ redux@3.7.2
├── lodash@4.17.4 deduped
├── lodash-es@4.17.4 deduped
├── loose-envify@1.3.1 deduped
└── symbol-observable@1.1.0
If you think the JavaScript world is insane, so that kind of thing is to be expected, let's see how the Java world compares.
A typical Java web backend will use some kind of framework like Dropwizard. If you want to see how many libraries you'll get just by making that choice, create a build.gradle file with the following contents:
repositories {
jcenter()
}
apply plugin: 'java'
dependencies {
compile "io.dropwizard:dropwizard-core:1.3.0-rc3"
}
task copyLibs(type: Copy) {
into "libs"
from configurations.runtime
}
Now, run the copyLibs task with Gradle and check how heavy the libs folder is:
➜ gradle copyLibs
Starting a Gradle Daemon (subsequent builds will be faster)
Download https://jcenter.bintray.com/io/dropwizard/dropwizard-core/1.3.0-rc3/dropwizard-core-1.3.0-rc3.pom
Download https://jcenter.bintray.com/io/dropwizard/dropwizard-parent/1.3.0-rc3/dropwizard-parent-1.3.0-rc3
.pom
......... hundreds of lines like that .........
BUILD SUCCESSFUL in 27s
1 actionable task: 1 executed
➜ du -sh libs
17M libs
Well, at least that's better than the Node example, but 17MB of jars is still quite a lot!! Let's see what dependencies are responsible for that:
➜ gradle dep --configuration runtime
> Task :dependencies
------------------------------------------------------------
Root project
------------------------------------------------------------
runtime - Runtime dependencies for source set 'main' (deprecated, use 'runtimeOnly ' instead).
\--- io.dropwizard:dropwizard-core:1.3.0-rc3
+--- io.dropwizard:dropwizard-util:1.3.0-rc3
| +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
| +--- com.google.guava:guava:23.5-jre
| | +--- com.google.code.findbugs:jsr305:1.3.9 -> 3.0.2
| | +--- org.checkerframework:checker-qual:2.0.0
| | +--- com.google.errorprone:error_prone_annotations:2.0.18
| | +--- com.google.j2objc:j2objc-annotations:1.1
| | \--- org.codehaus.mojo:animal-sniffer-annotations:1.14
| +--- com.google.code.findbugs:jsr305:3.0.2
| \--- joda-time:joda-time:2.9.9
+--- io.dropwizard:dropwizard-jackson:1.3.0-rc3
| +--- com.google.guava:guava:23.5-jre (*)
| +--- io.dropwizard:dropwizard-util:1.3.0-rc3 (*)
| +--- com.fasterxml.jackson.core:jackson-core:2.9.3
| +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
| +--- com.fasterxml.jackson.core:jackson-databind:2.9.3
| | +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
| | \--- com.fasterxml.jackson.core:jackson-core:2.9.3
| +--- com.fasterxml.jackson.datatype:jackson-datatype-guava:2.9.3
| | +--- com.google.guava:guava:18.0 -> 23.5-jre (*)
| | +--- com.fasterxml.jackson.core:jackson-core:2.9.3
| | \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
| +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.9.3
| | +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
| | +--- com.fasterxml.jackson.core:jackson-core:2.9.3
| | \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
| +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.9.3
| | +--- com.fasterxml.jackson.core:jackson-core:2.9.3
| | \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
| +--- com.fasterxml.jackson.module:jackson-module-parameter-names:2.9.3
| | +--- com.fasterxml.jackson.core:jackson-core:2.9.3
| | \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
| +--- com.fasterxml.jackson.module:jackson-module-afterburner:2.9.3
| | +--- com.fasterxml.jackson.core:jackson-core:2.9.3
| | \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
| +--- com.fasterxml.jackson.datatype:jackson-datatype-joda:2.9.3
| | +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
| | +--- com.fasterxml.jackson.core:jackson-core:2.9.3
| | +--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
| | \--- joda-time:joda-time:2.7 -> 2.9.9
| \--- org.slf4j:slf4j-api:1.7.25
+--- io.dropwizard:dropwizard-validation:1.3.0-rc3
| +--- io.dropwizard:dropwizard-util:1.3.0-rc3 (*)
| +--- org.hibernate:hibernate-validator:5.4.2.Final
| | +--- javax.validation:validation-api:1.1.0.Final
| | +--- org.jboss.logging:jboss-logging:3.3.0.Final
| | \--- com.fasterxml:classmate:1.3.1
| +--- org.glassfish:javax.el:3.0.0
| +--- org.javassist:javassist:3.22.0-GA
| \--- org.slf4j:slf4j-api:1.7.25
+--- io.dropwizard:dropwizard-configuration:1.3.0-rc3
| +--- io.dropwizard:dropwizard-jackson:1.3.0-rc3 (*)
| +--- io.dropwizard:dropwizard-validation:1.3.0-rc3 (*)
| +--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.9.3
| | +--- org.yaml:snakeyaml:1.18
| | \--- com.fasterxml.jackson.core:jackson-core:2.9.3
| +--- org.apache.commons:commons-lang3:3.6
| \--- org.apache.commons:commons-text:1.1
| \--- org.apache.commons:commons-lang3:3.5 -> 3.6
+--- io.dropwizard:dropwizard-logging:1.3.0-rc3
| +--- io.dropwizard:dropwizard-jackson:1.3.0-rc3 (*)
| +--- io.dropwizard:dropwizard-validation:1.3.0-rc3 (*)
| +--- io.dropwizard.metrics:metrics-logback:4.0.2
| | \--- io.dropwizard.metrics:metrics-core:4.0.2
| +--- org.slf4j:slf4j-api:1.7.25
| +--- org.slf4j:jul-to-slf4j:1.7.25
| | \--- org.slf4j:slf4j-api:1.7.25
| +--- ch.qos.logback:logback-core:1.2.3
| +--- ch.qos.logback:logback-classic:1.2.3
| | \--- ch.qos.logback:logback-core:1.2.3
| +--- org.slf4j:log4j-over-slf4j:1.7.25
| | \--- org.slf4j:slf4j-api:1.7.25
| +--- org.slf4j:jcl-over-slf4j:1.7.25
| | \--- org.slf4j:slf4j-api:1.7.25
| \--- org.eclipse.jetty:jetty-util:9.4.8.v20171121
+--- io.dropwizard:dropwizard-metrics:1.3.0-rc3
| +--- io.dropwizard:dropwizard-lifecycle:1.3.0-rc3
| | +--- org.slf4j:slf4j-api:1.7.25
| | +--- com.google.guava:guava:23.5-jre (*)
| | +--- org.eclipse.jetty:jetty-server:9.4.8.v20171121
| | | +--- javax.servlet:javax.servlet-api:3.1.0
| | | +--- org.eclipse.jetty:jetty-http:9.4.8.v20171121
| | | | +--- org.eclipse.jetty:jetty-util:9.4.8.v20171121
| | | | \--- org.eclipse.jetty:jetty-io:9.4.8.v20171121
| | | | \--- org.eclipse.jetty:jetty-util:9.4.8.v20171121
| | | \--- org.eclipse.jetty:jetty-io:9.4.8.v20171121 (*)
| | \--- io.dropwizard:dropwizard-util:1.3.0-rc3 (*)
| +--- io.dropwizard:dropwizard-jackson:1.3.0-rc3 (*)
| +--- io.dropwizard:dropwizard-validation:1.3.0-rc3 (*)
| +--- io.dropwizard.metrics:metrics-core:4.0.2
| \--- org.slf4j:slf4j-api:1.7.25
+--- io.dropwizard:dropwizard-jersey:1.3.0-rc3
| +--- io.dropwizard:dropwizard-jackson:1.3.0-rc3 (*)
| +--- io.dropwizard:dropwizard-validation:1.3.0-rc3 (*)
| +--- io.dropwizard:dropwizard-logging:1.3.0-rc3 (*)
| +--- org.glassfish.jersey.core:jersey-server:2.25.1
| | +--- org.glassfish.jersey.core:jersey-common:2.25.1
| | | +--- javax.ws.rs:javax.ws.rs-api:2.0.1
| | | +--- javax.annotation:javax.annotation-api:1.2
| | | +--- org.glassfish.jersey.bundles.repackaged:jersey-guava:2.25.1
| | | +--- org.glassfish.hk2:hk2-api:2.5.0-b32
| | | | +--- javax.inject:javax.inject:1
| | | | +--- org.glassfish.hk2:hk2-utils:2.5.0-b32
| | | | | \--- javax.inject:javax.inject:1
| | | | \--- org.glassfish.hk2.external:aopalliance-repackaged:2.5.0-b32
| | | +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
| | | +--- org.glassfish.hk2:hk2-locator:2.5.0-b32
| | | | +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
| | | | +--- org.glassfish.hk2.external:aopalliance-repackaged:2.5.0-b32
| | | | +--- org.glassfish.hk2:hk2-api:2.5.0-b32 (*)
| | | | +--- org.glassfish.hk2:hk2-utils:2.5.0-b32 (*)
| | | | \--- org.javassist:javassist:3.20.0-GA -> 3.22.0-GA
| | | \--- org.glassfish.hk2:osgi-resource-locator:1.0.1
| | +--- org.glassfish.jersey.core:jersey-client:2.25.1
| | | +--- javax.ws.rs:javax.ws.rs-api:2.0.1
| | | +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
| | | +--- org.glassfish.hk2:hk2-api:2.5.0-b32 (*)
| | | +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
| | | \--- org.glassfish.hk2:hk2-locator:2.5.0-b32 (*)
| | +--- javax.ws.rs:javax.ws.rs-api:2.0.1
| | +--- org.glassfish.jersey.media:jersey-media-jaxb:2.25.1
| | | +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
| | | +--- org.glassfish.hk2:hk2-api:2.5.0-b32 (*)
| | | +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
| | | +--- org.glassfish.hk2:hk2-locator:2.5.0-b32 (*)
| | | \--- org.glassfish.hk2:osgi-resource-locator:1.0.1
| | +--- javax.annotation:javax.annotation-api:1.2
| | +--- org.glassfish.hk2:hk2-api:2.5.0-b32 (*)
| | +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
| | +--- org.glassfish.hk2:hk2-locator:2.5.0-b32 (*)
| | \--- javax.validation:validation-api:1.1.0.Final
| +--- org.glassfish.jersey.ext:jersey-metainf-services:2.25.1
| | +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
| | \--- javax.ws.rs:javax.ws.rs-api:2.0.1
| +--- org.glassfish.jersey.ext:jersey-bean-validation:2.25.1
| | +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
| | +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
| | +--- org.glassfish.jersey.core:jersey-server:2.25.1 (*)
| | +--- javax.validation:validation-api:1.1.0.Final
| | +--- org.hibernate:hibernate-validator:5.1.3.Final -> 5.4.2.Final (*)
| | \--- javax.ws.rs:javax.ws.rs-api:2.0.1
| +--- io.dropwizard.metrics:metrics-jersey2:4.0.2
| | +--- io.dropwizard.metrics:metrics-core:4.0.2
| | \--- io.dropwizard.metrics:metrics-annotation:4.0.2
| | \--- org.slf4j:slf4j-api:1.7.25
| +--- com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:2.9.3
| | +--- com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:2.9.3
| | | +--- com.fasterxml.jackson.core:jackson-core:2.9.3
| | | \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
| | \--- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:2.9.3
| | +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
| | +--- com.fasterxml.jackson.core:jackson-core:2.9.3
| | \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
| +--- org.glassfish.jersey.containers:jersey-container-servlet:2.25.1
| | +--- org.glassfish.jersey.containers:jersey-container-servlet-core:2.25.1
| | | +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
| | | +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
| | | +--- org.glassfish.jersey.core:jersey-server:2.25.1 (*)
| | | \--- javax.ws.rs:javax.ws.rs-api:2.0.1
| | +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
| | +--- org.glassfish.jersey.core:jersey-server:2.25.1 (*)
| | \--- javax.ws.rs:javax.ws.rs-api:2.0.1
| +--- org.eclipse.jetty:jetty-server:9.4.8.v20171121 (*)
| +--- org.eclipse.jetty:jetty-webapp:9.4.8.v20171121
| | +--- org.eclipse.jetty:jetty-xml:9.4.8.v20171121
| | | \--- org.eclipse.jetty:jetty-util:9.4.8.v20171121
| | \--- org.eclipse.jetty:jetty-servlet:9.4.8.v20171121
| | \--- org.eclipse.jetty:jetty-security:9.4.8.v20171121
| | \--- org.eclipse.jetty:jetty-server:9.4.8.v20171121 (*)
| +--- org.eclipse.jetty:jetty-continuation:9.4.8.v20171121
| \--- org.apache.commons:commons-lang3:3.6
+--- io.dropwizard:dropwizard-servlets:1.3.0-rc3
| +--- org.slf4j:slf4j-api:1.7.25
| +--- io.dropwizard:dropwizard-util:1.3.0-rc3 (*)
| +--- io.dropwizard.metrics:metrics-annotation:4.0.2 (*)
| +--- io.dropwizard.metrics:metrics-core:4.0.2
| \--- ch.qos.logback:logback-classic:1.2.3 (*)
+--- io.dropwizard:dropwizard-jetty:1.3.0-rc3
| +--- io.dropwizard:dropwizard-logging:1.3.0-rc3 (*)
| +--- io.dropwizard.metrics:metrics-jetty9:4.0.2
| | \--- io.dropwizard.metrics:metrics-core:4.0.2
| +--- org.eclipse.jetty:jetty-server:9.4.8.v20171121 (*)
| +--- org.eclipse.jetty:jetty-servlet:9.4.8.v20171121 (*)
| +--- org.eclipse.jetty:jetty-servlets:9.4.8.v20171121
| | +--- org.eclipse.jetty:jetty-continuation:9.4.8.v20171121
| | +--- org.eclipse.jetty:jetty-http:9.4.8.v20171121 (*)
| | +--- org.eclipse.jetty:jetty-util:9.4.8.v20171121
| | \--- org.eclipse.jetty:jetty-io:9.4.8.v20171121 (*)
| \--- org.eclipse.jetty:jetty-http:9.4.8.v20171121 (*)
+--- io.dropwizard:dropwizard-lifecycle:1.3.0-rc3 (*)
+--- io.dropwizard.metrics:metrics-core:4.0.2
+--- io.dropwizard.metrics:metrics-jvm:4.0.2
| \--- io.dropwizard.metrics:metrics-core:4.0.2
+--- io.dropwizard.metrics:metrics-jmx:4.0.2
| \--- io.dropwizard.metrics:metrics-core:4.0.2
+--- io.dropwizard.metrics:metrics-servlets:4.0.2
| +--- io.dropwizard.metrics:metrics-core:4.0.2
| +--- io.dropwizard.metrics:metrics-healthchecks:4.0.2
| +--- io.dropwizard.metrics:metrics-json:4.0.2
| | \--- io.dropwizard.metrics:metrics-core:4.0.2
| +--- io.dropwizard.metrics:metrics-jvm:4.0.2 (*)
| \--- com.papertrail:profiler:1.0.2
| \--- joda-time:joda-time:2.9.1 -> 2.9.9
+--- io.dropwizard.metrics:metrics-healthchecks:4.0.2
+--- io.dropwizard:dropwizard-request-logging:1.3.0-rc3
| +--- io.dropwizard:dropwizard-jetty:1.3.0-rc3 (*)
| +--- io.dropwizard:dropwizard-logging:1.3.0-rc3 (*)
| \--- ch.qos.logback:logback-access:1.2.3
| \--- ch.qos.logback:logback-core:1.2.3
+--- net.sourceforge.argparse4j:argparse4j:0.7.0
\--- org.eclipse.jetty.toolchain.setuid:jetty-setuid-java:1.0.3
(*) - dependencies omitted (listed previously)
BUILD SUCCESSFUL in 0s
1 actionable task: 1 executed
Wholly crap!!! That's a lot of jars!!! 90, to be accurate.
Well, surely Java and JavaScript must be some of the worst offenders... but given it is just rational to try to avoid implementing functionality that already is implemented by libraries (which is nearly everything for the most popular languages), this proliferation of dependencies seems unavoidable.
For curiosity, I compared Java/JavaScript with Go and one of its heavier web frameworks, Beego, and was shocked to find out that this framework seems to have no external dependencies (even though it implements ORM, MVC, i18n and logging in the core module), weighs in at only 3,7MB, and a hello world, statically compiled web server (which, unlike Java and JS, does not require a runtime) compiles to around 11MB.
Not too bad! But that brings us to the curious case of Go dependency management (which may help explain how a large framework can be implemented without using libraries, even considering Go's stance to have a complete, large standard library).
You see, Go is very different from other languages in that it does not have a package repository! Until very recently, it did not even have a standard package manager.
Want to use a dependency in Go? Just run go get to download it from GitHub and start using it... For example, to use Beego, you first run this:
go get github.com/astaxie/beego
This downloads the source adds it to the GOPATH (where it can be found by the Go compiler). Now you can just import it in the source code with import github.com/astaxie/beego!
go get supports downloading from GitHub and a few other sources (e.g. BitBucket, LaunchPad), but it doesn't do any kind of version checks! It just gets the source currently in the default branch (it might look at tags with a Go version as well) and calls it a day.
This might sound like complete chaos! Every time you build your project in a new machine, you might get different code in your dependencies. The solution the Google Go team suggested was to not break backwards compatibility - because then, it doesn't matter when you build it, it will still work if it worked last time!
I was first shocked when I found out about that! But then I reasoned that Google engineers probably know what they are doing, even if it looks like they just came up with the most terrible solution to the hard problem of package management.
Have you ever tried to build software that is shared between hundreds of developers, maybe hundreds of teams? Keeping dependencies up-to-date in all projects so that everything works and remains free of security vulnerabilities is difficult. Some libraries may have no maintainer for a while, causing old versions of its transitive dependencies to spread like a fire.
But that's exactly the situation nearly everyone is at now. We all use hundreds of dependencies in our projects, some of which may not have been updated in a decade. We just can't keep up, this is a losing fight.
We must give up on trying to be heroic and accept the facts: updating all transitive dependencies of every library and project till the end of time is just unfeasible. We must find another way.
Here's how we can solve the problem:
No breaking changes at the binary/source level are allowed once a library is declared stable.
If a function is not ideal, write a new function that gets closer to perfection, but keep the old one with a "DEPRECATED" warning.
Don't ever remove anything. We all make bad decisions, but removing a function from a library will not solve anything.
When the burden of maintaining the legacy code becomes to heavy to carry, start a new library, with a new name, to obsolete the old one.
In the Java world, many of the most common libraries have been doing that for years, and it works great!
Check this gist that shows the evolution of the HTTP Components Client library. How hard can it be to design a HTTP client, right? Every version changed basically everything! But because they have been changing the package names from version to version, it is possible to run an application that depends on all 3 versions at the same time (which amazingly, can easily happen given how widespread this HTTP Commons is). Objects coming from one version will not have the same type than the equivalent Object coming from another, so any mismatches are caught by the type system auomatically (JS and other dynamic languages have the "advantage" that it will only blow up at runtime and you'll probably only ever notice the problem when it happens, hopefully during tests).
Same thing with Log4j, which changed its namespace (package) in version 2, effectively becoming a new library.
Unfortunately, we're humans so I acknowledge we cannot expect everyone to follow these rules... after all, even the Go community seems to have finally settled with a dependency manager (go dep). It works a little bit like npm and Cargo, pinning dependencies following version constraints present in the build file. But that solution still suffers from the problems I have mentioned above. Just because your version range should work according to semantic versioning, it doesn't mean it will. And as Gavin King, lead developer of Ceylon, said: version range compatibility is a lie. Remember, we can't trust people!
In the long term, the only viable solution is to automate the process of checking for backwards compatibility and stopping breaking changes from happening, so we don't need to trust people to achieve that.
In the Java world, there's already a tool, japicmp, that does that. Elm, a strict functional programming language for the web, aims to support automated version change checks at the language level! With that in place, you simply can't break backwards compatibility and cause havoc for all poor souls who thought it would be a good idea to rely on your software in their applications.
If you try hard, you actually can break compatibility by changing semantics... but that's just plain evil! If you do that, you should feel bad.
What about repeatable builds, you may ask. Well, perhaps what you're really asking for is archived builds, not the possibility to download all exact versions of all dependencies in every build?
To conclude, I acknowledge that we need to adapt to a world where versioning is a reality. I suggest just letting an automated tool bump our minor/patch versions every time we release (maybe every day, or several times a day). If we never bump our major version number and have automated checks in place to keep us honest, everyone can depend on version [1.0, 2.0) (or 1.* in semver language) forever without fear of being screwed, automatically receiving all fixes and improvements library authors add over time... even old examples in Stack Overflow will still work (if the example code becomes obsolete, the worst that will happen is that the developer using it will see a deprecation warning, but it will still work)! I am sure that if we do that, building software in 20 years will be a lot more pleasant and reliable than the current chaotic situation.