Posts

A 7MB native-image Java app that runs in 30ms and uses only 4MB of RAM!

posted May 28, 2018, 8:06 AM by Renato Athaydes   [ updated Jun 8, 2018, 11:18 PM ]

GraalVM, a new, polyglot compiler for Java and several other languages, is capable of generating native images which allow JVM-based languages to run, in theory at least, with a lot less bloat, both in terms of total executable size and memory, than when compared with a full blown JVM.

This is perfect for short-running executables such as CLIs and serverless implementations, but also great for micro-services which restart often as they can be started up in milliseconds.

As I was writing a CLI for RawHTTP, I thought this would be a perfect time to try out GraalVM and see if the improvements it promises are actually delivered.

In this post, I explain how I created a native image for the RawHTTP CLI using GraalVM. It fits in a single executable just over 7MB large, and can:

  • start up a HTTP server that accepts connection within a tenth of a second and uses only around 15MB of RAM while serving hundreds of requests.
  • send a HTTP request and show its response in less than 30ms, using only 4MB of RAM in total.
I also compare the performance of the native image with the plain Java application running on the JVM and with a commonly-used native tool: curl as a HTTP client, Apache httpd as a HTTP server.

Unfortunately, I did not manage to include Java 9/10 native images (I wrote about those before) in this comparison due to several issues I've had trying to convert my application to use Java 9 modules. It would be interesting to perform that comparison in the future.



RawHTTP CLI client on the JVM


Before we start, let me introduce RawHTTP: it's a zero-dependency, reflection-less Java library that implements the core HTTP/1.1 specification, RFC-7230. I wrote more about it in a previous blog post, but in summary, RawHTTP makes it easy to write both HTTP servers and clients by providing only the basic HTTP protocol primitives, leaving other things like server routing and client cookies to other libraries. It goes against the current trend in the Java world of using full-stack frameworks that make use of annotations and reflection for everything: it's just vanilla Java code. Because of that, it's extremely light and fast: the core jar weighs in at 102KB.

The RawHTTP CLI is another library that uses core RawHTTP to implement a CLI tool that can run as a HTTP server (serving a local directory) or as a HTTP client.

Before we get to the native image and GraalVM, let's see how the CLI normally works on the JVM.

If you want to follow along, download the rawhttp.jar file from JCenter. Running the jar requires that you have Java SE 8+ installed.

To run a HTTP server on port 8082 serving the public/ directory, type this:


java -jar rawhttp.jar serve public/ --port 8082


To test that the server is working, from another shell, use curl to get the contents of a file within the public directory:


$ curl -v localhost:8082/my-file
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8082 (#0)
> GET /my-file HTTP/1.1
> Host: localhost:8082
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/plain
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Length: 24
< Date: Mon, 28 May 2018 07:29:20 GMT
< Server: RawHTTP
<
Hello, this is my file.
* Connection #0 to host localhost left intact



We can also use the RawHTTP CLI, of course, to do the same thing:


$ java -jar rawhttp.jar send -t "GET localhost:8082/my-file"
HTTP/1.1 200 OK
Content-Type: text/plain
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 24
Date: Mon, 28 May 2018 07:31:27 GMT
Server: RawHTTP

Hello, this is my file.


If you run both of the commands above, you'll notice how curl runs a lot faster than java -jar.

We can actually measure exactly how long the commands take with the time built-in. I am using zsh with its very powerful time built-in, which can also show the memory consumption of a process, among other things, as explained here.

Here's the results when I run time on the above commands:

curl

curl -v localhost:8082/my-file  
0.01s  user 0.01s system 64% cpu 0.038 total

avg shared (code):         0 KB
avg unshared (data/stack): 0 KB
total (sum):               0 KB
max memory:                4932 MB
page faults from disk:     3
other page faults:         1475


java -jar

java -jar rawhttp.jar send -t   
0.34s  user 0.06s system 126% cpu 0.318 total

avg shared (code):         0 KB
avg unshared (data/stack): 0 KB
total (sum):               0 KB
max memory:                37568 MB
page faults from disk:     210
other page faults:         9549


In summary, curl runs in 0.038s, java -jar in 0.318s. curl uses around 4.9MB of RAM, java -jar uses 37MB.

That shows why the JVM is not generally great for CLIs.


RawHTTP CLI client in a GraalVM native-image


Not surprisingly, a Java process takes much more memory to run and needs a longer time to start up than a native tool like curl.

To make things more competitive, however, we can create a native image for the Java program using GraalVM's native-image command.

For a fat jar like the one we've been using, the command is very simple:


native-image -jar rawhttp.jar


This may take a minute or so... it looks like the GraalVM compiler is a little bit slow at the moment!

However, once it's done, you can try running the executable immediately and see what the result is:


$ ./rawhttp send -t "GET localhost:8082/my-file"
HTTP/1.1 200 OK
Content-Type: text/plain
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 24
Date: Mon, 28 May 2018 07:57:38 GMT
Server: RawHTTP

Hello, this is my file.


We can see it works, and feels very responsive! But just how fast is it? Let's time it:


./rawhttp send -t "GET localhost:8082/my-file"  
0.01s  user 0.03s system 34% cpu 0.125 total

avg shared (code):         0 KB
avg unshared (data/stack): 0 KB
total (sum):               0 KB
max memory:                4852 MB
page faults from disk:     597
other page faults:         838


The native process takes only 4.8MB and runs in 0.125s. Not bad!!

But I noticed that there were a lot of page faults from disk. It turns out that running the command again, something changes (filesystem cache??) and we get no page faults from disk, resulting in the process running quite a lot faster:


./rawhttp send -t "GET localhost:8082/my-file"  
0.01s  user 0.01s system 56% cpu 0.028 total

avg shared (code):         0 KB
avg unshared (data/stack): 0 KB
total (sum):               0 KB
max memory:                4836 MB
page faults from disk:     0
other page faults:         1431


Now, the process is actually about as fast and as memory-intensive as curl (running curl multiple times does not change its statistics significantly).

From the previous section:

In summary, curl runs in 0.038s, java -jar in 0.318s. curl uses around 4.9MB of RAM, java -jar uses 37MB.

The native Java application process ran in 0.028s and used 4.8MB.

This is quite amazing for a Java application and could revolutionize the applicability of JVM-based applications.

An important caveat worth mentioning is that the GraalVM native-image compiler still has significant limitations. For example, Java reflection is only partially supported and takes some effort to get working (which was not relevant for RawHTTP CLI as it does not use any reflection, but just about any Java framework, Netty for example, will have trouble running native because of that). Also, java security packages are not currently supported, so https connections will not work with the native image (which is a bummer for RawHTTP CLI, of course). But both of these limitations seem to be temporary, and when they are removed, GraalVM will seriously benefit the JVM world.



Native HTTP Server Performance


In the previous sections, we looked at how long it takes for the HTTP client to start up, send a HTTP request and print the response on the terminal, and how much memory the process needed.

But how about the HTTP server?!

We obviously don't want to know how long the HTTP server runs for (it should run forever!), but it's useful to know how long it takes for it to start up and accept a connection, as well as how much memory it uses.

To find out the time-to-first-connection, I wrote a small bash script that starts the server, then keeps trying to send it a HTTP request, measuring how long it takes for the first request to succeed.

It works on Mac OSX with coreutils installed... on Linux, just replace gdate with date and it should work:



#!/bin/bash

# command that tests that the server is accepting connections
CMD="curl localhost:8082 >& /dev/null"

START_TIME=$(gdate +%s%3N)

# start the server
../rawhttp serve . -p 8082 &
SERVER_PID=$!

STEP=0.001 # sleep between tries, in seconds
TRIES=500
eval ${CMD}
while [[ $? -ne 0 ]]; do
((TRIES--))
echo -ne "Tries left: $TRIES"\\r
if [[ TRIES -eq 0 ]]; then
echo "Server not started within timeout"
exit 1
fi
sleep ${STEP}
eval ${CMD}
done

END_TIME=$(gdate +%s%3N)
TIME=$(($END_TIME - $START_TIME))
echo "Server connected in $TIME ms"
#./rawhttp send -t "GET localhost:8082/hello"

kill ${SERVER_PID}



Running this script a few times, this is the output I get:


$ ./time-rawhttp.sh
Serving directory /Users/renato/programming/projects/rawhttp/rawhttp-cli-tests/. on port 8082
Server connected in 112 ms
$ ./time-rawhttp.sh
Serving directory /Users/renato/programming/projects/rawhttp/rawhttp-cli-tests/. on port 8082
Server connected in 53 ms
$ ./time-rawhttp.sh
Serving directory /Users/renato/programming/projects/rawhttp/rawhttp-cli-tests/. on port 8082
Server connected in 40 ms


Even though there's some variance, the server starts up rather quickly.

We can also measure the memory the server uses when bombarded with HTTP requests by running it with the time command and then using the ab tool to benchmark the server.

Start the server:


$ time ./rawhttp serve public/ --port 8082


From another shell, run ab:


$ ab -n 100 -c 2 localhost:8082/my-file


Here's the report I got from ab:


Server Software:        RawHTTP
Server Hostname:        localhost
Server Port:            8082

Document Path:          /my-file
Document Length:        24 bytes

Concurrency Level:      2
Time taken for tests:   0.058 seconds
Complete requests:      100
Failed requests:        0
Total transferred:      18600 bytes
HTML transferred:       2400 bytes
Requests per second:    1719.19 [#/sec] (mean)
Time per request:       1.163 [ms] (mean)
Time per request:       0.582 [ms] (mean, across all concurrent requests)
Transfer rate:          312.27 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       0
Processing:     1    1   0.2      1       2
Waiting:        1    1   0.2      1       2
Total:          1    1   0.2      1       2

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      1
  90%      1
  95%      2
  98%      2
  99%      2
 100%      2 (longest request)



Now, stop the server by entering Ctrl+C in its shell. Here's the report I got from time:


./rawhttp serve public/ --port 8082  
0.08s  user 0.14s system 0% cpu 4:27.58 total

avg shared (code):         0 KB
avg unshared (data/stack): 0 KB
total (sum):               0 KB
max memory:                16116 MB
page faults from disk:     935
other page faults:         3373


Note: When running ab multiple times, or just running 1000 requests instead of 100, the native server process crashed for me. That did not happen with the normal JVM process. I couldn't figure out a way to find out why the process was crashing as no crash dump seems to be generated when that happened, but I'd love to know how I can do that - I will update this blog post if I ever figure this out. Created an issue with the GraalVM project.

UPDATE (09th June 2018): it turns out the server was not closing a FileInputStream properly, causing the server to crash (it probably only works in the JVM because the finalize method would close the handle). As pointed out in the GitHub issue, just using FileInputStream is not a great idea anyway... after I updated the code to close the InputStream as soon as possible, and started using Files.newInputStream() instead, the server stopped crashing and can handle a very high load without problems.


Baseline: Apache HTTP Server Performance


To have a baseline for server performance, just like we used curl to baseline the HTTP client, let's use an Apache HTTP Server so that we have an idea of how the native HTTP server is performing.

Mac OS already comes with Apache installed, so all I had to do was to configure a Virtual Host and expose the public/ directory before getting the server to serve static files.

To measure the server startup, I've used the same bash script as before, but changing the line that starts the server:



# start the server
apachectl start &



I also changed how the server is killed:



#kill ${SERVER_PID}
apachectl stop




Results:

$ sudo ./time-rawhttp.sh
Password:
Server connected in 771 ms
$ sudo ./time-rawhttp.sh
Server connected in 417 ms
$ sudo ./time-rawhttp.sh
Server connected in 402 ms



ab report:

Server Software:        Apache/2.4.29
Server Hostname:        rawhttp
Server Port:            8081

Document Path:          /my-file.txt
Document Length:        24 bytes

Concurrency Level:      2
Time taken for tests:   0.081 seconds
Complete requests:      100
Failed requests:        0
Total transferred:      26900 bytes
HTML transferred:       2400 bytes
Requests per second:    1241.16 [#/sec] (mean)
Time per request:       1.611 [ms] (mean)
Time per request:       0.806 [ms] (mean, across all concurrent requests)
Transfer rate:          326.05 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     1    1   0.5      1       4
Waiting:        0    1   0.5      1       3
Total:          1    2   0.5      1       4
WARNING: The median and mean for the total time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      2
  75%      2
  80%      2
  90%      2
  95%      2
  98%      4
  99%      4
 100%      4 (longest request)



JVM HTTP Server Performance



Let's compare that with the normal, JVM-based process.

Change the line that starts the server in the bash script:



# start the server
java -jar rawhttp.jar serve public/ --port 8082 &




Run the scripts a few times:


$ rawhttp-cli-tests/time-rawhttp.sh
Serving directory /Users/renato/programming/projects/rawhttp/public on port 8082
Server connected in 651 ms
$ rawhttp-cli-tests/time-rawhttp.sh
Serving directory /Users/renato/programming/projects/rawhttp/public on port 8082
Server connected in 546 ms
$ rawhttp-cli-tests/time-rawhttp.sh
Serving directory /Users/renato/programming/projects/rawhttp/public on port 8082
Server connected in 507 ms


Now, start the server with the time command, run ab from a separate shell, then kill the server process as we did in the previous section.


time output:

java -jar rawhttp.jar serve public/ --port 8082  
1.90s  user 0.45s system 4% cpu 48.256 total
avg shared (code):         0 KB
avg unshared (data/stack): 0 KB
total (sum):               0 KB
max memory:                68068 MB
page faults from disk:     213
other page faults:         17220


ab report:

Server Software:        RawHTTP
Server Hostname:        localhost
Server Port:            8082

Document Path:          /my-file
Document Length:        24 bytes

Concurrency Level:      2
Time taken for tests:   0.126 seconds
Complete requests:      100
Failed requests:        0
Total transferred:      18600 bytes
HTML transferred:       2400 bytes
Requests per second:    794.57 [#/sec] (mean)
Time per request:       2.517 [ms] (mean)
Time per request:       1.259 [ms] (mean, across all concurrent requests)
Transfer rate:          144.33 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       2
Processing:     1    2   1.2      2       8
Waiting:        1    2   1.2      2       8
Total:          1    2   1.2      2       9

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      2
  75%      3
  80%      3
  90%      4
  95%      5
  98%      7
  99%      9
 100%      9 (longest request)



Conclusion



GraalVM's native-image produces binaries that look and feel like native applications written in native languages like C.

A summary of the numbers collected in the previous sections demonstrates that:


HTTP Client GET Request Performance:


 Memory (MB)
 Process time (ms)
 curl     4.93 38
 java -jar
 37.57 318
 native-image 4.84 28


HTTP client performance



HTTP Server Performance (100 requests, concurrency level = 2):


  Memory (MB)
 Startup time (ms)
 Requests/sec (mean)
 Apache Server
Unknown 530 1241.16
 java -jar
 68.07 568 794.57
 native-image 16.12 68.3 1719.19


HTTP server performance



Really amazing performance from the GraalVM native-image.

The only problem with GraalVM is that it still does not support all JVM features and seems to be a much less stable runtime at this point in time - May 2018 (though the crashes I had could have been caused by something I did wrong myself, I will keep this post updated).

However, once the compiler and runtime mature, there's no doubt that GraalVM is going to make it possible for JVM-languages to compete with native languages like Go, Haskell and maybe even Rust and C in some domains where, until now, the JVM memory- and startup-time- overhead, or even the necessity to install or ship a full JVM runtime, were considered unacceptable.

An interesting question to ask is how other native projects based on traditional JVM languages, such as Scala Native and Kotlin Native may be impacted, or perhaps even benefit, from GraalVM. Time will tell.



Final Note: I am having some problems publishing RawHTTP on JCenter and Maven Central... if you would like to run the benchmarks above, please build RawHTTP from source until that issue is resolved:


git clone git@github.com:renatoathaydes/rawhttp.git
cd rawhttp
git checkout back-to-java8
./gradlew fatJar # the jar is under rawhttp-cli/build/libs/rawhttp.jar


Building a mobile frontend for a Go application using Flutter.

posted May 3, 2018, 9:32 AM by Renato Athaydes   [ updated May 5, 2018, 9:35 AM ]

Some months ago, I decided to build a free and open-source CLI (command-line interface) password manager called go-hash using Go. It works great on desktop. The interface is easy to learn and I believe a password manager is a perfect fit for a CLI application (most password manager's GUIs I've used are ugly and just take unnecessary screen real-estate).

However, CLIs work well only on desktops (and thanks to Go's compiler, go-hash works on Windows, Mac and Linux seamlessly)... on mobile, that's another story. I couldn't find an easy way to run a terminal and install native applications on mobiles (actually, there is Termux for Android, and probably Termius would work on iPhones, but I am not sure I would actually enjoy using the terminal on my phone!). So, I decided to build a tiny mobile app for go-hash.

The problem is that, unfortunately, I have no experience writing apps for either Android or iPhone... and although I work as a Java/Kotlin developer, I've never used the Android SDK... and never done any Objective-C or Swift at all, let alone work with the iOS GUI framework - so the idea of learning all of it just to write a little app looked like far too much work.

That's why I decided to look into Flutter, which is advertised as "Google’s mobile app SDK for crafting high-quality native interfaces on iOS and Android in record time.". Sounds like just what I needed! Instead of writing 2 separate applications, one for each platform, Flutter makes it possible to write a single application in Dart (which is quite easy to learn for anyone who knows Java, JavaScript or Swift) that runs on both mobile platforms.

In this blog post, I will show how an existing Go application (as we'll see, with some adaptations) can be used from Flutter, so you can write the GUI side of the application in Dart (instead of one for Android, one for iOS), and the "back-end" mostly in Go (rather than duplicate the logic in Java/Kotlin and Objective-C/Swift), with only a thin glue-code layer written in the platform's native language (in our case, Kotlin on Android, Swift on iOS).




All language mascots and logos


Getting started with Flutter and the mobile toolchains


After installing Flutter and running flutter doctor, I got this report:


Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel beta, v0.3.1, on Mac OS X 10.13.2 17C205, locale en-US)
[✗] Android toolchain - develop for Android devices
    ✗ Unable to locate Android SDK.
      Install Android Studio from: https://developer.android.com/studio/index.html
      On first launch it will assist you in installing the Android SDK components.
      (or visit https://flutter.io/setup/#android-setup for detailed instructions).
      If Android SDK has been installed to a custom location, set $ANDROID_HOME to that location.
[!] iOS toolchain - develop for iOS devices
    ✗ Xcode installation is incomplete; a full installation is necessary for iOS development.
      Download at: https://developer.apple.com/xcode/download/
      Or install Xcode via the App Store.
      Once installed, run:
        sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
    ✗ Brew not installed; use this to install tools for iOS device development.
      Download brew at https://brew.sh/.
[✗] Android Studio (not installed)
[!] Connected devices
    ! No devices available

! Doctor found issues in 4 categories.


This is basically saying that you need to have the Android and iOS developer toolchain installed for Flutter to work. Installing all of this stuff (IDEs, phone emulators, CLI tools, package managers) takes time but is necessary for everything else to work later, so try to follow exactly the instructions shown on the Flutter website for your platform before proceeding.

After going through the whole setup process, I ran flutter doctor again a few times, fixing the errors it reported as I went (if you have trouble with Python's six dependency, see this), and after a few tries, everything seemed to be finally installed:



Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel beta, v0.3.1, on Mac OS X 10.13.2 17C205, locale en-US)
[✓] Android toolchain - develop for Android devices (Android SDK 27.0.3)
[✓] iOS toolchain - develop for iOS devices (Xcode 9.3)
[✓] Android Studio (version 3.1)
[✓] Connected devices (1 available)

• No issues found!



As we've just installed Android Studio (which is based on the IntelliJ platform), the next step is to to follow the Configure Editor guide, which describes how to install the Dart and Flutter plugins on Android Studio (or, if you prefer, VS Code - but this blog post uses only Android Studio).

Finally, we get to the Test Drive section of the Flutter guide, which is where we get to do more interesting things.

You can create a new app (named gohash in this case) by using the flutter command as follows:


flutter create -i swift -a kotlin gohash

The -i swift option tells Flutter to use Swift, not Objective-C, as the iOS language, and the -a kotlin option tells it to use Kotlin, not Java, as the Android language.

If everything worked correctly, we're ready to run the default app created by the Flutter plugin on the device emulator installed in the previous steps and even on a real device, if one happens to be connected.

Here's what the Android Studio screen looked like for me at this stage:


Android Studio screen


Trying to run the application with the play button (in the top toolbar) resulted in an error, but the error message told me what to do to fix:


Finished with error:

* Error running Gradle:
Unable to download needed Android SDK components, as the following licenses have not been accepted:
Android SDK Build-Tools 26.0.2

To resolve this, please run the following command in a Terminal:
flutter doctor --android-licenses


After running the magic command shown above and blindly accepting all Android licenses, I pressed the Play button again, and this time it worked! Except the application started running on my real phone, not the emulator as I had expected :). But that was a nice surprise!

I changed the values of some Strings in the code, saved it, and almost immediately, the app on the phone changed accordingly to reflect my changes. Pretty impressive!

To run the app in the phone emulator, so you don't have to keep the real phone tethered to the laptop all the time, just change the target device (the device emulator must be running to be shown in the drop-down):



Selecting emulator as target device



Now, the Android emulator opens the demo app instead:



Flutter Demo app running in the emulator


After starting up the iOS Simulator (open -a Simulator on Mac), we can also open our app on the emulated iPhone (notice that you can also run the app from the command line with flutter run at the root of the project):



iPhone running go-hash



Hot reload works both in the emulators and in the real phone: change the code, save it, and the changes automatically reflected on the running app.


Creating a Flutter Plugin to call Go code



Now that we've seen how to get a simple Flutter application running, it's time to find a way to use the Go code we're interested in from Flutter. The problem is that Go generates native code for the phone's chip architecture, and the integration with native code must be implemented in both Android and iOS via each platform's native bindings.

For this reason, we have to do two main things: first, generate bindings for Android and iOS (using the gomobile tool) to make the Go code available to Java/Kotlin and Objective-C/Swift, respectively... second, implement a Flutter Plugin which can, as we'll see, call the platform-specific code.

But let's start by creating a simple Fluter Plugin that illustrates how we can call platform-specific APIs using a single, common Dart API. After that, we can implement the iOS and Android glue code that calls the Go library via generated bindings. Finally, we'll modify the Flutter Plugin to expose the Go library to Dart code, so it can be consumed uniformly by the Flutter mobile app.


Create a Flutter Plugin


Flutter uses platform channels in order to integrate with platform-specific APIs. Even though it is possible to use them within the mobile app code base, doing that is messy because it mixes the mobile app's Flutter code with the glue-code required to call the Go bindings in both Android and iOS, making things more complex than they should be.

By writing a Flutter Plugin instead, we clearly separate the mobile app's code from the Go application bindings, which in the end gets exposed as just another Flutter package.

To create a Fluter Plugin is very easy. You can use Android Studio's wizard via File -> New -> New Flutter Project..., Select Flutter Plugin, then follow the prompts. Alternatively, just type this command in the command-line (the last argument is the name of the package you want to create):


flutter create -t plugin -i swift -a kotlin gohash_mobile


In the Android Studio Wizard, to use Kotlin and Swift, select the check-boxes as shown below:


Checkboxes to use Swift and Kotlin


If you used the command-line to create the project, open it in Android Studio to continue following along.

You should see something like this:



Initial Android Studio screen



You can see a Dart package is created under the lib directory. It simply exposes a variable called platformVersion which uses a MethodChannel to invoke the getPlatformVersion method in platform-specific code. The Android implementation can be found under android/src/main/kotlin/, the iOS code under ios/Classes/.

If you want, you can actually run the example app Flutter generated under the example/ folder to make sure everything is working under both iOS and Android!

Hint: to edit the iOS code using XCode, first build the project, then open it on XCode using these commands (as explained in this Youtube video):


cd example/
flutter build ios
open ios/Runner.xcworkspace/


The plugin code looks like this, so far:


 
import
'dart:async';

import 'package:flutter/services.dart';

class GohashMobile {
static const MethodChannel _channel =
const MethodChannel('gohash_mobile');

static Future<String> get platformVersion async {
final String version = await _channel.invokeMethod('getPlatformVersion');
return version;
}
}

lib/gohash_mobile.dart


Swift Sample Code
ios/Classes/SwiftGohashMobilePlugin.swift



package
com.example.gohashmobile

import io.flutter.plugin.common.MethodChannel
import io.flutter.plugin.common.MethodChannel.MethodCallHandler
import io.flutter.plugin.common.MethodChannel.Result
import io.flutter.plugin.common.MethodCall
import io.flutter.plugin.common.PluginRegistry.Registrar

class GohashMobilePlugin(): MethodCallHandler {
companion object {
@JvmStatic
fun registerWith(registrar: Registrar): Unit {
val channel = MethodChannel(registrar.messenger(), "gohash_mobile")
channel.setMethodCallHandler(GohashMobilePlugin())
}
}

override fun onMethodCall(call: MethodCall, result: Result): Unit {
if (call.method.equals("getPlatformVersion")) {
result.success("Android ${android.os.Build.VERSION.RELEASE}")
} else {
result.notImplemented()
}
}
}

android/src/main/kotlin/com/example/gohashmobile/GohashMobilePlugin.kt



This is just sample code to make it easier for us to get started. We need to replace it with the Dart API which will wrap the Go API we'll get via the generated Go bindings.

We'll come back to the Dart API later. First, let's create the Go bindings for each platform.



Generate Go bindings for iOS


golang

The gomobile tool can be used to create all-Go mobile apps, or to just generate bindings for Android (Java/Kotlin) and iOS (Objective-C/Swift) apps. In our case, as the idea is to use Flutter for the GUI, we'll just generate bindings so we can call the Go code from Flutter.

But for the gomobile tool to work, we first need to install the Android NDK (Native Development Kit). Using Android Studio, install the LLDB, CMake and NDK components as explained in the Getting Started with NDK guide.

After go-getting gomobile, run gomobile init pointing it to the NDK... on my Mac, the command looked like this:


gomobile init -ndk ~/Library/Android/sdk/ndk-bundle/


With both gomobile and the NDK installed, we should now be able to build the bindings for both iOS and Android.

The command to create iOS bindings for a Go package is the following (the last argument is the full name of the Go package you want to use):


gomobile bind -target=ios github.com/renatoathaydes/go-hash


Working around gomobile issues and limitations


First time I tried running gomobile, I found out that it's not possible to use a main Go package:


> Task :go:gobind
/Users/renato/go/bin/gomobile: binding 'main' package () is not supported


I did not design go-hash as a library, unfortunately, so I had to move some Go code around to make the functions I needed to use public in a non-main Go package. Once that was done, I tried again, just to get another error:


/Users/renato/go/bin/gomobile: go build -i -buildmode=c-shared -o=/var/folders/vz/dnr6kc0d4z76tk0mjxdbswxh0000gp/T/gomobile-work-518425925/android/src/main/jniLibs/armeabi-v7a/libgojni.so gobind failed: exit status 1
go build internal/race: mkdir /usr/local/go/pkg/android_arm_shared: permission denied
go build errors: mkdir /usr/local/go/pkg/android_arm_shared/: permission denied
go build runtime/internal/sys: mkdir /usr/local/go/pkg/android_arm_shared: permission denied
go build unicode/utf8: mkdir /usr/local/go/pkg/android_arm_shared: permission denied
go build sync/atomic: mkdir /usr/local/go/pkg/android_arm_shared: permission denied
go build unicode/utf16: open /usr/local/go/pkg/android_arm_shared/unicode/utf16.a: no such file or directory
go build unicode: open /usr/local/go/pkg/android_arm_shared/unicode.a: no such file or directory
go build math: open /usr/local/go/pkg/android_arm_shared/math.a: no such file or directory
go build runtime/cgo: mkdir /usr/local/go/pkg/android_arm_shared: permission denied


Hm... it seems that Go needs to create new directories under /usr/local/go, so I had to change the owner of this directory to avoid having to build with sudo (is there a better way to solve this??):


chown -R renato /usr/local/go


After this, I was hit with the type limitations of gomobile (not all Go code is supported), which meant I needed to make even more changes to the Go code to workaround those limitations. For example, here's the errors I got initially:


gomobile: /Users/renato/go/bin/gobind -lang=go,objc -outdir=/var/folders/vz/dnr6kc0d4z76tk0mjxdbswxh0000gp/T/gomobile-work-805394183 -tags=ios github.com/renatoathaydes/go-hash/gohash_db failed: exit status 1
unsupported, named type github.com/renatoathaydes/go-hash/gohash_db.State
unsupported, named type github.com/renatoathaydes/go-hash/gohash_db.State
unsupported const type uint8 for Argon2Threads
unsupported, named type github.com/renatoathaydes/go-hash/gohash_db.State


Long story short: the type limitations imposed by gomobile are still very crippling. time.Time is not supported, collections (such as map and slices) are not supported, uint8 (or any unsigned type) is not supported... struct return-values are supported only if returning a pointer (i.e. *MyStruct rather than MyStruct). So basically no Go package can be supported as-is.

For that reason, the only way to get around this problem is to create a (hopefully small) Go package exposing just enough API for the mobile app to function! The API cannot include any of the things mentioned above.

The biggest limitation is definitely lack of support for collections. I worked around that limitation by creating iterator types for the collections I wanted to expose.

For example, instead of exposing a function like this (won't work as it return a slice):


func Read(filepath string) ([]LoginInfo, error)


You can return an iterator instead (notice that there's no limit to non-exported symbol's types, hence it's ok to have slices inside private struct fields):


type LoginInfoIterator struct {
     contents []LoginInfo
     currentIndex uint
}
func (iter *LoginInfoIterator) Next() *LoginInfo {
     if iter.currentIndex < len(iter.contents) {
         item := iter.contents[iter.currentIndex]
         iter.currentIndex++
         return &item
     }
     return nil
}
func Read(filepath string) (*LoginInfoIterator, error) {
     // TODO
}



Once you have a compliant Go API, you can try to generate bindings for it again:


gomobile bind -target=ios github.com/renatoathaydes/go-hash/mobileapi


Even if the command appears to work, you must inspect the resulting bindings to see if there was any function or fields that were "skipped" due to type violations. For example, I had noticed these problems in my early attempts:



// skipped field LoginInfo.UpdatedAt with unsupported type: time.Time
...
// skipped field State.Data with unsupported type: map[string][]github.com/renatoathaydes/go-hash/gohash_db.LoginInfo
...
// skipped function ReadDatabase with unsupported parameter or return types


Make sure the functions and types you need to use were not skipped before proceeding.


Embed the generated iOS framework in the plugin


To include the Go bindings in the iOS project, you can simply drag-and-drop the Mobileapi.framework/ directory created by gomobile onto the XCode project tree, as explained here. After doing that, you can edit the Swift code in XCode and it will recognize the Mobileapi library (or whatever the name of your own library).

To make sure that Mobileapi.framework is available to users of the Flutter Plugin later, you also need to save it under the ios/Frameworks/ directory, then add the following line to the ios/gohash_mobile.podspec file:


s.ios.vendored_frameworks = 'Frameworks/Mobileapi.framework'


Generate Go bindings for Android


Assuming you successfully generated bindings for iOS, you have already made the necessary changes to your Go package to get gomobile to work properly. If you haven't done that, you should go back and read the previous section as you will most likely need to prepare your Go package (or even create a new package just for consumption by the mobile apps) before you can successfully generate bindings for it.

Even though we'll see a way to use Gradle to build the Go bindings, you can use this command to generate bindings for Android is the same as for iOS, just replace the -target=ios option with -target=android:


gomobile bind -target=android github.com/renatoathaydes/go-hash/mobileapi


This will generate 2 files: a jar containing the Java bindings' sources, and an AAR (Android Archive). You should inspect the generated sources to make sure all methods and types you want to use were correctly generated and not "skipped" (see the previous section for details).

To include the AAR in the Android project, open the android/ folder in another Android Studio window (so the project is recognized as an Android/Kotlin project), then follow these instructions. After doing that, you should be able to import the Go package's bindings from the Kotlin code, as we'll see in the next steps.

However, the recommended way to generate bindings for Android is to use Gradle to do it, so that every time the project is built, the bindings are generated again (the Gradle plugin should only re-generate the bindings if necessary, but currently it seems to not support that yet).

To do that, first create a new directory at android/mobileapi, then copy this build.gradle file into it:



plugins {
id "org.golang.mobile.bind" version "0.2.11"
}

gobind {
// the identifier of the Go package to be used
pkg = "github.com/renatoathaydes/go-hash/mobileapi"

/* GOPATH where the Go package is; check `go env` */
GOPATH = System.getProperty('user.home') + "/go"

/* Absolute path to the go binary */
// GO = "/PATH/TO/GO"

/* Optionally, set the absolute path to the gomobile binary if the
/* gomobile binary is not located in the GOPATH's bin directory. */
// GOMOBILE = "/PATH/TO/GOMOBILE"
}

android/mobileapi/build.gradle


Now, include this sub-project in the build:


rootProject.name = 'gohash_mobile'
include ':mobileapi'
// dir must be set explicitly so this file can be included in the example app build.
project(':mobileapi').projectDir = file('mobileapi')

android/settings.gradle


Working around Android build issues


This should be all you need to do to generate the bindings automatically, but due to this bug, Android Studio may not be able to "sync" the project, and you won't be able to see the Go bindings from the Android code. Also, due this other bug, you'll have to make a few other changes to the Gradle setup for the example app to be able to run.

See this commit for fixes for the former bug, and this one for the latter.


Expose the Go API on iOS

Swift logo


Now that we generated the Go bindings for iOS and included the generated library in the iOS project, we can use it from the Swift code. The generated bindings are written in Objective-C, but calling Objective-C from Swift is straightforward.

The Objective-C types we want to move from iOS (and later, Android) to Dart via platform channels must either be automatically serializable (e.g. null, bool, nums, String, Lists, Maps) or have custom serializers (which we must write ourselves), as described in the StandardMessageCoded documentation. But notice that if we use custom serializers, we need to write them for both iOS and Android, which can be costly. A simpler approach is to convert the native types, if needed, into the simple types which are automatically serializable.

For example, I decided to write the following Swift function to convert the go-hash database into a [String: [[String: Any]]] instance, which can be serialized automatically. This way, the same function can easily be implemented in Kotlin to produce the equivalent Map<String, List<Map<String, Any>>> instance. On the Dart side, we can create a simple model which can be built from the same structure regardless of whether it comes from Swift or Kotlin (similarly to if we had a JSON object as input).

Here's the conversion function in Swift:


Swift conversion function


And the updated implementation of the handle function of the Swift FlutterPlugin implementation:


Swift implementation of handle() method


This is all the Swift code we'll ever need to write! Or at least until we need to support more method calls, but for now, that's it.

Expose the Go API on Android



Kotlin logo


Similarly to the previous section, we need to implement the MethodCallHandler in order to delegate method calls to the Go package. gomobile generates Java bindings, but just like we were able to call the Objectve-C bindings from Swift in iOS, we can also easily call Java from Kotlin without limitations.

In this example, we need to produce an instance of Map<String, List<Map<String, Any>>> on calls to the getDb method, as we did in Swift. The Go bindings produce an actual Database object, so we need to convert that to the expected format:



private fun androidDatabaseFrom(db: Database): Map<String, List<Map<String, Any>>> {
val result = HashMap<String, List<Map<String, Any>>>()
val iterator = db.iter()
do {
val item = iterator.next()?.apply {
val entries = mutableListOf<Map<String, Any>>()
result[group] = entries
do {
val entry = next()?.apply {
val loginInfo = mapOf<String, Any>(
"name" to name(),
"username" to username(),
"password" to password(),
"url" to url(),
"description" to description(),
"updatedAt" to updatedAt())
entries.add(loginInfo)
}
} while (entry != null)
}
} while (item != null)

return result
}



Now we can easily write an implementation for the onMethodCall method (using a small helper function):



private inline fun <reified T> readArgument(args: List<*>, index: Int): T {
if (index < args.size) {
val argument = args[index]
if (argument is T) {
return argument
} else {
throw IllegalArgumentException("Argument at index $index " +
"has unexpected type: ${argument?.javaClass?.name}")
}
}
throw IllegalArgumentException("No argument available at index $index")
}

override fun onMethodCall(call: MethodCall, result: Result) {
if (call.method == "getDb") {
val args = call.arguments
if (args is List<*>) {
if (args.size == 2) {
try {
val dbPath = readArgument<String>(args, 0)
val password = readArgument<String>(args, 1)
val db = Mobileapi.readDatabase(dbPath, password)
result.success(androidDatabaseFrom(db))
} catch (e: IllegalArgumentException) {
result.error("BAD_ARGS", e.message!!, null)
} catch (e: Exception) {
result.error("NATIVE_ERR", e.message!!, null)
}
} else {
result.error("BAD_ARGS",
"Wrong arg count (getDb expects 2 args): ${args.size}", null)
}
} else {
result.error("BAD_ARGS", "Wrong argument types", null)
}
} else {
result.notImplemented()
}
}


Notice that this code is an almost-exact translation of the Swift code we wrote earlier (and looks like Swift has a slightly cleaner syntax than Kotlin, based on this small sample alone).


Implement the Flutter Plugin



Dart logo



Both iOS and Android now have identical implementations of the getDb method call, which uses the native Go bindings to do the actual work... and which we can use in Dart via a method channel. Both implementations return a type that in Dart looks like this:


Map<String, List<Map<String, dynamic>>>


However, it appears that generic objects that go through method channels lose the generic type arguments, hence we end up getting an instance of this instead:


Map<dynamic, dynamic>


That's, of course, not very convenient. It would be a lot nicer to expose a typed model for this on the Dart side.

For this reason, I decided to create conversion functions to go from Map<..> to GoHashDb, which will be the type we'll expose in the API of our Flutter Plugin. Here's the full Dart implementation of the plugin:



import
'dart:async';

import 'package:flutter/services.dart';

class GohashMobile {
static const MethodChannel _channel = const MethodChannel('gohash_mobile');

static Future<GohashDb> getDb(String filePath, String password) async {
final Map<dynamic, dynamic> db =
await _channel.invokeMethod('getDb', [filePath, password]);
return GohashDb.from(filePath, db);
}
}

DateTime _timestamp(dynamic value) =>
new DateTime.fromMillisecondsSinceEpoch(value as int);

class GohashDb {
final String filePath;
final List<Group> groups;

const GohashDb(this.filePath, this.groups);

static GohashDb from(String filePath, Map<dynamic, dynamic> map) {
List<Group> groups = new List<Group>();
map.forEach((key, contents) {
List<dynamic> contentList = contents;
var entries = contentList.map((e) => LoginInfo.from(e)).toList();
groups.add(new Group(key.toString(), entries));
});
return new GohashDb(filePath, groups);
}
}

class Group {
final String name;
final List<LoginInfo> entries;

const Group(this.name, this.entries);
}

class LoginInfo {
final String name, description, username, password, url;
final DateTime updatedAt;

const LoginInfo(this.name, this.description, this.username, this.password,
this.url, this.updatedAt);

LoginInfo.from(Map<dynamic, dynamic> map)
: name = map["name"] as String,
description = map['description'] as String,
username = map['username'] as String,
password = map['password'] as String,
url = map['url'] as String,
updatedAt = _timestamp(map['updatedAt']);
}

lib/gohash_mobile.dart


Hope we can agree that, by looking at this code sample, Dart is a really nice language to read (and I can tell it was very easy to write, even being new to Dart)! It's in my opinion, almost as nice as Kotlin or Swift to work with. Almost because the type-safety is much loser (took me a few trial-and-error attempts to get the types above right - seems that this is due to the fact that we get dynamic objects from method channels, which bypass type-safety entirely, so you get runtime errors rather than compile-time errors if you try to pass dynamic objects into a function with the wrong type) and it does not seem to offer protection against null-dereferencing (which bit me a few times), though there's a proposal to fix that.

Flutter Plugins by default include an example app that can be used for tests. I changed the example app to verify the plugin was working correctly, you can check my changes in this commit. But we'll skip that here so we can move on to the actual mobile application.


Publish the Flutter Plugin



Once we have a Flutter Plugin, we need to make sure to publish it so others can use it! As you might have guessed, the flutter tool can be used to do that.

Use this command to run a dry-run and make sure it all looks good:

flutter packages pub publish --dry-run

All files not ignored by git will be published. It seems to be common practice to include even the example app. It gets shown on the Flutter Plugin docs, so it's really easy for users to see how they can use your plugin. In our case, we also need to include the iOS Mobileapi.framework library, though on Android, if you followed the advice to let Gradle run gomobile, Gradle itself will generate the bindings and include it in the Android build when users of the Flutter Plugin import this package. That also means that users of the plugin will need to have Go installed. To avoid that, it may be better to just publish the aar file with the project and not use Gradle to generate it - either way seems fine depending on the situation.

To actually publish the package, run the command:

flutter packages pub publish

Interestingly, all you need to be able to publish a package is to be logged in to your Google account on the browser! The CLI will give you a link that you can open in the browser to get a token for publishing the package. Pretty cool.


Use the Flutter Plugin in the mobile app


Flutter logo


Finally, we can go back to the mobile app we'd created at the beginning of this article. Now, we can use the Flutter Plugin from the previous section as just any Flutter package!

Open the pubspec.yaml file. In the dependencies section, add the Flutter plugin (I named the Flutter Plugin gohash_mobile and gave it version 0.0.1), so your dependencies should look something like this:


dependencies:
gohash_mobile: ^0.0.2
flutter:
sdk: flutter


Every time you add/remove dependencies, you can run this command to let Flutter update its cache and config files:


flutter packages get


To get the Android Gradle build to work, you'll have to make some changes to work around the bug we met earlier.

Add this line at the top of the android/settings.gradle file (where mobileapi is the name of the Go-bindings sub-project in the Flutter Plugin we've created):


include ':mobileapi'


Add this as the very last line of the same file:


project(':mobileapi').projectDir = project(':gohash_mobile').file('mobileapi')


Once again, notice that these steps shouldn't be necessary, hopefully the Flutter team will fix the problem before the 1.0 release!

Now, the Android build should run without errors:


flutter build apk


To build in iOS, make sure to open the project in XCode first, select a Signing Team, then run the project from XCode itself before trying from Android Studio or the flutter CLI (not sure why, but it seems XCode creates some necessary config files). After that, you should be able to run this command successfully:


flutter build ios


The last thing to do is, of course, consume the Flutter Plugin API and create the actual mobile app's GUI for it. That's the fun part!

This is what I wanted my UI to look like (except it should not look like it was drawn by a 5-year-old!):



Mobile app sketch




And even if it's still early stages, the app turned out pretty good-looking IMHO (for my first mobile app ever):



go-hash mobile app screenshot



This is the relevant UI code (from lib/main.dart) that created the above screenshot:



_showEntry(LoginInfo loginInfo) {
showDialog(
context: context,
builder: (ctx) => SimpleDialog(
contentPadding: EdgeInsets.all(10.0),
children: [
Text('Username:', style: _boldFont),
Text(loginInfo.username),
Text('URL:', style: _boldFont),
Text(loginInfo.url),
Text('Last changed:', style: _boldFont),
Text("${loginInfo.updatedAt}"),
Text('Description:', style: _boldFont),
Text(loginInfo.description),
],
));
}

Widget _buildCopierIcon(IconData icon, String value) {
return GestureDetector(
onTap: () => Clipboard.setData(ClipboardData(text: value)),
child:
Container(padding: EdgeInsets.only(left: 10.0), child: Icon(icon)));
}

Widget buildEntry(LoginInfo loginInfo) {
return ListTile(
trailing: Icon(Icons.description),
title: Row(children: [
Text(loginInfo.name),
_buildCopierIcon(Icons.person, loginInfo.username),
_buildCopierIcon(Icons.vpn_key, loginInfo.password),
]),
onTap: () => _showEntry(loginInfo));
}

Widget _buildGroupBody(Group group) {
return Column(children: group.entries.map(buildEntry).toList());
}

ExpansionPanel _buildGroup(int index, Group group) {
return ExpansionPanel(
headerBuilder: (ctx, isExpanded) => Text(
group.name,
textAlign: TextAlign.start,
style: _biggerFont,
),
isExpanded: index == _selectedGroupIndex,
body: _buildGroupBody(group));
}

Widget _buildGroups() {
final panels = _database.groups
.asMap()
.map((index, group) => MapEntry(index, _buildGroup(index, group)));

return ExpansionPanelList(
children: panels.values.toList(),
expansionCallback: (index, isExpanded) =>
setState(() => _selectedGroupIndex = index));
}

@override
Widget build(BuildContext context) {
return MaterialApp(
home: Scaffold(
appBar: AppBar(title: Text('go-hash')),
body: _errorMessage.isEmpty
? SingleChildScrollView(
child: SafeArea(
child: Material(
child: _buildGroups(),
),
),
)
: Center(
child: Text("Error: $_errorMessage",
style: _biggerFont.apply(color: Colors.red))),
));
}



Conclusion


Dart and Flutter are very pleasurable to work with. Both the language and the UI widgets are very powerful, but still easy to learn. The fact that you can change nearly any code in the app and immediately see what happens on the UI, thanks to the hot-reloading feature, makes it possible to write the app interactively. That's a real boost to productivity.

I've worked or played around with quite a few UI frameworks before, from Java's Swing to JavaFX, GWT to React.js and Elm, and I believe that Flutter was clearly inspired by what came before it, and takes some great ideas from them.

Interacting with iOS- and Android-specific libraries and APIs, on the other hand, is not very nice. Platform channels work ok for simple cases, but as we've seen, passing generic types around, for example, exposes some nasty edges in the implementation.

Including native libraries (in our case, Go libs, but could as well be C/C++ or even Rust) with Flutter Plugins seems to be an area that still needs improvements, but with a little effort, things can already work, as they did for me... my main difficulty was actually having to learn the intricacies of embedding native libs in both Android and iOS, given that Flutter can't really do much to hide that away from Flutter developers.

The big let-down in this story, however, was gomobile. Even though the tool looks so promising, it's clearly not ready for serious usage yet - and given the low activity in the Github repo lately, looks like Google is not at all prioritizing developing it further. This decision does make sense, given they'd rather have people write the full mobile app code in Dart. But given the wealth of Go libraries available, it would be nice to make this integration work better in the future - that might actually help drive some more usage of Flutter in the future, who knows!?

But the one thing that's clear to me is that Flutter is here to stay, and it's already being used quite heavily (just look at the activity on the Gitter channel) even before it reached 1.0! Once it gets there, my bet is that it will easily become a main player in the mobile apps field.


Links





gohash_mobile_app on GitHub (this is the actual app - notice it's not ready yet! I need to finish the UI and some integrations like with DropBox so it actually becomes useful!)


The return of RPC - or how REST is no longer the only respectable solution for APIs

posted Mar 4, 2018, 11:55 AM by Renato Athaydes   [ updated Mar 4, 2018, 1:06 PM ]

With all the debate over REST APIs and the many difficulties awaiting on the path to REST nirvana (at least for the purists - for example, did you know that you're doing it wrong if you version your REST API and don't use HATEOAS), it's interesting to observe a certain resurgence of one of the oldest methods of writing distributed applications: RPC (Remote Procedure Call).

If you want to get stuff done and are not interested in this philosophical discussion about what REST is and what it's not, RPC can be a time-saver. And it can even be the right tool for the job!

But for some time, RPC was considered a bad word (not without reason, things like Java RMI and Corba made it look ugly, complicated and unreliable). Not anymore! There's a lot options available in the "RPC world" (if we can say there's such thing).

In this blog post, I investigate in detail some of those options with the objective of measuring them up against the standard REST approach that has become the default.

I've used Gradle to build the code and Java 8 and Groovy to implement the servers/clients, but whatever stack you use, the procedure to create RPC-based communications shouldn't be very different from this.

All code shown in this article can be found on GitHub.


gRPC


gRPC is Google's own take on the old RPC mechanism. With support for over 10 different languages, you can benefit from using the shiniest languages and frameworks just like with your REST-based code, but with the additional benefit of extra efficiency and an actual generated interface you can use directly in your code, in your programming language of choice (rather than having to manufacture HTTP requests yourself or hope that a good SDK is provided in the language you want to use).

It does require a lot of boilerplate to setup. Below, I describe the exact steps you would need to take to get a RPC server and client communicating via gRPC using Gradle (build system) and Java (notice that the procedure is nearly the same for any other JVM language).


1. Create a Gradle build file like this:



plugins {
id "com.google.protobuf" version "0.8.4"
id "java"
}

group 'com.athaydes.tutorials'
version '1.0-SNAPSHOT'

sourceCompatibility = 1.8

repositories {
jcenter()
}

dependencies {
compile 'io.grpc:grpc-netty:1.10.0'
compile 'io.grpc:grpc-protobuf:1.10.0'
compile 'io.grpc:grpc-stub:1.10.0'
testCompile 'junit:junit:4.12'
}

sourceSets.main.java {
srcDirs "$buildDir/generated/source/proto/main/grpc"
srcDirs "$buildDir/generated/source/proto/main/java"
}

protobuf {
protoc {
artifact = "com.google.protobuf:protoc:3.5.1-1"
}
plugins {
grpc {
artifact = 'io.grpc:protoc-gen-grpc-java:1.10.0'
}
}
generateProtoTasks {
all()*.plugins {
grpc {}
}
}
}

build.gradle



2. create a proto file (in a protobuffer-specific declaration language) that declares your RPC service(s) and data:



syntax = "proto3";

option java_package = "com.athaydes.tutorials.rpc.grpc.api";

// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {}
// Sends another greeting
rpc SayHelloAgain (HelloRequest) returns (HelloReply) {}
}

// The request message containing the user's name.
message HelloRequest {
string name = 1;
}

// The response message containing the greetings
message HelloReply {
string message = 1;
}

src/main/proto/helloworld.proto



3. Run gradle build to generate the Java code (you can configure it to generate other languages as well).


This will generate the source code you need to make RPC calls (both the service and the data you exchange).



4. Now, implement the server-side of the service:




package
com.athaydes.tutorials.rpc.grpc;

import com.athaydes.tutorials.rpc.grpc.api.GreeterGrpc;
import com.athaydes.tutorials.rpc.grpc.api.Helloworld.HelloReply;
import com.athaydes.tutorials.rpc.grpc.api.Helloworld.HelloRequest;
import io.grpc.stub.StreamObserver;

public class SimpleGreeter extends GreeterGrpc.GreeterImplBase {

@Override
public void sayHello( HelloRequest req, StreamObserver<HelloReply> responseObserver ) {
HelloReply reply = HelloReply.newBuilder().setMessage( "Hello " + req.getName() ).build();
responseObserver.onNext( reply );
responseObserver.onCompleted();
}

@Override
public void sayHelloAgain( HelloRequest req, StreamObserver<HelloReply> responseObserver ) {
HelloReply reply = HelloReply.newBuilder().setMessage( "Hello again " + req.getName() ).build();
responseObserver.onNext( reply );
responseObserver.onCompleted();
}
}

src/main/java/com/athaydes/tutorials/rpc/grpc/SimpleGreeter.java



5. Add the service to a server and run it in the server's main class:



package
com.athaydes.tutorials.rpc.grpc;

import io.grpc.ServerBuilder;

import java.io.IOException;

public class Server {

public static void main( String[] args )
throws IOException, InterruptedException {
io.grpc.Server server = ServerBuilder.forPort( 8081 )
.addService( new SimpleGreeter() )
.build();
server.start().awaitTermination();
}

}

src/main/java/com/athaydes/tutorials/rpc/grpc/Server.java



6. Now that you have a server, you can call it from your client code:




package
com.athaydes.tutorials.rpc.grpc;

import com.athaydes.tutorials.rpc.grpc.api.GreeterGrpc;
import com.athaydes.tutorials.rpc.grpc.api.GreeterGrpc.GreeterBlockingStub;
import com.athaydes.tutorials.rpc.grpc.api.Helloworld.HelloReply;
import com.athaydes.tutorials.rpc.grpc.api.Helloworld.HelloRequest;
import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;

public class Client {

public static void main( String[] args ) {
ManagedChannel channel = ManagedChannelBuilder.forAddress( "127.0.0.1", 8081 )
.usePlaintext( true ).build();
GreeterBlockingStub greeter = GreeterGrpc.newBlockingStub( channel );

HelloRequest joe = HelloRequest.newBuilder().setName( "Joe" ).build();
HelloRequest mary = HelloRequest.newBuilder().setName( "Mary" ).build();

HelloReply joesReply = greeter.sayHello( joe );
HelloReply marysReply = greeter.sayHello( mary );

System.out.println( joesReply.getMessage() );
System.out.println( marysReply.getMessage() );
}

}

src/main/java/com/athaydes/tutorials/rpc/grpc/Client.java



Notice the call usePlainText( true ) when creating the client's channel. That's just to avoid using TLS! The server in this example does not enable TLS, so if you copy this code, remember to do that... and don't blame me if you forget, this is just a hello-world example!



And that's it... run the Server class first, then the Client in a separate shell. You should see the expected hello messages:



Hello Joe
Hello Mary



You've probably never worked so hard to get a couple of hello world messages printed in the terminal, but hey, this is a distributed application now! And you didn't even once have to build a HTTP call!!!

Hm... maybe something else could do this but a little bit more easily?!

SOAP is definitely out of the question as using it could ruin our reputation (even though it didn't look too different from the above, to be honest... except when all the specs that developed around SOAP are taken into account, then yeah, we don't want to go back to that)... but how about its simpler, nearly forgotten predecessor, XML-RPC?! Well, let's have a look (if you hate XML so much you wouldn't even think, skip to the next section!).


XML-RPC


XML-RPC is an old technology. It was designed in 1998, according to Wikipedia, and later developed into SOAP. But despite that, it is really simple! So simple you can read the full specification in around 5 minutes.

Because of its simplicity and age, it has been implemented in every language imaginable (here are the libraries for JavaScript, PHP, Python, C and C++, Haskell, Go, .NET, OCaml, Common Lisp, Clojure, Rust, Swift and, of course, Java - or this one for Android).

It's amazing how XML-RPC already in 1998 brought us a way of making remote calls based on open internet protocols people were already familiar with, something that SOAP managed to totally mess up just a few years later...

Anyway, even really ancient technologies might sometimes turn out to still be useful, so let's have a look at what exactly XML-RPC looks like using Gradle and Java.


1. Create a Gradle build file:




plugins {
id "java"
}

group 'com.athaydes.tutorials'
version '1.0-SNAPSHOT'

sourceCompatibility = 1.8

repositories {
jcenter()
}

dependencies {
compile 'org.apache.xmlrpc:xmlrpc-server:3.1.3'
compile 'org.apache.xmlrpc:xmlrpc-client:3.1.3'
}

build.gradle


I've added dependencies on both xmlrpc-server and xmlrpc-client to that we don't need separate modules for the server and the client in this example.


2. Create a Handler class that you want to be called remotely.



package
com.athaydes.tutorials.xmlrpc;

public class Handler {

public String sayHello( String name ) {
return "Hello " + name;
}

public String sayHelloAgain( String name ) {
return "Hello again " + name;
}

}

src/main/java/com/athaydes/tutorials/xmlrpc/Handler.java


3. Create a Server.



 
package
com.athaydes.tutorials.xmlrpc;

import org.apache.xmlrpc.XmlRpcException;
import org.apache.xmlrpc.server.PropertyHandlerMapping;
import org.apache.xmlrpc.webserver.WebServer;

import java.io.IOException;

public class Server {

public static void main( String[] args ) throws XmlRpcException, IOException {
PropertyHandlerMapping mapping = new PropertyHandlerMapping();
mapping.addHandler( "Handler", Handler.class );
WebServer server = new WebServer( 8081 );
server.getXmlRpcServer().setHandlerMapping( mapping );
server.start();
}
}

src/main/java/com/athaydes/tutorials/xmlrpc/Server.java



4. Create a Client.




package
com.athaydes.tutorials.xmlrpc;

import org.apache.xmlrpc.XmlRpcException;
import org.apache.xmlrpc.client.XmlRpcClient;
import org.apache.xmlrpc.client.XmlRpcClientConfigImpl;

import java.net.MalformedURLException;
import java.net.URL;

public class Client {

public static void main( String[] args ) throws MalformedURLException, XmlRpcException {
XmlRpcClientConfigImpl config = new XmlRpcClientConfigImpl();
config.setServerURL( new URL( "http://127.0.0.1:8081" ) );

XmlRpcClient client = new XmlRpcClient();
client.setConfig( config );

Object[] params = { "Joe" };
String result = ( String ) client.execute( "Handler.sayHelloAgain", params );

System.out.println( result );
}

}

src/main/java/com/athaydes/tutorials/xmlrpc/Client.java


This Client call implementation is quite ugly, I am not sure why the Java library does not provide a Proxy wrapper around the XmlRpcClient so that calls to the remote method look like a normal call, but that's how things are currently.

I've implemented a Proxy wrapper anyway in a few lines of code, if you think it's useful, you can use it in your own projects:



package
com.athaydes.tutorials.xmlrpc.proxy;

import org.apache.xmlrpc.client.XmlRpcClient;
import org.apache.xmlrpc.client.XmlRpcClientConfigImpl;

import java.lang.reflect.Proxy;
import java.net.URL;

public class ClientProxy {

@SuppressWarnings( "unchecked" )
public static <T> T wrap( Class<T> handlerType, URL serverUrl, String handlerName ) {
XmlRpcClientConfigImpl config = new XmlRpcClientConfigImpl();
config.setServerURL( serverUrl );

XmlRpcClient client = new XmlRpcClient();
client.setConfig( config );

return ( T ) Proxy.newProxyInstance( handlerType.getClassLoader(),
new Class[]{ handlerType },
( proxy, method, args ) -> client.execute( handlerName + "." + method.getName(), args ) );
}

}

src/main/java/com/athaydes/tutorials/xmlrpc/proxy/ClientProxy.java


This proxy is very simple to use and makes the client code much simpler:



package
com.athaydes.tutorials.xmlrpc.proxy;

import java.net.MalformedURLException;
import java.net.URL;

public class SimplerClient {

public interface HandlerApi {
String sayHello( String name );

String sayHelloAgain( String name );
}

public static void main( String[] args ) throws MalformedURLException {
HandlerApi handler = ClientProxy.wrap( HandlerApi.class,
new URL( "http://127.0.0.1:8081" ),
"Handler" );
System.out.println( handler.sayHelloAgain( "Joe" ) );
}

}

src/main/java/com/athaydes/tutorials/xmlrpc/proxy/SimplerClient.java


Notice that we can only proxy via interfaces, so we needed to create a interface representing the remote service to be able to use the proxy... but you should use interfaces anyway when exposing remote services to avoid the client depending on the actual server implementation, so I don't see really this as a downside.




Build everything with gradle build.

Now we're ready to run the Server class in one shell, and the Client class in another. You should see the Hello again Joe message in the client, as expected.


It's difficult for me to see why XML-RPC is not so popular nowadays, or why people thought that it was not enough and some crazy complexity was necessary around it, so they created SOAP to solve whatever problem they think they had. As SOAP ultimately ended up being almost completely replaced with REST APIs, and that XML-RPC is much closer to REST than SOAP (in fact, you could architecture your XML-RPC API around REST principles), it seems that doing that was a serious mistake by the industry of the time.

Sure, XML is quite inefficient and verbose if performance is a requirement, but to solve that all they needed was a more efficient transport mechanism and serialization format (which is what gRPC did).

As you're probably aware, there's also a JSON-RPC specification, which is basically the same as XML-RPC except that JSON is used instead of XML, so I will not include it in this article.


GraphQL



Let's get this out of the way first: is GraphQL a RPC implementation? Well, some influential people certainly think so... and as I will show below, it definitely can be used like an RPC, so let's go with that!

GraphQL was developed by Facebook to make it easier for them to develop applications that required data from its servers in a variety of forms. It is described as "a query language for your API" in the home page and has been gaining substantial momentum recently.

As gRPC, GraphQL supports a dozen different languages. But how hard is it to use, both on the server and on the client side?

Only one way to find out! Let's implement a GraphQL Java HTTP server based on spark-java, and a Groovy client:


1. as usual, create a Gradle build file declaring the project's dependencies:



plugins {
id "java"
}

group 'com.athaydes.tutorials'
version '1.0-SNAPSHOT'

sourceCompatibility = 1.8

repositories {
jcenter()
}

dependencies {
compile 'com.graphql-java:graphql-java:6.0'
compile 'com.graphql-java:graphql-java-tools:4.3.0'
compile 'com.sparkjava:spark-core:2.7.1'
compile 'com.google.code.gson:gson:2.8.2'
runtime 'org.slf4j:slf4j-simple:1.7.25'
}

build.gradle


We'll use spark-java to wrap the GraphQL engine into a HTTP endpoint as that's probably the easiest way to do it in Java.

To convert the data objects to and from JSON, I chose Gson.



2. define your GraphQL schema. For example, here's mine:



schema {
query: QueryType
}

type QueryType {
sayHello(name: String!): Hello!
sayHelloAgain(name: String!) : Hello!
}

type Hello {
name: String!
message: String!
}

src/main/resources/hello_world.graphqls


GraphQL schemas can get pretty advanced. Its type system includes union types, enumerations, lists, interfaces, and non-null types (indicated by a ! at the end of the type name, as I used in the schema above for everything).



3. Write a Java definitions for the GraphQL types:



In this example, we make use of graphl-java-tools, which makes it considerably easier to use GraphQL, even though you could only use plain graphql-java and use a much lower-level API to feed data into the GraphQL engine.



package
com.athaydes.tutorials.graphql;

import com.coxautodev.graphql.tools.GraphQLQueryResolver;

public class Query implements GraphQLQueryResolver {

public Hello sayHello( String name ) {
return new Hello( name, "Hello " + name );
}

public Hello sayHelloAgain( String name ) {
return new Hello( name, "Hello again " + name );
}

}

src/main/java/com/athaydes/tutorials/graphql/Query.java



package
com.athaydes.tutorials.graphql;

public class Hello {

private final String name;
private final String message;

public Hello( String name, String message ) {
this.name = name;
this.message = message;
}

public String getName() {
return name;
}

public String getMessage() {
return message;
}
}

src/main/java/com/athaydes/tutorials/graphql/Hello.java



package
com.athaydes.tutorials.graphql;

import com.coxautodev.graphql.tools.GraphQLResolver;

public class HelloResolver implements GraphQLResolver<Hello> {

public String name( Hello hello ) {
return hello.getName();
}

public String message( Hello hello ) {
return hello.getMessage();
}

}

src/main/java/com/athaydes/tutorials/graphql/HelloResolver.java



4. Expose the GraphQL engine via HTTP:





package
com.athaydes.tutorials.graphql;

import com.coxautodev.graphql.tools.SchemaParser;
import com.google.common.reflect.TypeToken;
import com.google.gson.Gson;
import graphql.ExecutionInput;
import graphql.ExecutionResult;
import graphql.GraphQL;
import graphql.schema.GraphQLSchema;
import spark.Spark;

import java.util.Map;

public class Main {

public static void main( String[] args ) {
SchemaParser schemaParser = SchemaParser.newParser()
.file( "hello_world.graphqls" )
.resolvers( new Query(), new HelloResolver() )
.build();

GraphQLSchema executableSchema = schemaParser.makeExecutableSchema();
GraphQL graphQL = GraphQL.newGraphQL( executableSchema ).build();

Gson gson = new Gson();

// expose the GraphQL engine using a HTTP server
Spark.get( "/graphql-api", "application/json", ( req, res ) -> {
Map<String, Object> variables = gson.fromJson(
req.queryParamOrDefault( "variables", "{}" ),
new TypeToken<Map<String, Object>>() {
}.getType() );

ExecutionInput executionInput = ExecutionInput.newExecutionInput()
.query( req.queryParams( "query" ) )
.variables( variables )
.build();

ExecutionResult result = graphQL.execute( executionInput );
res.header( "Content-Type", "application/json" );
return gson.toJson( result );
} );
}

}

src/main/java/com/athaydes/tutorials/graphql/Main.java



The above implementation exposes a single /graphql-api endpoint which accepts GET requests with a query and variables query parameters. You could use POST as well, of course. Check the GraphQL docs for details on how GraphQL HTTP APIs are supposed to work.

At this point, you can build the server with gradle build.

If you're not into wiring everything together in your applications explicitly, you could use a framework with GraphQL support to do it for you.


5. Write a Groovy client (or use your favourite language):




@
Grab( "com.athaydes.rawhttp:rawhttp-core:1.1.0" )
import com.athaydes.rawhttp.core.*
import com.athaydes.rawhttp.core.client.TcpRawHttpClient

query = URLEncoder.encode '''query($n:String!) {
sayHelloAgain(name: $n) {
name
message
}
}''', 'UTF-8'

variables = URLEncoder.encode '{"n": "Michael"}', 'UTF-8'

client = new TcpRawHttpClient()

req = new RawHttp().parseRequest( """\
GET localhost:4567/graphql-api?query=$query&variables=$variables HTTP/1.1
Accept: application/json
User-Agent: rawhttp
""" )

println client.send( req ).eagerly()

client.close()

src/graphql-client.groovy



I used raw-http (a Java library with 0 dependencies that I wrote myself to make these kind of HTTP-based prototypes easy!) to create and send a HTTP request.

Notice how a GraphQL "query" can look just like a RPC call, with the difference that it describes what the response it wants back should look like. query is a static String, but it is accompanied by a variables object containing values to be substituted in the query (allowing for more dynamic behaviour and efficiency).



Run the server Main class, then run the client with groovy src/graphql-client.

It will print the full HTTP response provided by our GraphQL server:


HTTP/1.1 200 OK
Date: Fri, 02 Mar 2018 20:07:24 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(9.4.6.v20170531)

{"data":{"sayHelloAgain":{"name":"Michael","message":"Hello again Michael"}},"errors":[]}



Unfortunately, on the Java server, we had to write the data classes by hand, and on the client, I didn't bother using types for the model at all (we would have to convert the JSON to a Java/Groovy type, but that's left as an exercise to the interested reader). That's because, unlike with gPRC, generating the types does not seem to be the standard approach in GraphQL, even though it is supported via graphql-java-type-generator or the more polyglot graphql-code-generator (if you're into TypeScript, Swift or Scala, there's also apollo-code-gen). But all these type generators seemed clunky and under-documented to me, so I decided not to use them in this example.

So, in summary, GraphQL is very powerful but not very simple. Matching the server code to your data storage may be challenging (but there are some helper libraries and frameworks that may make this easier, and even databases that support GraphQL natively).  But losing any kind of type-safety and a language-idiomatic way to make calls on the client is a major bummer.

Anyway, GraphQL may be appropriate for clients that deal with complex data that changes frequently or back-ends that need to use several data sources to serve a single request, I guess.



Thrift


Thrift is Facebook's answer to Google's gRPC. It has been open-source since a whitepaper describing it was published in 2007 (when gRPC's predecessor, Stubby, was closed-source) and has now support for nearly 20 different languages. It is an Apache project since 2010.

To use Thrift with a Java, Gradle build, follow these steps:


1. Download and install the Thrift compiler:



Unlike gRPC, it seems that the Gradle plugin is not capable of automatically downloading the compiler, so we need to go to http://thrift.apache.org/download and install it manually. This can compilicate the build in CI servers.

The tutorial tells us to extract the tar ball, enter the thrift directory and run the following command:


./configure && make


What it doesn't say is that it will run for several minutes, as it builds and tests the Thrift implementation for several languages!

Well, nevermind... once you're done waiting, make sure you can run thrift from anywhere by adding a link to the executable:


ln -s $(pwd)/compiler/cpp/thrift /usr/local/bin/thrift


Apparently, if you're on Ubuntu, you can just run apt install thrift-compiler.



2. Create a Gradle build file:




plugins {
id "java"
id "org.jruyi.thrift" version "0.4.0"
}

group 'com.athaydes.tutorials'
version '1.0-SNAPSHOT'

sourceCompatibility = 1.8

repositories {
jcenter()
}

dependencies {
compile 'org.apache.thrift:libthrift:0.11.0'
}

sourceSets.main.java {
srcDirs "$buildDir/generated-sources/thrift/gen-java"
}

build.gradle



3. Create a Thrift file defining the data and service(s):




namespace java
com.athaydes.tutorials.thrift.api

service HelloService {
string sayHello(1:string name)
string sayHelloAgain(1:string name)
}

src/main/thrift/hello_world.thrift



4. Run gradle build to generate the Java files.



This will generate a single file at build/generated-sources/thrift/gen-java/com/athaydes/tutorials/thrift/api/HelloService.java containing all the definitions required for the implementation of the RPC server and client.


5. Implement the server-side service:



 package com.athaydes.tutorials.thrift;

import com.athaydes.tutorials.thrift.api.HelloService;
import org.apache.thrift.TException;

public class SimpleHelloService implements HelloService.Iface {
@Override
public String sayHello( String name ) throws TException {
return "Hello " + name;
}

@Override
public String sayHelloAgain( String name ) throws TException {
return "Hello again " + name;
}
}

src/main/java/com/athaydes/tutorials/thrift/SimpleHelloService.java



6. Implement the Server exposing the service:




package
com.athaydes.tutorials.thrift;

import com.athaydes.tutorials.thrift.api.HelloService;
import org.apache.thrift.server.TServer;
import org.apache.thrift.server.TThreadPoolServer;
import org.apache.thrift.transport.TServerSocket;
import org.apache.thrift.transport.TTransportException;

public class Server {

public static void main( String[] args ) {
try {
TServerSocket serverTransport = new TServerSocket( 7911 );

HelloService.Processor processor = new HelloService.Processor<>( new SimpleHelloService() );

TServer server = new TThreadPoolServer( new TThreadPoolServer.Args( serverTransport ).
processor( processor ) );
server.serve();
} catch ( TTransportException e ) {
e.printStackTrace();
}
}

}

src/main/java/com/athaydes/tutorials/thrift/Server.java



7. Implement the Client:



 
package
com.athaydes.tutorials.thrift;

import com.athaydes.tutorials.thrift.api.HelloService;
import org.apache.thrift.protocol.TBinaryProtocol;
import org.apache.thrift.protocol.TProtocol;
import org.apache.thrift.transport.TSocket;

public class Client {

public static void main( String[] args ) throws Exception {
TSocket tSocket = new TSocket( "localhost", 7911 );
tSocket.open();

TProtocol tProtocol = new TBinaryProtocol( tSocket );

HelloService.Client client = new HelloService.Client( tProtocol );

System.out.println( client.sayHello( "Mary" ) );
System.out.println( client.sayHelloAgain( "Mary" ) );

tSocket.close();
}

}

src/main/java/com/athaydes/tutorials/thrift/Client.java



As with the other examples, now you can run the Server in a shell and the Client in another, which should print the hello messages as expected:


Hello Mary
Hello again Mary


Thrift is really, really similar to gRPC. It's amazing that Google and Facebook both figured that the existing (at the time) RPC solutions were not enough and came up with something so similar, more or less independently (the Thrift whitepaper does mention Protobuffers but as it was closed-souce at the time, it's impossible to tell if they had access to the RPC design Google was using).

However, I do think that both of their solutions are sub-optimal in that they require more boilerplate than I think is justifiable for simple projects, including a custom IDL and its compiler, and that all code written to integrate with their frameworks cannot be re-purposed to use another RPC implementation without a lot of work.


Protobuf-TCP-RPC



Protobuf-TCP-RPC is a Java library I wrote myself based on Google's Protobuffer serialization format and TCP (the original idea was to implement an efficient Apache Aries' DistributionProvider for OSGi remote services). You might think that with gRPC, which is also a RPC based on Protobuffers, there should be no need for something like this, but I disagree.

First of all, gRPC only works with Protobuffer-generated data types, not with JVM types, which is quite annoying.

Secondly, having not only data types be generated by protoc but also the service base classes is quite a big limitation as all the code implementing the services needs to be specifically written for gRPC.

Hence, I thought that having a simpler, more JVM-friendly (but still usable in other platforms) RPC mechanism based on Protobuffers (which are great for serialization) was really needed. The disadvantage is that it's not as easy to use with non-JVM languages... but adding support for any language is simple enough using just protoc, given that the RPC implementation used by Protobuf-TCP-RPC is just a proto file describing a generic method call, and some kind of mapper from/to the language's data types, if desired.

Anyway, even though this is not nearly as mature and flexible as the other alternatives, it's simpler... here's how it works.



1. The Gradle build file:



 plugins {
id "java"
}

group 'com.athaydes.tutorials'
version '1.0-SNAPSHOT'

sourceCompatibility = 1.8

repositories {
jcenter()
}

dependencies {
compile 'com.athaydes.protobuf:protobuf-tcp-rpc:0.2.1'
}

build.gradle



2. Create a service interface representing the remote service:





package
com.athaydes.tutorials.protobuftcprpc;

public interface HelloService {
String sayHello( String name );

String sayHelloAgain( String name );
}

src/main/java/com/athaydes/tutorials/protobuftcprpc/HelloService.java


3. Create a server-side implementation of the service:




package
com.athaydes.tutorials.protobuftcprpc;

public class SimpleHelloService implements HelloService {
@Override
public String sayHello( String name ) {
return "Hello " + name;
}

@Override
public String sayHelloAgain( String name ) {
return "Hello again " + name;
}
}

src/main/java/com/athaydes/tutorials/protobuftcprpc/SimpleHelloService.java


4. Create a server exposing the service:



 
package
com.athaydes.tutorials.protobuftcprpc;

import com.athaydes.protobuf.tcp.api.RemoteServices;

import java.io.Closeable;

public class Server {

public static void main( String[] args ) throws Exception {
Closeable server = RemoteServices.provideService(
new SimpleHelloService(), 8081, HelloService.class );

System.out.println( "Type something to stop the server" );
System.in.read();
server.close();
}

}

src/main/java/com/athaydes/tutorials/protobuftcprpc/Server.java


5. Create a client that uses the service interface:




package
com.athaydes.tutorials.protobuftcprpc;

import com.athaydes.protobuf.tcp.api.RemoteServices;

public class Client {

public static void main( String[] args ) {
HelloService helloService = RemoteServices.createClient(
HelloService.class, "localhost", 8081 );
System.out.println( helloService.sayHello( "Joe" ) );
System.out.println( helloService.sayHelloAgain( "Joe" ) );
}
}

src/main/java/com/athaydes/tutorials/protobuftcprpc/Client.java



Build everything with gradle build. Run the Server class on a shell, and the Client on another. You should see the expected messages:


Hello Joe
Hello again Joe



This is, as far as I know, the simplest way to run RPC calls in the JVM, at least. If you need complex data types (i.e. custom data classes or structs), you need to create a proto file and generate the Java type for it, but if all you need is Java's basic types like int, float, double, char, boolean, String (coming soon, List, Set or Map of those), it will work without custom serialization logic.

I am even thinking of adding support for Kotlin data classes in a Kotlin-specific module, which would make this library much more powerful (and it's not difficult at all - if you would like to see this implemented, vote for it on GitHub).



Notable alternatives



This article is already a little bit longer than I think it should be (and I've already spent more time than is reasonable on it!), so without giving more details, here are some other alternatives you might want to look at:


Apache Avro


Apache Avro is a data serialization system that also supports RPC. It seems to be focused on dynamic languages and smaller payloads. As most alternatives, has its own custom IDL (defined with JSON), but defining one is optional.

It seems to have been developed for use in Apache Hadoop, and is also utilized in Apache Spark SQL.


JSON-RPC


As metioned in the XML-RPC section, JSON-RPC is quite similar to XML-RPC and for that reason, I decided to not describe it in any detail in this article.

If you're interested in learning more about it, have a look at simple-is-better.org's article about it. This website maintains a list of Java tools for working with JSON-RPC, including even a shell for experimentation.


Messaging systems


An alternative to REST and RPC which can be used to solve similar problems in distributed applications is a messaging system like JMS, ZeroMQ (highly recommended read even if you don't intend to use it!) and Apache Kafka (a really powerful system that can do a lot more than just implement the publisher/subscriber pattern).



Conclusion



REST is a great solution to distributed applications in many cases. As this article from 2013 shows, it was a welcome back-to-basics wake-up call to the industry at a time when complexity, much of it incidental, was starting to win over simplicity, causing applications to become difficult to write, maintain and evolve.

With REST, calling a remote service became as easy as sending a HTTP request.

But using REST for everything, including things like internal microservices used only by one or two other internal applications seems like a over-reaction. RPC is, and probably has always been, ideal for situations like this.

With the help of network-aware libraries like Netflix's Hystrix, RPC-based applications can be easier to develop and at the same time more efficient and reliable than a REST alternative.

If you ever need to break up a service into two simpler services, but you don't want to completely change how the different parts interact, do not hesitate to reach out for a RPC solution.

If your clients must be really flexible and customizable, and your data does look a little bit like a graph, GraphQL seems to be the way to go.

If performance is paramount and you want the option to use many different languages, go for either gRPC or Thrift.

You might even want to go with XML-RPC (or JSON-RPC) if you need to use some obscure languages or performance is not more important than simplicity for you.

To optimize for the absolute minimum network bandwidth, a good choice seems to be Apache Avro, specially if you are using dynamic languages.

If you're happy to stay in the JVM and use one of the many languages it supports, but don't want to rely on code-generation tools just to be able to call inter-process/remote services, then help me continue the development of protobuf-tcp-rpc (I think the library needs a better name, suggestions welcome!).



Thanks for reading.


Stop versioning libraries

posted Jan 16, 2018, 10:12 AM by Renato Athaydes   [ updated Jan 16, 2018, 10:18 AM ]

Versioning is one of the most basic concepts software developers have to deal with. But despite the apparent simplicity of the subject, we still haven't got it right!

Let me expand. We are used to a versioning scheme based on a set of numbers and a few qualifiers, so our versions look like 1.2.3 and 1.2.4-rc1. But have you ever stopped to think about what these versions really mean?

When you see a library that is on version 1.2.3, can you confidently claim that it is more mature than another library that is on version 1.0.0? Unfortunately, that's just impossible to tell. From real-world experience, I've come to believe that knowing a version number is almost completely useless. Version numbers are at the whim of library authors... they may not even follow semantic versioning, so version 1.2.0 might be completely different than version 1.1.0... or they might decide to skip a few major versions and go straight from 7 to 10!! But even if the library author follows semver religiously, which version of semver, given that semver itself is versioned?! They may decide to follow simver instead :D!

But seriously, when it comes to libraries (as opposed to apps/products, whose versions are normally meant for final users and marketing purposes, not dependency manager tools), I know you might be shocked by this and think that I have no idea what I am talking about, but try to give me a chance to explain it in the rest of this blog post... after all, one of the initial proponents of this idea was not me, but a really smart guy you may've heard of: Rich Hickey. Perhaps you'll start agreeing with this and join the cause, so in a not-so-distant future we might actually be able to build software confidently, even if we depend on hundreds of interdependent libraries!

This is my request to all library authors:

Don't ever make breaking changes in a library, regardless of the version change!
If breaking changes are unavoidable, change the library name instead.

That's because regardless of which language you use, there's no way to resolve versions correctly if different modules depend on different, incompatible versions of the same library.

I find it funny that some people believe that npm solves that. Or, if you're a little bit older, you might have thought that OSGi did it for Java 18 years ago. Well, they didn't! Because it's impossible.

OSGi will actually throw errors during startup if the two incompatible versions of a library are used by two different bundles and both can "see" each other (so, it actually solved a part of the problem by failing on startup, not runtime, but that doesn't help much and just got OSGi a bad rep), but in a node.js application, things might or might not work! Even though npm includes a module's dependencies into its own node_modules folder (so different consumers may use the correct version they require), it definitely won't work if objects from the two incompatible library versions happen to interact. This will cause difficult to debug runtime errors... that kind of error that only happens on a Saturday night with full moon.

To illustrate the problem, let's imagine we have a Javascript library, say lib1, that creates a tax summary given an income. This library was part of an effort to create tax reports, so the programmer thought it was a good idea to use a formatted String to describe the taxes... something like this:



function createTaxSummary(income) {
return {
"income": "Income = " + income,
"tax": "Tax = " + (income * 0.25),
}
}

exports.createTaxSummary = createTaxSummary;



Another library was used to actually generate a final report. Let's call it lib2. It uses lib1 to calculate the taxes, then returns a nice report:



const lib1 = require('../lib1')

function createReport(taxSummary) {
return "Tax Report\n----------\n" +
taxSummary.income.toUpperCase() + "\n" +
taxSummary.tax.toUpperCase()
}

exports.createReport = createReport;



The programmer made the line that contains the tax amount uppercase because that's the tradition in accounting circles.

The application code developed by another company should just call lib2 to generate the report and print it out, but lib2 did not have a way to add "extra" taxes that can be charged in some states... because that was only noticed when the release was already several weeks late, this feature was hacked into the application code by directly using lib1:



const
lib1 = require('../lib1')
const lib2 = require('../lib2')

const income = 100000;
const extras = 2500;

const summary = lib1.createTaxSummary(income);
var report = lib2.createReport(summary);
const extraTaxes = lib1.createTaxSummary(extras).tax;
report += "\nExtra taxes:\n" + extraTaxes.toUpperCase();

console.log(report);



The report now looked perfect!


Tax Report
----------
INCOME = 100000
TAX = 25000
Extra taxes:
TAX = 625



A few years later (the code was working, so no one had the time to go back "fix" the hack), someone noticed that the createTaxReport function was calculating the taxes directly. That doesn't make any sense! So the programmer changed lib1 so that it would provide a function just to calculate taxes, as well as fixing the tax summary object to contain only numbers, separating its logic from report logic... semantic versioning was being followed, so the version was bumped from 1.23 to 2.0. After all, this was a breaking change:



function calculateTax(income) {
return income * 0.25;
}

function createTaxSummary(income) {
return {
"income": income,
"tax": calculateTax(income),
}
}

exports.createTaxSummary = createTaxSummary;
exports.calculateTax = calculateTax;



The application developer, worried that the application was depending on an old versions of some libraries, decided to upgrade all versions. The change log on lib2 was pretty small, even though it was a major version change. It looked like the tax property was changed to a number, and that was pretty much it.

That's a one-line change.



const
lib1 = require('../lib1')
const lib2 = require('../lib2')

const income = 100000;
const extras = 2500;

const summary = lib1.createTaxSummary(income);
var report = lib2.createReport(summary);
const extraTaxes = lib1.createTaxSummary(extras).tax;
report += "\nExtra taxes:\nTAX = " + extraTaxes;

console.log(report);



Running it didn't work, though!


/lib2/index.js:6
      taxSummary.income.toUpperCase() + "\n" +
                        ^

TypeError: taxSummary.income.toUpperCase is not a function
    at Object.createReport (/lib2/index.js:6:25)
    at Object.<anonymous> (/app/index.js:8:19)
    at Module._compile (module.js:660:30)
    at Object.Module._extensions..js (module.js:671:10)
    at Module.load (module.js:573:32)
    at tryModuleLoad (module.js:513:12)
    at Function.Module._load (module.js:505:3)
    at Function.Module.runMain (module.js:701:10)
    at startup (bootstrap_node.js:193:16)
    at bootstrap_node.js:617:3


The author of lib2 is no longer maintaining it, so lib2 hadn't been updated with the changes. The only way to fix the error was to revert to the old version.

But what if there was a critical security bug in the old version? What if other libraries that were being used also expected the old summary format?

In this simple case (I admit it is contrived and looks artificial... sorry, that's the best simple example I could come up with - but can you be sure that something like this won't happen to you??) you might just have to fork the project(s) and patch it... but that could cost a lot of time. But even if this scenario looks unlikely, with a typical node_modules folder containing nearly 100MB of libraries, I think it should be almost unavoidable that something like this will happen at some point.

Just by using React/Redux, you get 40MB worth of dependencies:


➜  npm install --save react-redux
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN react-redux@5.0.6 requires a peer of react@^0.14.0 || ^15.0.0-0 || ^16.0.0-0 but none is installed. You must install peer dependencies yourself.
npm WARN react-redux@5.0.6 requires a peer of redux@^2.0.0 || ^3.0.0 but none is installed. You must install peer dependencies yourself.

+ react-redux@5.0.6
added 21 packages in 2.573s
➜  npm install --save react
npm WARN react-redux@5.0.6 requires a peer of redux@^2.0.0 || ^3.0.0 but none is installed. You must install peer dependencies yourself.

+ react@16.2.0
added 1 package in 0.419s
➜  npm install --save redux

+ redux@3.7.2
added 2 packages in 0.443s
➜  du -sh node_modules
40M     node_modules



Here's the full dependency tree:


➜  npm ls
lib3@1.0.0 /lib3
├─┬ react@16.2.0
│ ├─┬ fbjs@0.8.16
│ │ ├── core-js@1.2.7
│ │ ├─┬ isomorphic-fetch@2.2.1
│ │ │ ├─┬ node-fetch@1.7.3
│ │ │ │ ├─┬ encoding@0.1.12
│ │ │ │ │ └── iconv-lite@0.4.19
│ │ │ │ └── is-stream@1.1.0
│ │ │ └── whatwg-fetch@2.0.3
│ │ ├── loose-envify@1.3.1 deduped
│ │ ├── object-assign@4.1.1 deduped
│ │ ├─┬ promise@7.3.1
│ │ │ └── asap@2.0.6
│ │ ├── setimmediate@1.0.5
│ │ └── ua-parser-js@0.7.17
│ ├─┬ loose-envify@1.3.1
│ │ └── js-tokens@3.0.2
│ ├── object-assign@4.1.1
│ └─┬ prop-types@15.6.0
│   ├── fbjs@0.8.16 deduped
│   ├── loose-envify@1.3.1 deduped
│   └── object-assign@4.1.1 deduped
├─┬ react-redux@5.0.6
│ ├── hoist-non-react-statics@2.3.1
│ ├─┬ invariant@2.2.2
│ │ └── loose-envify@1.3.1 deduped
│ ├── lodash@4.17.4
│ ├── lodash-es@4.17.4
│ ├── loose-envify@1.3.1 deduped
│ └── prop-types@15.6.0 deduped
└─┬ redux@3.7.2
  ├── lodash@4.17.4 deduped
  ├── lodash-es@4.17.4 deduped
  ├── loose-envify@1.3.1 deduped
  └── symbol-observable@1.1.0


If you think the JavaScript world is insane, so that kind of thing is to be expected, let's see how the Java world compares.

A typical Java web backend will use some kind of framework like Dropwizard. If you want to see how many libraries you'll get just by making that choice, create a build.gradle file with the following contents:



repositories {
jcenter()
}

apply plugin: 'java'

dependencies {
compile "io.dropwizard:dropwizard-core:1.3.0-rc3"
}

task copyLibs(type: Copy) {
into "libs"
from configurations.runtime
}



Now, run the copyLibs task with Gradle and check how heavy the libs folder is:


➜  gradle copyLibs
Starting a Gradle Daemon (subsequent builds will be faster)
Download https://jcenter.bintray.com/io/dropwizard/dropwizard-core/1.3.0-rc3/dropwizard-core-1.3.0-rc3.pom
Download https://jcenter.bintray.com/io/dropwizard/dropwizard-parent/1.3.0-rc3/dropwizard-parent-1.3.0-rc3
.pom

......... hundreds of lines like that .........


BUILD SUCCESSFUL in 27s
1 actionable task: 1 executed
➜  du -sh libs
17M     libs


Well, at least that's better than the Node example, but 17MB of jars is still quite a lot!! Let's see what dependencies are responsible for that:


➜  gradle dep --configuration runtime

> Task :dependencies

------------------------------------------------------------
Root project
------------------------------------------------------------

runtime - Runtime dependencies for source set 'main' (deprecated, use 'runtimeOnly ' instead).
\--- io.dropwizard:dropwizard-core:1.3.0-rc3
     +--- io.dropwizard:dropwizard-util:1.3.0-rc3
     |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
     |    +--- com.google.guava:guava:23.5-jre
     |    |    +--- com.google.code.findbugs:jsr305:1.3.9 -> 3.0.2
     |    |    +--- org.checkerframework:checker-qual:2.0.0
     |    |    +--- com.google.errorprone:error_prone_annotations:2.0.18
     |    |    +--- com.google.j2objc:j2objc-annotations:1.1
     |    |    \--- org.codehaus.mojo:animal-sniffer-annotations:1.14
     |    +--- com.google.code.findbugs:jsr305:3.0.2
     |    \--- joda-time:joda-time:2.9.9
     +--- io.dropwizard:dropwizard-jackson:1.3.0-rc3
     |    +--- com.google.guava:guava:23.5-jre (*)
     |    +--- io.dropwizard:dropwizard-util:1.3.0-rc3 (*)
     |    +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
     |    +--- com.fasterxml.jackson.core:jackson-databind:2.9.3
     |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
     |    |    \--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    +--- com.fasterxml.jackson.datatype:jackson-datatype-guava:2.9.3
     |    |    +--- com.google.guava:guava:18.0 -> 23.5-jre (*)
     |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
     |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.9.3
     |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
     |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
     |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.9.3
     |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
     |    +--- com.fasterxml.jackson.module:jackson-module-parameter-names:2.9.3
     |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
     |    +--- com.fasterxml.jackson.module:jackson-module-afterburner:2.9.3
     |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
     |    +--- com.fasterxml.jackson.datatype:jackson-datatype-joda:2.9.3
     |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
     |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    |    +--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
     |    |    \--- joda-time:joda-time:2.7 -> 2.9.9
     |    \--- org.slf4j:slf4j-api:1.7.25
     +--- io.dropwizard:dropwizard-validation:1.3.0-rc3
     |    +--- io.dropwizard:dropwizard-util:1.3.0-rc3 (*)
     |    +--- org.hibernate:hibernate-validator:5.4.2.Final
     |    |    +--- javax.validation:validation-api:1.1.0.Final
     |    |    +--- org.jboss.logging:jboss-logging:3.3.0.Final
     |    |    \--- com.fasterxml:classmate:1.3.1
     |    +--- org.glassfish:javax.el:3.0.0
     |    +--- org.javassist:javassist:3.22.0-GA
     |    \--- org.slf4j:slf4j-api:1.7.25
     +--- io.dropwizard:dropwizard-configuration:1.3.0-rc3
     |    +--- io.dropwizard:dropwizard-jackson:1.3.0-rc3 (*)
     |    +--- io.dropwizard:dropwizard-validation:1.3.0-rc3 (*)
     |    +--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.9.3
     |    |    +--- org.yaml:snakeyaml:1.18
     |    |    \--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    +--- org.apache.commons:commons-lang3:3.6
     |    \--- org.apache.commons:commons-text:1.1
     |         \--- org.apache.commons:commons-lang3:3.5 -> 3.6
     +--- io.dropwizard:dropwizard-logging:1.3.0-rc3
     |    +--- io.dropwizard:dropwizard-jackson:1.3.0-rc3 (*)
     |    +--- io.dropwizard:dropwizard-validation:1.3.0-rc3 (*)
     |    +--- io.dropwizard.metrics:metrics-logback:4.0.2
     |    |    \--- io.dropwizard.metrics:metrics-core:4.0.2
     |    +--- org.slf4j:slf4j-api:1.7.25
     |    +--- org.slf4j:jul-to-slf4j:1.7.25
     |    |    \--- org.slf4j:slf4j-api:1.7.25
     |    +--- ch.qos.logback:logback-core:1.2.3
     |    +--- ch.qos.logback:logback-classic:1.2.3
     |    |    \--- ch.qos.logback:logback-core:1.2.3
     |    +--- org.slf4j:log4j-over-slf4j:1.7.25
     |    |    \--- org.slf4j:slf4j-api:1.7.25
     |    +--- org.slf4j:jcl-over-slf4j:1.7.25
     |    |    \--- org.slf4j:slf4j-api:1.7.25
     |    \--- org.eclipse.jetty:jetty-util:9.4.8.v20171121
     +--- io.dropwizard:dropwizard-metrics:1.3.0-rc3
     |    +--- io.dropwizard:dropwizard-lifecycle:1.3.0-rc3
     |    |    +--- org.slf4j:slf4j-api:1.7.25
     |    |    +--- com.google.guava:guava:23.5-jre (*)
     |    |    +--- org.eclipse.jetty:jetty-server:9.4.8.v20171121
     |    |    |    +--- javax.servlet:javax.servlet-api:3.1.0
     |    |    |    +--- org.eclipse.jetty:jetty-http:9.4.8.v20171121
     |    |    |    |    +--- org.eclipse.jetty:jetty-util:9.4.8.v20171121
     |    |    |    |    \--- org.eclipse.jetty:jetty-io:9.4.8.v20171121
     |    |    |    |         \--- org.eclipse.jetty:jetty-util:9.4.8.v20171121
     |    |    |    \--- org.eclipse.jetty:jetty-io:9.4.8.v20171121 (*)
     |    |    \--- io.dropwizard:dropwizard-util:1.3.0-rc3 (*)
     |    +--- io.dropwizard:dropwizard-jackson:1.3.0-rc3 (*)
     |    +--- io.dropwizard:dropwizard-validation:1.3.0-rc3 (*)
     |    +--- io.dropwizard.metrics:metrics-core:4.0.2
     |    \--- org.slf4j:slf4j-api:1.7.25
     +--- io.dropwizard:dropwizard-jersey:1.3.0-rc3
     |    +--- io.dropwizard:dropwizard-jackson:1.3.0-rc3 (*)
     |    +--- io.dropwizard:dropwizard-validation:1.3.0-rc3 (*)
     |    +--- io.dropwizard:dropwizard-logging:1.3.0-rc3 (*)
     |    +--- org.glassfish.jersey.core:jersey-server:2.25.1
     |    |    +--- org.glassfish.jersey.core:jersey-common:2.25.1
     |    |    |    +--- javax.ws.rs:javax.ws.rs-api:2.0.1
     |    |    |    +--- javax.annotation:javax.annotation-api:1.2
     |    |    |    +--- org.glassfish.jersey.bundles.repackaged:jersey-guava:2.25.1
     |    |    |    +--- org.glassfish.hk2:hk2-api:2.5.0-b32
     |    |    |    |    +--- javax.inject:javax.inject:1
     |    |    |    |    +--- org.glassfish.hk2:hk2-utils:2.5.0-b32
     |    |    |    |    |    \--- javax.inject:javax.inject:1
     |    |    |    |    \--- org.glassfish.hk2.external:aopalliance-repackaged:2.5.0-b32
     |    |    |    +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
     |    |    |    +--- org.glassfish.hk2:hk2-locator:2.5.0-b32
     |    |    |    |    +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
     |    |    |    |    +--- org.glassfish.hk2.external:aopalliance-repackaged:2.5.0-b32
     |    |    |    |    +--- org.glassfish.hk2:hk2-api:2.5.0-b32 (*)
     |    |    |    |    +--- org.glassfish.hk2:hk2-utils:2.5.0-b32 (*)
     |    |    |    |    \--- org.javassist:javassist:3.20.0-GA -> 3.22.0-GA
     |    |    |    \--- org.glassfish.hk2:osgi-resource-locator:1.0.1
     |    |    +--- org.glassfish.jersey.core:jersey-client:2.25.1
     |    |    |    +--- javax.ws.rs:javax.ws.rs-api:2.0.1
     |    |    |    +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
     |    |    |    +--- org.glassfish.hk2:hk2-api:2.5.0-b32 (*)
     |    |    |    +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
     |    |    |    \--- org.glassfish.hk2:hk2-locator:2.5.0-b32 (*)
     |    |    +--- javax.ws.rs:javax.ws.rs-api:2.0.1
     |    |    +--- org.glassfish.jersey.media:jersey-media-jaxb:2.25.1
     |    |    |    +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
     |    |    |    +--- org.glassfish.hk2:hk2-api:2.5.0-b32 (*)
     |    |    |    +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
     |    |    |    +--- org.glassfish.hk2:hk2-locator:2.5.0-b32 (*)
     |    |    |    \--- org.glassfish.hk2:osgi-resource-locator:1.0.1
     |    |    +--- javax.annotation:javax.annotation-api:1.2
     |    |    +--- org.glassfish.hk2:hk2-api:2.5.0-b32 (*)
     |    |    +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
     |    |    +--- org.glassfish.hk2:hk2-locator:2.5.0-b32 (*)
     |    |    \--- javax.validation:validation-api:1.1.0.Final
     |    +--- org.glassfish.jersey.ext:jersey-metainf-services:2.25.1
     |    |    +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
     |    |    \--- javax.ws.rs:javax.ws.rs-api:2.0.1
     |    +--- org.glassfish.jersey.ext:jersey-bean-validation:2.25.1
     |    |    +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
     |    |    +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
     |    |    +--- org.glassfish.jersey.core:jersey-server:2.25.1 (*)
     |    |    +--- javax.validation:validation-api:1.1.0.Final
     |    |    +--- org.hibernate:hibernate-validator:5.1.3.Final -> 5.4.2.Final (*)
     |    |    \--- javax.ws.rs:javax.ws.rs-api:2.0.1
     |    +--- io.dropwizard.metrics:metrics-jersey2:4.0.2
     |    |    +--- io.dropwizard.metrics:metrics-core:4.0.2
     |    |    \--- io.dropwizard.metrics:metrics-annotation:4.0.2
     |    |         \--- org.slf4j:slf4j-api:1.7.25
     |    +--- com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:2.9.3
     |    |    +--- com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:2.9.3
     |    |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
     |    |    \--- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:2.9.3
     |    |         +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0
     |    |         +--- com.fasterxml.jackson.core:jackson-core:2.9.3
     |    |         \--- com.fasterxml.jackson.core:jackson-databind:2.9.3 (*)
     |    +--- org.glassfish.jersey.containers:jersey-container-servlet:2.25.1
     |    |    +--- org.glassfish.jersey.containers:jersey-container-servlet-core:2.25.1
     |    |    |    +--- org.glassfish.hk2.external:javax.inject:2.5.0-b32
     |    |    |    +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
     |    |    |    +--- org.glassfish.jersey.core:jersey-server:2.25.1 (*)
     |    |    |    \--- javax.ws.rs:javax.ws.rs-api:2.0.1
     |    |    +--- org.glassfish.jersey.core:jersey-common:2.25.1 (*)
     |    |    +--- org.glassfish.jersey.core:jersey-server:2.25.1 (*)
     |    |    \--- javax.ws.rs:javax.ws.rs-api:2.0.1
     |    +--- org.eclipse.jetty:jetty-server:9.4.8.v20171121 (*)
     |    +--- org.eclipse.jetty:jetty-webapp:9.4.8.v20171121
     |    |    +--- org.eclipse.jetty:jetty-xml:9.4.8.v20171121
     |    |    |    \--- org.eclipse.jetty:jetty-util:9.4.8.v20171121
     |    |    \--- org.eclipse.jetty:jetty-servlet:9.4.8.v20171121
     |    |         \--- org.eclipse.jetty:jetty-security:9.4.8.v20171121
     |    |              \--- org.eclipse.jetty:jetty-server:9.4.8.v20171121 (*)
     |    +--- org.eclipse.jetty:jetty-continuation:9.4.8.v20171121
     |    \--- org.apache.commons:commons-lang3:3.6
     +--- io.dropwizard:dropwizard-servlets:1.3.0-rc3
     |    +--- org.slf4j:slf4j-api:1.7.25
     |    +--- io.dropwizard:dropwizard-util:1.3.0-rc3 (*)
     |    +--- io.dropwizard.metrics:metrics-annotation:4.0.2 (*)
     |    +--- io.dropwizard.metrics:metrics-core:4.0.2
     |    \--- ch.qos.logback:logback-classic:1.2.3 (*)
     +--- io.dropwizard:dropwizard-jetty:1.3.0-rc3
     |    +--- io.dropwizard:dropwizard-logging:1.3.0-rc3 (*)
     |    +--- io.dropwizard.metrics:metrics-jetty9:4.0.2
     |    |    \--- io.dropwizard.metrics:metrics-core:4.0.2
     |    +--- org.eclipse.jetty:jetty-server:9.4.8.v20171121 (*)
     |    +--- org.eclipse.jetty:jetty-servlet:9.4.8.v20171121 (*)
     |    +--- org.eclipse.jetty:jetty-servlets:9.4.8.v20171121
     |    |    +--- org.eclipse.jetty:jetty-continuation:9.4.8.v20171121
     |    |    +--- org.eclipse.jetty:jetty-http:9.4.8.v20171121 (*)
     |    |    +--- org.eclipse.jetty:jetty-util:9.4.8.v20171121
     |    |    \--- org.eclipse.jetty:jetty-io:9.4.8.v20171121 (*)
     |    \--- org.eclipse.jetty:jetty-http:9.4.8.v20171121 (*)
     +--- io.dropwizard:dropwizard-lifecycle:1.3.0-rc3 (*)
     +--- io.dropwizard.metrics:metrics-core:4.0.2
     +--- io.dropwizard.metrics:metrics-jvm:4.0.2
     |    \--- io.dropwizard.metrics:metrics-core:4.0.2
     +--- io.dropwizard.metrics:metrics-jmx:4.0.2
     |    \--- io.dropwizard.metrics:metrics-core:4.0.2
     +--- io.dropwizard.metrics:metrics-servlets:4.0.2
     |    +--- io.dropwizard.metrics:metrics-core:4.0.2
     |    +--- io.dropwizard.metrics:metrics-healthchecks:4.0.2
     |    +--- io.dropwizard.metrics:metrics-json:4.0.2
     |    |    \--- io.dropwizard.metrics:metrics-core:4.0.2
     |    +--- io.dropwizard.metrics:metrics-jvm:4.0.2 (*)
     |    \--- com.papertrail:profiler:1.0.2
     |         \--- joda-time:joda-time:2.9.1 -> 2.9.9
     +--- io.dropwizard.metrics:metrics-healthchecks:4.0.2
     +--- io.dropwizard:dropwizard-request-logging:1.3.0-rc3
     |    +--- io.dropwizard:dropwizard-jetty:1.3.0-rc3 (*)
     |    +--- io.dropwizard:dropwizard-logging:1.3.0-rc3 (*)
     |    \--- ch.qos.logback:logback-access:1.2.3
     |         \--- ch.qos.logback:logback-core:1.2.3
     +--- net.sourceforge.argparse4j:argparse4j:0.7.0
     \--- org.eclipse.jetty.toolchain.setuid:jetty-setuid-java:1.0.3

(*) - dependencies omitted (listed previously)


BUILD SUCCESSFUL in 0s
1 actionable task: 1 executed


Wholly crap!!! That's a lot of jars!!! 90, to be accurate.

Well, surely Java and JavaScript must be some of the worst offenders... but given it is just rational to try to avoid implementing functionality that already is implemented by libraries (which is nearly everything for the most popular languages), this proliferation of dependencies seems unavoidable.

For curiosity, I compared Java/JavaScript with Go and one of its heavier web frameworks, Beego, and was shocked to find out that this framework seems to have no external dependencies (even though it implements ORM, MVC, i18n and logging in the core module), weighs in at only 3,7MB, and a hello world, statically compiled web server (which, unlike Java and JS, does not require a runtime) compiles to around 11MB.

Not too bad! But that brings us to the curious case of Go dependency management (which may help explain how a large framework can be implemented without using libraries, even considering Go's stance to have a complete, large standard library).

You see, Go is very different from other languages in that it does not have a package repository! Until very recently, it did not even have a standard package manager.

Want to use a dependency in Go? Just run go get to download it from GitHub and start using it... For example, to use Beego, you first run this:


go get github.com/astaxie/beego


This downloads the source adds it to the GOPATH (where it can be found by the Go compiler). Now you can just import it in the source code with import github.com/astaxie/beego!

go get supports downloading from GitHub and a few other sources (e.g. BitBucket, LaunchPad), but it doesn't do any kind of version checks! It just gets the source currently in the default branch (it might look at tags with a Go version as well) and calls it a day.

This might sound like complete chaos! Every time you build your project in a new machine, you might get different code in your dependencies. The solution the Google Go team suggested was to not break backwards compatibility - because then, it doesn't matter when you build it, it will still work if it worked last time!

I was first shocked when I found out about that! But then I reasoned that Google engineers probably know what they are doing, even if it looks like they just came up with the most terrible solution to the hard problem of package management.

Have you ever tried to build software that is shared between hundreds of developers, maybe hundreds of teams? Keeping dependencies up-to-date in all projects so that everything works and remains free of security vulnerabilities is difficult. Some libraries may have no maintainer for a while, causing old versions of its transitive dependencies to spread like a fire.

But that's exactly the situation nearly everyone is at now. We all use hundreds of dependencies in our projects, some of which may not have been updated in a decade. We just can't keep up, this is a losing fight.

We must give up on trying to be heroic and accept the facts: updating all transitive dependencies of every library and project till the end of time is just unfeasible. We must find another way.

Here's how we can solve the problem:

  • No breaking changes at the binary/source level are allowed once a library is declared stable.
  • If a function is not ideal, write a new function that gets closer to perfection, but keep the old one with a "DEPRECATED" warning.
  • Don't ever remove anything. We all make bad decisions, but removing a function from a library will not solve anything.
  • When the burden of maintaining the legacy code becomes to heavy to carry, start a new library, with a new name, to obsolete the old one.

In the Java world, many of the most common libraries have been doing that for years, and it works great!

Check this gist that shows the evolution of the HTTP Components Client library. How hard can it be to design a HTTP client, right? Every version changed basically everything! But because they have been changing the package names from version to version, it is possible to run an application that depends on all 3 versions at the same time (which amazingly, can easily happen given how widespread this HTTP Commons is). Objects coming from one version will not have the same type than the equivalent Object coming from another, so any mismatches are caught by the type system auomatically (JS and other dynamic languages have the "advantage" that it will only blow up at runtime and you'll probably only ever notice the problem when it happens, hopefully during tests).

Same thing with Log4j, which changed its namespace (package) in version 2, effectively becoming a new library.

Unfortunately, we're humans so I acknowledge we cannot expect everyone to follow these rules... after all, even the Go community seems to have finally settled with a dependency manager (go dep). It works a little bit like npm and Cargo, pinning dependencies following version constraints present in the build file. But that solution still suffers from the problems I have mentioned above. Just because your version range should work according to semantic versioning, it doesn't mean it will. And as Gavin King, lead developer of Ceylon, said: version range compatibility is a lie. Remember, we can't trust people!

In the long term, the only viable solution is to automate the process of checking for backwards compatibility and stopping breaking changes from happening, so we don't need to trust people to achieve that.

In the Java world, there's already a tool, japicmp, that does that. Elm, a strict functional programming language for the web, aims to support automated version change checks at the language level! With that in place, you simply can't break backwards compatibility and cause havoc for all poor souls who thought it would be a good idea to rely on your software in their applications.

If you try hard, you actually can break compatibility by changing semantics... but that's just plain evil! If you do that, you should feel bad.

What about repeatable builds, you may ask. Well, perhaps what you're really asking for is archived builds, not the possibility to download all exact versions of all dependencies in every build?

To conclude, I acknowledge that we need to adapt to a world where versioning is a reality. I suggest just letting an automated tool bump our minor/patch versions every time we release (maybe every day, or several times a day). If we never bump our major version number and have automated checks in place to keep us honest, everyone can depend on version [1.0, 2.0) (or 1.* in semver language) forever without fear of being screwed, automatically receiving all fixes and improvements library authors add over time... even old examples in Stack Overflow will still work (if the example code becomes obsolete, the worst that will happen is that the developer using it will see a deprecation warning, but it will still work)! I am sure that if we do that, building software in 20 years will be a lot more pleasant and reliable than the current chaotic situation.



Impressions of Go: a wonderfully simple, weird, native language

posted Dec 28, 2017, 11:09 AM by Renato Athaydes   [ updated Jan 9, 2018, 2:13 PM ]

At my current job, I have joined a team that, among other things, is responsible for creating tools for developers. I'm a long-time JVM (Java/Kotlin/Groovy) developer, but I have to admit that for some kinds of tools we need to create, specially CLIs (command-line interfaces), the JVM (regardless of the language) is not the best choice. That's mostly because of the startup time of the JVM. It has gotten a lot better in recent JVM versions, but there's still a perceptible, even if under-a-second, lag to start even the simplest applications.

For some kinds of tool we might need to develop, a cross-platform, native language can be defintely a better choice.

I've been playing with some native languages like Haskell, Rust and Pony to see if I could find a language that fits that purpose, but unfortunately none of them made me feel satisfied.

Haskell and Rust, while being incredibly powerful tools, are very complex and I found myself being unable to become productive with them within a short time frame (several weeks).
I managed to write a couple of small projects in Rust (JGrab and the unfinished Vinegar), but it felt really constraining to me the whole time, without sign of getting much better with time.

Pony is another really interesting language, with a Python-like syntax, an Erlang-inspired actor-based runtime, a powerful type-system that can encode capabilities which make lock-free concurrency verifiably safe in a way similar to Rust, though using completely different means... but I just don't have the patience to write serious code using it due to the lack of tooling. It's extremely taxing for me to have to keep a browser open to check what method I can call, read basic documentation, read the standard lib code etc. (I've been spoiled by the wonderful tooling one gets to use with Java/Kotlin in IntelliJ).

That's one of the reasons I decided to finally try Go, the language developed by Google that has been a hit in the devops space, with arguably the most respected tools in this space written in Go: Kubernetes, Docker, rkt, several AWS utilities.

Go has been a hit with databases (CockRoachDB, InfluxDB) as well, and many other interesting, groundbreaking projects use Go! Just to give a few more examples of projects mostly written in Go that I felt excited about:

* IPFS - a huge effort towards a decentralized, permanent web.
* Keybase - end-to-end encryption for everyone! Supports chat, a distributed file system (public content can be viewed online) and Git.
* Hugo - static website generator that's becoming hugely popular (I know, I should be using it for this blog... hoping to switch to it soon).

That shows that a lot of cool stuff is being built with Go, which in my opinion is a great reason to give it a try.

Another reason for me, personally, is that Go is actively being used in another of our offices and there's been some talk about trying it in our office as well.

And that's how I got to write a mildly serious project in Go (the project is not ready yet, so I will not publish the link just yet... will update it as soon as I have something useful). UPDATE: the project is now useable and it's a CLI password manager called go-hash.

First impressions


At first look, Go code looks quite familiar, with a C-like syntax. The first not-so--obvious thing I noticed is how variables are declared:



i
:= 0



This declares a variable i with value 0. The type is inferred to be int.

You could also do this, equivalently:



var i int = 0



Or even just this:



var i int



Here, there's two things to understand... first, the type of the variable is declared after the variable name, so here i is the name of the variable, while int is its type.
Second, there's no need to explicitly provide an initial value!

That's because in Go, all types have a zero-value, so if you do not initialize a variable, it just takes that zero-value. For int, the value is, obviously enough, 0.

The next thing I noticed is that in many function declarations, the type of a variable seemed to be omitted:



func
HasPrefix(s, prefix string) bool { /* implementation */ }

 

That means that the variable type is the same as the one following it... so in this example, both s and prefix are of type string. bool is the return type of this function.

You might have noticed that most types start with a lower-case letter, and this function starts with a capital-letter. However, that's not really the case. In reality, any exported symbol in Go needs to start with a capital-letter, otherwise it is private to the scope where it is declared.
An exception to this rule seems to be primitive types, which always start with a lower-case letter, even though they can be used anywhere.

This, for me, is a strange choice because it makes it easy for types and variables names to conflict. I'm used to declare variables (for good or bad) with the same name as their types (but starting with lower-case), which in Go is impossible unless you export your type or the variable. The choice in Go, as the above function (taken from the strings package from the standard lib) shows, seems to be to use one-letter variables instead, which I am not a big fan of.

Anyway, the next thing to notice is that Go does not have classes. Instead, it offers a mix of interfaces and structs, similar to Rust (which has traits instead of interfaces).

A struct, as one might expect from other languages, is just a group of values:



type MyStruct struct {
     enabled bool
     name string
     values []int
}



It is interesting to notice that you can create an instance of a struct without providing values for all (or any) of its fields. The fields are just initialized to their zero-value if no value is provided:



m := MyStruct{
   enabled: true,
 }
 fmt.Printf("%+v\n", m)



Running this code prints this:


{enabled:true name: values:[]}


As you can see, the zero-value of a string is the empty string, and of an array (technically, a slice, actually), the empty array.

Interestingly, nil exists in Go but only pointers can be nil (as the zero-value of a pointer has to be something!) which seems to help limit the damage quite a lot.


The weird bits



The above code brings us to the first weird thing about Go (at least for me): when declaring a struct literal, you must write a trailing comma after each field value, including the last one.

That is, presumably, to be diff-friendly (so adding a new value won't cause the line above it to also change). But it's the first time I see this approach being used... I've seen other languages address this by formatting code differently, for example, Haskell and Elm prefer to just place the comma as the first character of a new line... but anyway, even though this is surprising and a bit weird, there seems to be a justification for it.

If we wanted to initialize the slice, we would need to learn about slice literals and the make built-in.

make is a built-in which cannot be written in Go itself. It can be used to create a slice, map or channel.

So, in order to provide an initial value for the values field of MyStruct, you could either give it a literal value, or use make to just allocate memory for it (in this case, with size 0, capacity 10):



    // literal slice
    m := MyStruct{
        values: []int{1, 2, 3},
    }

    // allocate array for slice with make
    n := MyStruct{
        values: make([]int, 0, 10),
    }



Creating a map with make looks like this (with a map literal also shown):



    // create map using make
    myMap := make(map[string]int)

    myMap["one"] = 1
    myMap["two"] = 2
    myMap["three"] = 3

    // create map literal
    mapLiteral := map[string]int{
        "one": 1,
        "two": 2,
        "three": 3,
    }



Looking at the map declaration above, you might think that Go supports generics. If you've ever read a discussion about Go anywhere, you would know that that's not the case! The Go creators famously refuse to add generics to the language, much to the despair of some people in every discussion around the topic.

The reason maps can be declared in Go is that they are a Go primitive, just like arrays/slices. It's not possible to create your own type that behaves similarly.

That's another weird thing about Go: the amount of constructs available to the user is much smaller than what the language itself makes use of.

So, while Go has no generics, it still has a generic-like thing: map. While there's no overloading in Go, built-ins can still behave as if there were, as in this example with the delete built-in, which removes entries from maps:



    myMap := map[string]int{
        "one": 1,
        "two": 2,
    }

    delete(myMap, "one")

    otherMap := map[int]string{
        1: "one",
        2: "two",
    }

    delete(otherMap, 1)



Declaring a function like delete in user code is just not possible. Notice, though, that unlike in most languages with generics, delete is type-safe: trying to delete a key with the wrong time is a compiler error.

To follow up on the delete built-in, it was very interesting to discover how to remove an item from a slice. Knowing about delete, I supposed that there would be something similar for slices. But there isn't! How to delete an item from a slice?

I bet you wouldn't guess it. This is how:



    slice := []string{"a", "b", "c"}
    i := 1

    // remove item at index i
    slice = append(slice[:i], slice[i+1:]...)



This idiom is used all over the place for this purpose because there's just no way to declare a function that does it for slices of any type (that would require generics) and the Go creators decided a built-in like delete was not necessary.

But at least they provided Go programmers with the append built-in, so they can just implement that themselves.

You might have notice the ... in the code above. We'll see what that means in a second, but before we move on, there's a few more weird thins I think I should mention!

Another feature that nearly every other language has but is missing in Go is enums.

To be fair, it's not completely missing, as there is a construct called iota which is supposed to compensate for that. I'll leave the details out (check the link for details), but here's, more or less, how you handle enumerated values in Go:



type Weekday int

const (
     Sunday = iota
     Monday
     Tuesday
    Wednesday
     Thursday
     Friday
     Saturday
)



Unfortunately, although this allows you to have something that resembles proper enums, this is not type-safe in the sense that you could pass -10 or whatever to any function that accepts a variable of type Weekday.

To finalize a section on the weird bits of Go, I couldn't do better than show how time format works in Go.

In Java and every other language I know of, time formatting is done via a String template with codes like H for hour and m for month, so you end up with templates like yyyy-MM-dd HH:mm:ss, which resolve to date-times like 2017-12-28 17:21:00.

Not so in Go! Because the traditional template may be confusing to newbies (for example, is mm month or minutes?), Go came up with a novel system.

Here's how to print the current time using the same format mentioned above with Go:



    println(time.Now().Format("2006-01-02 15:04:05"))



To understand why, I will just quote the Go docs:

These are predefined layouts for use in Time.Format and Time.Parse. The reference time used in the layouts is the specific time:

Mon Jan 2 15:04:05 MST 2006

which is Unix time 1136239445. Since MST is GMT-0700, the reference time can be thought of as

01/02 03:04:05PM '06 -0700

The cool bits


The append built-in is interesting because it shows how to use variadic parameters in Go. In the case we saw previously, to spread the slice into a variadic parameter.
Luckily, it is actually possible to declare a function that takes a variadic parameter:



func myAppend(s string, ss ...string) string {
     b := bytes.NewBufferString(s)
     for _, item := range ss {
         b.WriteString(item)
     }
     return b.String()
}



This function can then be used in any of the following ways:



    strings := []string{"dear ", "world"}

    println(myAppend("hello dear world"))
    println(myAppend("hello ", "dear ", "world"))
    println(myAppend("hello ", strings...))



Which prints hello dear world 3 times.

As mentioned briefly before, unlikely Java and most other object-oriented languages, Go does not have classes. But it still allows us to define methods for structs, maintaining the benefits of strong encapsulation!


By adding methods to structs, we can also make the struct implement an interface to achieve polymorphism -
a struct implements an interface implicitly, i.e. it can be used where an interface is required as long as it implements all methods of the interface.



For example, consider the MyStruct defined earlier. Here's its definition again:



type MyStruct struct {
     enabled bool
     name string
     values []int
}



We could define a method called Sum that sums over all values of a MyStruct instance with the following code:



func (m *MyStruct) Sum() (result int) {
     for _, item := range m.values {
         result += item
     }
     return
}



Here, I intentionally introduce quite a few features not mentioned earlier. For example, notice the return value is named (called result in this case), so we don't need to declare it in the function body or in the return statement at the end.

Also notice the for loop. Go supports simple indexed loops like most other imperative languages as well:



    for i := 0; i < 10; i++ {
        println(i)
    }



But the for loop in the previous example looks quite different. That's because that's a for-each loop.
It works like this: the range keyword iterates over data structures, returning two values. In the case of slices, an index and the current item.
In the Sum function, we throw away the index (because it is not needed in this case) by declaring it with _, using only the current item.

You cannot declare variables and not use them in Go, doing that causes a compiler error! That's why we need to say we don't care about the index by naming it _. I noticed that this feature saved me from writing quite a few bugs!

One more detail in the Sum function: notice that the parameter type is prefixed with a *. That indicates the variable is pointer to something of type MyStruct, very similar to C. And as in C, to deference a pointer, you would normally use the & character. But that's only required to call functions in Go, methods can be called without that!

Hence, this is how you can call the Sum method on an existing instance of a struct:


   
n
:= MyStruct{
        values: []int{1, 2, 3, 4, 5},
    }

    fmt.Printf("Sum of n: %d\n", n.Sum())



If Sum were a function, you would need the & symbol:



    fmt.Printf("Sum of n: %d\n", Sum(&n))



It is preferable to use pointers when passing structs around because otherwise Go would make a copy of the struct when passing it to the function or method (unless that's exactly what you need, even if it might be less efficient).

Before we move on, remember how pointers can be nil? So every time we handle a pointer, we need to be careful to handle nil, so our Sum function should really be something like this instead:



func (m *MyStruct) Sum() (result int) {
     if m == nil {
         return
     }
     for _, item := range m.values {
         result += item
     }
     return
}



Now, moving on... Go has a feature to defer the execution of a method invocation until the end of the function where it is invoked, regardless of whether the function panics (for Java developers, similar to throwing an Error).

This feature is commonly used to release resources in a safe manner. For example, suppose you need to write to a file. The common idiom is to defer a call to its Close method immediately after assigning it (and checking for an error):


    
    file, err := os.Create(filePath)
    if err != nil {
        return err
    }
    defer file.Close()



There's a few things you must understand to be able to use defer correctly, but the Go Tour explains that much better than I could, so just have a look at the Tour for the details (the Tour is a really great way to learn Go, by the way!).

This code snippet shows another cool thing about Go: its error handling.

Most functions that have a good likelihood of failing, like opening a file or reading from a socket, return both a value and an error. If an error occurs, then the error result will not be nil, otherwise you can safely use the success return value.

To propagate the error, just return it from your own function. To give up on error, call panic(err), which is meant to crash the program (though there are ways to recover upstream), similar to Exceptions in languages that have them.

The possibility to return multiple values from a function is really useful, and not only to return the tuple (success, error).

For example, this is a function I wrote in the project I am working on, currently:



func removeEntryFrom(entries *[]LoginInfo, name string) ([]LoginInfo, bool) {
     if i, found := findEntryIndex(entries, name); found {
         return append((*entries)[:i], (*entries)[i+1:]...), true
     }
     return *entries, false
}



It returns the new slice with the target entry removed if it is found, along with a boolean that tells the caller whether the entry was removed.

There's quite a few other cool things about Go, but to not make this blog post even longer, I should stop here and just mention goroutines and channels, which are the primitives used to achieve concurrency in Go.

Even though I haven't had the chance to use them yet, they seem quite approachable and are definitely one of the strongest points of Go.

Again, I will defer (sorry the pun) to the Go Tour (click on the link) for a short and easy explanation of how they work.



Conclusion


Go does justice to its reputation: it is a really simple language and runs quite fast.

On the one hand, the haters seem to be right about some things. Go really misses some very basic features that most modern languages have adopted: generics, enums with powerful tricks like in Kotlin and Rust, functional programming features (no sign of map/filter/reduce in Go - no doubt due to the lack of generics), and even a built-in package manager (though I really like the simplicity of go get, and go dep seems to be finally unifying the tooling around Go dependency management).

But on the other hand, the weaknesses of Go may play to its advantage: I read a lot of Go code while working on my first project with the language, and I could understand basically all of it without trouble! The simplicity of the language is essential for that to be possible.

The Go compiler is extremely fast. My project was compiling so fast sometimes I was not sure it actually did anything :) Only after the project grew to a couple of thousand lines of code, did I notice a small, sub-second delay when compiling!

There are excellent IDEs for Go: the free VSCode plugin is very good even for someone accustomed to the power of IntelliJ.

Jetbrains has a full-blown IDE for Go, Goland but it is not free... but I found out that because I already have a IntelliJ Ultimate license from work, I could just use the Go Plugin in IntelliJ Ultimate without extra cost (unfortunately, it seems that a license is required to install this plugin... I was not able to install it at home without the license - hence why I've been using mostly VSCode). The IntelliJ plugin is definitely better than VSCode, with lots of small conveniences missing from VSCode (automatic imports, refactoring, quick-fixes etc), but I was actually surprised at how close VSCode is to the IntelliJ experience!

All in all, I can definitely understand why so many successful products are written in Go: it is a very productive language, and even though it is native, it is still very easy to compile to multiple targets.

The simplicity of the language does annoy me at times when writing yet another old-fashioned for-loop, but that simplicity seems to be its greatest feature, after all.





Announcing RawHTTP - a JVM library for handling raw HTTP

posted Dec 10, 2017, 1:24 PM by Renato Athaydes   [ updated Aug 1, 2018, 11:14 AM ]

HTTP is the language of the internet. It is also the most commonly used communication protocol for REST APIs, which have become the backbone of most distributed applications.
That's probably not news to anyone. But something that a lot of people may not realize is that HTTP (up to version 1.1, more on that later) is a textual protocol you can quite easily write by hand!

For example, here is a GET request directed at the jsontest.com REST API that will return the headers we send out as a JSON object:


GET / HTTP/1.1
Host: headers.jsontest.com
Accept: application/json


Running this request results in the following response (shown exactly as it is returned by the server in bytes-form):


HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json; charset=ISO-8859-1
X-Cloud-Trace-Context: e7be085086214dffdeb6350ad37672ed
Date: Sun, 10 Dec 2017 18:33:53 GMT
Server: Google Frontend
Content-Length: 155

{
   "X-Cloud-Trace-Context": "e7be085086214dffdeb6350ad37672ed/9779089525915638501",
   "Host": "headers.jsontest.com",
   "Accept": "application/json"
}


As you can see, both the request and response are easy to read and write... one could argue you could write them easily by hand. And that's exactly why I decided to create RawHTTP.

RawHTTP is a library written in pure Java, with no dependencies, which lets you create requests and responses from their textual representation, letting you then turn them into bytes which can be sent to either a server in the case of a request, or a HTTP client in the case of a response!

As a side note: do you know the difference between a HTTP request and a HTTP response?

Well, only the first line (called start-line by the HTTP RFC). The rest of the HTTP message (either request or response) is exactly the same. Sure, requests and responses normally include different headers, but the message format is exactly the same:


start-line\r\n
header-name: header-value\r\n
...
\r\n
<optional body>

There is some complication in the framing of the optional body, but essentially that's all there is to a HTTP message.

Pretty neat.

So, back to RawHTTP! Given that HTTP messages are so easy to write, you might ask yourself why even bother creating a library to parse it.

That's a fair question. And the answer is that even though it is in fact very easy to write, it is also very easy to make mistakes that make your HTTP message completely invalid, which may cause a connection to hang (if you, for example, get the Content-Length header value wrong!) or to just be rejected right away by the recipient (if you forget to use \r\n as line-separators and carelessly use just \n).

If you can make sure you never make these mistakes (and many others, of course), then you can definitely just do the following, in Kotlin, for example:


All examples in this blog post are written in Kotlin, but should look similar in Java or any other JVM language.



val
request = "GET / HTTP/1.1\r\n" +
"Host: ip.jsontest.com\r\n" +
"Accept: application/json\r\n" +
"\r\n"

val response = Socket(InetAddress.getByName("ip.jsontest.com"), 80).use {
it.getOutputStream().write(request.toByteArray())
// TODO read the response
}


This works perfectly, but reading the response might be a little tricky because you need to be able to read the headers first to be able to determine how to read the body (if any), and as mentioned earlier, reading the body can be tricky if the sender decides to send it out in chunks for efficiency.

There are more reasons why you probably don't want to do it all by yourself, but let it suffice to say that deciding when you should attempt to read a HTTP message's body, which headers, status code, methods, impact on that decision, is not trivial to decide.

RawHTTP helps with all of these problems.

Here's how you would send a request AND read the response correctly, using RawHTTP:



val http = RawHttp()

val request = http.parseRequest("""
GET / HTTP/1.1
Host: headers.jsontest.com
Accept: application/json
""".trimIndent())

val response = Socket(InetAddress.getByName("date.jsontest.com"), 80).use {
request.writeTo(it.getOutputStream())
http.parseResponse(it.getInputStream()).eagerly()
}



Notice that RawHTTP keeps your HTTP message pretty raw, but it does fix a few things for you to make sure it is a little harder to make invalid requests and responses.

For one thing, you can use simple new-lines, no need to insist on using \r\n (though it works as well).
RawHTTP also inserts a final new-line for you automatically if needed.


You might have noticed the eagerly() method being called before we return the response? That's because, by default, HTTP message's bodies are not read unless you call asStream() and read the InputStream yourself, or you just call eagerly(), which does that for you. So, unless you only want to download the body of a message conditionally after processing the message's headers or you need custom behaviour, always call eagerly() after parsing a request/response to avoid surprises.


Other things RawHTTP does is make sure that you have the minimum set of headers necessary for your HTTP message to be generally valid, and set a HTTP version in the start-line if none is specified.

In the case of requests, you must make sure to include the Host header (at least in HTTP/1.1).

For the sake of convenience, you can simply specify your request as follows:


GET http://example.com/path/to/resource


RawHTTP will insert a Host header with value example.com and set the HTTP version to HTTP/1.1, so the end result should be this:


GET /path/to/resource HTTP/1.1
Host: example.com


If you do not want RawHTTP to fix the Host header for you (or fix new-lines), you can actually configure it to not do that! Just give an instance of com.athaydes.rawhttp.core.RawHttpOptions to the RawHttp constructor configured the way you want and you're done.

If your request needs to include a body, you can simply write the body after an empty line (don't forget to add the Content-Length header so the recipient of the message knows how to read the body - keep reading to see how RawHTTP can set a body from a String or File so you don't need to include it in the raw message):



val
response = RawHttp().parseResponse("""
HTTP/1.1 200 OK
Server: RawHTTP
Content-Length: 12

Hello World!
""".trimIndent())



Notice that just like you can stream a request to a server, you can stream a response to a client, just use its writeTo(OutputStream) method! That means that using RawHTTP, you can quite simply create working HTTP clients and servers!

But please beware that RawHTTP is still VERY low level. It concerns itself only with the format of the HTTP messages as defined by the HTTP/1.1 RFC (RFC-7230). It currently does not validate headers' names and values, for example, or support any kind of common HTTP extension features, such as cookies, caching etc. So it may not be a replacement to HttpClient or Jetty :)

RawHTTP also allows some HTTP/1.0 behaviour, such as not setting the Host header if the message has HTTP/1.0 version, and downloading the body until the connection is closed in case of a missing Content-Length and Transfer-Encoding headers, but it generally adheres to HTTP/1.1. In the future, I plan to add support for setting the message's version to HTTP/2.0 and then process it as such, but unfortunately HTTP/2.0 messages are binary rather than textual, so they will have to be converted from the HTTP/1.1 format rather than written raw.

If you just need a very lightweight way to send requests/responses, though, specially if you just want to test your client or server, RawHTTP may be ideal for your use-case.


One last thing about sending HTTP messages with bodies: RawHTTP messages have a replaceBody method that lets you, you guessed it, replace a HTTP message's body with the contents of a String (via StringBody) or a file (via FileBody).

When you use replaceBody, RawHTTP changes the Content-Type and Content-Length headers automatically.

All RawHTTP objects are completely immutable, so methods like replaceBody do not modify the instance they are called on, but return a new, modified version of the object instead.


For example, to use a file as the body of a HTTP response:



val response = RawHttp().parseResponse("""
HTTP/1.1 200 OK
Server: RawHTTP
""".trimIndent()
).replaceBody(FileBody(File("example.json"), "application/json", true))

println(response.eagerly())


Which, in the case of my example.json file, prints the following:


HTTP/1.1 200 OK
Server: RawHTTP
Content-Type: application/json
Content-Length: 47

{
  "hello": "Hi",
  "location": "Stockholm"
}



To finish off, notice that RawHTTP ships with a very basic HTTP client that makes it just a little more convenient to send requests and receive responses. Using the included client, called TcpRawHttpClient, your code would look like this:



val request = RawHttp().parseRequest("""
GET / HTTP/1.1
Host: headers.jsontest.com
Accept: application/json
""".trimIndent())

val response = TcpRawHttpClient().send(request).eagerly()



I am planning to write a very simple, asynchronous server implementation as well so that the library ships with both a functional client and server, but I haven't had the time to finish that yet... I also want to make sure the server implementation does a little bit more validation on the messages it accepts to avoid security issues, but in the spirit of the library's goals (namely, to provide the bare minimum HTTP implementation) it will still not take care of things like caching, CORS headers or anything on those lines... I hope this will be useful anyway as a test library or perhaps as the backbone of full-featured HTTP clients and servers in the future.

To use RawHTTP, just include it as a dependency of your project, it's on Maven Central and JCenter, see the RawHTTP Bintray page for details.

All source code shown in this blog post is available on GitHub.

Say no to Electron! Using JavaFX to write a fast, responsive desktop application.

posted Oct 3, 2017, 6:11 AM by Renato Athaydes   [ updated Oct 21, 2017, 7:42 AM ]

        This article has been translated to Russian!

Lately, there has been a lot of discussions (see here, here and here for a few examples - and today this one) in programming forums about Electron and its impact on the desktop app world.

If you don't know Electron, it's basically a web browser (Chromium) that hosts only your web application... as if it were a desktop application (no, it's not a joke)... that lets you use the web stack to develop cross-platform desktop applications.

Most new, hipster desktop apps these days are  built on Electron, including Slack, VS Code, Atom and GitHub DesktopThis is an extraordinary development.

We've been writing desktop apps for decades. The web, on the other hand, only really got started less than 20 years ago, and most of that time it was only used for serving documents and animated gifs, not creating full-fledged applications, or even simple ones!

To think that the web stack would be used to create desktop applications 10 years ago would be unthinkable. But here we are, in 2017, and a lot of intelligent people think that Electron is a great idea!

This is not as much a result of the superiority of the web stack for building applications (far from that, I don't think anyone disagrees that the web is a mess), as a failure of the current desktop UI frameworks. If people are preferring to ship a full web browser with their apps just so they can use great tools such as JavaScript (sarcasm) to build them, something must have gone terribly wrong.

So, what are these terrible alternatives that are losing out badly to the web stack?

I decided to have a look and build myself a real application with one of them.

Electron alternatives


If you don't mind having different teams developing for each popular OS, the options seem to be, currently, AppKit on MacOS, WPF on Windows (I am not a specialist in platform-specific development, so please let me know if other options are more popular nowadays).

However, the real competitors to Electron are the multi-platform frameworks. I believe the most common multi-platform frameworks today are GTK+Qt and JavaFX.

Gtk+


GTK+ is written in C but has bindings to many other languages as well. It was used to develop the beautiful GNOME-3 platform.

Qt


Qt seems to be the most often suggested alternative to Electron in the discussions I've seen... It's a C++ library but also has bindings in other languages (though it seems that none of them are supported commercially and it's hard to say how complete they are). It looks like a popular choice for embedded systems.

More alternatives!

UPDATE (2017-Oct-21): Thanks to comments from people on Reddit, I decided to add a link to a few more alternatives here:


Embeddable HTML/CSS/script engine for modern UI development.


Open source Python library for rapid development of applications
that make use of innovative user interfaces, such as multi-touch apps.


NW.js (previously known as node-webkit) lets you call all Node.js modules directly from DOM and enables a new way of writing applications with all Web technologies.


Lazarus is a Delphi compatible cross-platform IDE for Rapid Application Development. It has variety of components ready for use and a graphical form designer to easily create complex graphical user interfaces.


JavaFX


In this post, however, I will focus on development of a desktop application using JavaFX because I believe JavaFX and the JVM are great for developing desktop applications.

Whatever you think about the JVM, no other platform (except perhaps Electron itself!) is as easy to use for development on multiple platforms. Once you've created your jar, in any platform, you can distribute it to any user on any OS and it will just work.

With the large variety of languages that currently run on the JVM, language shouldn't be an issue: there will definitely be one that you like (even including JavaScript if you just can't help it), and you can use JavaFX with any JVM language without big issues. In this blog post, besides Java, I will show a little bit of Kotlin code as well.

The UI itself can be built with just code (in which case you have wonderful IDE support from IntelliJ, Eclipse or NetBeans, all excellent, free IDEs that probably beat any of the competitors and, by the way, probably the best examples of Java desktop applications), or using a UI visual builder: SceneBuilder (which can be integrated into IntelliJ) or NetBeans' Visual Debugger.


JavaFX history

JavaFX is not a new technology. It was first released in December 2008 and looked quite different from what it looks like today. The idea was to create a modern UI framework to replace the ageing Swing Framework which had been the official framework in the JVM since the late 90's.

Oracle almost messed up its future from the beginning by creating a special, declarative language that was supposed to be used to create application UIs. That didn't go down well with Java developers and it almost killed JavaFX.

Noticing that, Oracle decided to release JavaFX 2 in 2011 without the special language, but using instead FXML as an option to pure Java code (as we'll see below).

Around 2012, JavaFX gained some traction and Oracle spent a substantial effort trying to improve it and make it become more popular. With version 2.2, JavaFX became a fairly complete platform, but it still was not included in the standard Java runtime (though it was always shipped with the JDK).

Only with JavaFX 8 (the version change was just to match Java 8) did it become part of the standard Java runtime.

Today, JavaFX may not be a very large player in the UI world but it still has a good number of real-world applications, a quite a large number of related libraries, and has even been ported to mobile.


Creating a JavaFX application


For my application, a log viewer I called, un-inspiringly, LogFX, I chose to just use Java (because it's mostly pretty low level code and I wanted to focus on speed and small package size) and IntelliJ for an IDE. I almost went with Kotlin, but IntelliJ's Java support is just so great that having to write Java (or let IntelliJ do it for me would be a better description) was not an issue large enough to justify adding an extra 0.9MB to my packaged application.

I chose not to use FXML (the XML view-descriptor language) or a UI visual builder as the UI was really simple.

So, let's get started by looking at some code!

Java Hello World


A JavaFX application is just a class that extends javafx.application.Application and shows a JavaFX Stage.

Here's a JavaFX Hello World:


public class JavaFxExample extends Application {

@Override
public void start(Stage primaryStage) throws Exception {
Text helloWorld = new Text("Hello world");
StackPane root = new StackPane(helloWorld);
primaryStage.setScene(new Scene(root, 300, 120));
primaryStage.centerOnScreen();
primaryStage.show();
}

public static void main(String[] args) {
launch(JavaFxExample.class, args);
}
}
 src/main/java/main/JavaFxExample.java


On Mac, this will show something like this:


JavaFX Hello World


FXML+Java Hello World


If you have a hard time writing code for UIs and prefer using a markup language, here's the equivalent code + FXML:


 <?xml version="1.0" encoding="UTF-8"?>

<?import javafx.scene.layout.StackPane?>
<?import javafx.scene.Scene?>
<?import javafx.scene.text.Text?>
<Scene xmlns="http://javafx.com/javafx"
width="300.0" height="120.0">
<StackPane>
<Text>Hello world</Text>
</StackPane>
</Scene>
 src/main/resources/main/Example.fxml


public class JavaFxExample extends Application {

@Override
public void start(Stage primaryStage) throws Exception {
Scene scene = FXMLLoader.load(getClass().getResource("Example.fxml"));
primaryStage.setScene(scene);
primaryStage.centerOnScreen();
primaryStage.show();
}

public static void main(String[] args) {
launch(JavaFxExample.class, args);
}
}
 src/main/java/main/JavaFxExample.java


Notice that IntelliJ has support for FXML and will link its contents to the relevant Java code and vice-versa, highlight errors, auto-complete, manage imports, show documentation inline and so on, which is pretty cool... but as I said before, I decided to not use FXML as the UI I had in mind was very simple and quite dynamic.... so I will not show any more FXML. Have a look at the FXML tutorial if you're interested in it.

Kotlin+TornadoFX Hello World


Before we move on, let's see what this would look like with a modern language like Kotlin and its own library for writing JavaFX applications, TornadoFX:


class Main : App() {
override val primaryView = HelloWorld::class
}

class HelloWorld : View() {
override val root = stackpane {
prefWidth = 300.0
prefHeight = 120.0
text("Hello world")
}
}
src/main/kotlin/main/app.kt


Using Kotlin and JavaFX may be attractive to many teams, specially if you like type-safety (TornadoFX has a nice feature called type-safe stylesheets) and if adding up an extra 5MB to your application is not a big concern.

Even if the overhead is too much and you want to avoid including a framework (and having to deal with all its complexity and quirks), Kotlin works just fine with pure JavaFX as well.


Styling and theming JavaFX user interfaces



Now that we've seen how to get started with JavaFX, let's see how JavaFX applications can be styled.

Just as there are different layout approaches, there are different ways to style things in JavaFX.

Let's say we want to make the background dark and the text white as shown below:


Styled JavaFX Hello World


Programmatic and inline styles


One (painful, though type-safe) way is to do it programmatically:


root.setBackground(new Background(new BackgroundFill(
Color.DARKSLATEGRAY, CornerRadii.EMPTY, Insets.EMPTY)));

helloWorld.setStroke(Color.WHITE);


An easier way to do it programmatically is by setting the styles as in CSS:


root.setStyle("-fx-background-color: darkslategray");
helloWorld.setStyle("-fx-stroke: white");


Notice that IntelliJ, again, provides auto-complete support for the String values above.

If you're using FXML:


 <StackPane style="-fx-background-color: darkslategray">
<Text style="-fx-stroke: white">Hello world</Text>
</StackPane>

Same deal...

Using separate stylesheets


If you prefer to stay close to the web world and set styles using a separate stylesheet, JavaFX also allows that! That's the approach I decided to use because that allows me to style everything in a central place and even let users provide a custom stylesheet to change all styles to their own preferences.

To do that, first create a stylesheet:


.root {
-fx-base: darkslategray;
}

Text {
-fx-stroke: white;
}

src/main/resources/css/hello.css


Now, add the stylesheet to the Scene:


primaryStage.getScene().getStylesheets().add("css/hello.css");


And that's it.

Notice that the stylesheet sets not only the StackPane's background color to darkslategray, it changes the base color of the theme.

That means that all controls and "background" items will take a color that's based on this color. That's a pretty neat feature as you can set colors based on the base color, ensuring that if you ever change the base color, most things will look good automatically.

For example, a better stylesheet for our case would not set the text stroke to white, but to an "opposite" color to the theme's base color, so that the text is always readable:


-fx-stroke: ladder(-fx-base, white 49%, black 50%);


JavaFX stylesheets are pretty smart, check the CSS Reference Guide for more information.

Here's an example of a simple application where we replace the Text with a Button, with the default styles on the left, and with the stylesheet we just used above on the right:


JavaFX Button with and without styling
On the left: default JavaFX styles. On the right: using the custom stylesheet introduced above.


In my application, I wanted to have a dark theme by default, but also allow users to provide their own stylesheets so they could use whatever theme they like best.

This is what LogFX ended up looking like with the default theme:


LogFX default theme


Notice that I used FontAwesome's icons in the buttons. It was fairly simple to get the buttons styled with css. Just make sure to install the font first as early as possible with this instruction:


Font.loadFont( LogFX.class.getResource( "/fonts/fontawesome-webfont.ttf" ).toExternalForm(), 12 );


With a custom stylesheet, the look-and-feel can be completely changed. For example, here's an overly green theme in Linux Mint:



LogFX styled green in Linux


Although the good taste of doing this may be questionable, it shows that JavaFX allows for powerful styling, you can achieve pretty much anything your imagination comes up with.

To finalise, I would like to mention the cool effects JavaFX provides... I wanted to make a start screen that looked good with just a nicely formatted text.

JavaFX makes that easy. This is what I came up with (I based this on the GroovyFX example):


LogFX logo


And here's the stylesheet I used to achieve that:


Text {
-fx-fill: white;
}

#logfx-text-log {
-fx-font-family: sans-serif;
-fx-font-weight: 700;
-fx-font-size: 70;
-fx-fill: linear-gradient(to top, cyan, dodgerblue);
}

#logfx-text-fx {
-fx-font-family: sans-serif;
-fx-font-weight: 700;
-fx-font-size: 86;
-fx-fill: linear-gradient(to top, cyan, dodgerblue);
-fx-effect: dropshadow(gaussian, dodgerblue, 15, 0.25, 5, 5);
}


Very nice effects are available. See the tutorial for more details.

In the next sections I will discuss how to change your views, hot reload code and refresh stylesheets all at runtime.


Designing, debugging and hot code reloading


To write user interfaces without being able to instantly see what your changes do is next to impossible. For that reason, hot code reloading or some sort of UI builder is essential in any UI framework.

JavaFX (and the JVM itself) has a few solutions to this problem.

SceneBuilder


The first one is the SceneBuilder, a visual UI builder that lets you create FXML by drag-and-dropping UI components.


SceneBuilder screenshot

It can be integrated into all Java IDEs, making it easy to create new views.

I've used SceneBuilder before to create forms and similar complex views, but I usually just use it to sketch something quickly, then start editing the code by hand to polish it off.

If you do that and later re-open the view in SceneBuilder, it still works fine, so you can alternate how you change the view between manually coding it and designing it with SceneBuilder.

ScenicView


Once you've got the basic design ready, you can use ScenicView to see and edit your scene-graph while the application is running.

You can think of it as the equivalent of the browser's developer view.


ScenicView screenshot


To start it with your application, just download the jar and pass the option -javaagent:/path-to/scenicView.jar to the JVM.

ScenicView lets you change and remove Nodes, track events, and read Javadocs for the selected nodes.


JVM hot code reload


If you want to change the application code that is not directly related to the UI, you can use the Java debugger to hotswap code as your application is running. Basic support for code reloading exists in the Oracle's JVM, HotSpot, and I believe also in the OpenJDK JVM.

However, the built-in code reloading is pretty basic: you can only change the implementation of existing methods.

But there is a HotSpot VM extension called DCEVM (Dynamic Code Evolution VM) that allows you to do a lot more: add/remove methods and fields, add/remove classes, change the value of final variables and more. I wrote about this and other ways to reload code in a running JVM in another post.

I used this to develop LogFX and it worked great. Views that you don't close and re-open are not automatically changed when you reload code, but that's not a problem if you change something that goes into a Stage that can be closed and re-opened... besides, if you want to change the UI component only, you can use the ScenicView or just go back to ScenicBuilder and re-design it the way you want.  

To try DCEVM, you just need to install it and make sure you get the correct version for the version of the JVM you're using. After that, you can run your application with a debugger attached to it and every time you recompile it in the IDE, the code will be automatically reloaded into the running application.

In IntelliJ, you will see something like this when you change a class and re-compile (Cmd+F9 on Mac):


IntelliJ code reload



Refreshing stylesheets



JavaFX does not automatically refresh stylesheets. In LogFX, I wanted to make it possible to change the stylesheets and see immediately the effect of doing so in the application.

As LogFX is a log viewer, it has a pretty advanced FileChangeWatcher that I could use to watch stylesheets and reload them quite easily.

But this only works if the stylesheets come from a file, not from within the application jar itself.

As I already allowed users to specify a custom spreadsheet file to use, that was not a problem for me.

I used this feature during development, and it was really awesome. If you need this feature, you can implement your own file watcher or copy mine (it's open source after all).

To specify a stylesheet as a file (as opposed to a jar resource), you must use different syntax in Unix/Mac and Windows, unfortunately! I used this method to fix that problem:


private static String toAbsoluteFileUri( File file ) {
String absolutePath = file.getAbsolutePath();
if ( File.separatorChar == '\\' ) {
// windows stuff
return "file:///" + absolutePath.replace( "\\", "/" );
} else {
return "file:" + absolutePath;
}
}


This works in Mac, Windows and Linux Mint. But this was one of the only two problems I had related to differences in OS's (the other one was the icon in the system tray on Mac does not work, but there is an ugly workaround for that). JavaFX abstracts that away pretty well, most of the time!

Finally, when you detect a change in a stylesheet file, you can refresh it simply by removing it, then immediately adding it back:


Runnable resetStylesheet = () -> Platform.runLater( () -> {
scene.getStylesheets().clear();
scene.getStylesheets().add( stylesheet );
} );


This works pretty well. But if you don't want to build it yourself, ScenicView can also watch stylesheet in external files (but not inside jars), and TornadoFX also supports that, so you have a few choices there.


Conclusion


Writing an application in JavaFX was a pretty nice experience. I had a brief experience writing a JavaFX application professionally several years ago (when JavaFX was still in its early days, which is not the case anymore), so I certainly had a good head start... but I've also worked as a web developer and I find it hard to believe that anyone would prefer the webstack to working with a sane environment like the JVM.

The application I wrote, LogFX, works very well in my opinion, and it achieves the goals of being very fast, responsive and at the same time look good on all operating systems without changes... please try it yourself and let me know what you think:


curl -sSfL https://jcenter.bintray.com/com/athaydes/logfx/logfx/0.7.0/logfx-0.7.0-all.jar -o logfx.jar


Even though it is a fully functional app, the jar weighs in at just 303 KB. That's 0.3MB, and it includes a couple of pictures, the fontawesome TTF file and some HTML and CSS besides the Java class files!!

Of course, that does not include the JVM itself, but the JVM is not part of the application and can be shared between many apps! With Java 9, you can create native executables with just the JVM parts you need, though, so if your users would not be satisfied with just the jar, you can package it as a native app, as I showed in a previous blog post (a small native JVM app seems to take up around 35MB, or 21MB after stripping it).

To run LogFX you need around 50MB of RAM (not for LogFX itself, nearly all of that is used up by JavaFX itself). You can verify that by starting it with this command:


java -Xmx50m -jar logfx.jar


That's a world of difference from an Electron app, which typically needs 200MB just to open.

JavaFX is not perfect and there's many areas where some improvements are still needed. Distribution and auto-update is one of them. The current solution, JNLP and Java WebStart, seems to be poorly implemented, though there are community alternatives such as Getdown and FxLauncher, or if you want a proper native installer, the commercial solution is Install4J (notice that Install4J has free licenses for open source projects).

There is a lot of things I did not have time to mention in this already rather long blog post about JavaFX that I think you should check out if you're interested:


 

Spock-Reports - great test reports that you actually want to read

posted Sep 26, 2017, 11:34 AM by Renato Athaydes   [ updated Sep 26, 2017, 11:52 AM ]

Spock-Reports introduced in version 1.3.0 the possibility to embed source code in reports, a feature we call vivid reports.

This may sound like a crazy idea if you are not used to reading and writing Spock specifications! But trust me, the reports created when vivid report is enabled look awesome!

Spock specifications (or tests, if you prefer) tend to be highly readable. Many people have said that the code is normally clear enough that they'd prefer to skip the description block you would see in traditional Spock specifications and just show the code in reports. That's how vivid reports were born.

Just as importantly, Spock error messages (thanks mostly to Groovy power-assertions) are as good as they get.


Simple Spock Report
  A Spock Report for a failed test


When combined, awesome error messages and embedding the source code make these reports extremely useful (unlike reports from most other testing frameworks)!

For example, take this little test that is used to test spock-reports itself:


@Unroll
def "An @Unrolled spec with x=#x and y=#y"() {
setup:
"nothing"
expect:
x == 0
and:
"An error if y is 5"
if ( y == 5 ) {
throw new RuntimeException( 'y is 5' )
}

where:
x | y
0 | 1
2 | 3
0 | 5
}


It is obviously going to fail in the second (because x != 0) and third (because y == 5) iterations.

Here's what the report for the failed iterations look like:


Notice that when a test is Unrolled, spock-reports shows each iteration as a separate feature, unlike on the first report shown at the top of this blog post, where the test is example-based but not unrolled.


Older Vivid Report
 Example of a failed vivid report


This is what the reports look like in version 1.3.0 and 1.3.1.

In version 1.3.2, which is just about to come out, we take things even further and highlight the line of code that caused the test to fail!

Below, you can the same section shown in the report above, but with the line that caused the test to fail highlighted:



New Vivid Report
 Vivid report with line of code that caused the test to fail highlighted


This is really nice. Notice that even the line number is shown as a comment at the end of the guilty line.


Real-world project example


To give a more realistic example, let's look at a Specification for another side project of mine, LogFX. As a log viewer, LogFX needs to watch files for changes, and the FileChangeWatcherSpec tests the code that does it.

Below, we see what the default spock-report for the first feature test looks like:


Realistic standard spock-report
Real-world project report snippet example


This is good and nice for non-coders... but from experience, most actual consumers of this information are technical people, even if not only programmers... most of them would appreciate the further details provided by source code embedded in the report, as shown below (top summary omitted for brevity):



Real-world project report with source code
Real-world project report snippet with source code example


A lot more information is conveyed in this report, but it still looks like a report (as opposed to just printed code, for example)... and in my modest opinion, this is much more useful.

It allows teams to see what's tested, discuss not only the test plan but the implementation... as a side-effect, it might remind the developers writing the test to try to make the tests just a bit more readable as well, as this increases the visibility of the test code. Finally, when the test fails, you get right to the heart of the information you need:



Failure Real-world report with source code
Failed test real-world project report with source code


Even the stack-trace could be displayed if desired by setting the com.athaydes.spockframework.report.internal.HtmlReportCreator.printThrowableStackTrace property to true.

This failed test report is quite useful! The cause of the test failure is much easier to find if you can see exactly how the test is implemented. The fact that Groovy code can be quite compact and easy to read helps a lot... knowing exactly where the error happened without digging through build logs and loads of system output is really valuable as well.


Enabling vivid Spock reports in your tests


First of all, assuming you already use Spock, to start generating reports with spock-reports, just add a test dependency on spock-reports as shown below.

If you're not using Spock yet, head to the Spock documentation and do it now! You won't regret it.


Exclude transitive dependencies to avoid affecting your version of Spock and Groovy.
Notice that spock-reports 1.3.2 requires Spock version 1.1 at least.

Maven

<dependency>
    <groupId>com.athaydes</groupId>
    <artifactId>spock-reports</artifactId>
    <version>1.3.2</version>
    <scope>test</scope>
    <exclusions>
        <exclusion>
            <groupId>*</groupId>
            <artifactId>*</artifactId>
        </exclusion>
    </exclusions>
</dependency>


Gradle

testCompile 'com.athaydes:spock-reports:1.3.2', { transitive = false } 


After you run your tests now, you should find your reports under ${project.buildDir}/spock-reports/. The main report is at index.html.

As vivid reports are not enabled by default, to enable it, set the com.athaydes.spockframework.report.showCodeBlocks system property to true.

In Gradle, that's done by adding these lines to your build file:


test {
    systemProperty 'com.athaydes.spockframework.report.showCodeBlocks', true
}


You can also do that via a config file if you prefer, which allows you to configure just about anything about the reports. See the spock-reports README page for more information.




Faster command line tools with Kotlin

posted Aug 6, 2017, 5:35 AM by Renato Athaydes   [ updated Aug 6, 2017, 5:37 AM ]

Inspired by Faster command line tools with Haskell 
    which itself was inspired by Faster command line tools with Go
        (which inspired Faster command line tools with Nim)
        which itself was inspired by Faster Command Line Tools in D.

In this article, I will show how Kotlin (when run on the JVM, in this case) can be used to create fast command-line tools that run at similar speed to the fastest native languages.

Even though that is the main goal, I will also go beyond the basics to show how programs like this can be optimised to a high degree with the help of basic JVM tools and some intuition.

To recap from the original D blog post:

The original problem statement:

It’s a common programming task: Take a data file with fields separated by a delimiter (comma, tab, etc), and run a mathematical calculation involving several of the fields. Often these programs are one-time use scripts, other times they have longer shelf life. Speed is of course appreciated when the program is used more than a few times on large files.

The specific exercise we’ll explore starts with files having keys in one field, integer values in another. … Fields are delimited by a TAB, and there may be any number of fields on a line. The file name and field numbers of the key and value are passed as command line arguments.



From previous benchmarks I did on Kotlin, I know Kotlin is a pretty fast language... at least it is in the same league as Java and Go, which is a great thing!

Note: when I talk about Kotlin performance, I am talking about running Kotlin with the Oracle JVM, HotSpot. In particular, I've used Java version 8u141, Java HotSpot(TM) 64-Bit Server VM (build 25.141-b15, mixed mode).

Kotlin version 1.1.3-2 compiled for Java 8 (kotlin-stdlib-jre8)

However, the JVM is not known for good performance for small scripts like the one used in this experiment due to the slow startup time and the fact that it may not even have time to warm up properly (i.e. the JIT compiler does not have time to compile efficient machine code, so the JVM ends up just interpreting bytecode most of the time).

Anyway, let's get right into it and see how Kotlin and the JVM perform!


V1 - Initial Version


The first version of the code looks almost exactly the same as the original D code (allowing for syntax differences):


val delim = '\t'

private fun run(file: File, keyIndex: Int, valueIndex: Int) {
val maxFieldIndex = maxOf(keyIndex, valueIndex)
val sumByKey = mutableMapOf<String, Int>()

file.forEachLine { line ->
val fields = line.split(delim)

if (maxFieldIndex < fields.size) {
sumByKey.merge(fields[keyIndex], fields[valueIndex].toInt(), Int::plus)
}
}

if (sumByKey.isEmpty()) {
println("No entries")
} else {
val (key, maxEntry) = sumByKey.maxBy { it.value }!!
println("max_key: $key, sum: $maxEntry")
}
}



The boilerplate of parsing arguments and handling errors is omitted for brevity, but you can see it here.

To run the program, we type:

java -cp "build/libs/*" Max_column_sum_by_keyKt ngrams.tsv 1 2


Here's the result:


max_key: 2006, sum: 22569013
Total time: 3.37 sec


Note: all times reported in this article are self-reported by the Kotlin code, similar to what was done in the Go and Haskell articles.
Using the shell's time command to measure the time adds at most 0.1 seconds to all reported times. 
In any case, comparing the times obtained by different languages using different hardware
should be seen only as a means to get an idea of whether speeds are around the same order of magnitude.


Given the first D version ran in 3.2 seconds (and Go, in 3.49s), Kotlin is not looking too bad!

However, we can definitely improve on this!

Let's see what the poor man's profiler on the JVM, jvisualvm (with the Startup Profiler Plugin), has to say about our program so far:


profiler results 1


First of all, Kotlin seems to be introducing a lot of runtime null checks! We probably don't need that, so it would be nice to find out where that's being called from!


It is possible to disable null checks at runtime, but I won't do that as the options to do so are currently unofficial.


Using the Visual VM Sampler to inspect the call stack, the cause becomes obvious:


call stack



V2 - Java String::split


Kotlin's extension function CharSequence::split, which we use to split each line, seems to be guilty. Even though this function is much more powerful than Java's, all we need here is the simpler Java version, so let's replace the line:


val fields = line.split(delim)


With this:


@Suppress("PLATFORM_CLASS_MAPPED_TO_KOTLIN")
val fields = (line as java.lang.String).split(delim)


Because Java's split takes a String rather than a char, the type of delim had to be changed as well. See the full code here.

Result:


max_key: 2006, sum: 22569013
Total time: 2.73 sec


Nice! That's over 0.6 seconds off the first version with a one-line change!

And this dramatically changes the profiling characteristics of our application:


v2 profiling


The bottleneck now seems to be related to simply iterating over the file's lines.

Other than doing something like the Go guy did and iterate over the file bytes directly rather than the line Strings to solve that, we could try a few simpler things first that might help...

V3 - Wrap Integers into IntBox


Let's see if we can avoid any Integer boxing by implementing a int wrapper, thus avoiding lots of boxing/unboxing values to/from the Map with the results.

Here's the new code with just some simple modifications based on the introduction of the IntBox data class:


data class IntBox(var int: Int = 0) {
operator fun plus(other: Int) {
int += other
}
}

const val delim = "\t"

private fun run(file: File, keyIndex: Int, valueIndex: Int) {
val maxFieldIndex = maxOf(keyIndex, valueIndex)
val sumByKey = mutableMapOf<String, IntBox>()

file.forEachLine { line ->
@Suppress("PLATFORM_CLASS_MAPPED_TO_KOTLIN")
val fields = (line as java.lang.String).split(delim)

if (maxFieldIndex < fields.size) {
sumByKey.getOrPut(fields[keyIndex]) { IntBox() } + fields[valueIndex].toInt()
}
}

if (sumByKey.isEmpty()) {
println("No entries")
} else {
val (key, maxEntry) = sumByKey.maxBy { it.value.int }!!
println("max_key: $key, sum: ${maxEntry.int}")
}
}



As the box allows the code to always perform operations on primitive int's (new values are not put into the Map, the current box's value is just changed), this could run faster, in theory.

The result:


max_key: 2006, sum: 22569013
Total time: 2.52 sec


Hm.. not a huge difference, just a couple of tenths of a second (but faster). Running this a few times, the time does not vary significantly...

One thing we can do now to get to the bottom of this, is to include java.* classes in the profiler configuration (by default, they are excluded) and see exactly where time is being spent:


Profiler data with Java classes


Unfortunately, looks like now we're down to the lowest level Java methods taking up most of the time: copying array ranges (called by the split method), the String constructor itself (there's a lot of Strings being created!), String::indexOf (also from split)...

V4 - Faster Map


I had thought of trying to use a faster Map implementation, but the profiler's information does not seem to back up the assumption that doing that would help much!

But I am stubborn, and decided to try anyway... first, by explicitly instantiating a java.util.HashMap instead of using Kotlin's mutableMapOf function (which gives a LinkedHashMap).

Then, by using a heavyweight library called fastutil, which contains lots of fast implementations of primitive collections for Java (which we can use freely from Kotlin, of course)... I decided to try Object2IntOpenHashMap. It even has an appropriate method with signature

    int addTo(final K k, final int incr)

which I thought would give us at least a modest boost in speed (notice that this method allowed me to get rid of IntBox).

But, as usual with performance guesses, I was wrong!

Here's the result using HashMap:


max_key: 2006, sum: 22569013
Total time: 2.39 sec


And using Object2IntOpenHashMap:


max_key: 2006, sum: 22569013
Total time: 2.58 sec


Well, it turns out fastutil was not too fast on this problem (the library seems to optimise for large collections, not only speed)... but at least we're gonna take the tiny improvement of using HashMap, which brings the total time down another 100ms from the previous result to a respectable 2.39 seconds for a non-native, VM-based language (the only one in the articles I've read so far besides the mention of Python in the D article... it's worth mentioning that Kotlin Native is being developed, so Kotlin may not always rely on the JVM for best performance).

And the code so far remains extremely clean.

But we're still well behind D and Nim, and not quite better than Haskell yet.

V5 - Process bytes, not Strings


To make things even faster, it seems we'll need to bite the bullet and get rid of all those byte-to-String + String::split operations. They are not strictly necessary anyway, as most of the work can be done on the byte level, as shown in the Go article.

But how less readable and maintainable will the code become?!

Judge it by yourself:


const val delim = '\t'

private fun run(file: File, keyIndex: Int, valueIndex: Int) {
val maxFieldIndex = maxOf(keyIndex, valueIndex)
val sumByKey = HashMap<String, IntBox>()

val inputStream = file.inputStream()

val buffer = ByteArray(1024 * 1_000)
var fieldIndex = 0
val currentKey = StringBuilder(12)
val currentVal = StringBuilder(12)

fun startLine() {
if (currentVal.isNotEmpty()) {
sumByKey.getOrPut(currentKey.toString()) { IntBox() } + currentVal.toString().toInt()
}

fieldIndex = 0
currentKey.setLength(0)
currentVal.setLength(0)
}

inputStream.use {
while (true) {
val bytesCount = inputStream.read(buffer)

if (bytesCount < 0) {
break
}

(0 until bytesCount).forEach { i ->
val char = buffer[i].toChar()

if (fieldIndex <= maxFieldIndex) {
when (char) {
delim -> {
fieldIndex++
}
'\n' -> {
startLine()
}
else -> {
if (fieldIndex == keyIndex) {
currentKey.append(char)
} else if (fieldIndex == valueIndex) {
currentVal.append(char)
}
}
}
} else if (char == '\n') {
startLine()
}
}
}
}

if (sumByKey.isEmpty()) {
println("No entries")
} else {
val (key, maxEntry) = sumByKey.maxBy { it.value.int }!!
println("max_key: $key, sum: ${maxEntry.int}")
}
}


Result:


max_key: 2006, sum: 22569013
Total time: 1.39 sec


Really good speed up!

V6 - ByteArray conversion straight into Int


But notice that we're still wastefully converting every value into a String before parsing it as an int, something that can be avoided without much effort...

The code doesn't change too much from the last one shown above, so you can just click here to see the diff if you're interested!

Here's just the ByteArray-to-Int function used:


private fun ByteArray.toIntUpTo(maxIndex: Int): Int {
var result = 0
var multiplier = 1
((maxIndex - 1) downTo 0).forEach { index ->
val digit = this[index].toInt() - 48
result += digit * multiplier
multiplier *= 10
}
return result
}


And here's the result after doing that:


max_key: 2006, sum: 22569013
Total time: 1.18 sec


This is pretty close to the best D version!

But we still have one card up our sleeves with Kotlin!

V7 - Parallelise


Given that the current fastest implementation we have can be easily parallelised, there's no reason to not try doing that.

All we need is a Java's RandomAccessFile to be able to read different parts of the file concurrently, and some setup code to break the file up into partitions.

Here's the setup code:


private fun run(file: File, keyIndex: Int, valueIndex: Int) {
val maxFieldIndex = maxOf(keyIndex, valueIndex)

val partition1 = RandomAccessFile(file, "r")
val partition2 = RandomAccessFile(file, "r")

val fileLength = file.length()
var secondPartitionStartIndex = fileLength / 2L

partition2.seek(secondPartitionStartIndex)

while (true) {
val byte = partition2.read()
secondPartitionStartIndex++
if (byte < 0 || byte.toChar() == '\n') {
break
}
}

val firstPartitionResult = CompletableFuture.supplyAsync {
maxOfPartition(partition1, keyIndex, valueIndex, maxFieldIndex, secondPartitionStartIndex.toInt())
}

val secondPartitionResult = CompletableFuture.supplyAsync {
maxOfPartition(partition2, keyIndex, valueIndex, maxFieldIndex, fileLength.toInt())
}

val sumByKey = firstPartitionResult.join().toMutableMap()
.mergeWith(secondPartitionResult.join())

val maxEntry = sumByKey.maxBy { it.value.int }

if (maxEntry == null) {
println("No entries")
} else {
val (key, value) = maxEntry
println("max_key: $key, sum: ${value.int}")
}
}



Most of the code that was in the run function in the previous examples was moved to a new function called maxOfPartition, which only scans a partition of the file and returns the sumByKey Map only for its partition. After computing the maps for each partition in parallel, we merge them together to get the final result.

Why not use Kotlin coroutines?

Why did I not use Kotlin coroutines for this?

Kotlin coroutines are still experimental, but they are expected to become one of the bigger and coolest features of the language.

However, when I tried to write this code using coroutines, I ran into a compiler bug which made it impossible to use them, unfortunately.

If and when JetBrains addresses this bug, I might update this blog post with the code using coroutines, but for now, CompletableFuture it is.



Here is the result when reading two separate partitions of the file in parallel:


max_key: 2006, sum: 22569013
Total time: 0.71 sec


Even though I got some variance between different runs, the above result seems to be around the average for this code (slower runs ended in around 0.9 seconds, but the fastest runs were down at 0.67 seconds, with most runs close to the latter than the former).

So, even though we had to employ some pretty advanced stuff compared to the simple v1, we were able to increase the speed from over 3 seconds to well under 1 second. And, remember, we're processing a 192.4MB file.

So, I hope this shows that writing Kotlin command-line application that run fast is possible... and yes, you can get speeds similar to Go and D.


Bonus solution

Bonus solution: MappedByteBuffer

As a bonus solution, I decided to map the file bytes into a MappedByteBuffer, which makes reading files extremely fast!

I just replaced RandomAccessFile with MappedByteBuffer, making the necessary changes to compile, but otherwise keeping the last version of the code unchanged (i.e. two buffers running in parallel).

This is not a fair solution as it maps the whole file into memory rather than consuming small chunks of it at a time like the previous solutions did (which allows reading files of any size, not just whatever fits into memory).

In any case, if you don't need to worry about files that won't fit into memory, this would be the fastest thing to do.

Here's the result:

max_key: 2006, sum: 22569013
Total time: 0.58 sec


And that's our absolute best time!



Please leave your comments on Reddit.


Summary of results:

V1 - 3.37s

V2 - 2.73s (java.lang.String::split)

V3 - 2.52s (IntBox)

V4 - 2.39s (java.util.HashMap)

V5 - 1.39s (process bytes, not Strings)

V6 - 1.18s (ByteArray -> Int)

V7 - 0.71s (parallelise)

(Bonus version) - 0.58s (MappedByteBuffer)


The (gzipped) file used as an input can be downloaded from here:


Repository containing all the code shown in this blog post:






Kotlin's hidden costs - Benchmarks

posted Jun 25, 2017, 6:56 AM by Renato Athaydes   [ updated Oct 21, 2017, 10:27 AM ]

This article has been translated into Japanese!


A series of blog posts called Exploring Kotlin’s hidden costs , written by @BladeCoder, demonstrated how certain Kotlin constructs have a hidden cost.


The actual hidden cost is normally due to the instantiation of an invisible Object or the boxing/unboxing of primitive values. These costs are specially hard to see for a developer who doesn't understand how the Kotlin compiler translates such constructs to JVM bytecode.

However, just talking about hidden costs without putting some numbers on said costs makes one wonder how much they should actually worry about them. Should these costs be taken into consideration everywhere in the codebase, meaning that some Kotlin constructs should just be forbidden outright? Or should these costs only be taken into consideration in the tightest inner loops?

Even more provocatively, do these so-called costs actually result in performance penalties (given how the JVM actively optimises code at runtime, compiling it to efficient machine code based on actual usage, the answer to this question may not be as clear as it seems)?

Without putting numbers to the hidden costs, that's impossible to answer.

For that reason, I decided to write JMH benchmarks to try to quantify the actual costs of each Kotlin construct mentioned in all the 3 parts of that blog post series published so far.

Methodology and system


Some of the Kotlin constructs' costs mentioned in the blog posts can be directly compared to the equivalent Java construct. For example, the cost of Kotlin lambdas can be directly compared to the cost of Java lambdas. However, many Kotlin constructs don't have a Java equivalent, in which case instead of comparing the Kotlin construct with the equivalent Java version of it, I compare them with the author's suggestions to improve on the costly constructs.

The code is on GitHub, so anyone can run it in their own system to see if the numbers match (it would be very interesting to collect some results in the Reddit's comments) in different systems.

If you do run them, beware the full benchmark takes a few hours to run.

All results were collected using the following system:

Macbook Pro (2,5 GHz Intel Core i7, 16GB of RAM)

Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

The Kotlin version (1.1.3) and JMH (0.5.6) are the latest as of writing (see the pom.xml).

Update: Android ART benchmarks are available in a WillowTreeApps.com blog post!


Part 1

https://medium.com/@BladeCoder/exploring-kotlins-hidden-costs-part-1-fbb9935d9b62


Higher-order functions and Lambda expressions


In this first example, it was not very clear what cost the author was talking about. It appears that he is just referring to the cost of using Kotlin lambdas, given that Kotlin, by default, targets the Java 6 VM which doesn't have lambdas.

However, the advice given later in the post is specific to capturing lambdas, not just any lambdas, and specifically, not the one given in the example:


fun transaction(db: Database, body: (Database) -> Int): Int {
db.beginTransaction()
try {
val result = body(db)
db.setTransactionSuccessful()
return result
} finally {
db.endTransaction()
}
}


Which is used with the following syntax:

    
val deletedRows = transaction(db) {
it.delete("Customers", null, null)
}

The hidden cost here appears to be only the fact that a Function Object might be created when the transaction function is invoked above. But as the author himself notices, that's not the case for this particular function because it is not a capturing lambda, so a singleton Function instance is created and used on every invocation.

The only remaining cost mentioned in the post is not a runtime cost: the extra 3 or 4 methods created by the Function class generated by the Kotlin compiler.

Anyway, I decided to check whether there's a real runtime cost related to Kotlin lambdas when compared to Java 8's lambdas because I was left with the impression that such cost should exist from the description of the problem (the cost of using a capturing lambda, which the actor advises against in the advice given in this part of the post, will be benchmarked in Part 2, so read on).

The equivalent function was implemented by me using Java 8:


public static int transaction( Database db, ToIntFunction<Database> body ) {
db.beginTransaction();
try {
int result = body.applyAsInt( db );
db.setTransactionSuccessful();
return result;
} finally {
db.endTransaction();
}
}


The syntax for calling this function in Java 8 is just slightly different from Kotlin:


 int deletedRows = transaction( db, ( database ) ->
database.delete( "Customer", null, null ) );

And here is the results of the benchmark comparing the Kotlin version with the Java 8 version:

RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part1.KotlinBenchmarkPart1.javaLambda                                          thrpt       200  1024302.409     1851.789   ops/ms

c.a.k.part1.KotlinBenchmarkPart1.kotlinLambda                                        thrpt       200  1362991.121     2824.862   ops/ms






In the above chart, higher is better (more ops/ms).

Notice that this example should also show the overhead of using a Kotlin lambda which returns an Integer VS the specialized version (ToIntFunction) used in the Java 8 example.

However, the Kotlin lambda seems to be significantly faster than the Java lambda. So, the cost here seems to be negative, as Kotlin actually ran around 30% faster than Java! The mean error is a little bit larger for Kotlin than for Java, but that' not even visible in the above chart, which does attempt to show the error bar (but it's just too small to be seen).

Well, in any case, the solution offered by the author to make the Kotlin lambda less costly (even though it doesn't seem like you should worry about that) is to inline the function, which can be done by simply declaring the transaction function with the inline keyword.

When that's done, we get the following result:

RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part1.KotlinBenchmarkPart1.kotlinInlinedFunction                               thrpt       200  1344885.445     2632.587   ops/ms





As you can see, using an inline function in this example did not significantly improve the performance of the Kotlin lambda at all. If anything, it got just a little bit worse.

I have absolutely no idea why these results are the complete opposite of what the author and I, admittedly, expected. Looking at the benchmark code, I can't see anything that might be obviously wrong, so I am carefully confident that these figures are real.


UPDATE: See this discussion for possible reasons for the surprise results.


Companion Objects


As the author shows, companion objects seem to imply an overhead due to the generated synthetic getters and setters for access to class properties. In the worst case scenario, the first getter might need to even call a second getter, an instance method of the companion object, in order to get a simple constant value.

I decided to combine the companion object examples shown in the blog post to try to measure the cost of the worst case scenario using the following Kotlin class:


class MyClass private constructor() {

companion object {
private val TAG = "TAG"

fun newInstance() = MyClass()
}

fun helloWorld() = TAG
}

Here's the benchmarked Kotlin function:


fun runCompanionObjectCallToPrivateConstructor(): String {
val myClass = MyClass.newInstance()
return myClass.helloWorld()
}


The cost of the above Kotlin code is being compared against the equivalent, straight-forward Java implementation using just a simple static final String within the class itself:


class MyJavaClass {

private static final String TAG = "TAG";

private MyJavaClass() {
}

public static String helloWorld() {
return TAG;
}

public static MyJavaClass newInstance() {
return new MyJavaClass();
}
}

The Java method used was this one:


public static String runPrivateConstructorFromStaticMethod() {
MyJavaClass myJavaClass = newInstance();
return myJavaClass.helloWorld();
}

RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part1.KotlinBenchmarkPart1.javaPrivateConstructorCallFromStaticMethod          thrpt       200   398709.154      800.190   ops/ms

c.a.k.part1.KotlinBenchmarkPart1.kotlinPrivateConstructorCallFromCompanionObject     thrpt       200   404746.375      621.591   ops/ms





Again, Kotlin seems to have a better performance than Java, if only by a tiny margin this time.

Part 2


Local Functions


In this part of the blog post, the author theorizes about the hidden costs of Kotlin's local functions. The only cost, it seems, is the creation of a Function object for capturing functions, but not for functions that do not capture anything from its context.

To test that, we start with a Java local function, or lambda, that avoids boxing but captures one variable from the context:


public static int someMath( int a ) {
IntUnaryOperator sumSquare = ( int b ) -> ( a + b ) * ( a + b );

return sumSquare.applyAsInt( 1 ) + sumSquare.applyAsInt( 2 );
}

The exact same function is implemented in Kotlin as an example given in the blog post:


fun someMath(a: Int): Int {
fun sumSquare(b: Int) = (a + b) * (a + b)

return sumSquare(1) + sumSquare(2)
}


A second Kotlin version which avoids capturing anything from its context is also tried:


fun someMath2(a: Int): Int {
fun sumSquare(a: Int, b: Int) = (a + b) * (a + b)

return sumSquare(a, 1) + sumSquare(a, 2)
}


RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part2.KotlinBenchmarkPart2.javaLocalFunction                                   thrpt       200   897015.956     1951.104   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.kotlinLocalFunctionCapturingLocalVariable           thrpt       200   909087.356     1690.368   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.kotlinLocalFunctionWithoutCapturingLocalVariable    thrpt       200   908852.870     1822.557   ops/ms




By now, it should be no surprise! Kotlin again seems to beat Java by a tiny margin. And that means, once again, if there's any cost to Kotlin, that cost is negative!


Null safety


The Kotlin compiler adds a null-check on each public function's non-null parameter. But how much does that cost?

Let's see.

Here's the tested Kotlin function:


fun sayHello(who: String, blackHole: BlackHole) = blackHole.consume("Hello $who")


Blackhole is a JMH class that can be used to consume values during benchmarks, ensuring that the compiler cannot just avoid computing the value, making the benchmark meaningless.


And the Java baseline:


public static void sayHello( String who, BlackHole blackHole ) {
blackHole.consume( "Hello " + who );
}


RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part2.KotlinBenchmarkPart2.javaSayHello                                        thrpt       200    73353.725      155.551   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.kotlinSayHello                                      thrpt       200    75637.556      162.963   ops/ms






Again, the cost of using Kotlin is negative, or in other words, using Kotlin seems to help with performance over using Java, going against our predictions based on differences in the bytecode.

Note: I skipped benchmarking the Nullable Primitive Types Part because that is not, in my opinion, a hidden cost of Kotlin as compared to Java, as in Java nullable primitives would also incur the exact same boxing costs as in Kotlin.

Varargs


The cost of using varargs for method parameters, as the author points out, occurs only when you need to use the spread operator to use an existing array as a method argument, something that is not necessary in Java.

So, to test the overhead, we compare the following Java method call:


public static void runPrintDouble( BlackHole blackHole, int[] values ) {
printDouble( blackHole, values );
}

public static void printDouble( BlackHole blackHole, int... values ) {
for (int value : values) {
blackHole.consume( value );
}
}


With the equivalent Kotlin implementation:


fun runPrintDouble(blackHole: BlackHole, values: IntArray) {
printDouble(blackHole, *values)
}

fun printDouble(blackHole: BlackHole, vararg values: Int) {
for (value in values) {
blackHole.consume(value)
}
}


RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part2.KotlinBenchmarkPart2.javaIntVarargs                                      thrpt       200   173265.270      260.837   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.kotlinIntVarargs                                    thrpt       200    83621.509      990.854   ops/ms





Finally, a Kotlin hidden cost which you should definitely avoid! Using Kotlin's spread operator, which causes a full copy of the array to be created before calling a method, has a very high performance penalty (and that might increase with the size of the array). In our case, the Java version ran 200% faster than the seemingly equivalent Kotlin version.


Note: The Passing a Mix of Arrays and Arguments Part is also skipped because there's just no equivalent in Java to compare with.

Part 3


Delegated Properties

To measure the actual cost of using delegated properties in Kotlin, I decided to use the most efficient possible equivalent implementation in Java as a baseline. This may not be totally fair to Kotlin as the two things are not the same, and delegated properties enable using patterns that are just not possible in Java.

However, with this in mind, I think it is useful to know just what kind of cost this really entails, even if compared to a Java version that is specifically written for this one case.

The Java baseline uses the following, trivial classes:


class DelegatePropertyTest {

public static String stringValue = "hello";

public static String someOperation() {
return stringValue;
}

}

class Example2 {
public String p;

public void initialize() {
p = DelegatePropertyTest.someOperation();
}
}


As you can see, the caller must remember to call initialize on Example2 in order to initialize the p property!


public static void runStringDelegateExample( BlackHole blackHole ) {
Example2 example2 = new Example2();
example2.initialize();
blackHole.consume( example2.p );
blackHole.consume( example2.p );
}


The Kotlin code uses a delegate class for initializing the p property:


class StringDelegate {
private var cache: String? = null

operator fun getValue(thisRef: Any?, property: KProperty<*>): String {
var result = cache
if (result == null) {
result = someOperation()
cache = result
}
return result!!
}

operator fun setValue(thisRef: Any?, property: KProperty<*>, value: String) {
cache = value
}
}



class Example {
var p: String by StringDelegate()
}


And the Kotlin test function does roughly the same thing as the Java test function, except it doesn't need to explicitly initialize the Example class property:


fun runStringDelegateExample(blackHole: BlackHole) {
val example = Example()
blackHole.consume(example.p)
blackHole.consume(example.p)
}


RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part3.KotlinBenchmarkPart3.javaSimplyInitializedProperty                       thrpt       200   274394.088      554.171   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinDelegateProperty                              thrpt       200   255899.824      910.112   ops/ms





Here, we see that there's a small cost, in the order of 10%, associated with using Kotlin delegated properties when compared to manual Java property initialization.

Note: We skip Generic Delegates because, again, the cost they might incur is related to boxing/unboxing primitive types, not the feature itself.
We also skip the Lazy Delegate Part as that's not a hidden cost, just an informational section regarding how to correctly specify lazy delegates synchronization properties.


Ranges (indirect reference)


To find out the cost of using ranges, as Java does not currently have the equivalent concept at all, we compare the performance of the suggested solutions to the posed performance problems in most examples.

To start with, we compare the cost of using a range with at least on indirection, with just using a range directly where it's needed.

The code that uses indirection involves getting the range from a calling a getter:


private val myRange get() = 1..10

fun isInOneToTenWithIndirectRange(i: Int) = i in myRange


As opposed to using a range directly:


fun isInOneToTenWithLocalRange(i: Int) = i in 1..10


RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part3.KotlinBenchmarkPart3.kotlinIndirectRange                                 thrpt       200  1214464.562     2071.128   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinLocallyDeclaredRange                          thrpt       200  1214883.411     1797.921   ops/ms





Even though there seems to be a tiny cost associated with using indirect references to ranges, the cost is not significant.


Ranges (non-primitive types)


Another range cost found by the author is that, when Ranges are used with non-primitive types, a new ClosedRange instance is created even for locally declared ranges, as in this example:


fun isBetweenNamesWithLocalRange(name: String): Boolean {
return name in "Alfred".."Alicia"
}


Hence, the above should be more expensive than this:


private val NAMES = "Alfred".."Alicia"

fun isBetweenNamesWithConstantRange(name: String): Boolean {
return name in NAMES
}


RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part3.KotlinBenchmarkPart3.kotlinStringRangeInclusionWithLocalRange            thrpt       200   211468.439      483.879   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinStringRangeInclusionWithConstantRange         thrpt       200   218073.886      412.408   ops/ms





It turns out that, yes, it is better to use constant, non-primitive ranges, than locally declared ones, if you need the absolute best performance possible.

The cost of using local ranges instead of a constant range is around 3%, so this one is not something to worry too much about, though.

Ranges (iteration)


One more potential issue with ranges arises when we iterate over them.

Iterating over a primitive range should have zero overhead:


fun rangeForEachLoop(blackHole: BlackHole) {
for (it in 1..10) {
blackHole.consume(it)
}
}


However, iterating using the forEach method should have an overhead, according to the blog post:


fun rangeForEachMethod(blackHole: BlackHole) {
(1..10).forEach {
blackHole.consume(it)
}
}


As should iterating over a range created using a step:


fun rangeForEachLoopWithStep1(blackHole: BlackHole) {
for (it in 1..10 step 1) {
blackHole.consume(it)
}
}


RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part3.KotlinBenchmarkPart3.kotlinRangeForEachFunction                          thrpt       200   108382.188      561.632   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinRangeForEachLoop                              thrpt       200   331558.172      494.281   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinRangeForEachLoopWithStep1                     thrpt       200   331250.339      545.200   ops/ms




The above graph shows that using the forEach function for a Range, as the author predicted, should absolutely be avoided. It performs 300% slower than a simple for-loop!

On the other hand, using an explicit step does not seem to impact the performance of a range for-loop, contradicting the advice in the blog post.

Iterations: Collection indices


Finally, let's measure the cost of using indices on custom classes that are not optimised by the compiler.

For this example, we create a mock version of SparseArray:


class SparseArray<out T>(val collection: List<T>) {
fun size() = collection.size
fun valueAt(index: Int) = collection[index]
}


As suggested by the author, we extend it with a custom indices property:


inline val SparseArray<*>.indices: IntRange
get() = 0..size() - 1


Now, we iterate over the indices:


fun printValuesUsingIndices(map: SparseArray<String>, blackHole: BlackHole) {
for (i in map.indices) {
blackHole.consume(map.valueAt(i))
}
}


The better solution, according to the author, is to use lastIndex instead:


inline val SparseArray<*>.lastIndex: Int
get() = size() - 1



fun printValuesUsingLastIndexRange(map: SparseArray<String>, blackHole: BlackHole) {
for (i in 0..map.lastIndex) {
blackHole.consume(map.valueAt(i))
}
}


RESULT

Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part3.KotlinBenchmarkPart3.kotlinCustomIndicesIteration                        thrpt       200    79096.631      134.813   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinIterationUsingLastIndexRange                  thrpt       200    80811.554      122.462   ops/ms





Even though it might be slightly safer to use a range from 0 to lastIndex to iterate over a custom collection, the impact of using indices is really small, it seem to only run around 2% slower.

Conclusion


So, which features should you use without concern, and which ones should you avoid?!

The ones with a green tick mark below are the features that can, according to this benchmark, be used without concern (cost below 5%).

The red ones should be avoided if possible, unless performance is just a secondary concern.

In any case, I hope this analysis has demonstrated how, when it comes to performance, the only thing you can be sure about is that without measuring, you know nothing.




pass      Higher-order functions and Lambda expressions









No evidence was found to suggest that Kotlin lambdas and higher-order functions should be avoided. To the contrary, they seem to run faster than Java 8 lambdas.


pass      Companion Objects









Companion Objects have no significant performance costs that could be measured, so there's no reason to avoid them from a performance perspective.


pass      Local Functions








Local Functions, capturing or not, do not seem to affect the performance of Kotlin code.


pass
      Null safety









Kotlin null-safety checks appear to have a negligible performance impact that can be safely ignored.


fail       Varargs + Spread Operator

Kotlin varargs, when used with the spread operator, have a high performance cost due to an extra, unnecessary array copy. Avoid it if performance is a concern.


fail       Delegate Properties

Avoid delegated properties on performance-critical code. Even though the overhead is quite small, around 10%, this may be unacceptable in certain circumstances.


pass
      Indirect access to Range










No performance impact was observed from accessing Ranges indirectly.


pass
      Ranges (local, non-primitive types)










The cost of using local Ranges of non-primitive types is almost insignificant (measured to be around 3%), so only avoid in the most extreme circumstances where performance is the main concern.



fail
       Ranges (forEach function)

Absolutely avoid calling forEach on Ranges. The cost is extremely high at around 300%. Hopefully, the Kotlin team will be able to address this problem in time, but for now, it's a bad idea to use it.



pass
      Ranges (iteration with explicit step)










Using an explicit step does not seem to impact on the speed of an iteration over Ranges.


pass
       Use of indices on custom collection










Using the indices property on a custom Collection did not present a very significant cost over using a range to lastIndex. There is still a cost of around 2%, so using lastIndex may still be good advice for performance-critical applications.


All Results


Benchmark                                                                             Mode   Samples         Mean   Mean error    Units

c.a.k.part1.KotlinBenchmarkPart1.empty                                               thrpt       200  3540527.759    23025.839   ops/ms

c.a.k.part1.KotlinBenchmarkPart1.javaLambda                                          thrpt       200  1024302.409     1851.789   ops/ms

c.a.k.part1.KotlinBenchmarkPart1.javaPrivateConstructorCallFromStaticMethod          thrpt       200   398709.154      800.190   ops/ms

c.a.k.part1.KotlinBenchmarkPart1.kotlinInlinedFunction                               thrpt       200  1344885.445     2632.587   ops/ms

c.a.k.part1.KotlinBenchmarkPart1.kotlinLambda                                        thrpt       200  1362991.121     2824.862   ops/ms

c.a.k.part1.KotlinBenchmarkPart1.kotlinPrivateConstructorCallFromCompanionObject     thrpt       200   404746.375      621.591   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.javaIntVarargs                                      thrpt       200   173265.270      260.837   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.javaLocalFunction                                   thrpt       200   897015.956     1951.104   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.javaSayHello                                        thrpt       200    73353.725      155.551   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.kotlinIntVarargs                                    thrpt       200    83621.509      990.854   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.kotlinLocalFunctionCapturingLocalVariable           thrpt       200   909087.356     1690.368   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.kotlinLocalFunctionWithoutCapturingLocalVariable    thrpt       200   908852.870     1822.557   ops/ms

c.a.k.part2.KotlinBenchmarkPart2.kotlinSayHello                                      thrpt       200    75637.556      162.963   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.javaSimplyInitializedProperty                       thrpt       200   274394.088      554.171   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinCustomIndicesIteration                        thrpt       200    79096.631      134.813   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinDelegateProperty                              thrpt       200   255899.824      910.112   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinIndirectRange                                 thrpt       200  1214464.562     2071.128   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinIterationUsingLastIndexRange                  thrpt       200    80811.554      122.462   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinLocallyDeclaredRange                          thrpt       200  1214883.411     1797.921   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinRangeForEachFunction                          thrpt       200   108382.188      561.632   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinRangeForEachLoop                              thrpt       200   331558.172      494.281   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinRangeForEachLoopWithStep1                     thrpt       200   331250.339      545.200   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.javaStringComparisons                               thrpt       200   211488.726      450.531   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinStringRangeInclusionWithConstantRange         thrpt       200   218073.886      412.408   ops/ms

c.a.k.part3.KotlinBenchmarkPart3.kotlinStringRangeInclusionWithLocalRange            thrpt       200   211468.439      483.879   ops/ms



1-10 of 35

Comments