OSGiaaS-CLI - A CLI to run JVM-languages REPL and commands

Post date: Dec 17, 2016 8:9:59 PM

Every now and then, like most technical people, I feel the need to run some CLI (command-line-interface) commands to perform a quick, usually repetitive task. The operating system provides a lot of commands (ls, cat, cp, rm, grep...) that can be wired together to perform certain kinds of tasks (like deploying your application to Amazon, or even finding the cheapest flight somewhere!). There's even a venerable name for this: the Unix philosophy, though this approach can be used in any OS.

The Unix-way can make you very productive, so much so that some folks even think that Unix is an IDE because of that.

However, a lot of times you will find yourself trying to do something that cannot be easily done just with the existing tools you have in your CLI. In other words, you'll have to write some inline code directly in the CLI, or write your own command. And that's where the problem lies: doing either of those is not very convenient for JVM-languages programmers in any of the major Operating Systems.

You could of course write some bash scripts for the more complex logic you might need, or just write native executables in languages such as C or Rust. But that is quite an underwhelming proposition for people like me, who feel at home in the JVM world and expect their programs to run in any OS without changes, and don't want (or have time) to learn a whole new programming environment, using archaic languages like bash and awk, missing out on all the rich JVM libraries we've grown familiar with.   

That's why I have created the OSGiaaS-CLI Project.

The OSGiaaS-CLI Project is a collection of small libraries that, together, create a highly modular CLI based on OSGi services that are written in Java and other JVM languages, and can run code snippets as if inside a REPL.

Before we get into detail on how this works, here's an example of what you can do with the OSGiaaS-CLI (using a Java 8 lambda to process each line of input provided by the run ls command, which will be explained shortly):

>> run ls | java line -> line.contains("log") ? line : null

Whenever a command is shown with the >> prompt, it means it is typed within the OSGiaaS-CLI. If the >> prompt is not shown, the command is typed within a regular shell.

Because Java may get a little verbose, I normally use a more concise language such as Groovy for this:

>> run ls | groovy if (it.contains("log")) it

What this does is run the usual ls command we all know (to list files), then run the provided Groovy script on each line given by ls, returning it only if it contains the word "log" (returned Objects are printed by the java and groovy commands, like all other languages commands).

The easiest way to accomplish the task above would be to use grep: run ls | grep log.

The above are just simple examples to show how easy it is to use a JVM language in the CLI. 

We could use any other language supported by the OSGiaaS Project (currently: Java, Groovy, JavaScript and, experimentally, Frege, Clojure and Scala) directly in the CLI.

Just to prove the point, here's a silly example using a bunch of JVM languages on the same command pipeline (this example uses the CLI multi-line support, activated by starting with :{ and ending with :}):

>> :{  frege sum [1,2,3] |  groovy it.toInteger() + 2 |  java line -> Integer.parseInt(line) + 3 |  scala (line: String) => line.toInt + 4 |  clj (fn [line] (+ (read-string line) 5)) |  js function (line) { return line == 20; }  :}  true

Compiled commands can be written in any JVM language, including also Kotlin and, with some extra effort, Ceylon. And you don't even need to stop the CLI to reload your compiled commands, as we'll see in the end of this blog post! Just change the code, recompile and reload it.

As long as you have previous CLI experience, you'll feel right at home with this CLI as it is based on JLine, supporting command history, line-editing, common unix shortcuts (including vim/emacs modes), pipes (as shown above), tab auto-completion etc.

Getting started

The easiest way to get started if you have Gradle is to run the following build script's createOsgiRuntime task (just run gradle crOsgi), and then use the generated run script.

The detailed steps are explained below:

1. Save the file below as build.gradle:

plugins {

   id "com.athaydes.osgi-run" version "1.5.4"

}

repositories {

   mavenLocal()

   jcenter()

}

dependencies {

   // the osgiaas-cli core bundle

   osgiRuntime 'com.athaydes.osgiaas:osgiaas-cli-core:0.7'

   // OSGi Service Component Runtime implementation

   osgiRuntime 'org.apache.felix:org.apache.felix.scr:2.0.2', {

       exclude group: '*'

   }

}

runOsgi {

   bundles = [ ] // all bundles added as osgiRuntime dependencies

}

Alternatively, just run this command to download and save the file:

curl https://raw.githubusercontent.com/renatoathaydes/osgiaas/master/samples/osgiaas-cli-minimal.gradle --output build.gradle

2. In the same directory, run the createOsgiRuntime Gradle task (which can be abbreviated to just crOsgi):

gradle crOsgi

OSGiaaS uses the osgi-run Gradle plugin to create its runtime. This plugin takes your project dependencies, downloads them from JCenter and puts them all in a OSGi runtime which can be easily configured. Check the osgi-run documentation for more information.

3. Start OSGiaaS-CLI with the following from command:

In Linux/Mac:

bash build/osgi/run.sh

In Windows:

build/osgi/run

You should see the ASCII Art Logo of OSGiaaS CLI:

And that's it.

You are now ready to use the CLI.

Hit Tab to auto-complete almost anywhere. For example, try entering help (with a whitespace at the end), then Tab.

The first command you should try is ps (provided by the Felix Shell bundle):

This command shows all the libraries currently installed in the system.

The default OSGi runtime is Apache Felix (which is the system bundle above).

If you want to use a different runtime such as Equinox, change the build.gradle file by adding

configSettings = 'equinox' inside the runOsgi block as explained in the osgi-run documentation.

The Gradle build file we're using so far only declared the minimum set of dependencies the OSGiaaS-CLI needs to run. To see which commands are available you can just hit Tab:

The OSGiaaS-CLI documentation gives details on how to use each command it provides, but you can use the help command to see usage information about any command:

The ci (command introspect) command gives even more information about a command, including which bundle provides the command and the name of the implementing class (eg. ci -v alias).

Other commands you should probably get to know about right from the start are:

• shutdown - to exit the CLI.

• lr - list JMV resources.

• grep - filter text.

• highlight - highlight text.

• run - run a native command.

• use use a command, so you don't have to type it all the time. Very useful to use the CLI as a REPL, as we've seen.

Because the OSGiaaS-CLI can run native commands, you can use it to mix the OSGiaaS-CLI commands, as well as your own JVM-written commands, with native ones.

For example, to run netstat (the native command) and highlight all lines containing 72: (using the CLI's highlight command, written in Java):

To temporarily use the CLI as a native shell, you can use the use command (while a command is being used, all input entered by the user is passed as an argument to it):

While a particular command is being used, you can call other commands by prefixing them with `_`. That's actually how you stop using a command. You just call _use without any arguments, which means "use nothing".

To turn the CLI into a language REPL, just use a language command:

However, before you can use language commands, you need to install them! In the next sections, we'll see how to install more commands into the environment.

But before we move on, quit the CLI by running the shutdown command.

Depending on the OSGi runtime, you might need to enter stop 0, ie. stop the system bundle.

I like to add an alias for that:

>> alias exit="stop 0"

Now, you can exit by typing exit.

To remember this alias, add the above command to a init file at "${user.home}/.osgiaas_cli_init". This file is run every time the CLI starts up, so you can use it to customise the CLI.

My full init file looks like this:

~/.osgiaas_cli_init:

color prompt blue  prompt "osgiaas> "  color error yellow  alias exit="stop 0"  alias hl=highlight

Managing the environment using Gradle

The easiest way to manage which libraries are installed in your OSGiaaS-CLI environment is to use the Gradle file we used to get started.

For example, to add a popular Java library such as JavaSlang into the environment, just add a dependency on it via the Gradle file (the added line is shown in bold):

plugins {

   id "com.athaydes.osgi-run" version "1.5.2"

}

repositories {

   mavenLocal()

   jcenter()

}

dependencies {

   // the osgiaas-cli core bundle

   osgiRuntime 'com.athaydes.osgiaas:osgiaas-cli-core:0.7'

   osgiRuntime 'io.javaslang:javaslang:2.1.0-alpha'

   // OSGi Service Component Runtime implementation

   osgiRuntime 'org.apache.felix:org.apache.felix.scr:2.0.2', {

       exclude group: '*'

   }

}

runOsgi {

   bundles = [ ] // all bundles added as osgiRuntime dependencies

}

After saving the Gradle file, make sure to re-create your environment:

gradle crOsgi

If you want to ensure only bundles declared in the build file are installed, also run the clean task:  gradle clean crOsgi .

Now, when you restart the CLI, you'll find that the JavaSlang bundle is included in the environment.

In the same manner, you could add other useful commands:

Java command:

 osgiRuntime 'com.athaydes.osgiaas:osgiaas-cli-javac:0.7'

Groovy command:

 osgiRuntime 'com.athaydes.osgiaas:osgiaas-cli-groovy:0.7'

Js command:

 osgiRuntime 'com.athaydes.osgiaas:osgiaas-cli-js:0.7'

You get the idea!

Managing the environment via the CLI itself

Many commands exist only to manage the OSGi environment itself, like install (install a bundle from a URL into the system), start (starts a bundle), stop (stops a started bundle).

These commands allow you to take advantage of the incredible dynamism of an OSGi environment. If you have a jar you would like to install into the system, you can use the install command to do it:

>> install file:///Users/renato/jars/my-lib.jar

As the install command takes a URL, you can use it to get a jar from a remote server as well.

The install command does not automatically start the installed bundle. As the start command can also take a URL and install the bundle before starting it, you can just use start most of the time.

However, downloading Jar files in such a way can be quite inconvenient, especially when the artifact has transitive dependencies. Using a package manager such as Gradle or Ivy makes things much easier.

If you want to use Gradle for this, you can use the approach presented in the previous section (add osgiRuntime dependencies in the build file and they will be installed into the system when you build it again).

Another option offered by the OSGiaaS-CLI Project is to use Apache Ivy directly from the CLI!

You can install the ivy command on the CLI using the Gradle file by adding the following dependency:

Ivy command:

 osgiRuntime 'com.athaydes.osgiaas:osgiaas-cli-ivy:0.7'

If you prefer, replace your current build file with the "default" CLI build, which contains the Ivy command, besides debugging configuration:

curl https://raw.githubusercontent.com/renatoathaydes/osgiaas/master/samples/osgiaas-cli-default.gradle --output build.gradle

After that, re-build the project with gradle clean crOsgi, then start the CLI again.

Now, to retrieve any artifact that exists either in your local Maven repository or in JCenter (which includes Maven Central artifacts), you can use the ivy command. For example, to retrieve JavaSlang and immediately start it up, type the following:

>> ivy io.javaslang:javaslang | start

You can now easily play with JavaSlang (or whatever library you wanted to try out) using one of the CLI language modules, such as Groovy:

>> use groovy

Using 'groovy'. Type _use to stop using it.

>> import javaslang.collection.List

>> List.of(1,2,3).intersperse("-").mkString()

1-2-3

Check instructions on how to get the Groovy module working in the osgiaas-cli-groovy docs. Unfortunately, there's some small configuration that is necessary to get most of the language modules working, as most of them use non-standard classes not exported by the OSGi environment... the Groovy module requires sun.reflect, for example. Or just use one of the sample Gradle files to get you started.

Writing compiled commands:

To create a new, basic CLI command, you need to create a class implementing the org.apache.felix.shell.Command interface.

Here's the Java Hello World command, for example:

package com.athaydes.osgiaas.examples.java;

import com.athaydes.osgiaas.cli.CommandHelper;

import org.apache.felix.shell.Command;

import org.osgi.service.component.annotations.Component;

import java.io.PrintStream;

import java.util.List;

@Component( immediate = true, name = "hello-java" )

public class HelloJavaCommand implements Command {

   @Override

   public String getName() {

       return "hello-java";

   }

   @Override

   public String getUsage() {

       return "hello-java [<message>]";

   }

   @Override

   public String getShortDescription() {

       return "Prints a Hello World message or a custom message given by the user";

   }

   /**

    * This method implements the command logic.

    *

    * @param line full command provided by the user. Notice that this may be more than one line!

    * @param out  stream for the command output (prefer this to System.out)

    * @param err  stream for the command errors (prefer this to System.err)

    */

   @Override

   public void execute( String line, PrintStream out, PrintStream err ) {

       // break up the command line into separate tokens.

       // notice that the first part is always the name of the command itself.

       List<String> arguments = CommandHelper.breakupArguments( line );

       switch ( arguments.size() ) {

           case 1:// no arguments provided by the user

               out.println( "Hello Java!" );

               break;

           case 2:// The user gave an argument, print the argument instead

               out.println( "Hello " + arguments.get( 1 ) );

               break;

           default: // too many arguments provided by the user

               CommandHelper.printError( err, getUsage(), "Too many arguments" );

       }

   }

}

A complete command would also provide some options to the user, auto-completion and good documentation. The OSGiaaS Project provides a few facilities that make it quite easy to do so.

The ArgsSpec class can be used to specify which options a command may take, including documentation which can be used to auto-generate the command documentation in a standard way.

To show how that works, we will create a weather command which lets user see the current weather around the world, as well as the forecast for the coming week.

The weather command should take a -u option that allows the user to specify whether temperatures should be shown in Celsius (c) or Fahrenheit (f), and a -v option to declare the output should be verbose (verbose output includes the week forecast).

As shown in the osgi-run tutorial, there's a yahoo-weather-java-api project which can be used to consume the Yahoo Weather API easily from Java (or in the command we will write, Kotlin).

Using that Java project, the following shows an implementation of the weather command in Kotlin:

A few definitions are omitted for brevity, see the full code in the osgiaas-examples Github project.

@Component(name = "weather-command")

class WeatherCommand : Command {

   @Volatile

   var service: YahooWeatherService? = null

   companion object {

       val NAME = "weather"

       val VERBOSE_KEY = "-v"

       val UNITS_KEY = "-u"

       val argsSpec = ArgsSpec.builder()

               .accepts(UNITS_KEY, "--units")

               .withArgs("unit")

               .withDescription("Temperature units to be used (C for Celsius, F for Fahrenheit)")

               .end()

               .accepts(VERBOSE_KEY, "--verbose")

               .withDescription("Show verbose output")

               .end()

               .build()

   }

   @Activate

   fun start() {

       try {

           service = YahooWeatherService()

       } catch (e: JAXBException) {

           service = null

       }

   }

   override fun getName() = NAME

   override fun getUsage() = "$NAME ${argsSpec.usage} <location>"

   override fun getShortDescription() = """

   |Shows current weather and forecast for a given location.

   |

   |Options:

   |${argsSpec.getDocumentation("  ")}

   |""".trimMargin()

   override fun execute(line: String, out: PrintStream, err: PrintStream) {

       // freeze local service reference

       val service = service

       if (service == null) {

           err.println("Yahoo service could not be created. Restart this bundle.")

           return

       }

       val invocation = try {

           argsSpec.parse(line)

       } catch (e: IllegalArgumentException) {

           err.println(e.message)

           return

       }

       val location = invocation.unprocessedInput

       if (location.isBlank()) {

           CommandHelper.printError(err, usage, "Please provide a valid location")

       } else {

           val unit = if (invocation.hasOption(UNITS_KEY))

               parseUnit(invocation.getFirstArgument(UNITS_KEY))

           else DEFAULT_UNIT

           if (unit == null) {

               err.println("Invalid unit provided. Valid units are " +

                       "'${CELSIUS.unitKey()}', '${CELSIUS.name}' or " +

                       "'${FAHRENHEIT.unitKey()}', '${FAHRENHEIT.name}'")

           } else {

               val verbose = invocation.hasOption(VERBOSE_KEY)

               showWeatherAt(location, unit, verbose, service, out, err)

           }

       }

   }

}

Using ArgsSpec, we can easily create a class that exports the CommandCompleter service, which allows our command to provide auto-completion with almost no effort:

@Component(name = "weather-command-completer")

class WeatherCommandCompleter :

       CommandCompleter by WeatherCommand.argsSpec.getCommandCompleter(WeatherCommand.NAME)

The above example uses Kotlin delegation to implement CommandCompleter. In Java, you would have to implement the delegation yourself, of course.

Now, when the user types weather - followed by hitting Tab, all available options are displayed:

And finally, a sample run of the weather command:

This example, along with example commands written in many other languages, can be found in the osgiaas-examples GitHub repo.

The OSGiaaS Project took me around one year of planning, implementation and testing to bring to what it is now. It is by far my largest side-project ever! Even though there's a lot of things I am not completely happy about yet, I believe a lot of the modules, and the CLI itself in particular, can be incredibly useful already, and this is why I decided to publish the project on JCenter (and soon on Maven Central) and let other people know about it.

For more information about OSGiaaS, check the documentation on GitHub!