Inspired by Zach Bean's Quicklisp, I decided to challenge myself and see how fast I could create a webapp that showed as much data as possible from the quicklisp libraries (and anywhere else  I could semi-automate.) For lack of a more creative title, I'm going to call the lisp  package that I'm creating "ql-info-dump."

I'm going to start with the assumption that you have have a lisp implementation. Unfortunately, looking into system internals does vary from lisp implementation to lisp implementation. I'm writing this based on running Linux with SBCL 1.0.43, your mileage may vary.

This basic version does not require a database. Constructive criticism on anything in the tutorial is also appreciated. Unconstructive criticism will be filed in /dev/null with an automatic suggestion that you write the next tutorial.

Ok. So we are going to try to get the bare minimum webpage up and running and then start to build from there.

Load quicklisp, hunchentoot, cl-who, cl-ppcre,

We will need a few libraries from quicklisp, so get into quicklisp and run the following:


Hunchentoot is a webserver written in common lisp. Cl-ppcre is a  regex library for common lisp. cl-who is an html generating library for common lisp.

Test hunchentoot

So let's just test and make sure hunchentoot actually loads and works for us. We will load the hunchentoot-test demo, change to the hunchentoot-test package make an instance of the webserver and set it to the global parameter web-server and then actually start the webserver.


If you now point your browser to http://127.0.0.1:4242/hunchentoot/test/easy-demo.html you should actually see the hunchentoot server providing the demo page.

Note: If you are using a version of hunchentoot pre 1.2.1, then change "easy-acceptor" to "acceptor". Then ask yourself why you are using an older version.

You can play with this for awhile and then close this demo server.

Configure the Webserver and start it up

First we are going to create a quick and dirty page that just shows what packages are currently running in your lisp instance. We first create two parameters, one for the webserver and one for the port where hunchentoot will direct the pages. Note that I did change the port being used from 4242 to 8080 (just personal preference).

Assuming you want to have the hunchentoot server outputing in UTF-8 rather than iso-8859-1, then you need to set these parameters.


You can actually create a hunchentoot instance and have it running without assigning it to a parameter, but then you have no easy way to identify it when you want to stop and restart it.

So now, let's create a hunchentoot instance and start the webserver.

Define the package

At this point, if you click on http://localhost:8080 you should get the hunchentoot default page. We want to start with the hunchentoot equivalent of hello world, then build on that. I will go a little crazy, however, if I  have to preface every cl-who function with the package name, so lets define a package ql-info-dump which uses cl-ppcre, hunchentoot and cl-who. Other, more experienced, lisp programmers will argue that you should always preface functions with the package name. I have definitely found myself in situations where I would agree with them. You can decide for yourself. The source code version on the next page does preface the functions, just so you can see the difference.

Page Templates and Handlers

Hunchentoot has a really nice basic page macro called "define-easy-handler". In its simplest form, it takes a symbol name, a uri, a request type, a parameter list. In the following macro, I've extended it slightly to require a documentation string and insert a page template.

The page template provides the head and some basic body css and will eventually be modified to pick up some javascript. We then set up some functions to generate header, navigation, extra and footer stuff and the home page itself.

Stub Page Area Functions (fill out your divs)

The page template above indicates that we are going to have basically five areas of on the page:a banner,navigation, extra-stuff,footer and main content. Later on, we will also make a new template for a comparison page which will have the header, footer and two center  comparison columns.

Note that the template calls for a css file which we will get to shortly and also indicates that the character set encoding will be utf-8.

So, we need functions which will fill up these areas of the page. We can start with just something to fill in, but we'll add several lisp links
for the footer in case we want to jump off the page and go look for something.

CSS

Oh yeah. Now we need a css file. If we look at the page template above,  hunchentoot will expect a css directory and a css file named "base.css" Without an additional instruction to hunchentoot, however, it will not be able to find that file. Make a directory, and insert the following  basic css in it (or any other css you would prefer). 


Now we need to tell hunchentoot where to find the file. You can either tell hunchentoot where to find this specific file or, my preference,  you can tell hunchentoot what directory contains css files. This needs to be an absolute directory location. While I'm at it, I will also push the directory for xhtml documentation files.

Finally, the first blank page

Now we can create the homepage and everything should come together.

If you re-click on http://localhost:8080 you should get a really ugly, but live page.

What is actually running right now?

Now let's add some interesting stuff. What about showing all the currently loaded lisp packages in the navigation section? For that, we need to define a function to get a list of all the currently loaded "systems" or "packages". We'll return those names in a list, which can then be used in a redefined navigation function.


Notice that we sorted the system list in alphabetical order prior to returning it. If you reclick on http://localhost:8080 you should get a navigation bar that has been loaded with entries. You thought you only loaded hunchentoot, cl-ppcre and cl-who? So why are there so many other packages listed? Quicklisp also loads the dependencies, so everything that hunchentoot itself needs is also loaded.

We also redefined the navigation section to not only load the names, but also  wrap each in a href link. Since we have a link to display those functions,  let's do something with them.

Display package page (the very first version)

For that we need to define a page "display-package that takes a get parameter of name.


Note the leading slash in from of the url. Other than the fact we have the name of the package in the main content area, the rest of the page is using the template, so it looks exactly like the homepage.

We'll now define a function to list the exported functions from the package.


Now if we click on a package in the navigation bar, we get a list of  the exported functions in that package, with links to a new page  "display-function". Now we need to create that page.

Documented Functions? People really want documented functions?

As you play with this, you will notice that a lot of functions, even the exported functions, are not documented. Let's add a bit more information at the beginning and see what additional information we can get directly from the package or two other sources of lisp  information (clikie.net and lispdoc.com).

We will try to pull up the author, description and number of functions documented and undocumented which are exported by the package.
In many cases, the author and description slots in the asdf package are unbound, so we can use the "ignore-errors" function to ensure a nil
return rather than an error.

We will first create a helper function that takes a function list and returns three values, the total number of functions, the number of  functions with a documentation string and the number of functions without a documentation string. (In this case, we are only going to
be passing a list of the exported functions, not all package functions to the documented functions list.

Oops - a package with a plus

At this point, I've discovered that we don't properly handle the package "cl+ssl". That plus sign does not fit in well with standard url functions
and the plus tends to get treated as a space. If you click on the "cl+ssl" package in the navigation column, it will return exported functions for "cl" (the lisp system itself) because the space causes it to drop the "+ssl" part of the package name. There are a couple ways to deal with this. Right now I'll just take a quick and dirty solution and create a function that just  takes the name string and inserts plus signs back in where a space exists.

Available but not yet loaded

Let's change direction slightly and use that "Extras" area in the navigation bar provide links to additional information, starting with getting a list of all the available libraries from quicklisp.

Because there is no exported name method applicable to the quicklisp packages, we have to do a little bit of ugly text replacement to break out the name. We don't want to show the libraries which have already been loaded, so we use the "set-difference" function to reduce the total set of quicklisp libraries to the set of not currently loaded quicklisp libraries.


We gave a link to the library names for load-library. Right now we have no security whatsoever, so you would never do this on a production server and behind a firewall, but let's actually provide a function that asks  quicklisp to load the library and see if it now appears in the Navigation list.


Now see if you can click on an unloaded package and see the result.

Compare/Contrast 2 packages

Suppose we want to compare the function list of two packages. First we need a way to choose which packages to consider, so we will need a page with a form, two select lists and a submit button.

Given the length of some of the function-names, I've also decide to create another page-template (page-template-1), and defpage-macro (defpage-easy-d-1) which drop the navigation and extras side panel in order to get more room on the webpage. Then we create the compare-packages page using these new templates and page-macro, put a link to the compare packages page into the extras  function and we are on our way again.

Links to Source Files

So, you can now wander around the quicklisp libraries and look at the documentation strings and wonder why people don't write documentation strings for their exported functions. I know if I look at one of my functions six months after I wrote it, I prefer to have given myself a little bit of context before starting to read the source code. 

You will also notice that a lot of libraries do not put the author or license in the asdf definition. Some packages have no author or license anywhere, others put them in text files or in comments in the asd file. In other words, the data is dirty and we have pretty much reached the end of the easily auto-generated information.

So, you know the name of the function, but now you need to read the source code. In emacs running slime, you can hit Meta-. give the name of the function to  slime and it will automagically take you to the source code. We, however, are just surfing around the libraries in a browser, so what can we do to get to the text version of the source code (and check out whether there are any typical text files sitting around in the package?

The SBCL implementation of common lisp has functions that help here. I don't have other implementations, but they may or may not have similar functions. In any event, the following works on SBCL.

Consider the following function call and result from my box as I was writing this:


What this tells us is that the function "list-directory" in the cl-fad package can be allegedly found in the file "/usr/share/common-lisp/source/cl-fad/fad.lisp".

This seems a bit strange since all the source files for quicklisp packages are in "/home/user-name/quicklisp/dists/quicklisp/software". On my machine, it turns out that if the lisp implementation already has the package loaded, then quicklisp will not load a local quicklisp copy of that package. Testing this on some packages that I did not already have loaded returned the expected location for the package function.

The "sb-introspect:definition-source-pathname" function returns a pathname object, not the string version of the fully qualified file location.
However, Hunchentoot depends on cl-fad, which can return a namestring from a pathname object, so we already have that capability loaded.

Looking for the string which would give me the fully qualified path to the source code file for the hunchentoot function "authorization" can then be done as.


So going back to the display-function in our little webapp, let's add a link to the file where the function actually exists.


This gives you a link, which you an open by right-clicking on the link and opening it in a new page.

As mentioned previously, not every developer puts information into the asdf definition. Looking at the source directories, we notice a few
text files which are more common than others. For example LICENSE, COPYING, README, CHANGELOG, CHANGES, INSTALL, AUTHORS. So lets take those filenames and put links to them if they exist in the package directory.

We can get the pathname to the package directory using the asdf function system-source-directory. Let's create a function that returns the full file location if the file exists or blank otherwise.


Now we can go back to display package and insert the links if these files exist. We will also put in a link to the asd file. Unless someone deleted it after we loaded the library, that is one file we know should exist.

However, before we start, the function is getting too long for my taste.  So lets create another little helper macro which will deal with small
snippets of html, then use it for the package overview, and call that from the "display-package" function.

Dependencies

OK. Now let's add dependencies to our description page for packages. This will require a few more helper functions. Looking at these helper functions, the ensure-list and flatten functions were borrowed from Alexandria and the effective-dependencies functions were borrowed from Leslie Polzer. The gridify1 function is a relatively straightforward loop to build out a table grid of x columns and adding empty html cells to the end in order to even things out.


So given these helpers, we can add another some area to the "display-package" function.

Browsing source code revisited.

We've been looking at the package side of things for awhile. Let's go back to the display-function page and consider what would be interesting here. SBCL has a function in sb-introspect called "who-calls", which can tell us what other currently loaded functions call the currently displayed function. So let's use that to create a list of links to functions which call the displayed function.

Often, function A will call function B several times. Since we don't want to clog the screen with the same function link multiple times, we will just create a list of calling functions and push a new function on that list only if it is not already a member. The entire display-function code now looks  like this.

Note that at the moment we are sorting the calling functions in alphabetical order by name. You might choose to sort them by package or file location.

Now that we are getting display-function a bit more filled out, it is  getting too long for my taste, so I'm going to break it into a page-macro, a couple of html-snippet functions and a sorting helper function.