Java Abstract Syntax Trees

Java Binding is where you unmarshal an XML file into an Abstract Syntax Tree consisting of your own Java classes.  These are regular Java classes that obey certain conventions.

For every XML element with the name Elem, you must provide a Java class called Elem, and give it a default constructor;

For every XML attribute of Elem with the name attrib, the class Elem must provide:

• a private field called attrib having some Java type Type;

• a public method called Type getAttrib() that returns the value of this field;

• a public method called Elem setAttrib(Type val) that sets the value of this field and returns this, the target Elem object;

For every XML child-element of Elem, with the name Child, the class Elem must provide:

• a private field called child having the type Child;

• a public method called Child getChild() that returns the value of this field;

• a public method called Elem addChild(Child val) that sets the value of this field and returns this, the target Elem object;

Alternatively, if the children are to be stored as lists, the class Elem may provide:

• a private field called children having a list-type List<Child>;

• a public method called List<Child> getChildren() that returns the whole list;

• a public method called Elem addChild(Child child) that adds a child to the list and returns this, the target Elem object;

If desired, you may substitute Set-types or Map-types for these List-types, so long as the field is consistent with the methods that get and add to it.  For a Map-type field, the addChild() method must insert the child opposite some key, which is computed from the child object.

If the XML element called Elem also contains text, the class Elem should provide:

• a private field called content of the type String;

• a public method called String getContent() to return the value of this field;

• a public method called Elem setContent(String str) to set the value of this field and and returns this, the target Elem object.

The naming conventions for the getter and setter methods are significant:

•  for attributes they are based on the name of the attribute.  

• for children elements they are based on the type of the child class.

• for text content they have names based on the reserved word content.

This is how ASTReader distinguishes between the  dependent elements, attributes and text.

Reading an AST

ASTReader is the class to use for unmarshalling an XML file into an AST constructed from your own Java classes.  It has four main constructors:

ASTReader(File file, String encoding)

ASTReader(URL url, String encoding)

ASTReader(InputStream stream,
    String encoding)

ASTReader(Reader reader,
    String encoding)

These allow you to read from different kinds of source (from a file, from a web URL, or from another stream).  They all require the character encoding (e.g. "UTF-8") as a String.

ASTReader is in the top level package:

uk.ac.sheffield.jast

Reading from a File

Reading from a local file is easily done using the following Java code.  Here, we assume that the root element will be an object of the Java class called org.project.cat.Catalogue:

File file =
  new File("my/xml/catalogue.xml");

ASTReader reader =
  new ASTReader(file, "UTF-8");

reader.useDomain("project.org");
reader.usePackage("org.project.cat");

Catalogue catalogue =
  (Catalogue) reader.readDocument();

reader.close();

You need to specify the name of your domain, and then identify the name of the Java package within this domain that defines all the Catalogue-related classes.  This is so that ASTReader knows where to find the classes to instantiate.

Since the readDocument() method could return any kind of Object, you have to cast the type of this down to the expected Catalogue type.  After this, you can access any part of the Catalogue AST using the methods you provided.

Reading from a URL

Reading from a URL on the web is done in a similar way, using the following Java code:

URL url = new URL(
  "https://www.my.site/catalogue.xml");

ASTReader reader =
  new ASTReader(url, "ISO-8859-1");

reader.useDomain("project.org");
reader.usePackage("org.project.cat");

Catalogue catalogue =
  (Catalogue) reader.readDocument();

reader.close();

This shows that you can specify a different character encoding Latin-1, which used to be standard for the HTTP protocol (probably UTF-8 now).  Note that JAST raises an exception if the expected and actual encodings do not match.

After this, you may access any part of the Catalogue AST using the methods you provided in your own Java classes.

Java Binding

If the XML file specifies a default XML namespace, and also declares a java-binding, then you may skip the useDomain() and usePackage() methods.

A default XML namespace is declared as an attribute of the root XML element, for example:

xmlns = "https://project.org/"

A Java-binding is declared as a processing instruction immediately after the XML declaration, for example:

<?java-binding
    xmlns="org.project.cat"?>

With these in place, the code to read an AST is made much simpler:

File file =
  new File("my/xml/catalogue.xml");

ASTReader reader =
  new ASTReader(file, "UTF-8");

Catalogue catalogue =
  (Catalogue) reader.readDocument();

reader.close();

In this case, ASTReader works out the domain and package from the XML file, and does not have to be told this in the program.  You can choose either approach.

Multiple Packages

If your Java application is large, it will store classes in different packages.  You can direct ASTReader to map XML elements to classes from different Java packages, using XML namespace prefixes to identify the different packages.

The following shows how to unmarshal elements with the prefix util to classes from a package org.project.util, and unmarshal elements with the prefix main to classes from a package org.project.main in a large Product application:

File file =
  new File("my/xml/product.xml");

ASTReader reader =
  new ASTReader(file, "UTF-8");

reader.useDomain("project.org");

reader.usePackage("org.project.util", "xmlns:util");

reader.usePackage("org.project.main", "xmlns:main");

Product product =
  (Product) reader.readDocument();

reader.close();

The usePackage() method has two arguments here.  The first argument identifies the package, and the second identifies the XML namespace to map to this package.

Normally, you should define the XML namespaces in the XML file too (see next, below).

Multiple Bindings

You can achieve the same effect by defining multiple XML namespaces and Java bindings in the XML file.

A standard XML namespace includes a short prefix label after "xmlns".  All namespaces are declared as attributes of the root XML element, for example, the following declares prefixes util and main:

xmlns:util = "https://project.org/"
xmlns:main = "https://project.org/"

A Java-binding is declared as a processing instruction immediately after the XML declaration.  This time, the different namespaces are mapped to different Java packages:

<?java-binding
    xmlns:util="org.project.util"?>

<?java-binding
    xmlns:main="org.project.main"?>

With these in place, the code to unmarshal an AST is made much simpler:

File file =
  new File("my/xml/product.xml");

ASTReader reader =
  new ASTReader(file, "UTF-8");

Product product =
  (Product) reader.readDocument();

reader.close();

In this case, ASTReader works out the domain and packages from the XML file, and does not have to be told this in the program.

Writing an AST

ASTWriter is the class to use for marshalling a Java AST to an XML file.  It has three main constructors:

ASTWriter(File file, String encoding)

ASTWriter(OutputStream stream,
    String encoding)

ASTWriter(Writer writer,
    String encoding)

These allow you to write to different kinds of sink (to a file, or to another stream).  They all require the character encoding (e.g. "UTF-8") as a String.

ASTWriter is in the top level package:

uk.ac.sheffield.jast

Writing to a File

Writing to a stream that uses a different encoding is possible with the following code:

Catalogue catalog = ... // exists
OutputStream stream = ... // exists 

ASTWriter writer =
  new ASTWriter(stream, "ISO-8859-1");

writer.useDomain("project.org");
writer.usePackage("org.project.cat");

writer.writeDocument(catalogue);

writer.close();

This assumes that the existing output stream uses the Latin-1 encoding.  Note that JAST throws an exception if the expected and actual encodings do not match. 

You may also map AST classes from different packages to elements with different namespace prefixes (see above).

Metadata

Whenever an ASTReader unmarshals an XML file, it creates a data structure recording everything about the XML-to-Java mapping.  This may include how certain XML names have been normalised to ensure that they conform to legal Java names, or which XML namespaces were mapped to different Java packages used for the AST classes.  All of this information is stored in a single object of the type Metadata.

Whenever an ASTWriter marshals an AST back to XML, the easiest way of restoring the same Java-to-XML mappings is to use the same Metadata object that was constructed during unmarshalling.  This object can be transferred directly from the ASTReader to the ASTWriter:

File in = new File("my/xml/input.xml");
ASTReader reader = new ASTReader(in);

Catalogue catalogue = (Catalogue)
  reader.readDocument();

Metadata metadata =
  reader.getMetaData();

reader.close();

// do something with catalogue

File out = new
  File("my/xml/output.xml");

ASTWriter writer = new ASTWriter(out);

writer.setMetadata(metadata);

writer.writeDocument(catalogue);

writer.close();

This will ensure that the ASTWriter uses the same XML namespaces and java-bindings that were used by the ASTReader, which got this information from the original XML file.

Metadata supports limited XML-to-Java name conversion.  For example, an XML element with the name app:test-button could be mapped to a Java class called org.project.app.TestButton.

Metadata offers a full API to access different kinds of metadata.  See the Javadoc documentation.

Example XML

We develop an extended example below, to show how to design an AST for a particular XML data file.   We also show more about how AST classes may use inheritance, or be stored in polymorphic Lists.

<?xml version="1.0" encoding="UTF-8"?>

<?java-binding xmlns="org.project.cat"?>

<Catalogue xmlns="https://project.org/">
  <Film year="1976" rating="PG">

  <Title>Star Wars</Title>

  <Director>George Lucas</Director>

</Film>

<TVShow year="1965" rating="U">

  <Title>Thunderbirds</Title>

  <Director>Gerry Anderson</Director>

</TVShow>

<Film year="2007" rating="15">

  <Title>Transformers</Title>

  <Director>Michael Bay</Director>

</Film>

</Catalogue>

This defines an XML Catalogue for films and TV shows that share certain features, such as attributes for the year and rating, and also common child-elements for the Title and Director.

Examples for the AST classes needed for this XML dataset are provided in the package:

uk.ac.sheffield.jast.ast

Design the AST

Following the guidelines for designing AST classes, it is clear that we need the following Java classes:

• Catalogue - to model the whole catalogue

• Film - to model a film

• TVShow - to model a TV show

• Title - to model the title

• Director - to model the director

Both the classes Film and TVShow will require fields to store the attributes year and rating.  Similarly, both of these classes will require fields title and director to store objects of the types Title and Director.

We could provide a full set of methods in each class to set and get these fields.  But this seems wasteful, since in Java it makes more sense for Film and TVShow to inherit from a common abstract class, which we will call Show, which defines these in one place:

public abstract class Show {

  private int year;

  private String rating;

  private Title title;

  private Director director;

  ... // methods

  public Show() {}

}

public class Film extends Show {

  Film() {}

}

public class TVShow extends Show {

  TVShow() {}

}

Film and TVShow now inherit from the abstract class Show.  We will be able to add all the required methods for adding, setting and getting in this parent class.  The only thing we need to define locally in the subclasses is a default constructor for each type.

AST Methods

We focus now on the required public methods that must exist (for Film and TVShow), which we define in the parent class Show:

public abstract class Show {

  ... // fields

  public int getYear() { 

    return year;
  }

  public Show setYear(int year) {
    this.year = year;

    return this;

  }

  ... // similarly for rating
  public Title getTitle() {

    return title;

  }

  public Show addTitle(Title title) {

    this.title = title;
    return this;

  }

  ... // similarly for Director

}

This shows how the getter for year is called getYear() and the setter is called setYear();  similarly, the getter for the child Title is called getTitle(), but the setter for a child is called addTitle().

Note that the type of year is int, even though the XML only stores string values.  The readers and writers convert between XML string values and other strong Java types.  For primitive types, this is done using Java's type conversion methods.  For class types, the class must provide a constructor accepting a String.

AST Text

We focus now on the required public methods for classes that contain textual content.  These are Title and Director.  Following the guidelines, the Director class looks like this:

public class Director {
  private String content;
  public String getContent() {

    return content;
  }
  public Director setContent(String content) {

    this.content = content;

    return this;
  }

  public Director() {}

}

The field for storing textual content is always called content (mandatory name), and the getter is called getContent() and the setter is called setContent(), which must return the target object.

The design of the Title class is similar to this, with a content field and getter and setter methods.  Both Title and Director could also be defined as subclasses of a common abstract class.

AST Collections

We focus now on the required public methods for classes that contain collections of other classes.  In this example, Catalogue is the only class that needs to store a List of different kinds of Show.  

We may declare a polymorphic List<Show> field, and store in this objects of the arbitrarily mixed subtypes Film and TVShow:

public class Catalogue {
  private List<Show> shows;
  public List<Show> getShows() {

    return shows;
  }
  public Catalogue addShow(Show show) {

    shows.add(show);

    return this;
  }

  public Catalogue() {

    shows = new ArrayList<Show>();
  }

}

The field for storing the list of Shows is called shows; and this is also reflected in the getter-method name  getShows() that returns this list.

We could have provided separate adder-methods called addFilm() and addTVShow() to add these types to the list.  However, we may provide a single adder-method to do both:  Catalogue addShow(Show).

JAST is sensitive to inheritance relationships.  If it cannot find an adder for Film, it will look for an adder in superclasses of Film and will find one in Show.  The same method will add TVShow objects.

Catalogue provides a default constructor, which must correctly initialise shows to an empty list, here using the ArrayList type.

The type of shows could be declared as Set<Show> or even Map<String, Show>, so long as the getter methods returned a Set, or a Collection (of the map's values).  In the case of a Map, the adder method would need to compute some String key from the added Show object. This could be (for example) the name of the Show's Title.

Exceptions