This document describes the software infrastructure of the GIVE Challenge. It is targeted at developers of NLG systems who want to participate in the challenge and need to understand how their system will fit into the overall architecture.
The document is structured as follows. In Section #Overview we will give a high-level overview of the GIVE infrastructure. Section #Configuration describes how to build and configure an initial GIVE system that uses two demo NLG systems that come with the distribution. Section #Implementation goes into more detail about the internals of the NLG server infrastructure; it describes how your NLG server is implemented as a Java class, and what utility methods you can use.
During the challenge, we may occasionally release updated versions of the GIVE software. These will come with updated versions of this document, with special emphasis on things that changed. We would also like to draw your attention to the online documentation, in particular the Javadocs that come with the distribution and the Wiki.
The overall architecture of GIVE is split into three components:
A user who discovers GIVE on the Web and decides to give it a try will download the client and run it. This client will connect to the Matchmaker, which randomly selects one of the registered NLG servers and requests a new NLG server instance. The Matchmaker also randomly chooses a game world and sends it to the client and the new server instance. It then tells the client and the server instance how they can connect to each other and closes its connections; from this point on, the user tries to solve the given task following the instructions from the NLG server instance. When the game run is finished -- either because the user solved the task or lost, because the user cancelled the game run, or because the connection timed out, broke down, or the client crashed --, the NLG server instance reconnects to the Matchmaker, uploads a log of what happened in the game run, and terminates. The NLG server is now ready to accept a new connection from a client.
This entire process happens in software that we are providing for the challenge; you should not have to worry about it at all. Your primary job, as the developer of the NLG system, is to implement a Java class derived from give.nlgserver.NlgSystem which gets loaded by our server code. We will describe how to do this in #Configuration. If you ever run into problems that you feel are due to bugs in our part of the code, we urge you to report them in the GIVE participants forum. Although the backbone of the GIVE-2 code has been tested thoroughly in GIVE-1, the newer parts are still unpolished, and we hope to iron out the bugs and make things prettier in time for the challenge, in February.
With the participants' package, you have received a virtual environment (the "development world") that you can use for development purposes. This environment will not be used in the actual challenge in February; we will prepare new worlds (the "evaluation worlds") that no participant will have seen. However, we do promise that the evaluation worlds will not differ in dramatic ways from the development worlds. While there may be new object types, every new object will behave exactly like an existing object. For instance, there might be a piano object; but users will not be able to interact with it, and so it will behave exactly like a couch in the development world. You will get a week's time to adjust your NLG system to the evaluation worlds just before the evaluation period starts. Once the evaluation phase of the challenge starts, you are not supposed to change your NLG server any more. There is nothing we can do to stop you from doing it, but we appeal to your scientific honesty not to gain an unfair advantage over other participants by fitting your system to the evaluation worlds.
In order to build your own NLG system, you will need the following software:
You will also need the GIVE-2 tools distribution. The tools distribution is an archive that contains the Matchmaker, the Client, and the development worlds. After you unpack it, the give2-tools-V directory (where V is the version, initially 2.0) should contain the following files:
Finally, there is a GIVE-2 NLG demo server distribution, which contains two demo servers for GIVE-2. After unpacking, the give2-demo-nlgservers-V directory should contain the following subdirectories:
Notice that the NLG server distribution is only meant to get you started on your way to developing your own NLG systems; once you understand how the example servers work, you can delete them. However, you will be using the programs in the Tools Distribution, and when we make changes to the tools in this distribution, you will need to keep your copy updated; otherwise, they may become incompatible with the NLG server code into which your NLG system will plug into. We will announce changes to the Tools Distribution whenever they happen.
As of GIVE 2.1, the GIVE tools distribution contains the Map Viewer, which gives you a top-down view of a GIVE game world. Change into the main directory of the archive you just unpacked, and try this:
This will bring up a window that displays a 2D view of the development world. If you hover your mouse over an object in the world, you will be shown the name, type, and state of that object. If the object controls some other object, or is controlled by another object -- for instance, many doors are controlled by buttons in that pressing the button opens and closes the door --, the Map Viewer will display an arrow that points from the controlling object to the controlled object.
You can use the Map Viewer either to simply familiarize yourself with a given world. You can also use it as a tool that will support you in developing your own GIVE maps.
In order to familiarize yourself with the GIVE software, you should first build the demo NLG servers and then set up and run your own version of the development environment -- that is, run a demo NLG server, run your own copy of the Matchmaker locally, and run the client in such a way that it points to your local Matchmaker. You can do this as follows.
After the 3D models have been loaded, this should bring up a window in which you can move around with the cursor keys. Follow the tutorial to familiarize yourself with the controls, and then venture out of the first room to receive instructions from the (trivial) NLG system. This is where your NLG system will take over later. Note that you could run the Example NLG Server instead of the dummy server, and this is something you should try; but the example server uses a planner, and will be no fun to use until you've attached a decent planner (see below).
At any point (during, before, or after you're running the client), you can inspect the current status of the Matchmaker and the NLG server over a Web interface. If you open the URL http://localhost:8080/ (replace "localhost" by the name of the machine that runs your Matchmaker), you will see status information about the servers, worlds, and previous game runs that the Matchmaker knows about. Similarly, you can see information about the NLG server on http://localhost:8081. If you inspect the NLG server in this way while a client is connected (ideally from a different machine), you can observe how the NLG server spawns a new instance for this client, and you can watch the messages as the client and the server exchange them.
When you develop your own system, you will generally run your own local copies of the Matchmaker and the client (as above), and point the Matchmaker at your own NLG server. During the evaluation, you will then give us the contact details of your NLG server, and we will point the central Matchmaker at it. Feel free to look at the source code of the demo servers; they are very simple, but should give you an idea about what code you will have to provide.
Now that you have the system running in principle, let's have a look at the configuration files in more detail.
The file matchmaker-config.xml is the configuration file for the Matchmaker. Whenever you start the Matchmaker, it expects to find a file of this name in the same directory. This file consists of the following parts:
Notice that if you either omit the database element or the Matchmaker doesn't manage to connect to the given database (this is what happened earlier when you used the configuration file template from the distribution), the Matchmaker will still run; it simply won't store things in a database.
Now let's look at the configuration file for the NLG server. The NLG server expects a command-line argument that specifies the configuration file. Because this file encodes information about your own system, you need to pay much more attention to the information in it than you did with the Matchmaker. The file contains the following information:
The planner element is optional -- if you don't specify one, the system uses the builtin lazyff planner by default. As of GIVE 2.2, the "lazyff" is the fastest planner for the GIVE-2 planning problems we are aware of. If you want to use SGPLAN, you can obtain a Linux binary at the IPC-2008 website; we can also give you access to a MacOS version of SGPLAN 5.2.2.
It is up to you how many simultaneous instances you want to allow. There is a tradeoff: If you allow many simultaneous instances, you will collect a lot of data; but it will also increase the load on your CPU, and in some cases the rest of your generation infrastructure may not allow multiple simultaneous instances at all. However, in any case you must ensure that the ports on which your server and its instances listen to connections p to p+k must be accessible from the outside. It may be necessary to get your sysadmin to open some holes in your firewall for this.
While it is by no means necessary, it may be useful for you to attach your local Matchmaker to a database to store previous game runs more permanently. If you want to do this, your database has to conform to a certain structure. We refer you to http://code.google.com/p/give2/wiki/MysqlDatabaseSchema for details.
The above instructions were all written with a Unix-based operating system (such as Linux or MacOS) in mind. There is no reason why you shouldn't be able to run a GIVE server on Windows. However, the scripts in the scripts directory are all Bash scripts, which you will only be able to use on Windows if you run them from Cygwin. Alternatively, you can write Windows batch files to achieve the same functionality.
At this point, you know how to set up and run your own copy of the GIVE system, using one of the demo servers from the NLG server distribution. We will now explain how to build your own NLG system, and then go into some details about what classes and methods you can use from within your NLG system in order to access the world state and the plan. You can find more details about any of these classes in the Javadocs.
In order to implement your own NLG system, you need to write a Java class which is a subclass of NlgSystem. This is an abstract base class with a number of abstract callback methods that you need to implement. These callback methods are explained in more detail below. Conversely, you can call NlgSystem's send method to send a string to be displayed in the client at any time you like. send and the other methods you can use from within your NLG system are also described below.
Your NLG system is inserted into an NLG server by specifying the full name of your NlgSystem-derived class in the NLG server configuration file, as described above. You should then build a Jar file that contains your entire NLG system including the NlgSystem-derived class and all the classes that make up the GIVE-2 NLG server infrastructure. This sounds complicated, but if you use Maven to build your NLG server, we have simplified the process in the following way.
A Maven project is generally defined by a POM file, typically with the name pom.xml.
If you look at the POM file for the Example NLG System from the NLG
server distribution, you will find that it has a section at the top
that looks as follows:
To compile your own NLG system, you can simply copy this POM file into your own project directory and adjust the group ID, artifact ID, name, and version as needed. You can then run mvn install assembly:assembly to compile an executable Jar file that you can run with "java -jar blah.jar your-nlg-config.xml". This Jar file will be in the target subdirectory, and have a name consisting of your group and artifact ID, project version, and the prefix "-jar-with-dependencies.jar".
The standard POM file should take care of all dependencies into the GIVE-2 code, and download GIVE packages, automatically. In particular, every time we release a new version of the GIVE server infrastructure, Maven will download the new version and recompile your system against it. In general, you don't need to make any changes below the line that starts "if you don't care about Maven". However, you may want to add your own project dependencies or Maven plug-in configuration options below that line as needed. Feel free to do this, as long as the give2-* dependencies stay intact.
Notice that Maven requires you to follow a certain convention about the structure of your source code directories. This is explained here. If you cannot use Maven for some reason, please contact us, and we can work out a way to get you started anyway.
States of the virtual world are represented by objects of the class give.world.World. Your NLG system class inherits a method getWorld() which can be used to retrieve the current world object. This object is automatically updated when the user moves around or manipulates objects. You can query the world state by calling methods of World. In particular, you can retrieve all objects in the world by calling getUniverse; the type (button, wall, room, etc.), state, position, orientation, and attributes of each object by calling getType, getState, getPosition, getOrientation, and getAttributes; the player's position and orientation by calling getPlayerPosition and getPlayerOrientation; and the complete list of all known true atoms by calling getTrueAtoms. See the description of the GIVE game worlds for details.
Notice that although World has methods that change the world state, it makes no sense for you to call them. Such changes are not propagated back to the client, and would thus destroy the synchronization between the world state that the client (and thus the user) and your system assume.
Positions and orientations are represented in the world state as 3D floating-point coordinates. The position represents the center of the object. The orientation of an object, for objects where this makes sense, is the normal vector of the side of the object that faces the player when the player interacts with the object. For wall-like objects (doors, walls, safes), these are the sides that are visible to the player. To give you a sense of the dimensions of the world, the width of a door is 1.
You can use the 3D coordinates in the current world state as you see fit. However, a planner will typically not be able to process 3D coordinates; it only understands discrete logical constants. To bridge this gap, your class can access a discretizer by calling getDiscretizer(). The purpose of a discretizer is to translate back and forth between 3D coordinates and discrete names for locations and orientations.
For the current challenge, we have provided the class RegionDiscretizer. This discretizer splits the world into discrete regions according to the following rules:
You can ask the discretizer object to map 3D positions into region names. You can also ask it about adjacency relations between regions, and retrieve a RegionInfo object that gives you information about a certain region. See the Javadocs for details.
Your NLG system reacts to events in the virtual world by implementing several callback methods. If you look at the API of the class NlgSystem, you will see that this is an abstract class with five abstract methods. You must implement each of these methods. They will then be called by the rest of the GIVE server infrastructure at appropriate moments.
In detail, the role of these methods is as follows:
In all these methods except connectionDisconnected, you can throw an NlgServerException if an error occurs in your code. When you throw an exception, it is logged in the database, and the NLG server instance is then shut down in a controlled fashion.
Initialization code for your class can go either into connectionEstablished or into the constructor. You must define a public constructor that takes exactly three parameters, of types NlgServerInstance, World, and Discretizer. It must then pass these arguments on to the super constructor with the same parameters. You are welcome to do more in your constructor. However, you should be aware that the world and discretizer fields are not yet initialized at this point, and there is no network connection to the client yet. This means that any initialization code that depends on the virtual world belongs into connectionEstablished instead.
A new server instance object is created every time the NLG server serves a new client instance. This leads to the creation of a new object of your NLG system class, and therefore to a call to the constructor. In addition, the constructor is also called once in the very beginning, when the NLG server starts up; this call is distinguished by the fact that all arguments are null. If you want to perform expensive initialization code for your NLG system -- e.g., reading and preprocessing a grammar --, you can do this in this initial call to the constructor, and store your results in static fields which can then be accessed by each new instance of your class.
There are several utility methods which you can use in your NLG system.
The user moves around in and interacts with the virtual environment by means of actions. Whenever they execute an action by clicking on some object, such as pushing a button or picking up the trophy, the handleAction method of your NLG system gets called, as described above; it receives an instance of the action header as its first argument. In this instance, any parameters that refer to the positions of objects or the player in the world are instantiated with the exact 3D coordinates of this object, as an object of class Position. For instance, pushing a button is represented by the manipulate action. If the player clicks on the button b1, changing its state from "off" to "on" while the player's position is (1,0,3.5), then the handleAction method will be passed the term manipulate(b1,off,on,(1,0,3.5)) as its first argument.
The planner uses the same actions to compute a plan that the user should execute to achieve the goal. However, there are two important differences between the actions in the planner and the actions the user executes in the client. First, as we have already described above, the planner cannot deal directly with continuous 3D positions; it has to run a discretizer to split the map into discrete, disjoint rectangular regions, and it will only reason with these rectangular regions instead of the 3D positions. In the example above, assume that the position (1,0,3.5) is in the region with name "reg_room1_b1". Then the planner would compute a plan that contains the action instance manipulate(b1,off,on,reg_room1_b1). You can use the discretizer to map 3D positions to region names at any time.
The second difference is that the planner uses an action move to represent that the player moved from one region into another. This action has no analogue in the action the player actually performs in the 3D client: From the player's perspective, they can simply walk around in the world, and from the NLG system's perspective, player movements do not trigger handleAction calls. The way to find out about player movements in the NLG system is to inspect the current player position from within calls to handleStatusInformation; if you like, you can again discretize the player position to a region name and compare whether the player has entered the region that the plan specified.
The full list of actions is described as part of the specification of GIVE worlds.
Your NLG system will operate under rather stringent performance conditions. On the one hand, you can't take too much time between deciding that you want to say something and actually sending the message, because the user may turn or walk away while you are still generating. On the other hand, if we are as successful as we hope in recruiting experimental subjects online, you can expect many, perhaps simultaneous connections that will place a high computational load on your machine. You should take these factors into account while you're implementing your system.
One particular issue you should consider is splitting up your computation into separate threads. At the moment, all callback methods of your NLG system run in the same thread. This means that while your system is still busy reacting to one callback method, it won't process the next. If you have an expensive computation -- which, say, takes more than a second --, you should consider executing this computation in a new thread. The relevant methods in NlgSystem are thread-safe, so you shouldn't have to worry about concurrent access to send messing up the internal state of the server. However, you must be aware that exceptions that you throw in the new thread are not caught and logged by the GIVE infrastructure. Furthermore, runtime measurements are per-thread, i.e. in order to obtain meaningful runtime measurements, you must call logStartComputation and logEndComputation in the same thread that executes the computation.
When the server wants to shut down a server instance for any reason (e.g., because the user cancelled the client), it interrupts the message handling thread in which your callback methods run and releases the server instance. (See the documentation on Java threads to learn about interrupts.) At this point, new connections are accepted. This means that you want to check for isInterrupted frequently, and exit your callback method quickly if you discover the interrupt; otherwise a thread from an old instance of your NLG system might still continue running while the server has already spawned the next instance. It is your responsibility to react to interrupts in a correct and timely fashion.
One final caveat: Don't call System.exit in your code! This will terminate the entire server, and not just your instance. The correct way to terminate your instance is to throw an NlgServerException. This will cause the NLG server to shut your instance down in a controlled way, and will also log the error message you specify in the exception in the database for later analysis.
The GIVE infrastructure assumes that your NLG system can be
encapsulated in a Java class. This means that at least some part of
your system must be implemented in a language that compiles to Java
bytecode, such as Java, Scala, or Groovy. We are aware that many
participants have extensive NLG software which they want to use
although it isn't written in Java. In such cases, we recommend you
implement a thin Java (or Scala, or Groovy) wrapper that plugs into
the GIVE system as intended and then interfaces with the rest of your
generation system on the backend, e.g. through a socket. In this
case, it may be convenient for you to restrict the number of
simultaneous connections to one.