Hibbard, W. Building 3-D User Interface Components Using a Visualization Library. Computer Graphics 36(1), 4-7. 2002.
Building 3-D User Interface Components Using a Visualization Library
Bill Hibbard
University of Wisconsin - Madison
We usually think of visualization libraries and graphical user interface (GUI) libraries as two different things. GUI components have very stylized appearances, represent simple data as opposed to the complex data in typical visualizations, and emphasize information flow from user to computer as opposed to the flow from computer to user in visualizations. However, both GUI and visualization libraries are about visual communication between user and computer, and we can think of GUI components as visualizations of simple data that emphasize user interaction. For example, the slider in Figure 1 is a visualization of a real number, and the color table editor in Figure 2 is a visualization of three real functions (mapping pixel values to red, green and blue color components). This column describes an approach to building 3-D user interface components using the VisAD interactive visualization library. This is an open source system freely available from http://www.ssec.wisc.edu/~billh/visad.html.
Procedural versus Descriptive Visualization Libraries
VisAD is different from other visualization libraries. Most libraries define method or function calls that applications embed in procedures for explicitly constructing visual depictions of data. These library functions typically implement algorithms that compute isosurfaces and other geometry from data arrays, and algorithms for rendering such geometry into display frames. In contrast, VisAD defines Java class constructors and methods for describing data and display components via metadata that document their relation to world. Application procedures construct data and display components but do not explicitly construct data depictions. Generation of data depictions is done implicitly by the system when data components are linked to display components, based on analyses of descriptive metadata. The primary metadata of data are data schemas that define primitive type names (e.g., time, temperature, latitude) and units (e.g., seconds since 1 January 1970, degrees Kelvin, meters) for all numerical and text values in data components, plus the way that primitive values are grouped (e.g., latitude, longitude and altitude may be grouped as a 3-D vector), and functional dependencies between primitive types (e.g., temperature as a function of 3-D earth location). The primary metadata of displays are mappings from primitive data types to primitive display types such as animation, display spatial coordinates, display color coordinates and isocontouring. The combination of data schemas and display mappings of primitive types is sufficient for display components to automate the generation of data depictions.
Invertibility of Visualization Operations
In procedural visualization libraries where application programmers are free to specify arbitrary sequences or networks of operations transforming data into displays, it is difficult for the system to invert the sequence or network of operations in order to translate user gestures in displays back into operations on data. This problem is well known among the developers of data flow visualization systems.
However, in a system where display generation is based on an analysis of data and display metadata, the system can use that same analysis to define inverse generation of data changes from user gestures in data depictions. This is the basis for using VisAD to define 3-D GUI components.
One important element of GUI libraries is their ability to asynchronously notify applications of user gestures via callback methods. VisAD fills this need by providing a way to define methods that are invoked whenever selected data components are modified, as they may be in response to user gestures in data depictions.
Basic Component Types
A description of how to build GUI components as visualizations will require a more detailed understanding of VisAD. The system defines five basic kinds of components:
1. Data components: these may be simple real numbers, may be text strings, may be vectors of reals or other values, may be sets in real vector spaces, may be functions from real vector spaces to other data spaces, or may be complex combinations of these. They are mostly immutable in order to promote thread-safeness.
2. Display components: these contain visual depictions of one or more linked Data components. These may be 2-D or 3-D, may be a window on the screen or in a browser, or may be in an immersive virtual reality system. Display components update data depictions in response to changes in linked Data components.
3. Computational components: these execute user-defined methods in response to changes in linked Data components.
4. DataReference components: these are mutable components used to connect Display and Computational components to Data components, which are often immutable. In "X = 3" the number 3 is immutable and plays the Data role, whereas X is mutable (i.e., it can be changed to a value other than 3) and plays the DataReference role.
5. User interface components: these are traditional GUI widgets typically linked to Data or Display components, and should not be confused with the subject of this column.
Display and Computational components can be linked with DataReference (and hence with referenced Data) components on other machines, via Java RMI (Remote Method Invocation) distributed object technology. This facilitates collaborative visualization, in which a DataReference component on one machine is linked to Display components on the geographically distributed machines of multiple users. RMI is also used to enable applications to link groups of Display components on different machines, so changes in any are reflected in all.
Metadata for Data Components
Data components contain numerical and text values, plus metadata. The primary metadata is a data schema that defines names for primitive numerical and text values occurring in data, the way values are grouped into vectors, and functional dependencies among values. For example, a satellite image of Earth may be described as a functional dependence of radiance on pixel line and element coordinates, via the schema:
((line, element) -> radiance)
This function is approximated by a finite sampling at discrete pixels. The sampling metadata of a function may be a regular or irregular set in a real vector space. This function may also include metadata describing the Earth locations of pixels via an invertible coordinate transform:
(line, element) <--> (latitude, longitude)
Any real values may include units. For example latitude and longitude values may have units of degrees or radians. Function range values such as radiance may include metadata indicating missing values (caused by instrument or computational failures), or metadata defining estimates of errors.
A time sequence of images may have the schema:
(time -> ((line, element) -> radiance))
This function will define some finite sampling of time values, and may define units for time such as seconds since 1 January 1970.
A set of map boundaries may be described using the schema:
set(latitude, longitude)
VisAD defines many different classes for sets in real vector spaces, for regular and irregular topologies, for different domain dimensions, and for sets restricted to submanifolds with smaller dimension than their domains. For example, a set of map outline samples lie in a one-dimensional submanifold of a two-dimensional domain.
Metadata for Display Components
The depictions of Data components linked to a Display component are defined by a set of ScalarMap objects linked to the Display. These are mappings from primitive real and text types to what are called DisplayRealTypes. For example, the depiction of a time sequence of images and a map boundary overlay in Figure 3 is determined by the ScalarMaps:
time -> Animation
latitude -> YAxis
longitude -> XAxis
radiance -> RGB
Note the conventional GUI widgets in Figure 3 that allow the user to control time animation and the RGB color lookup table for radiance values. Each ScalarMap object has an associated Control object that provides a means to specify animation, color tables, contouring, flow rendering, 3-D to 2-D projection, etc. These Control objects can be linked to GUI widgets as in Figure 3, or may be manipulated by computations.
ScalarMaps for some DisplayRealTypes (e.g., XAxis, RGB) allow applications to control the linear mapping from primitive real data values to DisplayRealType values. If applications don't specify this mapping, then a system autoscaling algorithm determines an optimal default mapping to keep data depictions visible (e.g., to ensure that longitude and latitude values are mapped to XAxis and Yaxis values that are within the display screen).
ConstantMaps, which bind constant values to DisplayRealTypes, may be linked to Display components in just the way that ScalarMaps are. These allow applications to override default values for DisplayRealTypes, for example to control locations and colors of data depictions when they are not determined by ScalarMaps of any primitive data values.
A Simple 3-D GUI Component
Although the generation of data depictions is automated based on an analysis of data and display metadata, the system provides a way for applications to redefine that analysis and display generation. When a Data component is linked to a Display component, an object of a subclass of DataRenderer is used to analyze metadata and generate the depiction. There is a default subclass for each supported graphics API (e.g., Java3D and Java2D) but applications have the option of defining and using non-default subclasses. The default DataRenderer subclasses do not translate user gestures into data changes, because in the general case of data schemas and display ScalarMaps there may be no reasonable way to interpret user gestures as data changes.
The system includes a number of non-default DataRenderer subclasses that do translate user gestures into data changes. These first analyze a data schema and a set of ScalarMaps to make sure they are consistent with an interpretation of user gestures as data changes, and then implement that interpretation. For example, a Data component with schema:
(latitude, longitude, altitude)
and a Display component with linked ScalarMaps:
latitude -> YAxis
longitude -> XAxis
altitude -> ZAxis
will generate a data depiction as a simple point in 3-D display space where the user can modify data values by dragging the point. An analysis by an object of the DirectManipulationRendererJ3D class verifies and implements this way of interpreting gestures.
Figure 4 shows a display of a simple conical terrain surface in a 3-D box, with two large yellow points at opposite corners of the box. The yellow points are depictions of two 3 vector Data components linked to the Display component via objects of class DirectManipulationRendererJ3D, as described in the previous paragraph. These 3 vectors are linked to trigger a Computational component that modifies the linear mappings associated with the ScalarMaps for display spatial coordinates in order to keep the depictions of the two 3 vectors at the corners of the display box. This little network of Data, Display and Computational components and DirectManipulationRendererJ3D objects defines an embeded 3-D GUI component for rescaling 3-D display space.
Adding a ScalarMap:
latitude -> Shape
enables the application to define arbitrary shapes (via the Control associated with this ScalarMap) for the depictions of the two draggable points, in order to create custom cursors at the 3-D display box corners as illustrated in Figure 5. Note that although some primitive data type such as latitude had to be chosen for this ScalarMap, its Control is set in such a way that shape does not vary with latitude.
A More Complex 3-D GUI Component
In spatial data analysis applications it is often useful to apply analysis operations to restricted spatial regions. These regions may be defined in the data, for example within a political boundary, or may be defined by users based on their judgement. For this we need a GUI component that enables users to draw the outlines of regions as freehand curves. In VisAD, the CurveManipulationRendererJ3D class serves this purpose. It is a subclass of DataRenderer that requires a Data component with schema with the form:
Set(x, y)
It also requires ScalarMaps of x and y to spatial DisplayRealTypes. These may be two of XAxis, Yaxis and Zaxis, or they may be two coordinates in a non-Cartesian spatial coordinate system. The Data component will lie on a one-dimensional manifold embedded in the two-dimensional domain with coordinates x and y. User mouse movements are interpreted as samples along one-dimensional curves in (x, y) space. According to the ScalarMaps, the curve is embedded on a two-dimensional manifold in 3-D display space. Figure 6 is a snapshot of a curve being drawn on the two-dimensional manifold on the surface of a sphere. In this case, the ScalarMaps are:
x -> Longitude
y -> Latitude
Along with Radius these DisplayRealTypes define a 3-D spherical display coordinate system:
(Latitude, Longitude, Radius)
When a DataReference object is linked to a Display component, a number of ConstantMaps may be included which are applied only to the depiction of referenced Data component. In the example in Figure 6, a ConstantMap to Radius is used to specify which sphere defines the 2-D manifold where curves are drawn.
Applications can use CurveManipulationRendererJ3D for drawing on a nearly arbitrary 2-D submanifold of 3-D display space, by defining three DisplayRealTypes for a new coordinate system and defining a coordinate transform between these and Cartesian display coordinates. The 2-D submanifold is defined by a ConstantMap that fixes the value of one of these DisplayRealTypes, and by ScalarMaps of x and y to the other two. This 3-D GUI component can be used for freehand drawing in a wide variety of different applications.
Conclusion
The point of this column is that a visualization library can be as much about the flow of information from user to computer as the flow from computer to user, and hence can be used for defining 3-D GUI components. The key is that the library be a descriptive rather than procedural definition of visualization operations. In a descriptive library, the system derives the procedure for generating data depictions from an analysis of the description supplied by the application. It can just as easily derive a procedure for modifying data from user gestures on data depictions.