Communication Protocol

This page is a complete specification of client-server communications in the DataNav portal web application. It describes how the DataNav client-side apps -- DataNav Builder and DataNav Viewer -- send requests to DataNav Server over the HTTPS and HTTP protocols and defines the required format for the payload of each request and its commensurate response. With near code-level details, it is intended only for a developer audience. Portal authors and administrators can skip this entirely!

Access to the portal comes in two flavors: anonymous read-only access over HTTP, or authenticated read-write access secured over HTTPS. In both cases, the MIME content type is JavaScript Object Notation, or JSON (application/json). Builder supports both types of client-server communications, so you can explore a portal anonymously if you wish, or log into any portal on which you are a registered user. DataNav Viewer, as the name implies, offers only anonymous read access to the portal.

Regardless the type of request, the request and response payload are each packaged as a single JSON object. All request payloads include the req field, which holds the request ID as a 32-bit integer. Furthermore, any request requiring authenticated access includes the handle field, a positive 32-bit integer user handle which was assigned to the client at logon. The remaining payload content is request-specific and is detailed in the individual request-response descriptions below; request ID codes are listed in parentheses.

A normal response will include, at a minimum, a req field reflecting the original request ID and a result field set to the integer result code. In addition, any request that retrieves or modifies portal content will include a mod field specifying the 32-bit integer modification stamp for the portal's hub vault. Any time a change is made to portal content, this modification stamp is incremented. It provides a mechanism by which the client can detect whenever a change in the portal's content has occurred, rendering the client's cached content stale.

Additional fields in the response object vary with the request type and are again described in the comments for each request type. A failed response will have three or four fields: req, result, emsg, and mod.  The result field holds an integer result code other than 1 ("ok"), and emsg is a string describing why the request failed. The mod field is included only in content-related requests; if the response failed due to a timeout or a connection failure, this field is set to -1 and should be ignored.

Regardless the level of access, all client-server communications follow the usual request-response paradigm. The client app sends a payload as an HTTP or HTTPS POST request to DataNav Server, which parses the request payload, retrieves and/or modifies content in the portal's backing store as needed, then prepares and returns a response. The remainder of this page explains the purpose of each type of client request recognized by Server, along with the format of the request payload and the expected server response.

Client requests targeting the portal's registered users list

All client requests described in this section require authenticated access to the portal; they are sent over HTTPS and are issued by the DataNav Builder client only. Selected requests require administrative-level access.

LOGIN (request code = 100): Request to log onto portal.

An author must log onto the portal before s/he can mount or unmount an archive, toggle the archive's public/private flag, checkout an archive for revision, etcetera. An administrator must login to make any changes to the portal's registered users list. This request is sent to initiate a log-in "session". DataNav Server keeps track of the user handles for all log-in sessions; any HTTPS client request (other than the LOGIN request) lacking an active user handle will be rejected. 

LOGOUT (101): Request to log out of portal.

There is no additional request or response payload. Of course, the user handle in the request payload must identify a user that is currently logged onto the portal.

GETUSERS (102): Get the list of all registered portal users

DataNav Builder issues this request after a user logs onto a portal and parses the response payload to populate its summary of registered portal users. Users can check this list to see if another author is currently logged into the portal. An administrator will consult it when adding or removing users, or changing a user's access level.

ADDUSER (103): Add a registered user to the portal.

This request will fail if the new user's name is already taken, or if the supplied request data is invalid. It will also fail if the user issuing the request lacks administrator-level access. Only a portal administrator can add a new user to the portal.

RMVUSER (104): Remove a registered user from the online portal.

Only a portal administrator can remove a registered user from the portal. If that user is logged in at the time of removal, s/he is automatically logged out by DataNav Server -- so the next client request from that user will be rejected. Any portal archive that was owned by the removed user is automatically transferred to an archive collaborator or, if the archive's collaborator list is empty, to the administrator that issued this request.

ROLEUSER (105): Change the access level of an existing registered user.

The request will fail if the issuer lacks administrative-level privileges on the portal, or if the request data is invalid. 

PWDUSER (106): Change the password of the requesting user.

INFOUSER (107): Change the optional contact information of the requesting user.

Client requests that retrieve portal content

All client requests described in this section may be sent anonymously over HTTP or while logged in via HTTPS. These are the only requests issued by the DataNav Viewer client. When sent over HTTPS (DataNav Builder only), each request must include a valid user login handle in the handle field. When sent over HTTP (by Viewer or Builder), the handle field is still present but is set to -1 and is ignored by DataNav Server.

CHECKVER (10): Check client-server version compatibility.

This request is issued anonymously over HTTP by both the Viewer and Builder clients prior to attempting any other communication exchanges with DataNav Server. It is sent simply to verify that the client and server are both using the same version of the communication protocol. If your version of Builder is incompatible with the portal's current Server installation, then this request will fail with a result code of -13. Since Viewer is a Javascript-based web app that is downloaded from the portal server in the first place, it should always be compatible with the server (however, if your browser caches JS files, you might see a compatibility error). The current version number for the DataNav client-server communication protocol is 4 (as of app version 4.6.0).

GETVAULT (request code = 200): Get the portal's archive vault structure -- a skeletal summary of portal content that

includes everything a client needs in order to retrieve that content and present it to the end user.

The archives field in the response is a JSON array of length N, where N is the number of accessible archives in the portal. Each element, in turn, is a JSON object encapsulating a single archive's structural outline. For a figure archive, the object contains the following fields:

When the GETVAULT request is sent anonymously, the "meta" field contains only the first three elements. When sent by an authenticated client over HTTPS, it will contain the first 5 elements unless the archive is currently checked out of the portal for revision, in which case all 7 elements are present.

For a data archive ("hub"), the JSON object instead contains these fields:

Note that, when the GETVAULT request is sent by an anonymous client, then the response will only include those archives in the portal's vault that are marked as public -- an anonymous client does not have access to private archives. 

GETARCHINFO (201): Get textual summary information for a specified archive in the portal.

GETVIEWDEF (202): Retrieve the definition of a navigation view for a specified data hub in the portal.

The title and desc fields in the response hold the navigation view's title and HTML-formatted description, respectively. The fyp field is a JSON-formatted version of the FypML markup defining the view's template figure; it conforms to the latest schema for DataNav figures. Sending the template in JSON format makes parsing easier in the Javascript-based DataNav Viewer client.

The groups field is a JSON array of 0-4 JSON objects representing the view's instance configuration groups. Each such object has the following format: {name: string, blk: int32, attrs: JSONArray, ph: JSONArray}. The name field is the configuration group's reader-facing name, blk is its iteration block size, and attrs is an array of 1-6 strings listing the group's "attributes" or "search tags". The ph field is a mixed JSON array [id1, fmt1, iter1, ... idN, fmtN, iterN] containing the triplet (id, fmt, iter) for each template placeholder set assigned to the configuration group. Here id is the placeholder set's ID within the view template (a string), fmt is the data set format code (int32), and iter is an integer flag indicating whether or not data injected into the placeholder participates in an iteration over collection-type data (non-zero == iterable).

GETVIEWINSTANCES (203): Retrieve the entire instance list for a navigation view within a specified data hub.

Each entry in the instances array in a mixed-content JSON array defining a single view instance [g, M, P, av1, ..., avM, duid1, ..., duidP], where:

GETDATASETS (204): Retrieve one or more data sets from a portal hub's internal data repository.

For each data set successfully retrieved, the mixed-content JSON array in the datasets field contains the UID of the data set (int32), followed by the data set itself packaged as a base64-encoded binary string. Data sets are cached on the client side to the extent possible to minimize the number of client-server exchanges when a user is browsing hub data. Coordinate range information is included in the base64 string so that, when large data sets are decoded on the client side, it is not necessary to traverse the data to recompute that information.

The order of datasets retrieved may NOT match the order of the UIDs in the request field duids. That is why the UID of each retrieved set is included in the response. Also, if the download size is too large, DataNav Server will send back only 1 <= M < N data sets. In this case, the client must issue an additional GETDATASETS request for the remaining sets.

GETARCHFIG (205): Retrieve a member figure from the specified figure archive in the portal.

The title and desc fields in the response hold the archived figure's title and HTML-formatted description, respectively. Together, these serve as a legend when the figure is presented in the client's user interface. The fyp field is a JSON-formatted version of the FypML markup defining the requested figure; again, it conforms to the latest schema for DataNav figures.

Client requests that modify, upload, or download portal content

All client requests described in this section require authenticated access to the portal; they are sent over HTTPS and are issued by the DataNav Builder client only. These constitute the only means by which a portal's content can be changed.

SETARCHPUB (request code = 206): Change the public/private flag for an existing archive in the portal. 

This request will fail if the requesting user (identified by the login handle in the request payload) is not the designated owner of the specified archive.

The portal vault modification stamps both before and after the change (oldmod and mod, respectively) are included in the response so that the client that requested the change will be able to determine if any other portal update occurred after the request was issued but before it was actually executed on the server. The client compares the "before" stamp with its cached copy; if they are not the same, then some other change has occurred.

CHECKOUTARCH (request code = 207): Checkout an existing portal archive in order to make revisions to it. 

The requesting user must be the archive owner or one of its designated collaborators. If the request succeeds, the user gets a checkout lock on the archive; no one else can checkout the archive until that user checks in the revisions. The portal server makes a private copy of the archive, and all revisions are applied to that copy. The requesting user "sees" and edits the copy, while everyone else sees the current "live" version -- that is, the archive state prior to checkout. On check-in, the updated copy replaces the current version, and the checkout lock is released.

The client must check the vault modification stamp in the server's response against its cached copy to see if the portal's content has changed since the client issued the checkout request. If they don't match, the client must refresh its cached content. Otherwise, the client should mark the subject archive as checked out by the logged-in user that sent this request and cache the checkout time included in the response.

CHECKINARCH (request code = 208): Check in an existing portal archive that was previously checked out for revision. The private internal copy to which all revisions were applied replaces the current "live" version of the archive, and the checkout lock on the archive is released. 

The server will deny the request if the requesting user (as identified by the handle field) does not hold the checkout lock on the specified archive. As with SETARCHPUB, the portal vault modification stamps both before and after the check-in request was completed on the server (oldmod and mod, respectively) are included in the response so that the client that issued the request will be able to determine if any other portal update occurred while waiting for the server to respond.

EDITARCH (request code = 209): Submit a revision to an existing portal archive that is currently checked out by the requesting user. 

The server will deny the request if the requesting user (as identified by the handle field) does not hold the checkout lock on the specified archive. Also note that this request, if successfully fulfilled by the server, does NOT actually change the portal's content (its modification stamp is unchanged). This is because all revisions are applied to a private internal copy of the checked-out archive, which is accessible only to the user holding the checkout lock. The archive is updated -- and the portal's modification stamp incremented -- only when the revisions are checked back into the portal.

[NOTE: An archive revision which adds a data instance to a hub navigation view may be split over a sequence of consecutive EDITARCH client requests if the size of the data comprising the instance exceeds a certain "payload size threshold". In this scenario, the retval field is -1 for each successful request-response exchange except the last, which will hold the view instance just added to the private copy of the data hub on the portal. If any request-response exchange in the sequence fails, the client can assume the instance was NOT added to the hub; if the portal has not crashed, the client will have to re-send the entire sequence to try again.]

ARCHMETA (request code = 210): Get current meta-data for portal archives, or change an archive's owner or collaborator list.

Only an archive owner may transfer ownership to another user or add/remove archive collaborators. Thus, the server will deny the request if type > 0 and the requesting user (as identified by the handle field) is not the owner of the subject archive.

XFER_BEGIN (220): Begin a multi-request sequence that will transfer an entire archive (either data or figure archive) from the client to the remote portal or vice versa.

Since a typical archive (especially a data hub) will be quite large, it will take a relatively long time and many request-response exchanges to transfer an entire archive to or from a remote portal. There are two use cases involving an archive transfer:

If a problem occurs on the client side during either of these archive transfer operations, or if the user elects to cancel a transfer in progress, the client issues the XFER_CANCEL request to inform the portal server.

XFER_CHUNK (221): Transfer a file chunk from the ZIP file containing the backing store files for the archive that is currently being transferred to or from the remote portal.

Regardless the direction, the Builder client controls the progress of the archive transfer, "pushing" content to the portal or pulling content from it. Obviously, the chunks are streamed sequentially. If any chunk transfer request fails, then the transfer operation is automatically aborted on both the server and client sides.

XFER_CANCEL (222): Cancel an archive transfer operation in progress.

This request may be sent at any point after XFER_BEGIN, but before XFER_FINISH. If an archive mount or unmount operation is cancelled, there's no change to the portal.

XFER_FINISH (223): Complete an archive transfer operation in progress. The portal server's response depends on the type of transfer operation; see the description of the XFER_BEGIN request for details.

NOTE that this is the only request in the archive transfer sequence that actually alters the persisted content of the portal. As with SETARCHPUB,  the portal vault modification stamps both before and after the transfer operation was completed on the server (oldmod and mod, respectively) are included in the response so that the client that performed the transfer will be able to determine if any other portal update occurred during the transfer. This could easily happen, since archive transfers can take a significant amount of time to complete.

Server response result codes

Listed below are the various 32-bit integer result codes that may be sent in DataNav Server's response (the result field of the JSON response object) to a client request.