Data File Format
Whether recorded in Trial mode or Continuous mode, the data collected by Maestro is stored in a binary file as a series of 1KB records. The first record in the file is a header containing important summary data: the set of analog data channels that were saved, the total number of samples recorded on each of these channels, file version number, date, and other information. The remaining records in the file are either data records storing the recorded analog and digital data, or information records describing the stimuli that were presented when the data was recorded. Each such record type is identified by a unique "tag" in the first byte of the record. Analog data is compressed to reduce file size.
In addition to these Maestro-generated records, the in-house analysis program JMWork may append a number of different analysis action records onto the end of the data file that encapsulate any analysis actions taken by the user within the context of that program. For example, JMWork is commonly used to cut saccades from the behavioral data traces or place markers in the recorded timeline. By encoding such actions in a distinct record appended to the original data file, the effect of the action is attached to the data without altering its raw form. Furthermore, the next time the data file is loaded into JMWork, the actions can be automatically reproduced without user interaction.
While the format of the binary data file has changed over time, the changes have been carefully designed to avoid breaking offline analysis programs that need to read in and process the data files. For example, Maestro essentially inherited the data file format of its predecessor Cntrlx, then augmented that format by adding data fields to the unused portion of the header record and by creating new record types. While existing analysis code will not be able to access the new header fields and will ignore the new record types, it can still process the rest of the data file without failing. Analysis tools maintained by the Lisberger lab and described in this online guide are always updated as the data file format changes.
The pages in this section of the guide offer a detailed description of the Maestro data file format, including a history of the changes that have been made as the program has evolved. We describe each of the different file record types and list the actual C++ code that defines the records' data structure. We also outline the algorithm used to compress analog data and offer a code snippet that handles the decompression task. With this information, programmers should be able to write their own code to read in and parse data files generated by any version of Maestro or Cntrlx.
Relevant code files
Much of the information contained in this section is extracted from several C/C++ source files in Maestro's code base. These files define data structures and related constants that precisely layout the native format of the Maestro data file. For those willing to write their own code, these files are a must and are available for download:
cxfilefmt.h -- Defines a data structure for the header record, a generic data structure for handling any of the other types of data records, and related constants.
cxtrialcodes.h -- Lists all of the supported (and obsolete) Maestro trial codes. Data files recorded in Trial mode include one or more trial code records listing the sequence of codes defining the trial presented. Analysis programs can process the trial code records in order to reconstruct the trajectories of targets participating in the trial.
cxobj_ifc.h -- This file contains many constants and data structures for Maestro objects such as targets, trials, and stimulus runs. Included are several definitions upon which the data structures defined in cxfilefmt.h depend.
rmvideo_common.h -- This file contains those constants and data structures shared by Maestro and its framebuffer video server, RMVideo. The RMVTGTDEF structure encapsulates the parameters defining any RMVideo target, and it is part of the CXFILETGT structure which defines the format of target records in a Maestro data file (version >= 8).