Adding custom data to the job database

Custom data API methods allow a plugin to add its own information to a Survey Core job database.  This data may be exported as part of the job in JXL format, appear in the job review window, and be accessed and modified by the plugin.

What type of data?

All custom data that may be added to the database must be provided as XML using the tsc_XmlElement class.  This class holds an XML tree in an already-parsed form, which reduces the amount of processing required to access XML elements or to modify them.  Any data type may be represented, except binary.  While binary data can be encoded into some sort of character form, it is discouraged since the SC database is not optimised to deal with high volumes.

When a job is exported in JXL format, the custom data which has been added to the database will also be exported.  The custom XML will be embedded within the job XML at the locations where it was written by the plugin.  The JXL schema allows such XML nodes to contain "anything"; this means that schema validation is not performed for the content of custom XML data.

Three sorts of custom data record

The custom data may be added to the database in three different ways.  Each way is distinctly different from the others, and the difference is important in order to get the correct data design.  The following three ways are provided:

• • A "lone" database record, known as a Custom Record.  This is the simplest form and is stored chronologically.

  • A named entity (much like a Point) which supports lists, searching, and may be filtered. This is known as a Custom Entity.

  • Like the custom record, but attached to some existing entity such as a point.  This is known as an Custom Sub-record.

These names may be a little confusing, however they describe the same type of custom record which is treated a little differently depending on how it was originally added to the database.

Allowable operations

A Custom Record may only be appended to the database, in chronological sequence along with all other records.  Such records may be difficult to find again without significant (time consuming) sequential searching.  If the record's ID is known, it may be read directly, however the ID would have to be stored somewhere else - but this should be avoided because if the job is upgraded to a new SC version or exported and imported, the record IDs will often change.

A Custom Entity has a string key (that is, a name). A custom entity is stored to the database and indexed with it's key, which allows fast and easy retrieval by name. It is possible to obtain a list of custom entities, either by name or all entities of a given type (more about types later).  The key is not unique - multiple items may have the same key.  Custom entities may also be deleted and replaced.

A Custom Sub-record is added to an existing entity (such as a point or arc) and has no key, however its parent entity does have a name and is able to provide a list of the custom sub-records that have been attached to it.  The sub-records can be read, modified, or deleted from the parent entity.

Types and data separation

To prevent interference between the custom records of different plugins running at the same time, all custom data is classified by custom type.  The custom type identifies the record type by a name (for instance, "GiegerCounter"), a GUID to uniquely identify the type, and a version number.  The version number may be used for any purpose; it is ignored by Survey Core.  Additionally, the type identifies the name of the key for custom entities, and some other options such as whether the item is to be displayed in the Survey Core Job Review window.

A plugin may generate as many types as it requires.  It is expected that a different type is programmed for each different XML structure (ie, schema) that may be written to the database. The type also provides serialization and deserialization methods, so that a custom record may be defined as a class with member variables containing the values to be stored.

If a GUID is not provided by the plugin, then the plugin's application GUID is used instead.  This is adequate for most purposes.

Sharing custom data between plugins

If two or more plugins wish to share custom data records, they may do so by specifying the same GUID and name in the type.  The type's name and GUID uniquely identify the type and allow a plugin to access any custom record, provided the type information is known.

Custom database records and attached files

Custom database records can only be used in the main job database, and not in any attached file which may be selected in the Layer manager. Record snapshot lists, which generally do include records from attached files, will not contain custom records from a job which may have been attached by the user.

Design decisions : some use cases

The way the three different custom database records are used is dependent on the kind of data being stored.  Below are some examples of how various types of data may be handled:

• • Additional information attached to a stored point record, such as the level of radioactivity at the point.

    • • Use custom sub-records attached to the point.

      • Add one (or more) custom sub-records to the point, containing the radioactivity level and additional information about the measurement.

      • These values will be exported in a JXL file within the point to which they were attached.

      • The radioactivity levels may be read back in from the tsc_Point entity.

  • Logging details about equipment being used, such as the type and serial number of a geiger counter.

    • • Append a custom record to the database containing this information. 

      • Such a record will appear in an exported JXL file, which could be processed by specialised software.

  • Including information about each place visited, as a kind of header for a set of measurements.

    • • Add a custom entity to the database.

      • The entity will require a key - perhaps the place name (Chernobyl, Fukushima, etc).

      • Other relevant information about the place is included in the XML.

      • The information can be retrieved by the plugin supplying the place name as a key.

  • Adding configuration information, such as the setup or connection parameters for the geiger counter.

    • • Add a custom entity to the database.

      • The name (key) of the entity might be something like "GeigerConfig".

      • The xml should contain all information relating to the setup.

      • A list of all GeigerConfig entities may be obtained, which will be in the order they were originally written.

Classes and methods

Custom Records

• • tsc_DatabaseCustomType  Describes one custom record type.  All custom records must be subclassed from tsc_DatabaseCustomType.

  • tsc_DatabaseCustomXml   Represents a custom record read from the database.

  • tsc_CustomRecordArray   Contains an array of tsc_DatabaseCustomXml records.

Custom Entities

• • tsc_CustomEntity        Represents a custom entity read from the database.

  • tsc_CustomEntityList    Contains a list of custom entities as read from the database.

  • tsc_JobCustomEntities   A set of methods used to read, write, delete, and undelete custom entities. 

Methods in other classes for the suppport of custom XML

• • tsc_Database

    • • AppendCustomRecord() - appends a "lone" custom record to the database.

      • AddCustomSubRecord() - adds a custom sub-record to an existing entity, such as a point or line.

      • ReplaceCustomSubRecord() - replaces a custom sub-record on an existing entity.

      • RemoveCustomSubRecord() - removes a custom sub-record from an existing entity.

      •  

  • tsc_Entity

    • • GetCustomSubRecords() - obtains an array of the sub-records attached to the entity.

      • IsCustom() - returns true if the entity is a custom entity.

      •  

  • tsc_Observation                   Has methods to add custom sub-records to any observation before adding it to the database.

    • • AddCustomSubRecord() - Adds a custom record to the observation, to be stored when the observation is written.

      • ClearCustomSubRecords() - Removes all subrecords currently attached to the observation.

      •  

  • tsc_PointStakeoutStoreEventArgs

    • • AddCustomSubRecord() - Adds a custom record to the observation that is about to be stored by stakeout.

Job XML format

There are three locations within an exported JXL file where custom XML records can appear, plus there is a Type element which contains some meta-information about the custom records. 

While all custom XML is exported in a JXL file, only custom sub-records are imported, and only if the coordinate of the point the xml is attached to appears in the <Reductions> section.  This occurs if the point can be resolved to a Grid or WGS84 coordinate.

The easiest way to illustrate this is by example. The following xml was dumped from a TA job database:

A "lone" <CustomXmlRecord> can appear anywhere directly below the <FieldBook> element:

<CustomXmlRecord ID="00000028" TimeStamp="2012-11-29T16:48:51">

  <CustomTypeRecordId>00000027</CustomTypeRecordId>

  <TypeVersion>3</TypeVersion>

  <Creator>CustomTests_v2.10.0.1</Creator>

  <CustomContent>

    <Coffee>

      <Beans>Arabica</Beans>

      <Brew>Espresso</Brew>

    </Coffee>

  </CustomContent>

</CustomXmlRecord> 

A custom entity which can also appear anywhere directly below the <FieldBook> element:

<CustomEntityRecord ID="00000024" TimeStamp="2012-11-29T16:48:51">   

  <CustomTypeRecordId>00000022</CustomTypeRecordId>

  <TypeVersion>2</TypeVersion>

  <Creator>CustomTests_v2.10.0.1</Creator>

  <Deleted>false</Deleted>

  <KeyValue>second</KeyValue>

  <CustomContent>

    <TestElement Name="second">

       <inside>Some plugin stuff</inside>

    </TestElement>

  </CustomContent>

</CustomEntityRecord> 

A custom sub-record which appears inside a point, line, or arc. Appears under <FieldBook> and possibly also <Reductions>

<PointRecord ID="0000002b" TimeStamp="2012-11-29T16:49:04">

  <Name>P1</Name>

  <Code/>

  <Method>Coordinates</Method>

  <SurveyMethod>Fix</SurveyMethod>

  <Classification>Normal</Classification>

  <Deleted>false</Deleted>

  <Grid>

    <North>100.001</North>

    <East>200.02</East>

    <Elevation>300.3</Elevation>

  </Grid>

  <CustomXmlSubRecord> 

    <CustomTypeRecordId>00000022</CustomTypeRecordId> 

    <TypeVersion>2</TypeVersion>

    <Creator>CustomTests_v2.10.0.1</Creator>

    <CustomContent>

       <weather>

         <rain>none</rain>

         <wind>gale</wind>

         <sun>blazing</sun>  

       </weather>

     </CustomContent>   

  </CustomXmlSubRecord>

    ...

</PointRecord>

 A CustomTypeRecord will appear prior to the first custom record of that type. Type records are also directly below <FieldBook>:

<CustomTypeRecord ID="00000022" TimeStamp="2012-11-29T16:48:51">

   <TypeName>MyStuff</TypeName>

   <TypeVersion>2</TypeVersion>

   <Guid>278e7958-7818-497f-8a63-74eb426583e7</Guid>

   <Reviewable>true</Reviewable>

   <EntityKeyPath>@Name</EntityKeyPath>

   <Creator>CustomTests_v2.10.0.1</Creator>

</CustomTypeRecord>