- Dynamic Data. The fundamental goal is to be able to dynamically serialize any structure of data from point A to point B. It may be a client sending data over TCP/IP to a server. It may be an application storing a preferences file that it will re-load on startup. It may be the result-set of a "Select * from table" SQL query where you just get a table of data with whatever content was returned from the database. The point here is that the producer should be able to dynamically put any structure of data into the stream and the consumer should be able to read it into an in-memory exact replica of the original without having to necessarily know ahead of time what data is expected to be in that structure.
- Receiver does not require a schema. The serialization must be self describing. For example, Google's protocol buffers requires that the receiver and sender are built with a common .proto definition on what data may possibly be in the serialization. Any application can serialize a DataObject stream and any other application can read that serialization and get an exact duplicate of the original data structure without needing any kind of schema definition to interpret the stream.
- Not based on Code Generation. Some serialization implementations like protocoBuffers, Thrift, etc. are implemented by defining a schema definition (.proto file, a Thrift definition, etc.) and then running a program that reads the schema definition which then generates source code. You then use the generated source code in your application. Your compiled application then becomes fixed to that schema definition. There's no provision for dynamically adding in more data. Of course, they've done a good job letting you define more fields in the schema and the new code produced by a new schema is still inter-operable with the old code on the old schema. However, adding a new field somewhere in the data structure means that you need to re-generate the code. DataObjects aims to be dynamic so that an application that is built with the DataObject streaming could have the opportunity to work with new, dynamically added data without needing to be recompiled.
- Binary Serialization. Let's face it, parsing text sucks with all the error detection, text-to-datatype conversion, etc. It's generally slower to produce a text stream and it's definitely slower to parse a formatted text-orientated stream compared with a binary equivalent. One of the main uses of DataObjects is for high volume server to server communications and I'm convinced that one of the best ways to have scale-ability in a system is to start with efficiency in that system (efficient memory use, efficient CPU use, efficient networking). The answer isn't always "spin up another server". So, binary serialization it is. Why not JSON? JSON has some serious flaws dealing with data types. Further, the JSON serialization testing I've done is typically 5-7 times slower than using binary serialization techniques. In the end, the only thing JSON is great for is sending data to/from JavaScript. Note that even though I'm not a big fan of JSON, being able to work with JSON is a must. Let's face it, we are working in a JSON world right now. With that in mind, the JSON serialization implementation here is quite fast. Without getting to the assembly programming level, I did spend a significant bit of time researching a few different JSON parsing techniques and libraries and I compared their performances. By picking out a few of the techniques that seemed the fastest, improving on them a bit, and performing a wide variety of test cases (including extreme cases), I was able to get the JSON serializing to work faster than any of those that I compared against (including Google's JSON parsing in chrome). I'd love to tweak it even further, but it's really pretty good right now as it is.
- Non human-readable. The trade off in using binary serialization instead of a text-orientated serialization is that binary is considered to be not very "human readable". So, what does "human readable" mean? Does it mean looking at a file using notepad in Windows or vi in Linux. Well, you probably have something better like sublime, notepad++, or whatever your favorite text editor is. These are all applications that translate bytes of data into something on the screen that we can more easily comprehend with our eyes. However, using a text editor to view hierarchically structured data in text form (such as JSON) is usually not a very good experience, especially when it's "unformatted" JSON. Further, sometimes these 'text" serializations inject extra line-feeds, space paddings, etc. just to make it easier for a human to read in a text editor. I think that's wasteful, not to mention that the real experience sucks. For example, I can use notepad to view an XML file, or I can use XML Spy to view an XML file. I get a much better experience with XML spy. Notepad is designed to present text content to humans to view and interact with. XML Spy is designed to present XML content to humans to view and interact with. You always need an application to open and show you the contents of any file and that application should be suited to the type of data stored in that file. So, I've built a Windows application called the DataObject Editor which is designed to present DataObjects to humans to view and interact with. Since DataObjects can be serialized to/from JSON, CBOR, BSON, etc., it's also a handy tool if you are using these serializations. I hope to add more serialization formats as time goes by.
- Explicit Data Types. Some serialization mechanisms have a design goal to support a minimal set of data types. JSON is one example of such. It models null, true, false, number, string, array and object (a collection of name-value pairs). That's it. There's no differentiation between integers and floating point numbers, no concept of dates, GUIDs, etc. Because of the lack of many data types, people encode certain types of data into JSON using a formatted string value. Lots of people make up these formats and it's basically a free-for all. Sure, there are some common conventions that are out there for some of the data type such as dates and times, but it seems nothing solid is agreed upon by everyone simply because it's not in the underlying spec. I've come to learn that being explicit about modeling data is much more reliable than making implicit choices on how to treat data. When things are allowed to be implicit, different people make different assumptions and you run into compatibility problems. DataObjects attempts to be explicit as much as possible about the definition of different data types and I tried to model just about every (normal) data type I could think of and that I found in other serialization techniques. However, since DataObjects can serialize to multiple stream formats, there is inherently some translation/interpretation that we have to do internally too depending on the serialization format used. We still have to deal with the weaknesses of JSON so we must have a defined set of translation behavior when serializing to/from any one of the supported formats.
- Support Most Data Types Natively. Some serialization technologies out there only model data that can be easily and commonly modeled by most programming languages. For example, some serializing mechanisms don't have the ability to model an unsigned integer. These designers have taken the approach that for numbers, they didn't want to model unsigned numbers explicitly because some programming languages can't model unsigned numbers. They just have a number data type and it is inherently signed. DataObject data modeling kind of takes the opposite approach in that most atomic data types are explicitly modeled. The goal here is to be able to get data from some source like a MongoDB collection, or a query from the Oracle database, or from a C Struct, and the different types of data from those sources will have a proper data type modeled explicitly with the right data type. Having explicit data types allows you to modify some data and give it back to the original source and you will have proper translation back to that source. For example, DataObjects models the concept of an ObjectID that is exactly the same as a MongoDB ObjectID. JSON doesn't have that data type so you'd have to convert the BSON objectID to a string representation. Most of the time, I've seen it like this: 'ObjectId("57350b3718925a1e04db9c91")', or like this: "57350b3718925a1e04db9c91". The point is, whenever you have to translate from a well-defined data type to something more generic (usually a string) and then translate back, you run into problems. When you're in this situation, you end up writing code to look for certain patterns in strings to try to detect the intended data type and then converting those string representations into the proper data types. JSON is notorious for this problem. For example, if I have a floating point number that is put into JSON and that number is "12", translating that JSON 12 value back to it's original data type is guesswork, is it a floating point or is it an integer? You can't tell so you just guess. Therefore DataObjects tries to explicitly model as many typical data types as possible so we can regenerate values with strong data types from a serialization stream and to/from other systems and data exchange formats.
- Be compatible with all the others. I often need to work with structured data within an application where I'm interfacing with other systems using services of some sort. Sometimes the services are JSON, sometimes they are XML, sometimes I'm executing queries on a database or reading data from an ascii file produced by another system. Internally, I would like to be able to work with generic data using one object model and then be able to consume data from other sources into that object model and produce data to other targets from that same internal object model. DataObjects aims to be mostly translatable from everything and mostly translatable to everything. The goal here is to be able to directly translate from other common formats such as JSON, MessagePack, CBOR, BSON, etc. into the DataObjects object structure and also be able to translate from DataObjects to those other formats. This means that certain characteristics from those other formats need to be modeled in the DataObject object model. One example of this is the concept of attributes which is borrowed from XML. In DataObjects each DataObject has a data type and a value. However, it can also have one or more attributes applied to that value. For example, I may have a value that represents the height of a physical object. The dataType may be a floating point number with a value of 3.7. This value can also be given an attribute of "units=Meters". Another example is the concept of a "tag". A tag is a number applied to the data that follows to give a more refined meaning to that following data. For example, the data could be a string and a tag could be applied to it that tags that string as being a regular expressions. This concept is borrowed from CBOR.
- Efficiently stream to/from a languages' native objects. Even though DataObjects can dynamically model and stream data, we still want to be able to directly stream to and from an object instance within some OOP programming language as efficiently as possible, yet still be able to properly deal with data elements from the stream that may or may not align perfectly with said object. In other words, we won't just blindly copy bytes of data from the stream and assign them to the memory location of an object instance. However, we don't necessarily want to load the dynamic data into a generic object model in memory first, and then decide which object to instantiate and then plug the object's properties from the copy of data previously loaded into memory. In other words, let's not double consume memory if we can have a mechanism that lets us go directly from a stream into an object instance without using the blind byte-copying technique. This implementation allows me to take a real object instance (some TObject descendant) and make it a member of a DataObject somewhere in the hierarchy of that data within that DataObject.