Ever since I started dealing with these architectural problems concerning byte ordering and floating point formats, I have been struggling with issues in my present database format. Right now the real time data area is a large bitmap that contains the dynamic data. There is also an array of tag definitions that describe the data. The tag array describes the name, type, size of the data point (or tag) and also it's relative location in the real time database.
My original motivation in choosing this type of format was the idea that I would be able to transfer the entire real-time database very efficiently from one module or server to another. I could just write the whole thing all at once or in large chunks. This would work well if all the modules and servers were on the exact same type of hardware. Well the definition of OpenDAX contains the words "cross-platform" so this is a bad assumption. Some kind of data format conversion will have to be assumed between hosts that are on different types of hardware.
I'm starting to think that I should store the definition and the actual real time data in the same structure within the server. I started really having trouble with the original idea when I started considering data change events. Since most of the data handling functions worked with blocks of data within the real time database it was somewhat disconnected from the tag definition. So to handle data change events I would have to do some kind of reverse lookup from the handle into the database back to the tag definition to decide when to send events.
It's ironic that this event handling code was what forced me to reevaluate the message queue also. Since I have to do the lookup to handle these events anyway it starts to make sense to keep the data in the same structure with the definition and keep the event list in that structure as well. Another problem is tag locking and security. I've always known that I'd have to eventually deal with some kind of tag use count and locking mechanism so that modules could "own" a tag and that read/write permissions could be enforced. A tag could be removed from the system only when no modules left that had a lock on the tag, and security could be enforced.
Once I included the whole byte ordering, data format issues into this mix it started to make less and less sense to keep the data packed into that database. The data transport mechanisms in the system right now doesn't really care about the data type. They will have to deal with the data type to handle the proper conversion for communication.
The biggest advantage to the packed database is that it's very space efficient. Boolean values only take up one bit of space. 32 of them could be stored in every index of the array. This was also going to translate into lightning fast updates across the network. Of course this lightning fast transfer isn't going to work at all because of the data formatting issues. The storage advantage is probably not worth the added complexity, and the added complexity is considerable.
A final consideration is custom data types. It has always been a goal to include some mechanism for user defined data types (kinda like a structure in C) for OpenDAX. My brain always started to hurt when I thought about custom data types with the old method. I think they may be easier but I haven't really risked the brain pain on it yet. I'm sure that there is some really elegant, recursive type code that can be written for this.
I still have to work out exactly how I want to store this data. I'm leaning toward a pretty tight structure that contains the type, size and a pointer to the data. These would be stored in an array and the old handle would simply be the index into this array. Then I would have another array that is the same length that stores the name and a pointer (or index) to the first structure. This array would be sorted in alphabetical order or be some kind of hash table for speed. These arrays would be realloc()'d when necessary, which is actually the way I do it now anyway.