Okay so I grossly underestimated the amount of effort that it would take to change the system over to a sockets type IPC to replace the message queue. Actually replacing the message queue with a unix domain socket is not terribly difficult once the message buffering is done, but the the thought that I could use that same code for a remote module was naive.
Experienced network programmers may start laughing now. It's okay, this is my first time to try sending binary data over a network, and I figured it out BEFORE I wrote any code. For you Windows programmers out there let me explain the problem. Basically it's two things, endianess and structure packing.
The structure packing issue basically involves the fact that different compilers can store the structures in any way that they please. This could be a problem even with the message queue because the library and the server could be compiled with different compilers that would represent the structures differently. This is not likely in the existing system, but will be very likely in any distributed system with modules scattered all over the world.
The endian issue is the age old problem of different architectures storing floating point numbers and integers longer than 8 bits in different orders within memory. Floating point numbers are even worse because they don't even have to have the same format. If we jam those numbers onto the network socket in the order that we store them and we store them differently than the system on the other end then we'll lose our number.
Now how do I solve this problem? There are a few options. I can specify that the numbers will be stored in the tagbase in a certain format and then have the library convert the number before sending it across the socket to the module. Another method, is to specify a network protocol byte ordering. Most programs of this type will specify a network ordering of the different data types, then force the sender and the receiver to marshal and unmarshal the data as it goes into and comes out of the network socket.
I will probably go with the second idea simply because that is the way that I see all other programs that send binary data over a network working. The first idea is too painful to consider. Especially when the server is running on a machine that is opposite of the order of the tagbase. It would be a big performance hit to have to convert every integer and every float in every module every time it was transfered. If I use the marshal/unmarshal idea then I can simply ignore the whole thing on local modules and then take care with the remote modules.
The advantage to specifying the storage order in the tagbase is that my original idea of having multiple servers on different machines each sharing the same tagbase would be much simpler. Then the entire block of data could be sent from one to another without worrying about the byte order. I think that this idea will have to die. I've been leaning more toward simply having remote modules anyway. I think that is simpler to program and easier for end users to understand. I may implement a method that would allow servers to share parts of their tagbase across the network but that can be a separate issue.
I can speed this up by defining a method for the library to determine whether or not the server is on the same type of machine. Part of the module registration could have the module write some floating point values and integers to the server that are predefined. If the server (without unmarshaling) received the predefined numbers then it would know that the module was on the same type of architecture and the marshaling could be avoided. Part of the registration response would be to let the module know the answer to this question. Since the vast majority of these systems are likely to be x86 PCs then this would work well. It would save lots of clock cycles on a local module as well. There is no sense in converting an integer into big endian and then sending it through a local socket, only to have it converted back to little endian at the server when they are both on the same machine.
At first I wanted to ditch this whole sockets idea and run back to the message queue. The more I thought about it the more I realized that I will have to deal with these issues sooner or later. I might as well deal with them now. I am going to have to do more data type checking in the library than I originally thought I would have to do and low level data copying functions will have to go away. It was much better to discover this now than after I had 100,000 lines of module code written.