I've just finished implementing some pretty massive changes in the custom data type code in the server and library. I've also decided to start calling them Compound Data Types as well.
The changes that I made involved how the CDT's were created. Previously there were several messages that were sent to the server. The first created a data type by name, then members were added one by one and then the data type was finalized. This was simple to implement but it caused complexity in the supporting code. It allowed for recursion in the data types that had to be checked for and eliminated. There was also the problem of other modules trying to use the data type before the creating module was finished with it, so I had code in there to solve these problems.
I finally decided that this was too complex and the whole thing needed to be atomic. The CDT should be created with a single message to the server. Now the modules call dax_cdt_new() to allocate an opaque pointer to a data type. Then they call dax_cdt_append() to add members to the data type and then when the module is finished it calls dax_cdt_create() to send that definition to the server. A successful result from the dax_cdt_create() function means that the server is happy with the the data type, that it exists and is ready for use.
Compound data types can not be edited once they are created so this makes it impossible to create recursion in the members. This eliminated a lot of code. Also, since they are created atomically a lot of code to check for whether or not they were finished was eliminated. Another small but insidious problem involved how to back out of the creation of a data type when one part of it failed.
All of these problems went away when I changed the program to build the data types atomically on the server. I probably knew that when I did it to start with but the implementation of compound data types seemed so overwhelming at the time that I wanted to do everything the easy way. Sometimes the easy way is the hard way.