The DLL Hell - Problems and Solutions

• Download demo - 6.26 Kb

Abstract

In this article, I am going to touch on problems of DLL backwards compatibility, which are also well known as the �DLL Hell�.

I am going to list results of my own investigation and also refer to other investigators� results. At the end of the article, I will give my approach for solving one of the �DLL hell� problems.

Introduction

Once I�ve got a task to solve the problem with the DLL updates. Some company supplied users with DLLs that contained classes that developers used in their C++ applications. In other words, it was some kind of a SDK in the DLL. There were no restrictions on how to use these classes, so developers derived from them while creating their applications. Soon they realized that their applications started to crash after the DLL updates. Developers were required to rebuild their applications to get versions that can work with the new DLL.

Further I will provide you with my research results on this problem, with the information I have gathered from the external sources and finally I will give my solution for one of the �DLL Hell� problems.

Research results

As I have understood, the issue was in the modification of the base classes placed in the supplied DLL. I�ve overviewed several articles and found that the problem of the DLL backwards compatibility is not a new one but as a true researcher I�ve decided to make several experiments personally. As a result, I have got the following list of problems:

1. Adding of a new virtual method to a class exported from a DLL can cause the following problems:
   • If class already has a virtual method B and we are adding a new one named A before it, then we are changing the table of virtual methods. Now the first virtual method in the table will be A but not B and the client program which calls B will fail without recompilation as the call of B will cause the call of A and this is another method which possibly has other parameters and return type.
   • When a class doesn�t have virtual methods and none of its base classes have, then adding of a new virtual method to it or to its base class will cause adding of the pointer for the table of virtual methods. This will cause change in the class size. So the error will occur after a client program will allocate memory for the class (an amount of memory that was required before the class has been changed) and will try to modify some of the class' fields explicitly or implicitly. In this case, as the pointer to the table of virtual method is added to the beginning of the class, all addresses of the class fields are shifted and thus we will get an abnormal client program behavior or a runtime error.
   • In case when a class has virtual methods or any of its parent classes has, we can�t add virtual methods to classes exported from the DLL if they participate in the inheritance. We can�t add virtual methods not only to the beginning of the class declaration but also to the end of it. The problem is in shifting in the table of virtual methods. Note that even if you add a new virtual method to the end of the class declaration then the child�s virtual methods will be shifted.
2. Adding of a new field (of any kind) to a class declaration can cause following problems:
   1. Adding of a new field to the end of a class causes class size to change as in the case of adding of a virtual method to a class that didn�t have any. The client program will allocate less memory than it is required for a new class and this will cause referencing to the memory out of the class scope.
   2. Adding of a new field between existing ones is even worse. It will case all addresses of fields after the new one to be shifted and a client application will work with the incorrect addresses while working with the fields that are situated after the new field. We also have the problem as in the previous point here.

There are other reasons that can cause DLL backwards compatibility problems, so below is the list of solutions that are usually offered to overcome most of them.

DLL coding conventions in brief

This is the summary of solutions I�ve found in articles on the web and I�ve got while talking with different developers.

The following conventions of a DLL development are usually offered to support DLLs backwards compatibility:

1. Coding conventions:
   • Each class exported from a DLL or its base class should have at least one virtual method to add a virtual methods table to a class. This will allow adding of virtual methods to it later.
   • If you are going to add a virtual method to a class, add it after all other virtual methods� declarations. This will allow not to spoil the order of methods in the virtual methods table.
   • If you are planning to extend a class with the new fields later, then add there a pointer to the structure where you will add them. In this case while adding a new field, you will add it not to the class but to the structure and so the size of the class won�t change. Work with the structure as with the storage of additional class fields and add corresponding fields� accessors to the class declaration if it is required. In this case, the DLL must be linked to a client application implicitly.
   • To solve the problems stated in the previous points, you may also make the pure interface classes for all exported classes, but in this case it will be impossible to inherit these classes in the client application and it will be also impossible to make the hierarchy of exported classes in the DLL.
   • Distribute two versions of the DLL and LIB (Debug/Release) because if you will distribute only Release version then users who are developing clients won�t be able to debug their application as Release and Debug modes have different heap managers and thus if release DLL will allocate memory which will be freed in the debug client application (vice versa), it will cause a runtime failure. There is also another possibility to solve this problem. You may add methods to allocate and deallocate memory for the DLL classes to the DLL and also prohibit deallocation of the client�s application objects in the DLL. But it is not simple to maintain this solution.
   • Don�t change the default parameters in the DLL classes� methods as they are moved to the client�s code at compile time.
   • Beware of modification of inline methods.
   • Check that all enumerations don�t have default element values. As while adding/removing a new enumeration member you may shift values of the old enumeration members. That is why each member should have a unique value assigned to it. Also it should be documented if enumeration can be extended, because in this case client application developer should keep it in his mind.
   • Don�t change macros in DLL headers.
2. Use DLL versioning: In the case of major DLL changes it is good to have an ability to change the DLL name like Microsoft does with the MFC DLL. For example, there can be the following DLL name template: Dll_name_xx.dll Where xx is the number of the DLL version. Sometimes it is required to make big changes in the DLL, which break its backwards compatibility. In this case, the new name should be assigned to the DLL. While an installation to an OS, a new DLL, the old one will not be removed and clients compiled with it will still use it. At the same time, clients compiled with the new DLL will use the new one.
3. DLL backwards compatibility testing: There are too many possibilities to break the DLL�s backwards compatibility and that is why there should be some kind of backwards compatibility testing.

Below I am going to discuss the problem of virtual methods and I will also give a solution for one of its variants.

Virtual methods and inheritance

Let us investigate an example with virtual methods and inheritance:

/**********Exported class (dll) **********/
class EXPORT_DLL_PREFIX VirtFunctClass{
public:
    VirtFunctClass(){}
    ~VirtFunctClass(){}
    virtual void DoSmth(){
        //this->DoAnything();  

              // Uncomment of this line after the corresponding method 

              //will be added to the class declaration

    }
    //virtual void DoAnything(){} 

         // Adding of this virtual method will make shift in 

         // table of virtual methods

};

/**********Inherited class (client app) **********/
class VirtFunctClassChild : public VirtFunctClass {
public:
    VirtFunctClassChild() : VirtFunctClass (){}
    ~VirtFunctClassChild(){};
    virtual void DoSomething(){}
};

Let�s imagine that we have classes listed above. The first one is the my.dll member and the second one is the class defined in the client�s application. Now we will make some changes in the DLL class i.e. uncomment the following two lines:

//virtual void DoAnything(){}

and

//this->DoAnything();

This will symbolize some changes in the SDK classes.

Now if we won�t recompile the client�s application, then the VirtFunctClassChild class will not know anything about the new virtual method void DoAnything() that was added to its base class, so its table of virtual methods (vtable) will contain only the two methods in the following order:

1. void DoSmth()
2. void DoSomething()

This is not correct because the vtable should be as follows:

1. void DoSmth()
2. void DoAnything()
3. void DoSomething()

Then if after instantiation of the VirtFunctClassChild class, someone will call its void DoSmth() method, it will cause calling of the new virtual method void DoAnything(). The base class knows that it should be a second method in the vtable and so it will try to call it. Obviously this will cause calling of the void DoSomething() method of the child class as the vtable of the VirtFunctClassChild class contains this method in the second position and it doesn�t contain the void DoAnything() method at all !!!!

It is important to note that in case of prohibiting of inheritance from the DLL classes with adding of new virtual methods, we will still have the same problem because in this case (just imagine that there was no virtual void DoSomething(){} method in the VirtFunctClassChild class), the call of the void DoAnything() method from the base class will cause referencing to the empty memory as there will be only one member in the vtable.

Now it is obvious that there is a serious problem with adding of the new virtual methods but there is an ability to solve it in case when virtual methods are used to handle callbacks.

COM and others

Now it is time to tell that these problems are not only the well know ones and that there are not only special conventions that can help to solve them but that there are technologies that allow to avoid most of the DLL backwards compatibility problems. One of these technologies is Component Object Model (COM). So if you want to forget problems with your DLLs, use COM or any other appropriate technology.

Let us now return to the task that was given to me. The task I was talking about in the beginning of the article. It was to solve the backwards compatibility problem of one product that was a DLL.

I knew about COM and thus my first proposal was to start using COM in this project to overcome all the issues. This approach was rejected because of the following reasonable causes:

1. The product already had a COM server on one of its inner layers.
2. It is rather expensive to rewrite a large set of interface classes into COM.
3. As this product is a DLL library, there are a lot of applications that use it, so they did not want to force their customers to rewrite them.

In other words, what was required was to propose the most costless way to check the DLL backwards compatibility problem.

Of course, I should say that the main issue of this project was adding the new fields and virtual callback functions to the interface classes. The first problem is simply solved by adding a pointer to a structure (that will contain all new fields) to the class declaration and it is a common approach I have mentioned above. But the problem with the virtual callback functions was new. Further I propose the costless and effective way to solve it.

Virtual callback methods and inheritance

Let us imagine that we have a DLL with the exported classes someone has to inherit from, to implement virtual functions to handle callbacks in his application. We want to make minor changes in the DLL. This change should allow us to painlessly add new virtual call back functions to the DLL classes in the future. At the same time, we don�t want to affect applications that use the current DLL version i.e. all we can expect is that they will be once recompiled with the new DLL version but that should be done for the last time. So these are target settings and here comes the solution:

We can leave every virtual callback method as it is in the DLL's classes. All we have to remember is that adding of the new virtual method to any class definition may cause failure if a client application would not be recompiled with the new DLL version. We want to avoid this problem. The �Listeners� mechanism can help us here! If virtual methods, defined and exported from the DLL classes, are used to handle callbacks, then we can move new virtual methods to the separate interfaces.

Let�s look at the following example:

// Uncomment this definition to get sources after the DLL

// interface extension

//#define DLL_EXAMPLE_MODIFIED


#ifdef DLL_EXPORT
    #define DLL_PREFIX __declspec(dllexport)
#else
    #define DLL_PREFIX __declspec(dllimport)
#endif

/**********Exported class (dll) **********/
#define CLASS_UIID_DEF static short GetClassUIID(){return 0;}
#define OBJECT_UIID_DEF virtual short 
      GetObjectUIID(){return this->GetClassUIID();}

//Base interface for all callback handlers extensions

struct DLL_PREFIX ICallBack
{
    CLASS_UIID_DEF
    OBJECT_UIID_DEF
};

#undef CLASS_UIID_DEF

#define CLASS_UIID_DEF(X) public: static 
     short GetClassUIID(){return X::GetClassUIID()+1;}

#if defined(DLL_EXAMPLE_MODIFIED)
//Newly added interface extension

struct DLL_PREFIX ICallBack01 : public ICallBack
{
    CLASS_UIID_DEF(ICallBack)
    OBJECT_UIID_DEF
    virtual void DoCallBack01(int event) = 0;  //new call back method

};
#endif // defined(DLL_EXAMPLE_MODIFIED)


class DLL_PREFIX CExample{
public:
    CExample(){mpHandler = 0;}
    virtual ~CExample(){}
    virtual void DoCallBack(int event) = 0;
    ICallBack * SetCallBackHandler(ICallBack *handler);
    void Run();
private: 
    ICallBack * mpHandler;
};

It is easy to see that all we have to do to prepare the DLL�s classes for extension with the new virtual methods is to:

1. Add ICallBack * SetCallBackHandler(ICallBack *handler); method;
2. Add the corresponding pointer to each exported class definition;
3. Define 3 macros;
4. Define one common CallBackI interface.

I have added the ICallBack01 interface definition here as an example of how the CExample class can be extended with the new virtual callback method.

It is obvious that new virtual methods should be added to a new interface. One interface for each DLL update (i.e. if we want to add several new virtual callback methods to one class at the same time [for one DLL release], we should add them to one interface).

Each new interface (for one class) should extend the previous interface. In my example I have only one extension interface ICallBack01. If in the next release we will need to add a new call back virtual method to this class then we will create a new interface ICallBack02 that will extend the ICallBack01 interface in the way the ICallBack01 extends the ICallBack.

Macros in the code above are used to define methods required to check the interface version. For example when we are adding the new interface ICallBack01 with the new method DoCallBack01, we should check (in the CExample class) if we can call it on the ICallBack * mpHandler; member. This check should be done in the following way:

if(mpHandler != NULL && mpHandler->GetObjectUIID()>=ICallBack01::GetClassUIID()){
    ((ICallBack01 *) mpHandler)->DoCallBack01(2);
}

All these is made in the case when the new call back interface is added and the call back is inserted in the CExample class, so it is very easy.

Now you can see that the client�s applications would not be affected by these changes. The only thing will be required after the first DLL classes� update (adding of macros, base interface ICallBack definition, SetCallBackHandler method, ICallBack pointer) is the client application's recompilation.

Later when someone will add the new callback, he will do it via the new interface addition (like adding of the ICallBack01 interface in our example). It is obvious that such change would not affect anything, as the order of virtual methods will not be changed. So the client applications will work in the old manner. The only thing you should remember is that client applications will not receive the new callbacks until they implement the new interface.

It is also important to note that DLL users will still be able to work with it easily. This is the client�s class example:

#if !defined(DLL_EXAMPLE_MODIFIED)
//Before a new interface is added to the dll.

class CClient : public CExample{
    //Everything remains as it was

public:
    CClient();
    void DoCallBack(int event);
};
#else // !defined(DLL_EXAMPLE_MODIFIED)

//After the interface ICallBack01 was added client may (not necessarily)

//change his class in the following manner to handle new events

class CClient : public CExample, public ICallBack01{
public: 
    //The constructor changes

    CClient();
    //Everything remains as it was

    void DoCallBack(int event);
    //Add the new virtual method definitions 

    //(for methods inherited from the ICallBack01)

    void DoCallBack01(int event);
};
#endif // defined(DLL_EXAMPLE_MODIFIED)

Note that an example project is attached to this article. It is called Dll_Hell_Solution.

Example Project

In the example workspace, you will find two sub projects:

1. Dll_example - contains the example DLL sources
2. Dll_Client_example - contains the example DLL client sources.

The ./Dll_Hell_Solution/Dll_example/dll_example.h file contains a commented out DLL_EXAMPLE_MODIFIED macro definition. Uncomment it to emulate the DLL's classes update.

To check that everything works, perform the following steps one by one:

1. Build the Dll_example and then the Dll_Client_example projects (with DLL_EXAMPLE_MODIFIED undefined) to emulate the initial situation. Run client application.
2. Uncomment the DLL_EXAMPLE_MODIFIED macro definition. Build the Dll_example. Run the client application. Everything will work as it should!
3. Immediately rebuild the Dll_Client_example project. Run the client application. This will emulate its update that will allow to access the new functionality of the DLLs classes.

That is all about the example project. Just download and explore it to get all details.

Conclusion

In this article:

• I have represented my research on the DLL backwards compatibility problem.
• I have summarized the list of solutions proposed by different developers.
• I have offered my approach for solving the problem with virtual functions and inheritance.

I believe that this article will be helpful for anyone who has come across the �DLL Hell�.

References

While working on this article, I have used the following material:

• �DLL Maintenance Made Easy� by Geoff Vandegrift

If you have found that this is not the full list of references, then please notify me and I will immediately fix this problem.