Create dlls on Windows without declspec() using new CMake export all feature
CMake 3.4 will have a new feature to simplify porting C and C++ software using shared libraries from Linux/UNIX to Windows. Linux/UNIX developers are often surprised to learn that creating a shared library on Windows known as a DLL (dynamic linked library) requires changes to the source code or an explicit listing of all the symbols that the dll will export. The compilers on Linux/UNIX have the ability to export all symbols in a shared library automatically. On Windows, you must either use compiler directives __declspec(import) and __declspec(export) to declare which symbols are exported/imported from a shared library, or you must create a module definition text file (.def) with a list of all the symbols you want to export and pass that file to the linker.
With C libraries, creating a .def file by hand or automatically is not very difficult since you just have to list the names of all the functions in the library. However, with C++ code, name mangling and the sheer number of functions make crafting a .def file by hand nearly impossible. The standard workaround uses the preprocessor to conditionally insert __declspec(import) and __declspec(export) into the code. However, with a large existing C++ code base, it can be difficult and time consuming to edit all of the source code. CMake now has a feature which allows it to query .obj files that will make up a DLL and create a .def file automatically, in most cases without needing to modify the original source code.
The feature is implemented in CMake via a new target property, WINDOWS_EXPORT_ALL_SYMBOLS. When enabled, this property causes CMake to automatically create a .def file with all symbols found in the input .obj files for a SHARED library on Windows. The .def file will be passed to the linker causing all symbols to be exported from the DLL. For global data symbols, __declspec(dllimport) must still be used when compiling against the code in the DLL. The symbol is exported from the DLL correctly and automatically, but the compiler needs to know that it is being imported from a DLL at compile time. All other function symbols will be automatically exported and imported by callers. This simplifies porting projects to Windows by reducing the need for explicit dllexport markup, even in C++ classes. This property is initialized by the value of the CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS variable when a target is created.
To try this out on an existing project, run
cmake -DCMAKE_WINDOWS_EXPORT_ALL_SYMBOLS=TRUE -DBUILD_SHARED_LIBS=TRUE
on a CMake project. This should turn all add_library calls that do not explicitly specify build type into shared builds. If there are no global data variables in the project, all libraries will be built as DLLs with no errors. If you run into undefined symbols, check for global data like static data members of classes. The easiest way to handle the global data is to use the CMake GenerateExportHeader module like this:
add_library(mylibrary ${mylibrary_SOURCES})
# add these lines
include(GenerateExportHeader)
generate_export_header(mylibrary)
// edit the source like this:
#include <mylibary_export.h>
class myclass
{
static mylibrary_EXPORT int GlobalCounter;
…
Note, if you use GenerateExportHeader and still want static builds to work, you will need to add a -Dmylibrary_STATIC during static builds. See the generated mylibary_export.h for more details.
Real World Use Cases
After adding this feature to CMake, I tried it on two relatively large software projects VXL ( (http://vxl.sourceforge.net/) and ITK (www.itk.org). Although, it did take some time, it was much easier than adding dll markup to every source file. In the end the changes required fell into the following repeated issues and fixes:
-
Handling class static data members
-
Add dll import/export macros to the static data.
-
Eliminate the static data –
-
This can be done by making a static getter function with a local static variable.
-
-
-
Static const members
-
These are not exported by the new feature. In some cases to support older compilers, these are put in .cxx files and compiled into the library and are not just inlined in the .h files. If you have this case, you will need to use dllimport and dllexport markup and not just dllimport mark up as the auto export will not work.
-
-
Inherit from std::string or other templated classes with dllimport/dllexport mark up.
class mylibrary_EXPORT myclass public std::string { ..-
This can cause duplicate symbol errors in the parent templated class to show up in some cases. The fix for this is to remove the markup and let the auto-export do its thing.
-
-
Not having fully linked libraries. On many Unix systems the linker will allow for shared libraries to have unresolved symbols at create time, that are expected to be resolved at the link time of the executable. Windows dlls must resolve all symbols, so your library must explicitly link to everything it depends on. One way to find out where symbols exist in a big system if you get an undefined symbol at link to time is to look for the symbol in the .def files that are created. From a bash (cygwin or git) shell you could run something like this:
find . -name "*.def" | xargs grep symbol
-
Static data members of templated classes. It is best to just avoid these, or only use explicit instantiation or specialization for these. I was not able to get these to work at all.
Thanks
I would like to give a special thanks to Valeri Fine (code author) and Bertrand Bellenot and the rest of the ROOT team at CERN for contributing the .obj file symbols parsing code, and for testing the feature on their project. Enjoy!
Links
Nightly CMake with new feature: http://www.cmake.org/files/dev/cmake-3.3.20150721-g9cd2f-win32-x86.exe
Nightly Docs for feature:
http://www.cmake.org/cmake/help/git-master/prop_tgt/WINDOWS_EXPORT_ALL_SYMBOLS.html
ITK changes for this feature: http://review.source.kitware.com/#/c/20020
VXL changes for this feature: https://github.com/vxl/vxl/pull/4
Valeri Fine presented this approach and introduced the original code at Computing in High Energy Physics Berlin, Lichtenberger Congress Center April 7-11, 2977:
http://www.ifh.de/CHEP97/abstract/g340.htm
http://www.ifh.de/CHEP97/paper/340.ps.
This sounds really interesting. I just tried it out on our project. I got some errors coming from static members and static const members exactly as you explained. Otherwise it works well so good job.
However…
There is currently one big disadvantage of this. We use incremental linking to speed up linking of DLLs and make it independent of the main executable (so when there is a cpp code change in the DLL the main executable isn’t linked again which saves us lot of time). This important feature now stopped to work because the .def file is regenerated every time and so the import .lib is recreated again as well. This in turn means a dependency change for our executable so it continues to relink even when incremental linking of DLL is used. Quite frustrating:(
Do you think you could prevent this behavior by generating def file with different name (or in memory) and rewriting the original def file only when the content actually changed?
If an object file changes the .lib file is going to change regardless of the .def file changing or not. It will relink the .lib if any of the .obj files that go into it change. You could tell the build system to only build the dll you are working on. Then just run the exe without relinking it. However, I don’t think there is anything CMake can do automatically. The .lib and the .dll are created by the same link command.
Sorry, I am realizing I missed this thread. I need some time to catch up. Give me time. The solution does work for ROOT where we never use the incremental linking. ROOT system comprising >100 different packages (DLLs). Each one is small enough to be rebuild/ relinked with no incremental linking involved. On the other hands with ROOT when cpp code changes in DLL the main executable isn’t linked again. That has nothing to do with the incremental linking. The main has no direct formal dependency of any DLLs thank to the built C++ interpret.
I’ll try to think about the incremental link ‘use case’ to address your issue “def” file question
Mean time I would like to call your attention to the fact the ROOT builtin C++ interpret can be use with out ROOT as a part of ANY other C++ application.
So I;ll try to think about your question
thanks so much for this article, you just fixed a big bunch of work for me 😀
Just found out about this. Seems very interesting. Would this work if we throw some CUDA object files in the mix? I don’t see any reason why it would not work but maybe there are some limitation I am not seeing now?
“For global data symbols, __declspec(dllimport) must still be used when compiling against the code in the DLL.”
Can you let me know what does “global data symbols” means? I trying to understand when exactly dllimport is still needed for optimized performance
Global data symbols are things like static member variables or global data. Basically anything that is not a function.
class foo
{
pulblic:
static int Counter; // this is global data
Thanks for the response, fortunately static member variables is not common in my code, does that mean performance wise
class foo
{
public:
void my func()
and
class foo
{
public:
__declspec(dllimport) void my func()
is the same with this new feature? Or does the performance difference as explains in https://msdn.microsoft.com/en-us/library/zw3za17w.aspx still applies?
I think it would have the performance difference mentioned. The main benefit is that you do not have to mark up the code. This can be very helpful if you are porting a large code base from a compiler/OS that does not require mark up like this.
Got it, for non performance critical code this feature eliminates the need for a DLL macro and allows the code to be ported to windows as a shared library without any change.
It would be great to have a new cmake feature that does dllimport so we can completely say bye to DLL macros!
Not sure there is much CMake can do about this. It is a compiler level thing. Although, I am not sure how much of a performance hit there actually is with modern computers.
Hi, This is an awesome feature, and I love it. Thank you. Quick question:
Let’s say I have two independent apps/exes “appA” and “appQt” that depend on dlls “dllA”, and “(dllA, dllQt)” respectively. That is “appA needs dllA” and “appQt needs (dllA, dllQt)”, with implied Qt app and dll.
I assume that using this feature on “dllA” and “dllQt” means that building “appA” and “appQt” would be seamless. Are there any restrictions, for instance, does CMake expect appA and appQt to be in the same VS Solution? I assume that there is no problem with Linux builds.
Thanks again for this wonderful feature!
That should work fine.
Indeed the above aspect works fine for me, and saved a great deal of time (Thank you again, my code looks clean now)! But as my Qt app is becoming more complex, I am running into your observation on “static const members” – I have no choice here, because that’s what Qt does.
I have Qt signals emitted by one dll that must be connected to Qt slots in another dll through a Qt connect object. For this aspect, Qt generates moc_XXXX files from my class, and one of the meta objects in the generated file is a static const struct. Where do I put the “_declspec (dllexport) staticMetaObject” and the corresponding import? For prototyping purposes, I tried putting the dllexport in moc_XXXX file and dllimport in the class that expects to use the staticMetaObject, but that does not work.
Any pointers is appreciated. Thanks.
Ps: once my prototype works, I will find a way to generate these export/import declspecs, because I *cannot* put them into the generated moc_XXXX files.
I discovered that using the standard DLL export/import macro approach on the class itself as: “class dllMACRO XXXX : public QSomeThing { Q_OBJECT /*…*/};” works fine. I was afraid of trying it because I thought my “dllMACRO” would somehow interfere with what CMake is trying to do. Here “dllMACRO” translates to dllexport when used by the native DLL, and to dllimport for the clients.
CMake will just skip over symbols that are already being exported via markup in the code, so that should work.
It is very useful for me!!
And i have met with one doubt!
How can I implement the same for run time..
So what my project is,
Iam having 2 Dll separately, so with the exe i want to link the dll according to the usage..
so if there is any possibility of doing the same for this idea?
Thank you in advance 🙂