10. Dlopened modules

It can sometimes be confusing to discuss dynamic linking, because the term is used to refer to two different concepts:

1. Compiling and linking a program against a shared library, which is resolved automatically at run time by the dynamic linker. In this process, dynamic linking is transparent to the application.
2. The application calling functions such as dlopen that load arbitrary, user-specified modules at runtime. This type of dynamic linking is explicitly controlled by the application.

To mitigate confusion, this manual refers to the second type of dynamic linking as dlopening a module.

The main benefit to dlopening object modules is the ability to access compiled object code to extend your program, rather than using an interpreted language. In fact, dlopen calls are frequently used in language interpreters to provide an efficient way to extend the language.

As of version 2.2.6, libtool provides support for dlopened modules. However, you should indicate that your package is willing to use such support, by using the LT_INIT option `dlopen' in `configure.ac'. If this option is not given, libtool will assume no dlopening mechanism is available, and will try to simulate it.

This chapter discusses how you as a dlopen application developer might use libtool to generate dlopen-accessible modules.

10.1 Building modules to dlopen

On some operating systems, a program symbol must be specially declared in order to be dynamically resolved with the dlsym (or equivalent) function. Libtool provides the `-export-dynamic' and `-module' link flags (see section Link mode), for you to make that declaration. You need to use these flags if you are linking an application program that dlopens other modules or a libtool library that will also be dlopened.

For example, if we wanted to build a shared library, `hello', that would later be dlopened by an application, we would add `-module' to the other link flags:

burger$ libtool --mode=link gcc -module -o hello.la foo.lo \
                hello.lo -rpath /usr/local/lib -lm
burger$

If symbols from your executable are needed to satisfy unresolved references in a library you want to dlopen you will have to use the flag `-export-dynamic'. You should use `-export-dynamic' while linking the executable that calls dlopen:

burger$ libtool --mode=link gcc -export-dynamic -o helldl main.o
burger$

10.2 Dlpreopening

Libtool provides special support for dlopening libtool object and libtool library files, so that their symbols can be resolved even on platforms without any dlopen and dlsym functions.

Consider the following alternative ways of loading code into your program, in order of increasing "laziness":

1. Linking against object files that become part of the program executable, whether or not they are referenced. If an object file cannot be found, then the compile time linker refuses to create the executable.
2. Declaring a static library to the linker, so that it is searched at link time in order to satisfy any undefined references in the above object files. If the static library cannot be found, then the compile time linker refuses to create the executable.
3. Declaring a shared library to the runtime linker, so that it is searched at runtime in order to satisfy any undefined references in the above files. If the shared library cannot be found, then the dynamic linker aborts the program before it runs.
4. Dlopening a module, so that the application can resolve its own, dynamically-computed references. If there is an error opening the module, or the module is not found, then the application can recover without crashing.

Libtool emulates `-dlopen' on static platforms by linking objects into the program at compile time, and creating data structures that represent the program's symbol table. In order to use this feature, you must declare the objects you want your application to dlopen by using the `-dlopen' or `-dlpreopen' flags when you link your program (see section Link mode).

Structure: struct lt_dlsymbol { const char *name; void *address; }

The name attribute is a null-terminated character string of the symbol name, such as "fprintf". The address attribute is a generic pointer to the appropriate object, such as &fprintf.

Structure: struct lt_dlsymlist { const char *originator; const lt_dlsymbol symbols[]; }

The originator attribute is a null-terminated character string, naming the compilation unit that symbols were preloaded on behalf of. This is usually the basename of a library, `libltdl.la' has a corresponding originator value of `libltdl'; if the symbols are for the benefit of the application proper, then originator is `@PROGRAM@', though Libtool takes care of that detail if you use `LTDL_SET_PRELOADED_SYMBOLS'.

Variable: const lt_dlsymlist * lt_preloaded_symbols

An array of lt_symbol structures, representing all the preloaded symbols linked into the program proper. For each module `-dlpreopen'ed by the Libtool linked program there is an element with the name of the module and a address of 0, followed by all symbols exported from this file. For the executable itself the special name `@PROGRAM@' is used. The last element of all has a name and address of 0.

Some compilers may allow identifiers that are not valid in ANSI C, such as dollar signs. Libtool only recognizes valid ANSI C symbols (an initial ASCII letter or underscore, followed by zero or more ASCII letters, digits, and underscores), so non-ANSI symbols will not appear in lt_preloaded_symbols.

Function: int lt_dlpreload (const lt_dlsymlist *preloaded)

Register the list of preloaded modules preloaded. If preloaded is NULL, then all previously registered symbol lists, except the list set by lt_dlpreload_default, are deleted. Return 0 on success.

Function: int lt_dlpreload_default (const lt_dlsymlist *preloaded)

Set the default list of preloaded modules to preloaded, which won't be deleted by lt_dlpreload. Note that this function does not require libltdl to be initialized using lt_dlinit and can be used in the program to register the default preloaded modules. Instead of calling this function directly, most programs will use the macro LTDL_SET_PRELOADED_SYMBOLS.

Return 0 on success.

Macro: LTDL_SET_PRELOADED_SYMBOLS

Set the default list of preloaded symbols. Should be used in your program to initialize libltdl's list of preloaded modules.

#include <ltdl.h> int main() { /* ... */ LTDL_SET_PRELOADED_SYMBOLS(); /* ... */ }

Function Type: int lt_dlpreload_callback_func (lt_dlhandle handle)

Functions of this type can be passed to lt_dlpreload_open, which in turn will call back into a function thus passed for each preloaded module that it opens.

Function: int lt_dlpreload_open (const char *originator, lt_dlpreload_callback_func *func)

Load all of the preloaded modules for originator. For every module opened in this way, call func.

To open all of the modules preloaded into `libhell.la' (presumably from within the `libhell.a' initialisation code):

#define preloaded_symbols lt_libhell_LTX_preloaded_symbols static int hell_preload_callback (lt_dlhandle handle); int hell_init (void) { … if (lt_dlpreload (&preloaded_symbols) == 0) { lt_dlpreload_open ("libhell", preload_callback); } … }

Note that to prevent clashes between multiple preloaded modules, the preloaded symbols are accessed via a mangled symbol name: to get the symbols preloaded into `libhell', you must prefix `preloaded_symbols' with `lt_'; the originator name, `libhell' in this case; and `_LTX_'. That is, `lt_libhell_LTX_preloaded_symbols' here.

10.3 Linking with dlopened modules

When, say, an interpreter application uses dlopened modules to extend the list of methods it provides, an obvious abstraction for the maintainers of the interpreter is to have all methods (including the built in ones supplied with the interpreter) accessed through dlopen. For one thing, the dlopening functionality will be tested even during routine invocations. For another, only one subsystem has to be written for getting methods into the interpreter.

The downside of this abstraction is, of course, that environments that provide only static linkage can't even load the intrinsic interpreter methods. Not so! We can statically link those methods by dlpreopening them.

Unfortunately, since platforms such as AIX and cygwin require that all library symbols must be resolved at compile time, the interpreter maintainers will need to provide a library to both its own dlpreopened modules, and third-party modules loaded by dlopen. In itself, that is not so bad, except that the interpreter too must provide those same symbols otherwise it will be impossible to resolve all the symbols required by the modules as they are loaded. Things are even worse if the code that loads the modules for the interpreter is itself in a library - and that is usually the case for any non-trivial application. Modern platforms take care of this by automatically loading all of a module's dependency libraries as the module is loaded (libltdl can do this even on platforms that can't do it by themselves). In the end, this leads to problems with duplicated symbols and prevents modules from loading, and prevents the application from compiling when modules are preloaded.

,-------------.    ,------------------.    ,-----------------.
| Interpreter |---->     Module------------>   Third-party   |
`-------------'    |     Loader       |    |Dlopened Modules |
                   |        |         |    `-----------------'
                   |,-------v--------.|             |
                   ||  Dlpreopened   ||             |
                   ||    Modules     ||             |
                   |`----------------'|             |
                   |        |         |             |
                   |,-------v--------.|    ,--------v--------.
                   ||Module Interface||    |Module Interface |
                   ||    Library     ||    |     Library     |
                   |`----------------'|    `-----------------'
                   `------------------'

Libtool has the concept of weak library interfaces to circumvent this problem. Recall that the code that dlopens method-provider modules for the interpreter application resides in a library: All of the modules and the dlopener library itself should be linked against the common library that resolves the module symbols at compile time. To guard against duplicate symbol definitions, and for dlpreopened modules to work at all in this scenario, the dlopener library must declare that it provides a weak library interface to the common symbols in the library it shares with the modules. That way, when libtool links the Module Loader library with some Dlpreopened Modules that were in turn linked against the Module Interface Library, it knows that the Module Loader provides an already loaded Module Interface Library to resolve symbols for the Dlpreopened Modules, and doesn't ask the compiler driver to link an identical Module Interface Library dependency library too.

In conjunction with Automake, the `Makefile.am' for the Module Loader might look like this:

lib_LTLIBRARIES = libinterface.la libloader.la

libinterface_la_SOURCES = interface.c interface.h
libinterface_la_LDFLAGS = -version-info 3:2:1

libloader_la_SOURCES    = loader.c
libloader_la_LDFLAGS    = -weak libinterface.la \
                          -version-info 3:2:1 \
                          -dlpreopen ../modules/intrinsics.la
libloader_la_LIBADD     = $(libinterface_la_OBJECTS)

And the `Makefile.am' for the `intrinsics.la' module in a sibling `modules' directory might look like this:

AM_CPPFLAGS             = -I$(srcdir)/../libloader
AM_LDFLAGS              = -no-undefined -module -avoid-version \
                          -export-dynamic

noinst_LTLIBRARIES      = intrinsics.la

intrinsics_la_LIBADD    = ../libloader/libinterface.la

../libloader/libinterface.la:
        cd ../libloader && $(MAKE) $(AM_MAKEFLAGS) libinterface.la

For a more complex example, see the sources of `libltdl' in the Libtool distribution, which is built with the help of the `-weak' option.

10.4 Finding the correct name to dlopen

After a library has been linked with `-module', it can be dlopened. Unfortunately, because of the variation in library names, your package needs to determine the correct file to dlopen.

The most straightforward and flexible implementation is to determine the name at runtime, by finding the installed `.la' file, and searching it for the following lines:

# The name that we can dlopen.
dlname='dlname'

If dlname is empty, then the library cannot be dlopened. Otherwise, it gives the dlname of the library. So, if the library was installed as `/usr/local/lib/libhello.la', and the dlname was `libhello.so.3', then `/usr/local/lib/libhello.so.3' should be dlopened.

If your program uses this approach, then it should search the directories listed in the LD_LIBRARY_PATH(9) environment variable, as well as the directory where libraries will eventually be installed. Searching this variable (or equivalent) will guarantee that your program can find its dlopened modules, even before installation, provided you have linked them using libtool.

10.5 Unresolved dlopen issues

The following problems are not solved by using libtool's dlopen support:

• Dlopen functions are generally only available on shared library platforms. If you want your package to be portable to static platforms, you have to use either libltdl (see section Using libltdl) or develop your own alternatives to dlopening dynamic code. Most reasonable solutions involve writing wrapper functions for the dlopen family, which do package-specific tricks when dlopening is unsupported or not available on a given platform.
• There are major differences in implementations of the dlopen family of functions. Some platforms do not even use the same function names (notably HP-UX, with its shl_load family).
• The application developer must write a custom search function in order to discover the correct module filename to supply to dlopen.

This document was generated on July, 20 2009 using texi2html 1.76.