Dynamic Object File Loading

Introduction
Binary classlibraries (packages)
Individual binary classes
C libraries / C modules
Architecture specific notes
- AIX (3.2.5) shared objects
- Solaris shared objects

Introduction

On systems which support dynamic loading of object modules, (currently, these are: SGI, Linux (both a.out and ELF based), Unixware, SUNOS4.x), ST/X allows that binary files are loaded and unloaded dynamically to the running system - without a need to leave the session.

Binary classlibraries (packages)

Binary class libraries consist of a group of classes, which are contained in a single object file and loaded/unloaded together.
Class libraries can be statically included in the initial executable (the whole system that surrounds you typically is) or, alternatively, added dynamically to an executing image.

Loading

To load a class library, select it in the fileBrowser, and execute the fileIn menu function.
The system automatically detects the file as a binary object (in contrast to a smalltalk source file) and loads it as a binary object.

Programmatically, this can be done with:

    Smalltalk fileIn:'path-of-the-binary-file'

The pathName argument must specify the binary objects complete path, including any suffixes (such as ".so" or ".dll").
This is of course not portable across different architectures, both due to differnet directory names, and since filename extensions are highly system specific.
If you can place your class libraries into some standard directory (i.e. a directory named "binary" somewhere in the searchPath), you should use a more portable interface, which only requires the binaries base name, and uses the architecture specific filename extension:

    Smalltalk fileInClassLibrary:'name-of-the-library'

The searchPath is the standard ST/X searchpath, which can be defined via the shell environment or in the "startup.rc" file. Thus, a programmer using dynamic class libraries should write:

    Smalltalk fileInClassLibrary:'libFoo'.

and place the corresponding "libFoo.so" or "libFoo.sl" or "libFoo.dll" file into one of the binary directories.
(you can also ask the ObjectFileLoader class about the systems preferred file extension and construct the files full name manually; try "ObjectFileLoader sharedLibraryExtension").

When loaded, all classes contained in that package are installed, and #initialize messages are sent to the classes in the order of inheritance (i.e. superclasses get this message before any subclasses). Classes of which the superclass is missing (i.e. a binary class of which the superclass is in another binary file) will not be installed; however, these are rememebred to be installed and initialized as soon as the missing superclass is loaded. If the missing superclass is marked as an autoloaded class, ST/X tries to load it first, by searching for an appropriate binary (along the searchPath).

If the classLibrary contains any class named after a class in the executing image, the old class will remain physically present as an object, but will be no longer accessible via the global name table (i.e. "Smalltalk at:name" will return the new class).
This allows existing instances to continue to function correctly, in case the instance layout changes, or invalid code is loaded, or the load failed for some reason.

If a class library is reloaded (i.e. it was already loaded), the old code is unloaded (see below) before the new code is read.

Unloading

Once loaded, the system keeps track of which objects are loaded and which classes/methods refer to instructions in that library.
The finalization mechanism is used to detect when the last reference to the object file ceases to exist (i.e. there are no more methods/classes from that module in the system) and the module is automatically unloaded.

This happens, if you remove those classes in the browser AND there are no more instances of them in the system.

You can also force a class to be unloaded, by opening the launcher's file-modules dialog, selecting a package and pressing the unload button. By the way: this dialog also shows a packages contents and the classes revision numbers - useful to validate a correct version has been loaded.

Existing methods which still refer to this binary are no longer executable. (the browser marks them as ** unloaded ** and a method invocation will raise an invalidCode exception).

Since the unload is automatic in normal circumstances, no special programming interface exists for unloading.
(However, see the "ObjectFileLoader" class for low level access to loaded modules and load/unload functionality.
This classes protocol is private and subject to changes, and we do not guarantee this to remain unchanged forever.)

Before the actual unlaoding is performed, all classes in the module receive a #deinitialize message, to give them a chance to perform any required cleanup actions.

You can also arrange for #aboutToUnload change notifications to be sent - to either the unloaded classes or any other watcher object.
To receive these, the interested object must be a dependent of the ObjectFileLoader and check for a #aboutToUnload argument in the #update:with:from: method. The second with:- argument will be the class which is about to be unloaded.
If the unloaded class is itself interested, it should check this argument for identity to itself, before performing any cleanup actions. (which is why the #deinitialize mechanism is probably easier to use ...).

Errors during the load

While loading, new classes are validated against their superclasses definitions and the installation of a class or a number of classes may be suppressed.

The error most likely to occur is that a compiled class inherits from some other class, which has changed its definition (i.e. added or removed an instance variable) in the meanwhile, and the subclass has not been recompiled.
This situation is detected, by comparing the classes signatures, and ignoring any class(es) from the package, for which this check fails.
Other (valid) classes are not affected and the package may be partially loaded.

Therefore, as a programmer, it is a good idea to check for all expected classes to be present after the load.

Snapshot images & dynamic libraries

When a snapshot image is saved, sufficient information about all loaded binaries is kept and saved with the image, to allow those modules to be reloaded at restart time.

Therefore, be careful when removing any binary class libraries - there could still be references from some passive snapshot images to them !!

If some binary is no longer present or loadable at restart time, all methods depending on it are invalidated (similar to the above described unload behavior).
This means, that the image is still runnable, as long as those missing methods are not invoked. If they are, you'll end up in the debugger.

Individual binary classes

The required actions to load object files containing individual classes (as opposed to packages) are the same as described above.
However, the Makefiles as created by the configuration process (or stmkmp) are not prepared to generate such files.
Therefore, you have to either manually change the "Make.proto" files, or by generating a loadable file via command line linker invocation.

The details vary among system architectures:

System command comment
Linux a.out -- can directly load ".o" files --
Linux ELF ld -shared -o file.so file.o
SGI ld -shared -o file.so file.o
SUNOS4.0 -- can directly load ".o" files --
Solaris ld -G -B dynamic -o file.so file.o
Unixware ld -G -o file.so file.o
NeXTStep 3.x dynamic loading not (yet) supported
HPUX 9.x dynamic loading not supported
HPUX 10.x dynamic loading not (yet) supported
AIX 3.2 cc -bI:.. -bE:file.exp -bM:SRE file.o -o file.so requires librun.so & librun.exp
REAL/IX dynamic loading not supported
NT link /DLL /OUT:file.dll /DEF:file.def file.lib need to create/edit a .def file
OSF/1 dynamic loading not supported
OpenVMS dynamic loading not supported

System	command	comment
Linux a.out		-- can directly load ".o" files --
Linux ELF	ld -shared -o file.so file.o
SGI	ld -shared -o file.so file.o
SUNOS4.0		-- can directly load ".o" files --
Solaris	ld -G -B dynamic -o file.so file.o
Unixware	ld -G -o file.so file.o
NeXTStep 3.x		dynamic loading not (yet) supported
HPUX 9.x		dynamic loading not supported
HPUX 10.x		dynamic loading not (yet) supported
AIX 3.2	cc -bI:.. -bE:file.exp -bM:SRE file.o -o file.so	requires librun.so & librun.exp
REAL/IX		dynamic loading not supported
NT	link /DLL /OUT:file.dll /DEF:file.def file.lib	need to create/edit a .def file
OSF/1		dynamic loading not supported
OpenVMS		dynamic loading not supported

Since the existing makefiles ``know'' how to create loadable classLibraries (if supported by your architecture), we recommend placing all binary classes into a classlibrary - even if it only consists of a single object file.
If you insist on creating loadable modules consisting of a single class, you have to create those ``manually'' on some systems.

C libraries / C modules

Beside smalltalk class binaries, ordinary c object modules and libraries can be dynamically loaded with:

    |moduleHandle|

    moduleHandle := ObjectFileLoader
			loadDynamicObject:'binary-object-path'.

the individual functions are extracted with:

    |functionHandle|

    functionHandle := moduleHandle
			getFunction:'nameOfCFunction'.

and finally, that function is callable via:

    functionHandle call

There are various call methods provided, for passing different types and number of arguments to the C function. See the ExternalFunction classes protocol for detailed information.

Like with class libraries, these modules can be unloaded, making the corresponding function handles become invalid (calling them leads to a primitiveFailure exception) and are reloaded when a snapshot image is restarted.

If the binary contains a function named "__<fileName>__Init()", that function is called when the module is loaded. This allows initialization code to be performed automatically, before any other function calling takes place.
This function is also called when the binary is reloaded at image restart time.

In addition to initialization, this function can also signal failure by returning a negative integer value (i.e. -1). In this case, the module will be immediately unloaded and failure will be reported to the caller on the smalltalk side.
To report success, the init function must return a value >= 0.

A similar de-init function named "__<fileName>__deInit()" is called (if present in the binary) when the module is unloaded, allowing the C code to perform any required cleanup.

The deinit function is always called - even if the unload is due to the init functions failure return.

Example c files usable for dynamic load experiments are found in "doc/coding/cModules".

Architecture specific notes

Some systems require special actions, for shared objects to be created; the following only applies to specific systems.

AIX (3.2.5) shared objects

On aix (3.2.5), the linker needs an exports file (which defines the symbols of a shared object, which are to be exported), and also import files which define the names to be imported from other shared objects.

The makefiles as provided create export files, which must be kept aside the corresponding classLibrary file (if you move/copy shared objects to other machines, do not forget to copy those as well).
The exports file of the runtime library has been created by hand, and is found in "librun/librun_aix.exp".

At link time, these exports files must be present and reachable.

We recommend, creating a directory named: "/usr/local/lib/smalltalk/lib", where all shared libraries and export files are copied to (especially, the librun-exports file).

To create a shared object, use the command (or make-rule):

    cc -bI:/usr/local/lib/smalltalk/lib/librun_aix.exp
       -bE:libName.exp
       -bM:SRE
       -o libName.so
       libName.o

after that, "libName.so" is ready to be loaded via fileIn.

If individual methods containing inline C-code are to be accepted in the browser, the system executes a command similar to the above.
Here, you also have to make certain, that "librun/librun.so" and "librun/librun_aix.exp" have been copied to "/usr/local/lib/smalltalk/lib" - otherwise, the linkage operation will fail.

Solaris shared objects

Dynamic class libraries must be compiled with the -fPIC compiler option; otherwise, a segmentation violation may occur when the object is loaded. The configuration as delivered already contains the required definitions, so that Makefiles when generated from a Make.proto will have this option set.
If the dynamic object needs other dynamic libraries, make certain that those will be found at load time - either by adding the path to the library to the LD_LIBRARY_PATH shell variable, or by placing a copy/symbolic link to those libraries into your current directory, or by adding a -Rpath option to the link command of your class library.

<cg at exept.de>

Doc $Revision: 1.28 $ $Date: 2021/03/13 18:24:51 $

Dynamic Object File Loading

Contents