Magic DLLs

Magic allows you since long to call external functions residing in a DLL (Windows Dynamic Link Library) by means of the the function UDF() (User Defined Function) and the operation CALL UDP (User Defined Procedure).

Before Version 8 ,which added support for "standard DLLs", these DLLs however had to obey special rules defined by Magic and they were usually solely built for usage within Magic applications. Its were said to be "Magic DLLs" or "Magic extension modules" then.

The requirements for a Magic DLL are:

• the DLL needs to export a special function named MAGIC_BIND (which has to return a pointer to a "extension module definition" as defined in USER_DLL.h)
• calling convention for all functions in a Magic DLL made visible to Magic thru the extension module definition needs to be: _cdecl

Besides there are restrictions regarding the supported data types. For a complete list see the page about supported "data types".

Long time only fundamental data types and pointers to them were supported. V9 added the (Magic) data type "BLOB" to the list of data types possible to pass as parameter into a DLL. With this and together with the buffer manipulation functions it is also possible to use elaborated data types (structures) as parameters for DLL functions now.

Version 8 (advent of CallDLL) added support for "Standard DLLs" to the tool. Before it were only Magic DLLs which were supported in Magic. Some more explanation regarding "Standard DLLs" and how they are used you'll find on Arch's page about Windows API. It also shows how tu use buffer manipulations to fill the elaborated data type "structure".

Version 8.3 then added a special "@" notation to the 'module_name.function_name' specifiers required for CALL UDP. Since then it is also possible to use Standard DLLs with CALL UDP, which is particularly important since it is the CALL UDP operation only which allows a function residing in a DLL to change any of its parameters. Functions inside a DLL invoked by any of the Magic DLL functions can only return a value.

Version 8.3 also added support for another calling convention (FASTCALL) and added functions (+ setting for CALL UDP) to explicitely specify the required calling convention.

Basically, since then, there is no requirement for "Magic DLLs" anymore in terms of functionality. They are however still in use and developers sometimes do still add a MAGIC_BIND export to their own DLLs. One reason for that is that it saves you to specify the parameter string as first argument in UDF() and CALL UDP; Magic can read that from the DLL itself and compare actual with defined parameters, do some type checking , ...

Famous examples for Magic DLLs or "Magic extensions" as they are sometimes referred to, are Craig Martin's Hotfudge and GET.DLL from Dominique STÉPHAN. These two extensions which you find in many Magic/eDeveloper applications however go beyond of simply adding functions to Magic which were hard or even impossible to achieve with Magic. These two DLLs also "subclass" (or superclass) parts of Magic which effectively allows you to change look & feel and some of the behaviour of Magic applications during runtime.

Just recently mgBoost∞ has been started and it is (intended to be) a series of libraries (DLLs for Windows; shared libraries for Unix) with eDeveloper enhancments. The C/C++ source code there - which is freely available for download - might give some more insight in development of "Magic DLLs". The libraries you find in the mgBoost package do however not adhere strictly to the requirements for "Magic DLLs". Actually they pursue a "hybrid" approach.
All the interface is exported and all exported functions use callinc convention stdcall. The libraries do however have a MAGIC_BIND as well and therefore the functions can be used with UDP (cnv: Standard) and UDFS().

By wrapping API functions in a Magic DLL it is also possible to workaround some subtle issues with alignment of structures immanent since BLOB parameters and buffer manipulation functions are there.

A different approach for solving this issue would be COM and ActiveX. Check the information available on data alignment for more information on this issue.

---

Development considerations (notes for Magic extension builders)

The SDK for Magic DLLs basically is a C header file named USER_DLL.H + some sample source code (written in C as well). To get this you need to check option "User defines procedures" during installation of Magic/eDeveloper. Its possible to add this to the list of installed features when you already have Magic installed.

MAGIC_BIND

Magic DLLs need to have in their data section a special data structure filled with information about the "extension module". Besides few other things this structure contains an pointer to an array of "function descriptions" - the (so called) "function description table". A function description contains everything Magic needs to dynamically invoke this function during runtime and check actual parameters against defined parameters. Besides the address a function description contains the function's names, the number and type of parameters and the return value type (if any).

It is a special function named "MAGIC_BIND" then which has to return a pointer to the extension module definition. MAGIC_BIND is the only function in a Magic DLL which needs to be explicitly exported and which is required to have a well-know signature.

The only thing which is hardcoded in this concept (besides MAGIC_BIND) is the calling convention used for invocation of these functions. It has to be (per definition): _cdecl. If you built your own Magic DLL, you need to ensure that this calling convention is used (for C/C++ its the default anyway).

If you choose to also explicitely export all the other public functions in your DLL you get a "hybrid interface". These DLLs can be used from within other programming languages (as long as they support _cdecl) as well.



Since a Magic DLL contains a function description table, there's no need to tell Magic about the signature of any of these functions. Magic will read these from this table. This is different from what you have to do when you invoke a function within a "Standard DLL". Regardless what function/operation is used to invoke a function from a standard DLL (UDF, UDP, CallDLL, ...), the first parameter (following the module.function_name) always has to be a string which tells Magic about number and type of parameters. See StandardDll for more information about this.

UDF vs. UDP

Magic distincts between user defined functions and - procedures. User defined procedures (UDP) are not able to return any value (would be a void function in C/C) but are able to return values in their parameters. User defined functions (UDF) can return a value but are not able to change any of their parameters. Thats the fundamental difference.

eDeveloper v.10 meanwhile renamed CALL UDP operation to "Invoke UDP"; Magic v.8 added variants of UDF() to the list of available functions, namely UDFS() and UDFF(). In similar way 8.0's CallDLL() has been brought in society with functions CallDLLS() and CallDLLF(). The variants of UDF/CallDLL are mainly about supporting different calling conventions. Functions from a Magic DLLs however still can only be invoked with UDF() and CALL UDP (v10: Invoke UDP). This is because only these support calling convention "C" (_cdecl).

The restrictions for UDF also hold for CallDLL (and its variants).

Note: the concept of strictly distinguishing between prodedure (UDP - NOT possible to return a value but able to alter parameters) and functions (UDF - possible to return a value but NOT able to alter parameters) has been wekened meanwhile. You can use CALL UDP operation to invoke functions from Standard DLLs and thes functions can do both. They can alter parameters and they can return a value. This was required because this is how it is with DLLs. Since you cannot specify a return variable in CALL UDP operation, you need to put the Magic variable which will receive the return value as last parameter in the parameter selection box of CALL UDP.

pragma pack(1)

There's were plenty of discussion on the list regarding the requirement of having to pack your data structures on byte boundary when building a DLL for usage within Magic.

Note that the advise often given: "Change your compiler settings to alignment 1 Byte" is not completely correct and that you should avoid this for performance and compatibility reasons.
Reason for all this is mis-aligned (non naturally aligned) structure definitions in the USER_DLL.H and a wrong pragma pack instruction there.

Consider following:

#pragma pack(1)
 struct
{
Ushort use_cnt; /* only for Magic use */
void USER_FAR *ini_func_addr; /* called upon initialization */
void USER_FAR *trm_func_addr; /* called upon termination */
Ushort func_cnt; /* number of functions in module */
FUNC_DSC USER_FAR *func_dsc_tbl; /* function table */
Uchar USER_FAR *module_name; /* name (label) of module */
Ushort context_hdl;
} EXT_MODULE;
#pragma pack (4)

In above structure a 32-Bit compiler would insert two pad bytes after use_cnt (which is 2 Bytes) to naturally align ini_func_addr (which is a 4 Byte pointer) to a multiple of its size.

use_cnt is a 16 bit value which starts at offset 0 of this structure. It allocates Byte 0 and 1. The 4 Byte ini_func_addr would be naturally aligned at offset 4 (allocating bytes 4, 5, 6 and 7).
pragma pack(1) aligns it on byte boundaries however. ini_func_addr now starts at offset 2 (allocating bytes 2, 3, 4 and 5). Its said to be "mis-aligned".

Since Magic compiles compiles those parts of its binaries which read above structure with the pragma pack instructions as given above, you need to do this as well. You cannot use a different alignment (by e.g. removing the pragma pack instructions). If you do, Magic is not able to read your EXT_MODULE definition correctly anymore, loading your DLL (extension module) will fail.

No need however to use data alignment 1 Byte for all your code. You just need it to access the mis-aligned data structures in USER_DLL.H.

The pragma pack (4) shows you that you can change alignment temporarily and set back to different values afterwards. pragma pack (4) is wrong however for 32-Bit Windows. Default there is alignment on 8 Bytes boundaries.

You should replace the "#pragma pack (4)" by "#pragma pack ()" if your compiler supports this or (better) pragma pack push(1)/pop() instrucions. After EXT_MODULE is aligned on one byte boundaries, this will effectively switch back to default alignment for your platform (or whats currently active). Notation for your compiler may be different. Check the available documentation on pragma pack and data alignment.

Note: When working with mis-aligned data structures as above and forbidding the compiler to align that naturally for you by pragma pack instructions, you may get serious performance problems on 32-Bit platforms. On a 64-Bit platform your program most likely will crash and experience even more severe performance problems if you instruct it to add the require fix-up code and handling for you. See MSDN on SEM_NOALIGNMENTFAULTEXCEPT for more information on this issue.

fix-up code is the code the compiler or the system adds to be actually able to access the mis-aligned ini_func_addr from the structure above. Modern RISC based CPUs can't do that. The fix-up code therefore will load two WORDS (first WORD being bytes with offsets 0, 1, 2 and 3; second WORD being bytes with offsets 4, 5, 6 and 7). Afterwards data is extracted and shifted together to get the required WORD value (which is stored at offsets 2, 3, 4 and 5) and store that in a CPU register. You can imagine that this is a tremendous extra effort. Naturally aligned data can be moved with a single CPU instruction to a register.