Using GT.M external calls to access shared libraries

Many times, when developing MUMPS applications, you may come upon a situation where you need to use a bit of functionality exposed by a Linux shared library. FIS GT.M provides support for this via its external call mechanism. The syntax and semantics are a bit odd, so we’ll step through the implementation of a wrapper for some of the  trigonometry functions exposed by the C standard library. This example will be produced with the intention of being complete and usable.

Assumptions on the Reader

I will assume that you have a reasonably recent GT.M release configured on a Linux system. I will assume that you have access to a bash shell prompt (for those of you using VistA, a captive account which takes you directly into a VistA entry point is not sufficient for this tutorial, as you will be creating several files in the Linux filesystem with tools which don’t exist in or are inaccessible from the GT.M programmer mode. If this applies to you, please see your local guru for help).

I will assume that your local GT.M routines are in $HOME/p, that $gtmroutines is set accordingly, that you have a $HOME/lib directory available, and that your GT.M environment is set up to the point where you can read and set MUMPS globals.

I will also assume that you have working knowledge of MUMPS and C programming, including basic knowledge of pointers and their use for the latter language.

I will also assume that you have gcc and make installed, and that you or your system manager has made them available in your search path. If you are working through this example on your own machine, here are some instructions for getting gcc and make running:

Ubuntu

$ sudo apt-get install build-essential

Other Distributions

For other distributions, please search Google.
First, let’s talk a bit about the overall architecture of the GT.M external call mechanism.

Architecture

The GT.M external call mechanism uses a layered architecture. The GT.M runtime looks in the GTMXC_yourcalltable environment variable to find the location of a .xc file which contains the GT.M to C mappings. The .xc file also contains a path to a shared library (a file ending with the .so extension) in which the external routines are defined. Here’s an abstract overview of what the architecture looks like:

GT.M Callout Architecture

Figure 1.1: Abstract GT.M external call architecture

If we were creating new functionality without trying to access an existing shared library, the wrapper_library.so and wrapped_library.so pieces of the stack would likely be replaced with a single .so file containing the new functionality.

In this case, we’ll derive from this abstract architecture a more concrete architecture to apply to our trigonometry wrapper:

Concrete Architecture

Figure 1.2: Concrete architecture for our applicaton

GT.M Type System

When working with external calls to non-MUMPS shared libraries in GT.M, you need to first come to terms with the fact that you are going to end up writing C wrapper functions for every function you use. Unfortunately, GT.M lacks the intelligence to directly call external functions of arbitrary types and parameter lists, and requires an external call table to map the weakly-typed data of C to the untyped data of MUMPS.

Return Values

In order for an external function to be callable by GT.M, it can only return one of three types:

  • gtm_long_t (a long integer)
  • gtm_status_t (an int)
  • void (function does not return a value)

For our purposes, we will use gtm_status_t for as the return values’ types. This will allow us to return a 0 value from our wrapper functions when successful. Returning a nonzero value will indicate to GT.M that an error has occurred. For the sake of clarity, we will leave extensive error handling as an exercise to the reader.

Parameters

Each parameter can be either an input parameter (GT.M passes the value of this parameter to C) or an output parameter (GT.M passes a reference to this parameter to C, which populates it with a return value). Parameters can be any of the types listed in Chapter 11 of the GT.M UNIX Programmer’s Guide. For our purposes, we will be using gtm_double_t* for both our input parameters and output parameters.

External Call Table

Okay, we’re now ready to look at the format of the external call table (trig.xc in our example). The first line must be a full UNIX path to the shared library that GT.M will call, for example:

$HOME/lib/trig.so

This will tell GT.M to look in /home/your_username/lib/trig.so when trying to resolve the functions defined within the trig.xc external call table.

The remainder of the external call table is a list of definitions which map C functions to GT.M routines; one per line. Ours will look like this:


sin: gtm_status_t m_sin(I:gtm_double_t*, O:gtm_double_t*)
cos: gtm_status_t m_cos(I:gtm_double_t*, O:gtm_double_t*)
tan: gtm_status_t m_tan(I:gtm_double_t*, O:gtm_double_t*)

Using the sin function, let’s break down the format of one of these lines:

  • sin: is the name by which this function will be referred when called by our MUMPS code
  • gtm_status_t is the data type which will be returned by our C wrapper
  • m_sin is the name of our C wrapper function
  • I:gtm_double_t* specifies that the first parameter of our C wrapper is a pointer to a double-precision floating point value. The I specifies that this parameter is used for input to our C wrapper. In this case, this is the number to which the sin function will be applied.
  • O:gtm_double_t* specifies that the second and final parameter of our C wrapper is a pointer to a double-precision floating point value. The O (output) specifier indicates that GT.M will be passing a variable by reference for this parameter, and that our C wrapper will be populating it with a return value.

Here’s the completed external call table, trig.xc:


$HOME/lib/trig.so
sin: gtm_status_t m_sin(I:gtm_double_t*, O:gtm_double_t*)
cos: gtm_status_t m_cos(I:gtm_double_t*, O:gtm_double_t*)
tan: gtm_status_t m_tan(I:gtm_double_t*, O:gtm_double_t*)

C Wrapper Functions

For our C wrapper functions, there are a couple of important conventions to note:

  • The first parameter to each wrapper function must be an int. Although this is not (and must not be) specified in the external call table (trig.xc), it must be included in each wrapper function. GT.M will automatically pass to this parameter a value indicating the total number of parameters passed to our wrapper function. It is essentially the GT.M external call mechanism’s own implicit version of argc.
  • We must tell the preprocessor to include gtmxc_types.h which is located in $gtm_dist

Let’s look at the complete trig.c:

#include <math.h>
#include "gtmxc_types.h"

gtm_status_t m_sin(int c, gtm_double_t *x, gtm_double_t *out)
{
    *out = sin(*x);
    return(0);
}

gtm_status_t m_cos(int c, gtm_double_t *x, gtm_double_t *out)
{
    *out = cos(*x);
    return(0);
}

gtm_status_t m_tan(int c, gtm_double_t *x, gtm_double_t *out)
{
    *out = tan(*x);
    return(0);
}

The points that bear further discussion are the function definitions, such as m_sin(int c, gtm_double_t *x, gtm_double_t *out), and the pointer assignments, such as *out = sin(*x);

The function definitions are unique in that they use the typedefs (defined in $gtm_dist/gtmxc_types.h) for the return type and parameter types. This should facilitate portability among the various flavors of UNIX and Linux that GT.M supports.

The assignments use pointers so that the output value (in this case, gtm_double_t *out) can be accessed from within the GT.M environment. The assignment *out = sin(*x); means that we are assigning the value of the sin function of the data pointed to by x into the memory location pointed to by out. This is what allows GT.M to retrieve the value from within its native environment. When dealing with C pointers, I find it useful to read the “flow” of the operation from right-to-left.

MUMPS Routine

Next, we will build a MUMPS routine to hide our use of the GT.M external call mechanism. This is a good idea in case you ever need to port your MUMPS application to a platform that uses different mechanisms for external calls, such as InterSystems Cache’.

Here is our MUMPS routine, trig.m:

trig ;; trigonometry wrappers

sin(num)
 new result
 do &trig.sin(num,.result)
 quit result

cos(num)
 new result
 do &trig.cos(num,.result)
 quit result

tan(num)
 new result
 do &trig.tan(num,.result)
 quit result

Although you could call the external routines directly, the wrapper-within-a-wrapper approach provides more readable code, hides the call-by-reference from the programmer using your routines, and gives you MUMPS code that is more idiomatically representative of the 1995 MUMPS ANSI standard.

The uniqueness of this routine is in the &wrapper.function() syntax. The &trig.function() syntax instructs GT.M to check the $GTMXC_trig environment variable to find the external call table it should use to execute function(). In official GT.M parlance, trig is a package. There is also a $GTMXC environment variable, which points to the callout table for what is referred to as the “default” package, but in the interests of portability and modularity, we will not cover its use here.

Compiling and Linking

Now that we have our MUMPS code (trig.m), our callout table (trig.xc), and our C wrappers (trig.c), we can create a Makefile to compile and link our wrapper functions into a shared library. Please note that this makefile is specific to Linux and may or may not work on other UNIX or UNIX-like operating systems.

CFLAGS=-Wall

all: trig.so

trig.so: trig.o
        gcc $(CFLAGS) -o trig.so -shared trig.o -lm

trig.o: trig.c
        gcc $(CFLAGS) -c -fPIC -I$(gtm_dist) trig.c

clean: 
        rm trig.so
        rm trig.o

install:
        cp trig.so $(HOME)/lib

Let’s break this down line by line:

  • CFLAGS=-Wall

This line gives us a variable for flags that we will always pass to the C compiler. -Wall instructs the compiler to enable all warning messages. This should always be used, as it will help you to write cleaner code.

  • all: trig.so

This is the first rule of the Makefile, which will be invoked automatically if make is run with no command-line arguments. It simply says that the rule “all” depends on “trig.so” to be considered complete. So, if “trig.so” does not exist, make will then scan the Makefile to find a rule to use to build “trig.so”

  • trig.so: trig.o

This is the rule for building “trig.so”. It simply means that trig.so depends on the existence of “trig.o” in order to build it. If “trig.o” does not exist, make will search for a rule to build “trig.o”

  • gcc $(CFLAGS) -o trig.so -shared trig.o -lm

This is the command necessary to generate “trig.so” from “trig.o”. The “-o” flag tells the linker to name the output “trig.so”. The “-shared” flag tells the linker that we wish to generate a shared library from “trig.o”. “-lm” tells the linker that this production depends on libm.so (the “-l” flag automatically prepends “lib” onto the library we specify. “-llibm” does not work).

  • trig.o: trig.c

This is the rule from building “trig.o” from “trig.c”. It informs make that trig.o requires trig.c in order to be built. Since there is no rule in this Makefile for generating “trig.c”, make will look for “trig.c” in the current working directory.

  • gcc $(CFLAGS) -c -fPIC -I$(gtm_dist) trig.c

This is the command necessary to generate “trig.o” from “trig.c”. The “-c” flag instructs gcc to compile, but not link, the specified file. The “-fPIC” flag instructs gcc to generate position-independent code, which means that the symbols in the object file may be relocated and resolved at load time rather than link time, which is necessary for shared libraries. The “-I$(gtm_dist)” flag tells the compiler that it should search $(gtm_dist) for header (.h) files. This is necessary because gtmxc_types.h will not likely be in a place that the compiler knows about. $(gtm_dist) is an environment variable which is required in order for GT.M to function, and will always contain the path in which gtmxc_types.h is located.

Putting it all together

When you have trig.m, trig.c, trig.xc, and Makefile typed in, run the following commands from the shell prompt:

$ cp trig.m $HOME/p/
$ make
$ make install
$ export GTMXC_trig=$HOME/lib/trig.xc
$ mumps -dir
GTM> w $$sin^trig(2.53)

GT.M will respond by writing .574172 to the screen.

Where to go next

Refer to the GT.M UNIX Programmer’s Manual for more information on advanced uses of the GT.M external call mechanism.

I hope this article proves useful, and welcome your feedback!

Advertisements

8 thoughts on “Using GT.M external calls to access shared libraries

  1. Great explanation!

  2. The section on writing the MUMPS routine doesn’t explicitly say it’s named “trig.m” as a file, so that has to be deduced implicitly when you refer to it in the section about putting everything together. Good, clear, complete article, otherwise. Thanks. Have you done one on the topic of calling the GT.M facilities from another language? I understand there’s already a port to Python. I’ll want to work from Perl, for a project I have in mind.

    • John,

      I clarified the filename of the MUMPS routine. Thanks for the suggestion!

      I haven’t done one on calling GT.M from another language, but you would need to use one of the call-in facilities. GT.M provides call-in APIs for both Java and C. To get into it from Perl, you’d need to write a Perl extension in C and use the C call-in API. I have implemented a generic set of C functions that expose familiar MUMPS operations (get, set, kill, data, order, etc.) via the GT.M call-in interface. It might be a starting point for your project, if you could adapt it to work with Perl:

      https://github.com/CoherentLogic/gtmci-data-api

      • Hey, thanks so much! That’ll be a big help, along with my looking at the Python interface I’ve heard already exists. Are you familiar with that one? I assume it’s a bit similar to what you have. I just haven’t looked in any detail yet.

      • Oh, that’s odd. I think maybe the clock isn’t set accurately on your web server.

      • John,

        I’m not familiar with the Python interface, although my colleagues have said that there is not just one, but several. I only very recently started picking up the Python language.

        The time zone was set incorrectly for my blog, hence the wonky times. I fixed that, too.

        If you need any help with gtmci-data-api, feel free to hit me up here or in the #cfmumps IRC channel on Freenode. You may need some guidance for setting it up, as I haven’t written much documentation for it. I can help with GT.M call-in issues, but I’m afraid I wouldn’t be much help with adapting it for Perl, as I’ve never extended Perl in C. Used to do Perl when employed by General Dynamics, but I’ve breathed since then.

  3. John,
    This is a great write-up of the issues for an external library that essentially provides a pure-functional interface. Thank you for writing it.

    Have you dealt with an external interface to any C code that needs to have some dynamic data that it creates and frees (presumably with some form of malloc() and free() ) as it is processing ?

    I think if the dynamic data is totally freed between calls to the library that your pure-functional approach can still work.

    Mly question is re the situation where data is created on the initial call into the package and then is consulted by all the other functions of the package and essentially is unchanged between calls.

    I thought about using a static variable in C, but as this creates a shared object file, I think that has only one copy for the many possible callers from more than one GT.M process. My desire is to have a local copy for each GT.M calling process, but have the memory that is allocated continue to exist after the call from GT.M is completed. As there is no way to specify a “on EXIT” function to call, I don’t know how that memory would be freed if there are no more calls to the package from the GT.M code. Hopefully the HALT command in GT.M will return all of the used memory for this process back to the Operating System.

    Thanks for Info,
    Dave Whitten
    713-870-3834

    • Hi Dave,

      The static variable approach should work, as shared libraries only share their text section, AKA code section. Linux’ loader sets the permission bits on memory pages containing code so that their contents are immutable, and maps the same code section (and same memory address space) of shared libraries into the address space of each process that accesses the shared library.

      Data sections and the global offset table, as well as any other part of the shared library that is writable, are process-private. Linux’ loader maps data sections for shared libraries to unique, non-conflicting address spaces for each process that accesses the shared library.

      GT.M provides its own malloc function (I believe it’s called gtm_malloc()) for use in situations like these, as GT.M must track more information about memory in its process space than is afforded by libc-provided versions of malloc().

      HALTing a GT.M process results in all of that process’s memory being freed, even if GT.M has memory leaks of its own, as Linux itself will free anything unused once the owning process terminates.

      Let me know if you have any more questions!

      Thanks,
      John Willis

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s