Using C libraries from Julia

Keith Rutkowski

Keith Rutkowski
October 22, 2019

Using C libraries from Julia
Improving the user's experience when using C libraries from Julia by using CBinding.jl to avoid the pain points.

The Julia programming language

We have been developing software in Julia for over 5 years now, and had been writing C and C++ code for over 15 years before that. Julia is a superb language and package ecosystem for science, engineering, research, and even software development in general. The language also very elegantly empowers users to take advantage of compile-time optimization capabilities without requiring an understanding of complicated (and nearly unreadable) template meta-programming techniques like C++ does (for those not familiar with that, just take a quick glance at the source code for CGAL or Eigen).

While Julia has solved some rather challenging problems, it is still a relatively young language. Many organizations already trusted solutions written in C or C++ and while Julia presents a powerful new capability, some of the features that they rely on are missing or not yet mature enough. Often users wish to use their existing C-based software from within Julia and migrate their implementations to Julia as time or funding permits.

Julia’s built-in facilities for binding with C libraries are thoroughly explained in Calling C and Fortran Code in the Julia documentation. Please start there if you are not yet familiar with them since this article assumes you have at least a basic working knowledge of interfacing C with Julia.

Interfacing C with Julia: a painful endeavor

Even though Julia provides facilities for using C libraries, just about every practical C API exceeds those facilities, and the Julia documentation notifies developers of this fact. Unfortunately, this is one place where some of the new users encounter a bit of tool friction when adopting Julia. Having worked through these challenges, we thought it would be beneficial to highlight the following pain points so that you can avoid them.

Wow! That seems like a long list of rather common C constructs which are not represented in Julia, doesn’t it? But there is some good news! We have developed a package, CBinding.jl, to add proper support for C (ANSI C, C89, C99, C11, and C18) constructs to Julia. Let’s have a look at each pain point in detail before we present how our package brings proper support for C to Julia.

Pain Point #1: No support for union

You can get a near approximation of a union if you know, a priori, the field that will have the greatest size (potentially including padding). When translating your fields to Julia, declare the Julia field to be only of that type.

— Calling C and Fortran Code, Julia 1.2 Documentation

Many C libraries use union, so it is frustrating to integrators when there is no support for them in Julia. A union specifies the subset of types that a field may have, and it can be used as an idiom to mimic polymorphism, which the following code illustrates.

struct Number {
  enum {
  } kind;

  union {  (1)
    int i;
    unsigned int u;
    double d;
1 A nested union that has the size of a double and causes the struct to have padding inserted between the enum and union to satisfy memory alignment requirements.

In order to use Number, Julia code must drop the type information of the nested union and replace it with a Cdouble (the largest of its fields). The code using the struct must use reinterpret calls along with other byte-level intrinsic operations to try to get the behavior they desire. This is not a very elegant solution for users to encounter.

Pain Point #2: No support for enum

…​if T is an enum, the argument type should be equivalent to Cint or Cuint…​

— Calling C and Fortran Code, Julia 1.2 Documentation

Enumerations are as ubiquitous as unions in C API’s. Julia provides the @enum macro to define Julia’s enumerations which are, unfortunately, not compatible with C’s enumerations. A user is directed by the documentation to use integer primitive types and a bunch of constant definitions to mimic the behavior of C. The CEnum.jl package claims to provide a "C-compatible" enumerations for Julia, so we used it initially to interface with any enum defined in a C header file. However, we eventually discovered that it doesn’t entirely comply with C enum semantics, and the following example is one that exposes that.

enum Choice {
  CHOICE_LAST = 0xffffffff,  (2)
1 An enum having multiple labels for the same value.
2 CHOICE_LAST and CHOICE_UNDECIDED are distinctly different values and change the size of the enum.

The package is not able to represent enumerations having multiple labels for the same value, and it doesn’t correctly determine the integer type used for storing an enumeration (which is actually derived from the enumeration’s values). An enum could be of type Cint, Cuint, Clonglong, or Culonglong depending on its set of values, but CEnum.jl leaves that detail to the user to know and work around.

Pain Point #3: Nested struct or union is not supported

When mirroring a struct used by-value inside another struct in C, it is imperative that you do not attempt to manually copy the fields over, as this will not preserve the correct field alignment. Instead, declare an isbits struct type and use that instead.

— Calling C and Fortran Code, Julia 1.2 Documentation

Nesting a struct or union is pretty standard in C API’s. In fact, Julia’s header file offers many examples of composition using both constructs, like this one:

struct _jl_taggedvalue_bits {
  uintptr_t gc:2;  (1)

struct _jl_taggedvalue_t {
  union {  (2)
    uintptr_t header;
    jl_taggedvalue_t *next;
    jl_value_t *type;
    struct _jl_taggedvalue_bits bits;  (3)
1 Usage of an integer bit field within a struct.
2 An anonymous/unnamed nested union within a struct.
3 Storage of a struct within a nested union.

Actually, that small snippet of code demonstrates several C features which have no representation in Julia. It uses bit fields, an anonymous nested union, and composition of a struct within a union. A more simplistic illustration of a nested struct would look like this:

struct Inner {
  int i;

struct Outer {
  struct Inner i;  (1)
  int j;
1 The nesting of an Inner object within Outer presents a challenge to Julia developers.

In Julia, a nested struct must be immutable, otherwise the outer struct will store a reference to the nested struct. If the nested struct is also used as a top-level struct in the C API, then it must be declared mutable. To compensate for the contradiction in mutability, both variants of the type must be defined in Julia. The following code shows the approach recommended by the Julia documentation to support the simple C code above.

mutable struct Inner  (1)

struct _Inner  (2)

mutable struct Outer
1 Definition of the mutable form of Inner for use as a top-level object.
2 An identical definition for an immutable form of Inner for use within Outer.

This simple example begins to reveal how confusing and unmanageable the code can become when porting many nested aggregate types to Julia.

Pain Point #4: Anonymous struct, union, and enum are not supported

Unnamed structs are not possible in the translation to Julia.

— Calling C and Fortran Code, Julia 1.2 Documentation

Anonymous fields are also frequently used in types defined by C API’s. They let developers avoid name pollution, particularly when the type is only used once in a very specific context. The first pain point serves as an example using both an unnamed enum type and an unnamed union type. All unnamed types must be given names for their Julia representation.

Pain Point #5: No support for alignment strategies

Packed structs and union declarations are not supported by Julia.

— Calling C and Fortran Code, Julia 1.2 Documentation

Using a packed alignment strategy is a feature that rarely affects C development, but it is one that system-level code relies on. The following example of this comes from the ALSA SoC header file:

struct snd_soc_tplg_tlv_dbscale {
  __le32 min;
  __le32 step;
  __le32 mute;
} __attribute__((packed));  (1)
1 Use of __attribute__((packed)) on the definition of a struct to prescribe its alignment strategy.

Usage of __attribute__((packed)) on the struct informs the compiler to tightly pack its fields, and it also enables tightly packed arrays of the struct. It is like telling the compiler to disregard any performance impacts cause by memory misalignment in order to minimize the memory footprint of the struct. There is no capability for changing the alignment behavior of Julia types, just as the documentation says, so interfacing a system-level API, like ALSA, is prohibitively difficult.

Pain Point #6: No support for integer bit fields

In general, integer bit fields are used to reduce a type’s memory footprint, and their usage is common in system-level development where bit-twiddling is a standard modus operandi. Actually, we can look in Julia’s own header file for an example of something the language itself cannot properly represent:

typedef struct {
  uint16_t how:2;  (1)
  uint16_t ndims:10;
  uint16_t pooled:1;
  uint16_t ptrarray:1;
  uint16_t isshared:1;
  uint16_t isaligned:1;
} jl_array_flags_t;
1 Several integer bit fields pack into a single uint16_t in this struct.

Another property of integer bit fields is that when they are declared using signed integers they use \(N\)-bit two’s complement, where \(N\) is the number of bits specified by the field. This detail is rarely used by C API’s, and few developers know about the behavior. Therefore, it illustrates that when no authoritative solution exists in Julia, developers will implement C features on their own that are likely not compliant with C.

Pain Point #7: No support for type-safe function pointers

T (*)(…​) (e.g. a pointer to a function) ⇒ Ptr{Cvoid}

— Calling C and Fortran Code, Julia 1.2 Documentation

A popular C idiom is to accept user-provided callback functions in order to enable functional composition and customization within the library’s algorithms. Another common idiom is to store function pointers within a struct to create a form of object-oriented interfaces. We can find some good examples of these idioms, shown below, in C frameworks like GLFW, SDL, and Libav.

/* from GLFW/glfw3.h */
typedef void (* GLFWerrorfun)(int, const char*);
GLFWerrorfun glfwSetErrorCallback(GLFWerrorfun callback);  (1)

/* from SDL2/SDL_thread.h */
typedef uintptr_t (__cdecl * pfnSDL_CurrentBeginThread)(
  void *, unsigned,
  unsigned (__stdcall *func)(void *),
  void * /*arg*/, unsigned, unsigned * /* threadID */
typedef void (__cdecl * pfnSDL_CurrentEndThread) (
  unsigned code
SDL_Thread * SDL_CreateThread(
  SDL_ThreadFunction fn,
  const char *name,
  void *data,
  pfnSDL_CurrentBeginThread pfnBeginThread,  (2)
  pfnSDL_CurrentEndThread pfnEndThread

/* from libavcodec/avcodec.h */
typedef struct AVBitStreamFilter {
  const char *name;
  const enum AVCodecID *codec_ids;
  const AVClass *priv_class;
  int priv_data_size;
  int (*init)(AVBSFContext *ctx);  (3)
  int (*filter)(AVBSFContext *ctx, AVPacket *pkt);
  void (*close)(AVBSFContext *ctx);
  void (*flush)(AVBSFContext *ctx);
} AVBitStreamFilter;
1 Setting a user-provided type-safe error callback function pointer for GLFW internal code to use.
2 Using function pointers for the creation of a thread in SDL is type-safe and specifies the calling convention as well (__cdecl and __stdcall).
3 An example of a struct holding type-safe function pointers for filters in Libav.

As directed by the Julia documentation, a user should simply use Ptr{Cvoid} to store function pointers, but that leads to fairly unsafe and fragile code. When using a type-safe, and high-level language like Julia, a user would rather be productive with their development time instead of managing type information or debugging segfaults.

Pain Point #8: Lack of support for variadic functions

…​variadic functions of different argument types are not supported.

— Calling C and Fortran Code, Julia 1.2 Documentation

The last pain point that we will highlight is the lack of support for variadic functions. Variadic functions allow users to pass arbitrary numbers and types of arguments to them. A classic example of this comes from the C standard library itself:

​int printf(const char *format, ...);​  (1)
printf("%d %s %f\n", 1234, "string", 1.5);  (2)
/* prints "1234 string 1.500000" to the terminal */
1 The printf function signature.
2 An example of calling printf with an arbitrary number of arguments of various types.

Not a lot of C libraries use variadic functions, but some very important libraries (like GLib, GTK, Gstreamer, and others) use them prolifically. When a critical function from such a library cannot be called from Julia, it makes some, or possibly all, of the features of that library inaccessible to Julia users.

The pains go away with CBinding.jl!

So now that you have a good sense of the struggle that a Julia developer faces when interfacing a C library, we wish to introduce you to our work on CBinding.jl. Below, we show our pain-free implementations of all of the pain point examples presented earlier. You will notice that they all have a very natural representation in Julia, yet they still look very similar to their original C form. Additionally, all of these capabilities are fully supported without requiring a Julia developer to do the work of the compiler.

using CBinding
# disclaimer: exact syntax and usage of CBinding may evolve slightly as it continues to mature

# from pain point #1  (1)
@cstruct Number {    # struct Number {
  kind::@cenum {     #   enum {
    INTEGER,         #     INTEGER,
    UNSIGNED,        #     UNSIGNED,
    FLOAT            #     FLOAT
  }                  #   } kind;
  @cunion {          #   union {
    i::Cint          #     int i;
    u::Cuint         #     unsigned int u;
    d::Cdouble       #     double d;
  }                  #   };
}                    # };

# from pain point #2  (2)
@cenum Choice {                     # enum Choice {
  CHOICE_NONE = 0,                  #   CHOICE_NONE = 0,
  CHOICE_FIRST,                     #   CHOICE_FIRST,
  CHOICE_SECOND,                    #   CHOICE_SECOND,
  CHOICE_LAST = 0xffffffff,         #   CHOICE_LAST = 0xffffffff,
  CHOICE_UNDECIDED = -1,            #   CHOICE_UNDECIDED = -1
}                                   # };

# from pain point #3  (3)
@cstruct _jl_taggedvalue_bits {    # struct _jl_taggedvalue_bits {
  (gc:2)::uintptr_t                #   uintptr_t gc:2;
}                                  # };
@cstruct _jl_taggedvalue_t {       # struct _jl_taggedvalue_t {
  @cunion {                        #   union {
    header::uintptr_t              #     uintptr_t header;
    next::Ptr{jl_taggedvalue_t}    #     jl_taggedvalue_t *next;
    type::Ptr{jl_value_t}          #     jl_value_t *type;
    bits::_jl_taggedvalue_bits     #     _jl_taggedvalue_bits bits;
  }                                #   };
}                                  # };
@cstruct Inner {                   # struct Inner {
  i::Cint                          #   int i;
}                                  # };
@cstruct Outer {                   # struct Outer {
  i::Inner                         #   struct Inner i;
  j::Cint                          #   int j;
}                                  # };

# from pain point #5  (4)
@cstruct snd_soc_tplg_tlv_dbscale {    # struct snd_soc_tplg_tlv_dbscale {
  min::__le32                          #   __le32 min;
  step::__le32                         #   __le32 step;
  mute::__le32                         #   __le32 mute;
} __packed__                           # } __attribute__((packed));

# from pain point #6  (5)
@ctypedef jl_array_flags_t @cstruct {    # typedef struct {
  (how:2)::uint16_t                      #   uint16_t how:2;
  (ndims:10)::uint16_t                   #   uint16_t ndims:10;
  (pooled:1)::uint16_t                   #   uint16_t pooled:1;
  (ptrarray:1)::uint16_t                 #   uint16_t ptrarray:1;
  (isshared:1)::uint16_t                 #   uint16_t isshared:1;
  (isaligned:1)::uint16_t                #   uint16_t isaligned:1;
}                                        # } jl_array_flags_t;

# from pain point #7  (6)
@ctypedef GLFWerrorfun Ptr{Cfunction{Cvoid, Tuple{Cint, Cstring}}}         # typedef void (* GLFWerrorfun)(int, const char*);
glfwSetErrorCallback(callback::GLFWerrorfun)::GLFWerrorfun = ...           # GLFWerrorfun glfwSetErrorCallback(GLFWerrorfun callback);
@ctypedef pfnSDL_CurrentBeginThread Ptr{Cfunction{uintptr_t, Tuple{        # typedef uintptr_t (__cdecl * pfnSDL_CurrentBeginThread)(
  Ptr{Cvoid}, Cuint,                                                       #   void *, unsigned,
  Ptr{Cfunction{Cuint, Tuple{Ptr{Cvoid}}, STDCALL}},                       #   unsigned (__stdcall *func)(void *),
  Ptr{Cvoid}, Cuint, Ptr{Cuint}                                            #   void * /*arg*/, unsigned, unsigned * /* threadID */
}, CDECL}}                                                                 # );
@ctypedef pfnSDL_CurrentEndThread Ptr{Cfunction{Cvoid, Tuple{              # typedef void (__cdecl * pfnSDL_CurrentEndThread) (
  Cuint                                                                    #   unsigned code
}, CDECL}}                                                                 # );
SDL_CreateThread(                                                          # SDL_Thread * SDL_CreateThread(
  fn::SDL_ThreadFunction,                                                  #   SDL_ThreadFunction fn,
  name::Cstring,                                                           #   const char *name,
  data::Ptr{Cvoid},                                                        #   void *data,
  pfnBeginThread::pfnSDL_CurrentBeginThread,                               #   pfnSDL_CurrentBeginThread pfnBeginThread,
  pfnEndThread::pfnSDL_CurrentEndThread,                                   #   pfnSDL_CurrentEndThread pfnEndThread
)::Ptr{SDL_Thread} = ...                                                   # );
@ctypedef AVBitStreamFilter @cstruct {                                     # typedef struct AVBitStreamFilter {
  name::Cstring                                                            #   const char *name;
  codec_ids::Ptr{AVCodecID}                                                #   const enum AVCodecID *codec_ids;
  priv_class::Ptr{AVClass}                                                 #   const AVClass *priv_class;
  priv_data_size::Cint                                                     #   int priv_data_size;
  init::Ptr{Cfunction{Cint, Tuple{Ptr{AVBSFContext}}}}                     #   int (*init)(AVBSFContext *ctx);
  filter::Ptr{Cfunction{Cint, Tuple{Ptr{AVBSFContext, Ptr{AVPacket}}}}}    #   int (*filter)(AVBSFContext *ctx, AVPacket *pkt);
  close::Ptr{Cfunction{Cvoid, Tuple{Ptr{AVBSFContext}}}}                   #   void (*close)(AVBSFContext *ctx);
  flush::Ptr{Cfunction{Cvoid, Tuple{Ptr{AVBSFContext}}}}                   #   void (*flush)(AVBSFContext *ctx);
}                                                                          # } AVBitStreamFilter;

# from pain point #8  (7)
printf = Cfunction{Cint, Tuple{Cstring, Vararg}}(Clibrary(), :printf)    # int printf( const char *format, ...);​
printf("%d %s %f\n", 1234, "string", 1.5)                                # printf("%d %s %f\n", 1234, "string", 1.5);
1 Definition and use of a C-style union is now very natural and easy to do in Julia.
2 Proper type, size, and value support for C-style enum is now available in Julia.
3 Specifying named and unnamed nested struct, union, and enum constructs in Julia mirrors the same expressions in C.
4 The alignment and packing strategies of struct and union can be specified in Julia to be compatible with C API’s.
5 Representing even the most challenging of integer bit fields is now trivial to do in Julia.
6 Complete type-safety and calling convention safety for C function pointers is added to Julia.
7 Julia users can now simply add a Vararg to a function pointer’s argument type list to specify a variadic function and invoke it the same way as in C.

CBinding.jl is the first complete solution for interfacing C API’s from Julia. We believe the package is a critical addition to the Julia ecosystem and it will make any Julia-C integration achievable. We will be publishing future blog posts to demonstrate interfacing real-world C libraries from Julia using our package.

If you are considering the transition to Julia, but have several C libraries you depend on, let us help! Analytech Solutions offers many years of experience working with both Julia and C, and we can streamline your transition process. Please contact us for more information!

Keith Rutkowski Keith Rutkowski is a seasoned visionary, inventor, and computer scientist with a passion to provide companies with innovative research and development, physics-based modeling and simulation, data analysis, and scientific or technical software/computing services. He has over a decade of industry experience in scientific and technical computing, high-performance parallelized computing, and hard real-time computing.