WAAF-CFFI - pass lisp arrays to foreign land


Allows Lisp arrays to be passed to foreign functions by copying their contents to malloc'ed foreign space, using only CFFI.


  • Uses an internal set of optimized copying functions to and from foreign space to avoid consing.   The speed is about 200 megabytes per second of float copying in SBCL (if you set the optimization to zero as described in MANUAL.txt).
  • It seems that the only consing in SBCL is 4 words per foreign pointer.
  • A full set of CFFI types.  
  • No dependancies other than CFFI.  WAAF knows nothing about details of lisp implementations.
  • Automatic allocation and deallocation of foreign memory in an UNWIND-PROTECT.
  • Documentation - there's a detailed MANUAL.txt  
  • Other macros in package
    • WITH-ARRAY-AS-FOREIGN-POINTER - pass lisp array to foreign land
    • WITH-ARRAYS-AS-FOREIGN-POINTERS - a plural version of above
    • SBDEFINE-ALIEN-ROUTINE - define an alien routine in SBCL/CMUCL style, allowing pointer arguments
    • WITH-LISP-VALUES-AS-FOREIGN-POINTERS - put individual lisp values like ints and floats into pointers
  • A test package WAAF-CFFI-TESTS
  • packaged as waaf-cffi.asd and waaf-cffi-tests.asd


  • CFFI - The Common Foreign Function Interface


           (U PU :FLOAT            ;; ARRAY  POINTER-VAR  CFFI-TYPE
           :LISP-TYPE SINGLE-FLOAT ;; promise U is a single-float array
           :START 2                ;; begin at index 2 of of Lisp array
           :END   7                ;; last index used is 6
           :COPY-TO-FOREIGN   T            ;; put contents of Lisp into foreign memory
           :COPY-FROM-FOREIGN T)           ;; copy back from FFI space 
         ;; at this point, PU is a foreign pointer containing data copied
         ;; from the array, of type :FLOAT.

       ;; at end, all foreign memory is deallocated, and U has been copied
       ;; back from foreign space, but the 0th element of U is untouched
       ;; because START was 1, not 0
     ;; The options :START :END :COPY-TO-FOREIGN :COPY-FROM-FOREIGN are optional.  
     ;; By default, :LISP-TYPE is T and any array can be given.


There exist a number of packages for passing arrays to foreign functions.  Some pass pointers to lisp arrays, others allocate foreign memory and copy lisp arrays to it.  This package takes the approach of allocating foreign memory.  It is a reasonable approach because

  1. Lisp implementations want you to do this, because they provide malloc, free, and pointers.  The other approach sometimes involved undocumented internals.  The simple approach, one hopes, won't break.
  2. Passing Lisp memory rather than copying runs into issues like pinning arrays against garbage collection, or freezing the GC, or worrying that Lisp type X is the same as foreign type Y.  Here, we let CFFI do the hard work of worrying about various lisp implementations.   There should be no maintenance once it works, as long as CFFI is up to date.
  3. Copying is fast enough for just about all purposes. On SBCL it doesn't cons.  On a slower lisp, the lisp is slower for everything, so why worry about copying? Copying is of O(N) and the work done in foreign space is usually  O(Na) where a>=1.  
  4. More and more lisps are 64 bit.  A lack of foreign memory should no longer be a problem. 
However, if you are allocating the maximum possible amount of memory, then perhaps copying is not the approach to take.  If you are copying tiny arrays, then the copying overhead might become comparable to the foreign workload, and and the consing to allocate the pointers might become an issue.

J Klein,
Sep 12, 2010, 4:11 PM