Static Executables with SBCL

Written on 2021-01-05 13:00:00

UPDATE: Make sure to see my follow up post to this).

Common Lisp is an amazing language with many great implementations. The image based development paradigm vastly increases developer productivity and enjoyment. However, there frequently comes a time in a program's life cycle where development pauses and a version must be delivered for use by non-developers. There are many tools available to build an executable in Common Lisp, most of which follow the theme of "construct a Lisp image in memory, then dump it to disk for later reloading". That being said, none of the existing methods fit 100% of my use cases, so this post is dedicated to documenting how I filled the gap by convincing SBCL to generate completely static executables.

Background

There are a variety of reasons to want static executables, but the most common ones I run into personally are:

I want to archive my executables. I want to have a version of my executables saved that I can dig up at any point in the future, long after I've upgraded my OS (multiple times), and run for benchmarking purposes, to test if old versions exhibited specific behavior, etc. without needing to recompile.
I want to enable someone to reproduce my results exactly. This is important for reproducibility in academic contexts. Also, some computing contests that conferences organize prefer static executables so they can run tests on their hardware without needing to set up a complicated run time environment.
I want to make it trivial for someone to install my software. With a static executable, all anyone running on Linux needs to do is download a single file, chmod +x it, and copy it onto their path (preferably after verifying its integrity, but, let's be honest, fewer people do that than should).

There certainly are issues with static executables/linking in general. If you are unaware of what they are, I highly encourage you to read up on the subject before deciding that static executables are the be-all-end-all of application delivery. Static executables are just another tool in a developer's toolbox, to be pulled out only when the time is right.

I'll pause at the moment for a clarification: when I say static executable I mean a truly static executable. As in I want to be able to run ldd on it and have it output not a dynamic executable and I do not want it to call any libdl functions (such as dlopen or dlsym at runtime). While some existing methods claim or imply that they make static executables with SBCL (such as CFFI's static-program-op or manually linking external libraries into the SBCL runtime while building it), they by and large mean they statically link foreign code into the runtime, but the runtime itself is not a static executable.

I have yet to find a publicly documented method of creating a fully static executable with SBCL and it's not too hard to understand why. Creating a static executable requires statically linking in libc and the most common libc implementation for Linux (glibc) does a half-assed job at statically linking itself. While it is possible, many functions will cause your "static" executable to dynamically load pieces of glibc behind your back. Except now you have the requirement that the runtime version must match the compiled version exactly. That defeats the whole point of having a static executable!

For that reason, musl libc is commonly used when creating a truly static executable is important. Unfortunately, musl is not 100% compatible with glibc and for a while SBCL would not work with it. There have been various efforts at patching SBCL to run with musl libc throughout the years, but the assorted (minor!) changes finally got merged upstream in SBCL 2.0.5. This laid the groundwork necessary for truly static executables with SBCL.

Patches

Enough with the blabber, show me the code!

I am maintaining a fork of SBCL that contains the necessary patches. There is a static-executable branch which will always contain the latest version. I plan to rebase this branch on new SBCL releases or on top of upstream's master branch if it looks like I'm going to need to do some extra legwork for an upcoming release. There will also be a series of branches named static-executable-$VERSION which have my patches applied on top of the named version, starting with SBCL 2.1.0.

The patch for any SBCL release is also located at https://www.timmons.dev/static/patches/sbcl/$VERSION/static-executable-support.patch. There is a detached signature available at https://www.timmons.dev/static/patches/sbcl/$VERSION/static-executable-support.patch.asc signed with GPG key 0x9ACF6934.

I would love to get these patches upstreamed, but they didn't get much traction the last time I submitted them to sbcl-devel. Admittedly, they were an early, less elegant version that hadn't seen much use in the real-world. My hope is that other people who desire this capability from SBCL will collaborate to test and refine these patches over time for eventual upstreaming.

Quickstart

Given that most people aren't using musl libc on their development computer, the quickest, easiest way to get a static executable is to build one with Docker. After getting the patchset, simply run the following set of commands in the root of the SBCL repo. This will use the clfoundation/sbcl:alpine3.12 Docker image (another project of mine for a future post) to build a static executable and then copy it out of the image to your host's file system.

docker build -t sbcl-static-executable -f tools-for-build/Dockerfile.static-executable-example .
docker create --name sbcl-static-executable-extractor sbcl-static-executable
docker cp sbcl-static-executable-extractor:/tmp/sb-gmp-tester /tmp/sb-gmp-tester
docker rm sbcl-static-executable-extractor

You should now be able to examine /tmp/sb-gmp-tester to see that it is a static executable:

$ ldd /tmp/sb-gmp-tester
     not a dynamic executable

If all goes well, you should also be able to run it, see the sb-gmp contrib tests all pass (fingers crossed), and realize that this worked because libc, the SBCL runtime, and libgmp were all statically linked!

The file README.static-executable (after applying the patchset) has an example of building locally and a set of docker commands that doesn't require tagging images and naming containers.

How does it work??

This approach requires that the target image be be built twice: once to record the necessary foreign symbols, and then again with the newly built static runtime. I can, however, envision ways around this for a sufficiently motivated person.

One way could be to modify the (already in-tree) shrinkwrapping recipe to handle libdl not being available at runtime. I abandoned this approach largely because the shrinkwrapping code is written for x86-64 and does a lot of things with assembly (which I do not know). It is important for me to have static executables on ARM as well. A second way could be to patch out or otherwise improve the check that the runtime version used to build the core matches the runtime version used to run it. I didn't go this approach as it would certainly lead to difficult to debug issues if used incorrectly, plus the Lisp code in the core would need to check the presence/usefulness of libdl functions at runtime.

So, how does this patchset work and why does it require two passes? Apologies to the SBCL devs if I completely butcher the explanation of SBCL internals, but here it goes anyways!

Lisp code routinely calls into C code, whether it is to a runtime provided function, a libc function, or another library the user has linked and defined using the sb-alien package or the portable counterparts in CFFI. In order to mediate these calls from the Lisp side, SBCL maintains a linkage table. This table has two components. First is a Lisp-side hash table that maps foreign names (and an indicator of if it is data or a function) to an integer. The second is a C-side vector that contains either the address of the symbol (in the case of data) or the opcodes necessary to call the function (e.g., by JMPing to its address).

The C-side vector is populated by looking up the symbol's address using dlsym. This lookup generally happens under two possible scenarios. First, when the Lisp code defines a foreign symbol it wants to be able to call or read. Second, every time the runtime starts, it populates the C-side entries for every symbol contained in the core's hash-table. This second case is how SBCL handles the dynamic linker changing the address of symbols in between core dumps.

This reliance on dlopen and dlsym is so baked into SBCL at this point that, even though the code is nominally conditioned on the internal feature :os-provides-dlopen, I was unable to build a working SBCL without it (before these patches, of course).

With these patches, you first build your Lisp image that you want to deliver like normal. Then, you load the file tools-for-build/dump-linkage-info.lisp into it. Next, you call sb-dump-linkage-info:dump-to-file to extract the Lisp side linkage table entries into a separate file (filtered to remove functions from libdl). Once you have this file, you rebuild SBCL, this time with the intention of creating a static runtime. To do this, you should provide the following:

The environment variable LINKFLAGS should contain -no-pie -static in order to build the static runtime.
Any additional libraries you need should be specified using the environment variable LDLIBS.
You probably want to set the environment variable IGNORE_CONTRIB_FAILURES to yes.
You need to pass the file containing the linkage table entries to make.sh using the --extra-linkage-table-entries argument.
Build without the :os-provides-dlopen and :os-provides-dladdr features. One way of doing this is to pass --without-os-provides-dlopen and --without-os-provides-dladdr to make.sh.

During the build process, the contents of the --extra-linkage-table-entries file are inserted into the cold SBCL core during second genesis and a C file is autogenerated containing a single function that populates the C side of the linkage table using the address of every symbol. This C file is the built into the runtime and called while the runtime boots, before it starts executing the core. This means that, if the runtime is a dynamic executable, the system linker will patch up all the references we need at runtime without SBCL needing to call dlsym explicitly. If the runtime is a static executable, then the symbols are statically linked for us and nothing needs to be done at runtime.

Issues

Given how new this approach is, you will certainly run into issues. Many systems that load foreign code will blindly assume that libraries can be linked in at runtime and will fail to work (silently or loudly) if that assumption is not met. Some libraries already have their own homebrew ways of dealing with this. For instance, if the feature :cl+ssl-foreign-libs-already-loaded is present, the cl+ssl system will not attempt to load the libraries. To deal with this issue in a more principled way, I strongly recommend patching systems to use CFFI's (relatively) new canary argument to define-foreign-library.

CFFI itself also has some issues with this arrangement because it dives into some sb-alien internals that simply aren't present on #-os-provides-dlopen. I currently fix this in a kludgy way by commenting out most of %close-foreign-library in src/cffi-sbcl.lisp, but if more people start building static executables, we'll need to come up with a better way of handling it.

Next Steps

I would love to get feedback on this approach and any ideas on how to improve it! I strongly believe that better support for building static executables with SBCL should be upstreamed and I doubt I am alone in that belief. Please drop me a line (etimmons on Freenode or daewok on Github/Gitlab) if you have suggestions.

Personally, I have used earlier iterations of these patches to build static executables for some of my grad school work. My next real deployment of these patches will likely be to build CLPM with them and providing static executables starting with v0.4.