Roswell and Walled Gardens

Written on 2022-01-04 19:00:00 UTC

Recently, Eitaro Fukamachi has been sharing blog posts about Roswell, "a launcher for a major lisp environment that just works." Like many in the CL community, I've heard of Roswell and even dabbled with it a bit. I'm not sure how many people actually use Roswell, but I do know it's non-negligible.

Roswell certainly solves some real problems for folks, but I could never get into it myself. There are two primary reasons for that. First, I use a Linux distro with that a) stays relatively up to date with upstreams and b) makes it trivial to carry my own patches to CL implementations (which I frequently do). Second, Roswell feels like a walled garden to me (I doubt this was an intentional decision by its authors, however).

The purpose of this post is to dig more into the second reason. This is mostly for my own benefit. I have not really progressed beyond broad "feelings" on this subject and I'd be doing myself and the Roswell authors a disservice if I keep not using it based on mere feelings without some concrete issues backing it up. Perhaps it will benefit others as well by finding others with concerns similar to mine and getting a concrete set of issues laid out that we could work on contributing fixes for.

Roswell authors: If you read this, please know this isn't meant to be a dig at you. I'm writing this as a sincere effort at exploring why I don't like Roswell with an eye toward coming up with solutions that would make it more palatable to me and (hopefully) others with a similar mindset.

User/Environment Intercession

My core complaint is that Roswell interposes itself between the user and the CL environment in a highly visible and intrusive way: through the ros executable. Let's look at what this means in terms of both being an implementation manager and scripting.

Implementation Manager

Roswell bills itself largely as an implementation manager. It makes it trivial to install just about any version of any major CL implementation on any computer. That's a huge win for folks running Debian or Ubuntu LTSs (as they tend to have packages that are extremely out of date) or on odd arch/OS combinations (if binary packages are not provided, Roswell can build the implementation for you).

But what does it mean to be an implementation manager? To me, that means after installing the implementation, I should be able to use it freely, as if it were installed by my native package manager. So let's give that a try:

user@rocinante:~$ ros install sbcl-bin
No SBCL version specified. Downloading sbcl-bin_uri.tsv to see the available versions...
[##########################################################################]100%
Installing sbcl-bin/2.2.0...
Downloading https://github.com/roswell/sbcl_bin/releases/download/2.2.0/sbcl-2.2.0-x86-64-linux-binary.tar.bz2
[##########################################################################]100%
Extracting sbcl-bin-2.2.0-x86-64-linux.tar.bz2 to /home/user/.roswell/src/sbcl-2.2.0-x86-64-linux/
Building sbcl-bin/2.2.0... Done.
Install Script for sbcl-bin...
Making core for Roswell...
Installing Quicklisp... Done 7169

user@rocinante:~$ export PATH="/home/user/.roswell/bin:$PATH"
user@rocinante:~$ sbcl
zsh: command not found: sbcl

Huh. Well that's disappointing. It seems that the only (out of the box) way to run an implementation is via ros run.

user@rocinante:~$ ros run
* (find-package :ql)
#<PACKAGE "QUICKLISP-CLIENT">

What does this mean? Virtually everything CL related needs to know you use Roswell. Switching from an OS-managed SBCL install to Roswell-managed? Better update your SLIME/Sly config to use ros run instead of sbcl. Writing documentation for a cool hack? Better include directions for Roswell as well as stock implementations (or hope that your users are confident enough in CL to figure it out on their own). You know those bad jokes that go something like "How do you know if someone is X? Don't worry, they'll tell you!"? This kind of feels like a real-world instantiation of that.

Not only that, but Roswell is imposing its opinions on its users. See that #<PACKAGE "QUICKLISP-CLIENT"> in the REPL? That certainly doesn't come from my .sbclrc, so where does it come from? Let's look at what ros run invokes under the hood:

user@rocinante:~$ ps aux | grep sbcl
user       43354  0.4  0.5 1238788 93548 pts/1   Sl+  10:27   0:00 /home/user/.roswell/impls/x86-64/linux/sbcl-bin/2.2.0/bin/sbcl --core /home/user/.roswell/impls/x86-64/linux/sbcl-bin/2.2.0/lib/sbcl/sbcl.core --noinform --no-sysinit --no-userinit --eval (progn #-ros.init(cl:load "/etc/roswell/init.lisp")) --eval (ros:run '((:eval"(ros:quicklisp)")))

Yikes. It looks like ros run modifies your CL image a decent amount by default. Not only does it load its own init file at /etc/roswell/init.lisp (while ignoring your own!), it also loads the Quicklisp client for you. And it's not obvious here, but the QL client it loads is located at ~/.roswell/quicklisp/, not the standard ~/quicklisp/ folder.

I dislike this for several reasons. First, I'm definitely biased here, but Quicklisp isn't the only dependency management solution out there. Second, this can make providing support to Roswell users a nightmare. If something goes wrong with one of my programs on a Roswell user's computer, I need to become an expert in Roswell to help them! Third, it really rubs me the wrong way that it just blithely ignores a user's standard customization file by default. Fourth, I think almost every feature added on top of the vanilla implementation should default to off. That reduces cognitive burden when worrying about if Roswell will ever change a default on me when they add a new feature.

So, how do we get a bog standard REPL from Roswell? Based on the help messages, there is an option to disable loading QL and most of Roswell's own init files. But there's no option to load the default RC files or skip /etc/roswell/init.lisp. So the best we can do seems to be:

user@rocinante:~$ ros run +Q +R --load /etc/sbclrc --load ~/.sbclrc

Which ends up invoking SBCL as:

/home/user/.roswell/impls/x86-64/linux/sbcl-bin/2.2.0/bin/sbcl --core /home/user/.roswell/impls/x86-64/linux/sbcl-bin/2.2.0/lib/sbcl/sbcl.core --noinform --no-sysinit --no-userinit --eval (progn #-ros.init(cl:load "/etc/roswell/init.lisp")) --eval (ros:run '((:load "/etc/sbclrc")(:load "/home/user/.sbclrc")))

Not terrible, but not great either.

Solution?

We already have a standard way for a user to specify to shadow programs of the same name: the $PATH variable. This is already used by other programming language environment managers out there. Let's take a look at RVM, which is probably the closest analog to Roswell that I know of.

user@rocinante:~$ rvm install 2.6.9
[OUTPUT CUT]
user@rocinante:~$ which ruby
/home/user/.rvm/rubies/ruby-2.6.9/bin/ruby

That's nice! Every tutorial or piece of documentation out there that uses Ruby should Just Work. No need to modify anything because you're using RVM-managed Ruby instead of an OS-managed one.

So, Roswell can keep its opinionated setup if it really wants to, no matter how much I disagree with it (hey, that's how opinions go). But I think it would do its users a great service if installing an implementation also placed that implementation on the PATH, with the standard name. The easiest way of doing it is probably a shell script that looks something like:

#!/bin/sh

# A hypothetical Roswell command that resolves which version of SBCL we should
# use. This could look at config files, envvars, whatever.
_SBCL_PATH="$(ros which sbcl)"
exec "$_SBCL_PATH" "$@"

UPDATE: Opened an issue to discuss this more.

Scripting

Let's turn to scripting now: the other big place where it feels like Roswell is making a land grab and then building a wall around it.

First, let's get this out of the way: each CL implementation has different CLI options, some of them are persnickety about order, and it really sucks. This does make writing portable scripts difficult and is something that really needs improvement.

But, again, with Roswell's solution we see it forcing itself between the user and the CL implementation.

First, let's consider a script that works only on SBCL. Starting that script with the following is a great way of doing that (assuming you don't care that Busybox's env doesn't support the -S option).

#!/usr/bin/env -S sbcl --script

If Roswell added its managed implementations to the PATH, this would even work with Roswell! As it currently stands though, you need to start with something like:

#!/bin/sh
#|-*- mode:lisp -*-|#
#|
exec ros -- $0 "$@"
|#

You additionally need to define a main function which Roswell calls for you. Explicitly calling a function in the file, whether it be main or another, might work (so you could use sbcl --script or similar which does not automatically call main for you), but I suspect it'd break ros build and might result in some weird error messages if you ever return from that function.

With this approach we run into many of the same issues as above. Support is difficult for projects maintained by non-Roswell users, it requires a Roswell and non-Roswell version of any given script, and you're subject to Roswell's opinionated defaults.

The defaults issue is particularly thorny here, as I have seen Roswell scripts in the wild that depend on the existence of the ros or ql packages and functions they export. This means that folks using SBCL can't count on being able to do sbcl --load some-script.ros --eval '(main)' and have it work. This does not smell like portability to me.

Solution?

The CL community really needs a portable way of running scripts across multiple implementations and OSes. Unfortunately, I don't have a concrete solution in mind. If I did I would have started trying to implement it already!

I think cl-launch is pretty nice, but have issues with its insistence on loading ASDF and upgrading it. ASDF is not needed for every script under the sun and I'd sometimes prefer to use a specific version of ASDF I ship with the script instead of whatever the end-user has lying around in their ~/common-lisp/asdf/ folder. I also dislike that it's written as a shell script, which makes it a non starter on Windows.

If Roswell aimed to be less of a monolith, perhaps its scripting facilities could be broken out into a separate project and adapted to call implementations directly instead of via ros. This might be a tough sell though, given the current defaults load a decent chunk of Roswell code into the image.

Honestly, I worry that there's never going to be a single implementation of CL scripting that satisfies everyone. Which leads to the next point...

Importance of interfaces

I don't know if you've noticed or not, but directing CL'ers (myself included) is a lot like herding cats. If you tell them to do one thing you'll have a couple follow along, some start then get lost on the way, some start to explore different options and then do what you suggested (or get lost), and some that do the exact opposite of what you want/invent a new way of doing it purely out of spite (I jest about the spite part... mostly).

Given Roswell's current state, if someone told me that I had to install Roswell to run their fancy program which is run through a .ros script, then, by God, I will find a different way to run it.

Roswell is a black box to me. I don't know what utilities are loaded with ros run or when running a script. And even if I did, Roswell could choose to change them at any time. Similarly, I expect a certain contract from the implementation's CLI when I run it, which ros run breaks.

If we instead had some community specifications that described things like "CL scripts" (written with an honest attempt at considering competing needs and desires) and some project told me "you can run this script with any CL script runner that conforms to v1 of the CL script spec. Oh, by the way, Roswell contains one such implementation," I'd be much more likely to just say "OK" and install Roswell to get it to work. Knowing that there's the possibility I could make my own implementation of a conforming CL script runner would make me more likely to follow the crowd in the short term and then split off later if I really needed to.

I speak only for myself, of course, but my gut tells me a lot of CL'ers would feel the same way. Maybe it's because it's the way our favorite language is designed :).

Anyways, this post has grown too long. But it has achieved its primary purpose of helping me organize my thoughts on this topic. Now I can idly day-dream about "CL script" specs and how to get Roswell to install unsullied CL implementations on the user's PATH.