On recent posts, I mentioned that I have a Chromebook, and that I would like to run Debian on devices that ship with old Linux versions. The Samsung ARM Chromebook is such a device, it has Linux 3.4, and that's still what I am running on it most of the time.

After an upgrade of Debian, systemd stopped working, and it took me some time before I could look into it. In order to boot, I had to temporarily use System V init. The bug involves the use of new interfaces, bugs in such interfaces, and how versioning would not be a solution to such cases. Cases like this are not common, so it's no excuse to doing the right thing when developing software, like considering portability, API and ABI maintenance and not bundle. But they expose some of the challenges when trying to support different OSes on top of old versions of Linux, or maybe even new versions.

The new version of systemd included in Debian Jessie uses a feature of Linux of checking the mount ID of a file by inspecting the mnt_id line in the fdinfo files. fdinfo files are not that new, they have been present since 2007, Linux 2.6.22. The mnt_id line, however, is from Linux 3.15, from 2014. So, a fairly new feature. But systemd falls back to other methods if the feature is not present.

However, during the last few weeks of development of Linux 3.4, a bug was introduced to fdinfo handling, which will cause its access to fail in a way that will cause systemd to halt. It simply will refuse to mount the most basic filesystems and not proceed. The bug was fixed not much longer than two weeks after that, but that window of time was enough to have it on a shipped product, one that was not using systemd, let alone one that required a feature not yet present on Linux.

But now I wanted to run this new systemd on this old Linux version. Being too early in the boot process, systemd even ignored some of the debug options and I had to resort to other methods of debugging. As I am running a kernel I built by myself, it wasn't hard for me to have a fixed kernel up and running after I found out the problem. Unfortunately, not everyone will be able to do it by themselves. And the point of getting different OSes working on these many varied devices is scaling. If every device needs to be tested, we lose this scalability.

The other problem I found was that this old Linux version didn't load firmware files by itself, and requested help from userspace, which used to be the job of udev. Well, udev now does not support loading firmware, and the kernel must do it by itself. I managed to backport that feature too. But now, it sounds like I should get upstream support for this device.

Unfortunately, this means not working on the scalability problem, but going back to get upstream support for a lot of devices out there, which is very hard to scale without the necessary people to do the work. I am certainly going to do that, because it's the right thing to do. But not as many people will benefit from that.

But I think it has been interesting to look into the possible problems we might find when trying to make Debian and other OSes working on old Linux versions. I know even older Debian versions won't boot fine on Linux 2.6.32, one of the LTS versions of Linux. I have one other device which runs on it, and I am also working on getting better upstream support for it. Let's see if I can get some news on that soon.