SkiffOS: cross-compiled container initramfs wrapping NixOS

Hi all,

NixOS is really cool. I was hacking around with it a bit, and built a Docker-based setup for setting it up cross-architecture.

The SkiffOS project is built around Buildroot to make it easy to cross-compile a minimal Linux-based initramfs for hosting containers.

It supports many embedded Linux boards, containers, VMs, etc.

There’s a tool available called “skiff core” which automatically compiles and starts docker images / containers on the target device, and configures users, which are dropped into the container on connect or login - according to a yaml config file.

I added NixOS as a skiff-core configuration, so the system (i.e. a Raspberry Pi) will boot, then compile Nix from source, then use it to build NixOS and install it into a Docker image, and then finally start a Docker container with NixOS inside and map a incoming user “core” to be “dropped in” to the container on connect.

I wanted to share in case someone finds this useful.

The first part of this is a multi-stage dockerfile for compiling Nix and a NixOS container, which is useful on its own independently from Skiff:

The second part is a configuration core/nixos for Skiff, which you can select like this:

# compiling
git clone --recursive https://github.com/skiffos/skiffos
export SKIFF_WORKSPACE=pi
export SKIFF_CONFIG=pi/4,core/nixos
make compile

# flashing
sudo bash
export SKIFF_WORKSPACE=pi
export PI_SD=/dev/sdN # change to your SD
make cmd/pi/common/format
make cmd/pi/common/install

Or for a Docker container:

export SKIFF_CONFIG=virt/docker,core/nixos
make compile
# build docker image
make cmd/virt/docker/build
# start skiff container
make cmd/virt/docker/run
# exec a shell
make cmd/virt/docker/exec
# inside the shell - switch to nixos
su - nix

I know Nix has its own mechanisms for this already, but Skiff provides a dead-simple workflow for cross-compiling robust setups for any environment, and this particular trick of building Nix in a Dockerfile makes initial setup of that part (and disaster recovery) also very easy. Perhaps this has some benefit in certain circumstances over the direct install approach. It also allows one to test multiple systems in parallel, or even distros in parallel.

Here’s a little asciinema demo of connecting to this running on a pi4:

https://asciinema.org/a/vO4KnAW3TLAfm9zkiR0QTSJIl

Let me know if this is useful to anyone.

Best regards,
Christian Stewart (paralin)

7 Likes

This is really cool, Christian. So you have extended SkiffOS or you created SkiffOS? Can you talk a bit about your use-case for this? Want to make sure I understand. Thanks. :slight_smile:

Buildroot is used to make embedded Linux builds, cross-compiled with familiar KConfig and Make tools. Skiff is designed to compile an initramfs with Docker and minimal dependencies to connect to networking and be ssh reachable, and produce a nearly identical environment across underlying compute boards.

You can then run any operating system inside Docker - Ubuntu, Alpine, NixOS, Gentoo, etc…

I am currently using it for robotics, where it’s critical that the system always boot to a known good state, even in the face of hardware failure, often because the computer unit will be embedded deep within the robot and extracting the SD card or EMMC to repair a broken boot is usually very difficult.

1 Like