Building a (non-NixOS) Linux image using Nix

I’m very interested in using Nix to manage embedded Linux builds in place of something like Buildroot or Yocto. I’ve made Linux builds from scratch before, and I’m familiar with that end of the process but I’m a Nix noob and am not sure about the best way to go about making such a build file. To be clear, I am not trying to make a NixOS image with Nix, but rather define a custom, minimal Linux distribution in Nix build files.

Building the cross-compilation tools and the kernel seems fairly straightforward with Nix. The part I’m stuck on is building and installing packages for the target image. In my experience, in a normal Linux build you do this by setting up a skeleton structure for the root file system (and copy GCC and binutils into that initial root). Then you chroot into the new root file system, and build & install all software you want in the image from there.

The concept of a chroot seems to be nonsensical in Nix, since everything gets built into the nix store anyway. At the same time the need for a chroot seems unnecessary, since all builds are inherently isolated from the host system. The question remains: how do I transfer the packages built in Nix into the root file system image file? The idea I currently have is to build all packages in Nix as normal and then to use a bash script to copy them out of the nix store into the root file system. This doesn’t feel very idiomatic though, is there an obvious approach I’m missing?

So usually, all packages internally set up a partial FHS structure, and copy files into there. You can easily merge multiple packages into one using buildEnv.

For example, let’s say you want hello, nom and nvd in one directory. Their individual directory trees look like this:

# nix build --print-out-paths nixpkgs#hello | xargs tree
/nix/store/8fpvkfwr8fm91xlzznsgh3g1fcw0hfnh-hello-2.12.1
├── bin
│   └── hello
└── share
    ├── info
    │   └── hello.info
    └── man
        └── man1
            └── hello.1.gz
# nix build --print-out-paths nixpkgs#nvd | xargs tree
/nix/store/s2bdnl5ybyj14ac5l1v81lslc33dk39p-nvd-0.2.4
├── bin
│   └── nvd
└── share
    └── man
        └── man1
            └── nvd.1.gz
# nix build --print-out-paths nixpkgs#nix-output-monitor | xargs tree
/nix/store/w2sgpxmzyrsnz81xbskdrsvhki754pmw-nix-output-monitor-2.1.4
├── bin
│   ├── nom
│   ├── nom-build -> nom
│   └── nom-shell -> nom
└── share
    ├── bash-completion
    │   └── completions
    │       └── nom.bash
    └── zsh
        └── site-functions
            ├── _nom
            ├── _nom-build
            └── _nom-shell

Now, you could write a nix file like this:

{ pkgs ? (import <nixpkgs> {}) }:
pkgs.buildEnv {
  name = "myLinuxRoot";
  paths = with pkgs; [
    hello
    nom
    nvd
  ];
}

And if you build it, you get all the trees merged into one:

# nix build --print-out-paths -f test.nix | xargs tree
/nix/store/ymkam57gbaixhh8l58vvwn87n59aizj3-myLinuxRoot
├── bin
│   ├── hello -> /nix/store/p3vrcf0z38979rclg1923vwi4vcrpgq1-hello-2.12.1/bin/hello
│   ├── nom -> /nix/store/i15qmf2nqi7k0j5qkfqn3w87qyb02dml-nom-2.6.1/bin/nom
│   └── nvd -> /nix/store/l5wx7fl5yx3y7qxpm96p80qz1znimwjp-nvd-0.2.3/bin/nvd
└── share
    ├── info -> /nix/store/p3vrcf0z38979rclg1923vwi4vcrpgq1-hello-2.12.1/share/info
    └── man
        └── man1
            ├── hello.1.gz -> /nix/store/p3vrcf0z38979rclg1923vwi4vcrpgq1-hello-2.12.1/share/man/man1/hello.1.gz
            └── nvd.1.gz -> /nix/store/l5wx7fl5yx3y7qxpm96p80qz1znimwjp-nvd-0.2.3/share/man/man1/nvd.1.gz

Now of course, these are all symlinks to the store, and even if you could resolve those, the binaries in here link to other store paths, so you’ll have to copy the contents of this directory plus all the store paths that it depends on (this is also called the “closure”) to the image.

Luckily, there’s nix copy, which computes the closure and copies it in its entirety into a new nix store, so you can do something like this:

# nix copy --no-check-sigs --to /Volumes/TestDrive -f test.nix
[1/17/20 copied (37.9/145.7 MiB)] copying path '/nix/store/9pj4rzx5pbynkkxq1srzwjhywmcfxws3-python3-3.12.5' to 'local'
# nix build --print-out-paths -f test.nix
/nix/store/ymkam57gbaixhh8l58vvwn87n59aizj3-myLinuxRoot
# cp -r /nix/store/ymkam57gbaixhh8l58vvwn87n59aizj3-myLinuxRoot/* /Volumes/TestDrive

Let’s inspect the result:

# ls /Volumes/TestDrive
bin/  nix/  share/
# tree /Volumes/TestDrive/{bin,share}
/Volumes/TestDrive/bin
├── hello -> /nix/store/p3vrcf0z38979rclg1923vwi4vcrpgq1-hello-2.12.1/bin/hello
├── nom -> /nix/store/i15qmf2nqi7k0j5qkfqn3w87qyb02dml-nom-2.6.1/bin/nom
└── nvd -> /nix/store/l5wx7fl5yx3y7qxpm96p80qz1znimwjp-nvd-0.2.3/bin/nvd
/Volumes/TestDrive/share
├── info -> /nix/store/p3vrcf0z38979rclg1923vwi4vcrpgq1-hello-2.12.1/share/info
└── man
    └── man1
        ├── hello.1.gz -> /nix/store/p3vrcf0z38979rclg1923vwi4vcrpgq1-hello-2.12.1/share/man/man1/hello.1.gz
        └── nvd.1.gz -> /nix/store/l5wx7fl5yx3y7qxpm96p80qz1znimwjp-nvd-0.2.3/share/man/man1/nvd.1.gz
# ls /Volumes/TestDrive/nix/store
6ak0yz6g90ccc8qhyxadk61zf8abhji7-sqlite-3.46.0/         iafzjk7zbkqaszqfp6n006vvxjrpn4f6-bash-5.2p32/
8j28a2jx7mf16sm86hkp6bl7kwmg748s-libffi-3.4.6/          kpjz6vd2hfapkfkd9znkn64m0c7kz2dj-expat-2.6.2/
9cpzwxa2fmkdl2da43968idffwfpcs53-zlib-1.3.1/            l5wx7fl5yx3y7qxpm96p80qz1znimwjp-nvd-0.2.3/
9pj4rzx5pbynkkxq1srzwjhywmcfxws3-python3-3.12.5/        p3vrcf0z38979rclg1923vwi4vcrpgq1-hello-2.12.1/
a8p4iis95j6zxx0k11pqfndcyffdd96p-ncurses-6.4.20221231/  qn1jp772yh5fg93krh9wiq24w0dpvvky-gdbm-1.24-lib/
chrhxhmvf38lnbnzqi3pv6f2c66pjnbf-iana-etc-20240318/     sny39q6mg5swbahpmd97wway99hxhx6p-libxcrypt-4.4.36/
cngakhdqpl5w0dz6ivplg40czi8jbqi7-readline-8.2p10/       xdlykb948l5dwi4wd3vpff6kmdxcbnm0-openssl-3.0.14/
fh6k84p2mqx5mhmmdbz8dqk7hgz11pcw-tzdata-2024a/          xpcf6byi2caqx3dsy174y2s9ra30sgjk-mailcap-2.1.54/
gcawqxmi4x42v9hfsgy89hamx1m6scxj-bzip2-1.0.8/           xrq1dzjsvrn86ad6i3r52f6bb0vwh2ar-xz-5.6.2/
i15qmf2nqi7k0j5qkfqn3w87qyb02dml-nom-2.6.1/             ymkam57gbaixhh8l58vvwn87n59aizj3-myLinuxRoot/

As you can see, all dependencies were added to the nix store on the thumb drive and linked in the FHS directories. Instead of putting this on a drive, you can of course also put it into a directory and build an image from that or whatever floats your boat.

The --no-check-sigs argument for nix copy is required if you didn’t set up signing store paths you’ve built yourself (which most people don’t), but is safe to use when you copy stuff from your own store to a local directory.

DO NOT USE THIS when copying from a remote store!

You might be able to also automate these steps down to building a disk image with a derivation and a custom builder script, but running nix inside nix derivations is a bit of an edge-case that is disabled by default. You can enable it with the experimental feature recursive-nix.

Recursive Nix isn’t required for this task if you use pkgs.closureInfo.

Ah yes, I forgot about that. It seems that nix copy also populates the database on the target, which is useful in some cases, but in general, this seems to work quite alright:

{ pkgs ? (import <nixpkgs> {}) }:
let
  bareRootEnv = pkgs.buildEnv {
    name = "myBareLinuxRoot";
    paths = with pkgs; [
      hello
      nom
      nvd
    ];
  };
  closureInfo = pkgs.closureInfo { rootPaths = [ bareRootEnv ]; };
in
pkgs.stdenvNoCC.mkDerivation {
  name = "myLinuxRootWithNixStore";
  phases = ["installPhase"];
  installPhase = ''
    mkdir -p $out/nix/store
    cp -r ${bareRootEnv}/* $out
    cat ${closureInfo}/store-paths | xargs -I_ -n1 cp -r _ $out/nix/store
  '';
}

Output:

# nix build --print-out-paths -f test.nix
/nix/store/67yk6r6ki2im68892610p15i9s2ydsm6-myLinuxRootWithNixStore
# ls /nix/store/67yk6r6ki2im68892610p15i9s2ydsm6-myLinuxRootWithNixStore
bin/  nix/  share/
# ls /nix/store/67yk6r6ki2im68892610p15i9s2ydsm6-myLinuxRootWithNixStore/nix/store
6ak0yz6g90ccc8qhyxadk61zf8abhji7-sqlite-3.46.0/         i15qmf2nqi7k0j5qkfqn3w87qyb02dml-nom-2.6.1/
8j28a2jx7mf16sm86hkp6bl7kwmg748s-libffi-3.4.6/          iafzjk7zbkqaszqfp6n006vvxjrpn4f6-bash-5.2p32/
9cpzwxa2fmkdl2da43968idffwfpcs53-zlib-1.3.1/            kpjz6vd2hfapkfkd9znkn64m0c7kz2dj-expat-2.6.2/
9pj4rzx5pbynkkxq1srzwjhywmcfxws3-python3-3.12.5/        l5wx7fl5yx3y7qxpm96p80qz1znimwjp-nvd-0.2.3/
a8p4iis95j6zxx0k11pqfndcyffdd96p-ncurses-6.4.20221231/  p3vrcf0z38979rclg1923vwi4vcrpgq1-hello-2.12.1/
aa6zqh84jgw8vkzs2ghhhmkjzjbx4xf3-myBareLinuxRoot/       qn1jp772yh5fg93krh9wiq24w0dpvvky-gdbm-1.24-lib/
chrhxhmvf38lnbnzqi3pv6f2c66pjnbf-iana-etc-20240318/     sny39q6mg5swbahpmd97wway99hxhx6p-libxcrypt-4.4.36/
cngakhdqpl5w0dz6ivplg40czi8jbqi7-readline-8.2p10/       xdlykb948l5dwi4wd3vpff6kmdxcbnm0-openssl-3.0.14/
fh6k84p2mqx5mhmmdbz8dqk7hgz11pcw-tzdata-2024a/          xpcf6byi2caqx3dsy174y2s9ra30sgjk-mailcap-2.1.54/
gcawqxmi4x42v9hfsgy89hamx1m6scxj-bzip2-1.0.8/           xrq1dzjsvrn86ad6i3r52f6bb0vwh2ar-xz-5.6.2/

Bender saying "Neat!" and taking a picture

This is incredibly helpful and exactly what I was looking for, thank you both!!

I don’t mind the nix folder in the root holding the copied nix store, but out of curiosity is there a command that will follow the symlinks and copy items out of the store, such that I could have a “normal” root file system? Ultimately the reproducibility (ergo storing the package hashes) I want is on the host and not very useful on the target, since the target will never update individual packages.

recently i built an os from scratch using nix and it was a very pleasant experience

there are a few things you can do to solve your package issue like the others have discussed (buildEnv), etc… at one point i just added a symlink from the nix store to parts of /usr which worked fine

i specifically wanted to mention that building a kernel is great with nix, we have awesome tooling for that, and the same for building initrd stuff

good luck and i am sure you will have a great time :partying_face:

Just like to mention not-os for building images for resource-constrained systems:

1 Like

There is no simple solution to this. As I said, compiled packages internally link to dependencies (be it libraries or other executables) in the nix store. You could of course try to use patchelf (and whatever else nixpkgs uses to actually make this happen in the first place) to reverse all of this and get a regular FHS back, but that will be a lot of work and a brittle solution at best.