Simple deployment strategy using nix-copy-closure

Development Deployments
1

Hello!

I’ve been working superficially with NixOS and Nixops for a few years but only recently tried to develop a deep understanding of the ecosystem. One of my goals is to replace Nixops with a simpler and more transparent deployment system, and I wanted to share the method I’ve come up with here. Now that I have it working, it’s hard to imagine wanting to deploy any other way, so I’m interested to see what others think.

In this specific example I’ll be cross-compiling for a Raspberry Pi system.

Side note: I am vaguely aware of nixos-rebuild --target-host, which maybe overlaps with a lot of what I’m doing here. Perhaps the main advantage of my method is that you only have to enter your remote sudo password once?

I’ll go through the main parts of the deployment setup below:

The basic idea

I have a configuration.nix describing a Raspberry Pi system. I want to build the whole system profile locally by simply running nix-build, copy the result to the remote system, then run nixos-rebuild --switch there and have it switch to the new profile without having to build anything on the slow Pi processor.

Pinning nixpkgs

Pinning is essential so that you can build the correct versions of all the packages locally. There are a lot of ways to do it, but I use this (both in default.nix and configuration.nix) to select a recent 24.05 commit:

  pinned-nixpkgs = fetchTarball {
    sha256 = "sha256:1rzdqgs00vzw69m569li3c6yvkdlqf7zihiivi4n83lfqginr7ar";
    url = "https://github.com/NixOS/nixpkgs/archive/0b8e7a1ae5a94da2e1ee3f3030a32020f6254105.tar.gz";
  };

  pkgs = (import pinned-nixpkgs { });

In configuration.nix, pkgs definition also requires an system argument to specify the Raspberry Pi architecture.

(I do not use fetchFromGitHub here to avoid depending on multiple versions of nixpkgs.)

Setting up your local OS for cross-compilation

To make cross-compilation for Raspberry Pi work, you must have qemu installed, and nix needs to know to use it. On NixOS, put this in your config:

boot.binfmt.emulatedSystems = "aarch64-linux";

With Ubuntu/Debian you install the qemu-user package and put extra-platforms = aarch64-linux in nix.conf.

Running nixos-rebuild as a nix expression

The wiki has a pretty clear explanation of what nixos-rebuild does internally:

  • Build the config.system.build.toplevel derivation of the current configuration. This can be manually done by:
$ # without Flakes
$ nix-build <nixpkgs/nixos> -A config.system.build.toplevel -I nixos-config=path/to/configuration.nix

It’s just evaluating the expression in nixpkgs/nixos/default.nix with some particular arguments. And you can cross-compile by specifying the system argument, like this:

  # This locally cross-compiles the configuration for Raspberry Pi.
  profile =
    (import "${pinned-nixpkgs}/nixos" {
      configuration = ./configuration.nix;
      system = "aarch64-linux";
    }).config.system.build.toplevel;

Note that I am using the pinned version of nixpkgs to run nixos-rebuild.

Packaging everything up

To make it easy to copy everything to the remote system, I collect all the items necessary for a deployment into a little package:

  • configuration.nix
  • link to pinned nixpkgs
  • link to the cross-compiled profile
  • a script called nixos-rebuild-switch that switches to the profile using nixos-rebuild --switch with appropriate arguments

and the link to that package is the output of nix-build. So now I can build and deploy with a series of commands like this:

result=`nix-build`
nix-copy-closure --to raspberry-pi.local $result
ssh -t raspberry-pi.local sudo $result/nixos-rebuild-switch

As a final touch, the script links this package itself as a GC root at /etc/nixos/current-configuration. The guarantees that everything, particularly configuration.nix, will stick around in an obvious place in case you need it later. At any time, for example, I can just abandon the deployment system, copy configuration.nix into its normal location, and go back to editing it locally like on a normal NixOS installation.

For the complete proof of concept code:

Further questions

With this deployment strategy, a few packages seem to still get built on the remote machine, such as nixos-manual-html. So I think I must have not quite figured out how to replicate nixos-rebuild exactly.

4 Likes