Nix-rebuild too slow on Raspberry Pi 3B+

Help
1

I thought using NixOS on my Raspberry Pi would be perfect to get all my little home services configured declaratively in a neat way. It has been, however, a painful experience since each nixos-rebuild takes at least 4 minutes, which really sucks when you have to debug the configuration of an application running as a systemd service.
Not to mention when something more significant is changed, like new packages. Then it takes even longer.

Is there a way to not have to do this every time, like not having to rebuild to change a service configuration temporarily? Or to make it faster like doing rebuilds on a faster computer?

2

I’ve been using deploy-rs which seems to work so long as stuff is in the cache, but I’ve not quite figured cross compiling yet for stuff that isn’t (compiling under qemu-user is also a pretty miserable experience!)

The other thing to try might be to add a bit of swap - without it executable pages get evicted from memory under pressure which can really slow stuff down; adding a little swap allows infrequently used heap memory to be considered for paging out alongside.

3 Likes
3

I use binfmt to build the required image file on my host, from which the pi can then boot.

binfmt.emulatedSystems = [ "aarch64-linux" ];

The disadvantage here is that a new image has to be created each time changes are made, which is then written to the SD card.

4

Thanks, I’ll check that out!

I already have a 1GB swap file and it doesn’t seem to surpass 50% usage so I guess it is being enough-ish? Using an SD card as RAM is terrible enough hahah, but before I added it, I couldn’t rebuild at all

5

I did this the first time so I could have swap and not have to build a whole lot of the stuff I knew I’d need on the Pi, but doing it this way every time would be much slower unfortunately

6

You don’t need an additional tool to deploy a configuration to another system.
nixos-rebuild can already do that on its own.

I’m using flakes but if you use man nixos-rebuild you can probably adapt it to your needs.

4 Likes
7

Oh nice, that seems to be exactly what I wanted!
Can you shed some light on how I’d do it from an x86_64 host? Simply adding boot.binfmt.emulatedSystems = [ "aarrch64-linux" ]; and importing <nixpkgs/nixos/modules/installer/sd-card/sd-image-aarch64-installer.nix> like I did for creating the SD card image doesn’t seem to be the way to go and, after trying out a few things, I don’t have much idea on how to proceed

8

I’m doing exactly the same as @Nebucatnetzer and adding that line was enough. Can you be a bit more specific what is not working?

9

I don’t know how your config looks but here you have a typo. aarch64 is with one r.

AFAIR I didn’t need anything else.

The only other thing I can add is to use nixpkgs.hostPlatform, that option made my configuration significantly easier because I didn’t have to juggle system in my flake anymore.

1 Like
10

Ok thanks for sharing your configs. I realized you were defining boot.binfmt.emulatedSystems on your main machine configuration, not on the RPi’s like I was doing (I’m not daily driving NixOS, so I simply have to have the required Qemu packages installed and extra-platforms = aarch64-linux on nix.conf). I was defining it there because that’s I did that to create the initial SD card image… So removing that line was what made it possible for me. Before I removed it, I was getting the error:

error:
       … while calling the 'head' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixpkgs/lib/attrsets.nix:1:35741:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixpkgs/lib/modules.nix:1:34707:
       … while evaluating the option `system.build.toplevel':

       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixpkgs/nixos/modules/system/activation/top-level.nix':

       … while evaluating the option `assertions':

       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixpkgs/nixos/modules/system/boot/systemd.nix':

       … while evaluating the option `systemd.services':

       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixpkgs/nixos/modules/system/boot/binfmt.nix':

       … while evaluating the option `boot.binfmt.registrations':

       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixpkgs/nixos/modules/system/boot/binfmt.nix':

       (stack trace truncated; use '--show-trace' to show the full, detailed trace)

       error: assertion '(system != (pkgs).stdenv.hostPlatform.system)' failed
       at /nix/var/nix/profiles/per-user/root/channels/nixpkgs/nixos/modules/system/boot/binfmt.nix:1:14439:

which I assume was because it doesn’t make sense to activate emulation for the running architecture?

I also had to add the line

nix.settings.trusted-users = [ "@wheel" ];

to the RPi configuration, even if using nixos-rebuild --use-remote-sudo. Otherwise I’d get:

error: cannot add path '/nix/store/22q0z1sh36lp6467d1vv2616ryz7n920-system-units' because it lacks a signature by a trusted key

Btw, I didn’t need to add nixpkgs.hostPlatform, which I didn’t really understand what would do.

Anyways, it all works now and I can install lolcat at a blazingly fast 2 minutes! Still not great but definitely much better. Thanks for the help!

There was no typo in my config btw hahah, I probably typed that instead of copy pasting.

11

Ah yes you only need that option on an x86 machine which builds for an aarch64 machine.

It defines which architecture your system has.
Before this you would often see things like the following.
Which made it quite annoying to have a flake with multiple architectures because you basically needed the same thing again just for the different architectures.
Of course there are ways around it but then the code gets more complicated.
With hostPlatform you can define the architecture on the host and don’t need to define system in your flake.

system = "aarch64-linux";
  pkgs = import inputs.nixpkgs {
    inherit system;
    config = {
      allowUnfree = true;
    };
};

inputs.nixpkgs.lib.nixosSystem {
  inherit pkgs system;
  specialArgs = {
…
12

To be clear, do NOT pass pkgs or system to lib.nixosSystem, this is just an example of the old way of writing things.

1 Like