Rebuild NixOS Offline

This will only work if you didn’t add any new packages since your last build.

Does it? I can’t rebuild my system even with an identical configuration.
When the caches are cold Nix always tries to fetch these packages:

c-ares-1.17.2
curl-7.79.1
libkrb5-1.18
libssh2-1.10.0
nghttp2-1.43.0
openssl-1.1.1m

which seem to be dependencies of curl. I’m confused because
nix why-depends --all /run/current-system on any of these says there is no dependency.

EDIT: I figured these are native build inputs of a fetchzip derivation I have.

It seems the core issue is that Nix tries to rebuild some (fixed output?) derivation X which needs Y at build time, however Y is not available and hasn’t been cached.
Can we just not rebuild X in this case?

2 Likes

To summarize the topic, either of the two options passed to nixos-rebuild switch --flake will allow to rebuilt the system offline offline, given all the dependencies are already present in the Nix store

  • --option substitute false
  • --option binary-caches ""
1 Like

There’s also --offline I believe.

FWIW, system.includeBuildDependencies is (now) a thing on NixOS 23.05 and unstable.

It keeps all your build-time closures, so you can rebuild your system as long as you don’t introduce totally new components (sources).

9 Likes

system.includeBuildDependencies works, but it’s crazy expensive: it adds several GB of dependencies for even a pretty minimal headless system, so pretty much unfeasible.

I really don’t understand why Nix couldn’t just reproduce the same system build, again, given nothing changed in either configuration.nix or the channel. This is extremely annoying.

3 Likes

It always irritated me that NixOS cannot be reconfigured offline, like, sometimes I have to change some configuration while on a train and it takes forever.

I get the feeling that this is one of those cases where we started taking for granted the constant availability of a network connection without thinking it through… Isn’t this also the issue that has led to the custom solution presented in the Nix in Space thread?

7 Likes

NixOS can be reconfigured offline though? Some Nix commands will accept --offline, and for others you can use --option substituters "" to disable checking the cache.

1 Like

Those don’t work, see the comments above for the problem. Also note that --offline is part of the experimental Nix CLI and is not accepted by nixos-rebuild.

It sounds like you’re garbage collecting a build-only time dependencies between rebuilds. Nix doesn’t keep build-only time dependencies alive by default. You can fix this with keep-outputs = true in nix.conf. This works because the system closure keeps the derivation closure alive (meaning the .drv files for build time deps), but the derivations don’t keep the build time outputs alive without keep-outputs = true.

I didn’t know about keep-outputs, I’m not sure I understand how it compare to system.includeBuildDependencies. If it won’t cause to download additional dependencies (like sources for every single package installed) it may be what I’m looking for. I wonder why it’s not the default…

It’s sort of a lite version of system.includeBuildDependencies, in that it doesn’t make them part of the closure; it just keeps them from being GC’d. So it doesn’t force things all the way back to the bootstrap compiler to be on your system; it just keeps them alive if they’re already there. And that’s why it isn’t the default. It has the potential to keep a very excessive amount of paths alive from the GC.

1 Like

And I assume I can’t enable it just for one build, right?

It’s not a “build” thing. It’s a GC thing. So no.

I think it might be a sensible investment to implement a way to pin build dependencies more selectively than by using system.includeBuildDependencies: I’m OK if I can’t change some fancy javascript-heavy service like home-assistant or hedgedoc, but I definitely need the ability to change things like networking.hosts, services.dnsmasq.settings, or networking.wg-quick.interfaces while being offline (because I’m totally the kind person who’ll switch to a configuration with the broken dns, reboot, and call nix store gc). I’ll also find it annoying if changing a line in the neovim config should start downloading things.

These changes should map into rewriting a bunch of text files and not require much more than bash, but I’m pretty sure that on many occasions they need more: this might be because of extra dependencies used to generate or validate the configs, or because new configs result in building new wrappers, or because of literally anything else since I’m not sure anyone is really keeping track of this?

5 Likes

I made a script to ease the download of sources by providing a list, and also keep a link so they don’t get garbage collected :+1:

8 Likes

I was in a situation where I wanted to connect to a wifi network from my raspberry pi, but all of the --options above still resulted in nixos-rebuild requiring internet.

I had wpa_supplicant installed, so I used wpa_passphrase <ssid> | sudo tee /etc/wpa_supplicant.conf to create the wpa_supplicant config, then manually ran sudo wpa_supplicant -iwlan0 -c/etc/wpa_supplicant.conf, which successfully connected. I then changed to a different tty to run the nixos-rebuild switch that had my wifi config declared.

I had to manually run wpa_supplicant because the systemd unit only uses the immutable nix-generated config in /nix/store.

I build (cross-compile)/download rasperrypi packages, specified in my configuration, on a different machine (in the same network), and deploy onto the pi using deploy-rs

Mirroring nicpkgs is a matter of 3 or 4 GiB…

Nixpkgs is just a (huge) git repository…

To actually mirror the binary chace you need some hundreds of TiB, given the figures that have been announced during the summers S3 bucket crisis.

2 Likes

Just wanted to drop a blog post in here to supplement the discussion: Get all your sources! - Linus's blog but as @rnhmjoj says system.includeBuildDependencies is expensive. Also linking nix.conf - Nix Reference Manual for those that want to learn more about the “lite” option

Thank you for this great tip. --option substitute false also works on nix-darwin and fixed my problem.

1 Like