I’m trying to build an offline setup using an SBC, so I’m using sdImage.
But after booting, I’d like to change the configuration, such as configuring wifi, by changing the configuration.nix, and rebuilding.
Is that possible?
Is there anything I have to do configuration-wise for that to work?
Theoretically. In practice it can be a little tricky to ensure you have all the store paths you need for a given configuration change.
Small changes like updating a config file will usually work, though you might have to add the --offline flag.
Configuration changes should largely not be necessary, but you’ll want to avoid garbage collection, look into nix’ configuration to add stuff like gc-keep-outputs - which settings to use (or if you want to simply not run garbage collection) depends on your use case.
That said…
If you’re planning this kind of task, you’re probably better off using tools that support runtime configuration changes for those kinds of things. E.g. use NetworkManager (which can be configured declaratively while still retaining the ability to create ad-hoc profiles at runtime using the ensureProfiles option) over systemd-networkd.
Also keep in mind that a nixos-rebuild can be very memory-hungry. Depending on your SBC it can either be impossible or unworkably slow. You’ll be better off running nixos-rebuild ... --target-host theSBC from a beefier machine such as a laptop so everything gets built on that powerful device and only the necessary paths then get copied to the SBC. As long as you e.g. built the sdImage on that laptop, little changes such as network settings should’t require an internet connection. You can use the local network or even a direct LAN connection to the SBC for that.
Yeah, for that type of deployment you should also consider A/B-style image deployments with a RO root FS instead of rebuilding on-host; you can pretty easily create system images with NixOS.
See the fancy new build-image subcommand, as well as the good ol nixos-rebuild build-vm for testing pre-deployment!
I’m finding that the --offline flag doesn’t work, even when what I’m changing shouldn’t require new dependencies, e.g. changing a network configuration.
I’m conflicted, because I really like that NixOS is immutable, but only to the extent that you can then change your configuration and reconfigure your system.
I thought that Nix would fetch anything that it needs into a local cache, and that any subsequent rebuild wouldn’t need to go online.
But perhaps I am misunderstanding how substitution works?
Do I understand correctly that, when you rebuild, you actually build each and every package from scratch, but can use a substitution binary cache when your build inputs are identical.
But, that the default configuration doesn’t keep a local copy of every NAR file, and the substitution cache, which is why the system must download the entire system from the cache on every rebuild?
I understand that this saves disk space, but it seems overly reliant on the remote cache.
How can I configure it to keep a local cache?
Yes, I realize now that manual configuration is better for anything that needs to be dynamic, e.g. network/wifi configuration.
I’m used to rebuild taking 5 seconds, not 5 minutes, and also not pulling from remote cache. I’m not sure if I’m doing something wrong.
I am indeed discovering now how painfully slow the rebuilds are!
I’m currently using serial only, but I agree that I should use ethernet instead for provisioning, rather than trying to configure it correctly on-device.
However, it’s also an exercise for me in building offline systems, and I was a bit surprised by how dependent it was on internet.
I agree, I’m excited that Nix has so much functionality around this, yet also struggle with getting it to work 
My current struggle has been in getting the hardware to work (sdimage, serial, bootloader, emmc, usb network devices, etc.), which is all untestable/irrelevant in a VM setting.
For example, I was trying to configure a USB wifi device to act as an AP for both 2.4G and 5G, which caused hostapd to fail since the device only supported single-band, which caused wpa_supplicant to fail, which caused all network devices to fail, which meant I couldn’t rebuild. The solution was deleting the hostapd.conf and stopping the service, then rebuilding with single-band.
Yes.
If you don’t specify the --offline flag, nix will still attempt to ask remote caches for binaries though, and when those then don’t return the expected messages nix tells you that the cache is broken, rather than understanding that your network is offline. That’s all --offline is for.
No, that’s not how nix works. That’d be stupid and defy the purpose of the cache. You’d never end up downloading any cached binaries.
Nix computes .drv (“derivation”) files, from which it then figures out which archives to download and turn into store paths. To my knowledge NAR archives are only used to transfer things over the wire to prevent encoding shenanigans.
Yeah, that’s my point. It’s a little hard in practice to figure out if a given deployment does or does not require additional dependencies.
There’s always a chance that you’re depending on something like the -release channels, or maybe -small. That, or hydra happens to have fallen behind for a specific package you depend on this week. Or maybe you added an overlay without fully appreciating the impact. All of these would really trivially lead to build output differences, and hence rebuilds - but “obvious” things like that aren’t the only potential cause.
E.g., for “changing a network configuration”, there’s a good chance you end up calling out to something like pkgs.writeText or pkgs.writers.json, and those functions might then in turn use utilities like jq.
Those utilities might not actually be part of your runtime system, and hence deleted during garbage collects (or never deployed to the device in the first place if you deployed using a utility that uses nix-copy-closure). That then would result in a system that cannot rebuild itself; jq is a build dependency, not a runtime dependency.
As I say, there are some nix.conf settings that can control garbage collection behavior, though I don’t think any of them work for controlling what nix-copy-closure deploys.
Most likely, but nix is also very memory hungry. With a sufficiently small machine you may well be thrashing. Hard to say what’s going on without more details about your hardware/configuration.
Or maybe you’re actually rebuilding some C code or something.
But yeah, you’re going through the pains of actually trying to figure out how to deploy an airgapped, low-compute environment. NixOS is frankly just not designed for this use case, so you’ll have even more papercuts than usual (though the end result can probably be very nice if you put in the effort).
liminix or such might be a better fit, depending on your use case.
Thanks @TLATER ,
I’ve concluded that:
dd writing to a microsd card on MacOS should write to /dev/rdiskX (80 MB/s) and not /dev/diskX (8 MB/s)
provisioning remotely makes more sense:
nix build .#nixosConfigurations.nixos-example.config.system.build.toplevel
ID=$(readlink -f result)
nix copy --to ssh://root@192.168.0.10 ./result
ssh root@192.168.0.10 ${ID}/bin/switch-to-configuration switch
And ‘nix copy’ only transfers the delta, which is fast.
IIUC, building a system requires a different set of inputs than activating a system (that is pruned of unnecessary inputs).
I think the ideal would be if the system config had a simple setting like “nixos.self-rebuildable = true” which would include any build dependencies into the output.
Similarly, is there a way to build a builder, i.e. toplevel output that includes the build inputs for something else?
If you want nicer wrappers around this, there are a lot. I recommend deploy-rs, but many people use Colmena, and there are many other deployment tools that essential just do the same build->copy->switch over ssh.
There is such an option, it’s called system.includeBuildDependencies. Beware, it drastically increases the closure size, though, as it actually includes everything needed to rebuild the entire thing from scratch.
That is basically what the above option is doing for the special case of the nixos toplevel. See the implementation: