Hey, I’m looking into the feasibility of using Nix (as a package manager, probably not the full NixOS) in an environment where the following constraints are critical:
Builds need to be reproducible, and easily testable before deployment
There is no “internet” connectivity. There is communication, but the device can’t send any arbitrary IP requests. Everything needs to be transferred over manually via custom tooling.
So of course, as a NixOS user myself, it feels like Nix might be perfect for this use case.
The main workflow that I want to run by people here who are more familiar with Nix internals than me is:
Having a mirror of the satellite’s Nix state on Earth
Whenever changes are needed, Earth’s Nix state is rebuilt as needed, *including* garbage collection
The /nix/store folder is diffed with the previous Earth build, to see what new was added
All the new /nix/store items are transferred to the satellite, along with the new configuration flake
The satellite runs rebuild on the new flake. All of the needed /nix/store stuff is there, so nothing new needs to get downloaded.
For some more context, the device is a Nvidia Jetson, which runs a custom debian distro called Jetpack, and running NixOS on it is infeasible. However, running the Nix package manager is likely fine, along with some other custom activation scripts to automate other behavior on the device.
Currently, the only solution I’ve seen my workpace and other companies use for satellite linux management is manually transferring files and bash scripts to update the non-reproducible state, and pray that nothing breaks. I believe this needs to change.
What do you think? Is there any important Nix internal stuff I could make use of, or some details I’m missing?
For that you could engineer your own approaches, or you could make your own operating system by using Nix just as a tool and language for building and describing your OS as the following have,
I still believe that NixOS is a better fit for the use-case you described than Debian, but making your own minimal OS with Nix gives you even more control and will likely result in a lot less data to transfer.
For some context, I’m not worried about the initial data transfer, it’s more about incremental updates. I’m pretty sure it would be ok to pin the NixOS/nixpkgs version to a specific stable release, and include that in the payload at launch, along with some common libraries like various CUDA versions. After that, any service updates should be fairly small, unless I’m misunderstanding. Especially if it’s garbage collected, and no build dependencies remain.
How tight are your bandwidth/storage constraints? E.g. if you have a bit to spare, you can have Nix on-device for GC, and transfer (as an archive to unpack) a chunk of a binary cache containing the new derivation and the needed new paths (no on-device Nix evaluation, Nix evaluation is horrible).
In my experience, «whatever Linux based OS fits the device booting + all the actual tooling in a single Nix buildEnv» works pretty well for constrained settings (it is also the cheaper upgrade option from your current setup so you can polish the tooling without giving up the existing last-ditch-debugging options).
As far as I’m aware, the upload bandwidth constraints are in the hundreds of megabytes, and the storage constraints are in the hundreds of gigabytes. I plan to have Nix (hopefully NixOS) on-device, and it should be able to do its own GC after transfers. As far as I can guess (haven’t tested yet), I can just copy across any *new* /nix/store folders to the device, and it won’t require any building to preform the rebuilds?
You may not have the power to make sweeping changes in the existing system, but if the system were custom and minimal, and all of the programs it was running were statically compiled, you can perform binary diffs and easily send single files to /bin on the satellite in order to update the whole system’s functionality. It just depends how you want to construct the system. The mass rebuilds caused by small changes like to openssl with nixos may not be suitable long term for sending updates to the machine, but may still be better than debian, (not sure since I don’t know what programs are being used), making the approach of engineering your own operating system more likely to succeed and provide the benefit you’re looking for.
True, though having Nix manage services and guarantee that there won’t be dependency conflicts helps. I’m pretty sure the bandwidth will be enough to manage Nix stuff.
Oh also, a lot of clients’ code is python, because of course it is. It’s running on a jetson so people like using python to write any GPU/ML code. Makes me suffer, but Nix does help.
guarantee that there won’t be dependency conflicts
Making your own OS using Nix, like the examples above doesn’t mean that dependency conflicts are re-introduced somehow, that was already solved by using Nix to build the system.
Also nixos-jetpack’s FODs are probably a more convenient and reliable way to manage the nvidia drivers than jetpack, whether you use it to deploy nixos or just re-use the approach bootstraping your own thing.
If I’m allowed a bit of shameless self promotion, there’s also the talk from the previous NixCon too (Nix: a space odyssey) (:
The 2023 one was just me quickly putting something together last minute to fill in for a speaker that could not make it, so it ends up being a bit of an extension of what I had said the year prior.
That’s so awesome! Would you have any pointers to existing open source foundation that I could build from, so I don’t have to reinvent too much? I hope I can get my work to be open source in the end too.
While I’ve tried to push for open sourcing at least parts of our work not much has come of it (yet?), so the only pointer that I can give you is the already mentioned jetpack-nixos. I’ve tried making my own version of that but their version came out faster than my own and it did things cleaner so we ended up basing on top of that. Beyond that it’s not much more than configuring NixOS on any other hardware.