I am confused about the reproducibility of a system through flakes

Help
1

Sorry if the title is not very clear. I was not sure how to summarize my doubt in a single sentence that made sense.

I’m looking for an explanation on the philosophy behind having your system configurations as a flake stored in a git repository so you can track your changes and do your “configuration as code” and start “treating your machines as cattle” thingy.

So far, I have only used Nix for defining my macOS configurations, and to be honest, I never used my two Mac machines as daily drivers at once. I always stick to one of them and keep iterating my configuration on that single machine.

The idea was to expand my flake into a kind of all-my-systems store, including some VMs of my home lab, so I don’t have to take notes on how to get some VMs back to the intended state.
The idea is that I can have everything in my repository, and all that they need when I get a new laptop or a new whatever is just to use my flake to set everything up. That is the goal, and that is what attracted me to Nix.

However, every time I check how to set a new NixOS machine to use your flake (which I haven’t done yet) I am greeted it with some steps that, in my opinion, go against this philosophy. You are supposed to:

  1. Set a new NixOS machine.
  2. Copy that specific machine’s hardware configuration into your flake.
  3. Store that hardware configuration into your flake.

How is that reproducible? What if I spin up another VM that I want to be exactly the same configuration, but I prefer to use a different network card, or I wanted to have two network cards? Then my Nix configuration, which is only focused on user-land things, will need a new hardware definition added to my flake? This part is what confuses me and makes me think that this is not the right way to “treat your hosts as cattle”. It feels more like “have a spreadsheet with all your pets” to me.
This gets even more confusing if you decide to use an actual physical machine, where the hardware will be very specific to that machine. If you then decide to change it because, let’s say, your actual physical machine broke, you will have to modify the hardware section of your flake to accommodate your new machine?

Can someone clarify me if people use nix and then keep a curated set of pets? Or how does people do this?

2

I’m not any expert by any means, but you don’t have to do that flow of install nixos → generate hardware-configuration → copy to flake.

You can use something like nixos-anywhere, with disko to handle disk partitioning. That’s what I did when I set up a NixOS VPS with Hetzner. That system lives in my “homelab” flake, which effectively now is a “all-my-servers” flake.

How is that reproducible? What if I spin up another VM that I want to be exactly the same configuration, but I prefer to use a different network card, or I wanted to have two network cards? Then my Nix configuration, which is only focused on user-land things, will need a new hardware definition added to my flake? This part is what confuses me and makes me think that this is not the right way to “treat your hosts as cattle”. It feels more like “have a spreadsheet with all your pets” to me.

There’s no getting around adding machine-specific hardware modules. But yeah, you’d just update hardware-configuration.nix or wherever appropriate in that host’s config. You could also extract out logic into a module.

Does that help answer your question? I maintain a couple laptops and ~six or so servers with NixOS, so I can answer some questions around deployment.

3 Likes
3

Note that another sense of “reproducibility” w/rt flakes comes from purity.

Pre-flakes, the intended way to manage a system’s version of nixpkgs was with channels, which were stateful per-device. Hence, the same config on two devices could produce different results by virtue of using different versions of nixpkgs.

A flake-based config, by describing and locking its inputs, is (in theory) wholly reproducible from just the flake itself, with no per-system state. This purity can be violated by, for example, having a relative local path as an input, but for the most part the idea that “the flake is a self-contained description of the entire system” holds.

Such reproducibility can be achieved without flakes with tools like npins, which serve a similar “describe and lock dependencies” purpose. However, flakes come with purity enforced by default, while there is (to my knowledge) no such enforcement available without flakes.

One other thing to address is the strict meaning of “reproducible” w/rt Nix. While Nix is meant to (and often does) make replicating a configuration on a new device easy, “reproducible” is moreso in a functional sense. Derivations, by specifying expected hashes for their requirements (such as downloaded source code), and being built in isolated environments, should be bit-by-bit reproducible, such that two people building the same version of the same package can reproduce the exact same result (ignoring, for example, -march=native optimisation or similar hardware-linked variance).

This sense of reproducibility is why, for example, Guix is so valuable for the scientific community. I can publish a derivation for the code used by [whatever paper] with the understanding that another Guix user can (in theory) exactly reproduce my results.

2 Likes
4

As with all things there is a spectrum. I guess I don’t understand what a cattle nixos looks like on diverse hardware? If you have a different disks on each computer, different cpus etc, how do you think that should look? It’s very easy to go full cattle on AWS for instance, but even then: you can have instances with local block devices, multiple attached ebs volumes, extra nics. You manage that typically by injecting metadata about the hw into your config (for instance I manage all vms with terraform, and inject json into the userdata script that nixos modules read to configure themselves). The nixos installer hardware scan effectively does that on your local hardware, but it’s not the only way to do things. If you have an identically labelled set of hard drives on all your computers, you can keep that as shared. If you only use the same CPU family, again all shared. You can even reject nixos modules that depend on hardware, and your create your own that can do more dynamic setup, s.t. that if you have a kernel with the superset of modules you need, then udev rules + systemd can do setup on the fly.

Also see facter for another take on solving this.

The world is your oyster, and with nix shall you open it.

3 Likes
5

In practice, I think what’s confusing you is that the answer to this is largely “yes”.

If you’ve heard people referring to their personal devices as “cattle”, that is because they don’t understand the pets vs cattle analogy.

NixOS is however very good at declaratively describing pets, to the point that managing their similarities and dissimilarities becomes feasible.

Most of this power comes from the module system - if you learn to use it, you can create some wicked maintainable pets.

Swapping out some hardware config while maintaining more interesting software config identically is trivial, and both hardware setups can be maintained alongside each other in the same codebase without interfering, at any level of granularity. If you haven’t played around with lib.mkForce, or noticed the merge rules for e.g. types.listOf, you’ve probably not fully realized the implications of NixOS’ module system design - it really is a huge part of what makes the nix ecosystem so good at what it does.

Of course, if your laptop gets stolen, you’ll have to buy a new laptop and do some manual provisioning steps. This is the hallmark of a “pet” - however, a good NixOS config does limit this provisioning to only configuring things bespoke to this pet, rather than the full OS, which makes it significantly more achievable - especially if you use e.g. preservation to minimize drift. Still a pet, but it’s like you got an automated feeder.

Full “cattle”-style configuration is of course feasible, but you’ll have to lean heavily on systemd and networking features, and actually provision at least similar hardware (and write a custom installer). This is not what people do in their homelabs.

All of this has almost nothing to do with flakes, reproducibility, or even nix itself, it’s all developed in NixOS. All those properties just help upstream develop and maintain NixOS in the first place, end users can ultimately largely ignore that stuff and still get benefits from NixOS as a deployment tool.

I believe that this is why weird warts are so common in random configurations; people often don’t understand what they’re breaking (or sometimes just don’t care in the short term), and the results are often not immediately visible.

That’s a misunderstanding of “pure eval”. Pure eval means that your nix code will be evaluated from within the nix store.

Without pure eval your nix files are allowed to be anywhere, but the resulting deployment script is still generated with functional “purity” - you get fully reproducible results.

8 Likes
6

that’s a misunderstanding …

Interesting, thank you for the correction. Perhaps I articulated myself poorly - my understanding was that arguments like --impure for nixos-rebuild were for flakes only, and that there was no way to achieve pure eval w/out using flakes (indeed, I’ve definitely done some impure evals w/ my flakeless setup, for example when using npins environment variables).

Do you know of a way to perform pure eval of a non-flake config (using, eg., part of the nix3 CLI)?

7

You technically can, but not really no. This is one of the big gripes with flakes, and why people so often say that they have features unnecessarily tied to them, which could benefit the greater Nix ecosystem.

3 Likes
8

Even in an environment that’s as “cattle” as it can be, there’s a few files that must be unique to each host, such as /etc/machine-id and the SSH host keys. And, if you want to be able to move an existing host to new (matching) hardware 100% seamlessly – which, as I understand it, is a big piece of the point of “cattle” configuration management – you have to have those files be part of the system derivation somehow.

(In my own system derivations I go out of my way not to derive SOPS machine keys from SSH host keys, for this exact reason. I don’t understand why all the guides tell you to do that. Yes, it avoids a difficult and poorly automated key management step, but it’s setting you up for a catastrophe later.)

3 Likes
9

Bit-for-bit reproducibility is a very high bar, and while projects like https://reproducible.nixos.org/ attempt to tackle this and binary caches can hide/mitigate/shim bit-for-bit-style reproducibility, the use of nix alone is not sufficient to claim bit-for-bit style repro, especially because many build chains and software are not written to be easily bit-for-bit repro. See "NixOS is not reproducible" for more details and perhaps a better place to actually have the discussion if more commentary is important. I’m only commenting in this thread because I think as a community it does a disservice to advertise claims that later do not prove out. That can cause a lot of loss of faith in the ecosystem.

Nix does provides dependency-graph reproducibility, as you note, which is already great for most practical purposes and very hard to achieve in other ecosystems. A great win! Definitely a huge reason to use nix, and I don’t see a better tool currently.

10

Just don’t use impure language constructs, in particular <…> lookup paths.
This is exactly what --pure-eval will enforce in addition to closing down network access.

In practice it’s often convenient to have some controlled impurity though, for instance toString ./some/file for local helper scripts or (rarely) passing builtins.currentSystem at entry points.

This only half true, i.e. the worst kind of wrong, and a misunderstanding that keeps getting perpetuated. Yes it was originally intended like that, but in ancient times. Nobody would seriously have suggested doing that in say 2017.

Channels are a distribution format for Nix expressions (a directory with particular things in it, including a Nix expression), are still being produced by Hydra, in practice have persistent identifiers (example), can still be consumed by flakes, and are only problematic when references to them are managed outside of expressions.

So what beginners indeed often did in the distant past — due to lack of well-documented and easy-to-find alternatives — is use nix-channel --update which would download a channel foo to some file, and then refer to it as <foo> in their expressions. This of course would fall apart on the next machine which may not have that downloaded channel at all, have it downloaded at a different time and thus have a contents, or get out of sync on the next nix-channel --update. After a while people would figure out a still barely documented fact: when building a channel from a release branch is done, Hydra pushes the respective commit back to a Nixpkgs branch corresponding to that channel (which a while ago we decided to call channel branch and at least finally wrote that down somewhere). One can just download the tarball of that commit from GitHub with builtins.fetchTarball with a hash, and even the URL is stable. npins just automates the boilerplate around storing those references in a JSON file and fetching the latest commits.

So in fact, combining npins and --pure-eval gives you that purity part.

Flakes add a bunch more stuff, such as caching evaluation of top-level attributes and hard-wiring the semantics of some of them into CLI commands such as nix flake check and nix run. The latter can also be done with a combination of classic commands, very light scripting, and the module system (which is slower to evaluate but gives you a rather strong type system and the associated machinery to automatically check your expressions for consistency).

4 Likes
11

I’m not sure I understand what you want to happen. If you have a machine with one network card and one with two, and you want to use the same configuration for both, how do you expect to use the second network card?

If you’re using by-uuid identifiers for your hard drives, your hardware-configuration.nix will never be identical for any two machines. But you can share other configuration between them through imports, like for instance your list of installed packages.

Maybe what you want is some way to automatically generate a hardware-configuration.nix per machine, according to some rules?

12

For the record, this is sort of why I don’t actually consider using UUIDs for disks to be the best idea in a lot of cases. It’s probably still the best general purpose recommendation, but it does fundamentally mean things aren’t generic. Or, to put it another way: UUIDs are state, not configuration. Sometimes state leaks into configuration, and that’s ok, and NixOS at least makes that very explicit. But in terms of having really generic logic in your nix code, there’s a strange link between eval/build-time and runtime information. For the eval/build-time code to be generic, the runtime code has to be generic as well; we can’t make a NixOS module that detects your hardware needs automatically because that would be impure. And making the runtime logic generic is where it gets stateful. e.g. NetworkManager discovering NICs and managing them statefully, or other distros probably consider /etc/fstab to be state rather than config like NixOS does, but those both still rely on generic runtime components, namely the kernel being capable of arbitrary NIC and FS drivers.

One of the options for a generic config that doesn’t have UUIDs is boot.initrd.systemd.root = "gpt-auto";, which causes the root FS and ESP to be found at runtime based on which device the system booted from in the first place. Conceptually, this is much cleaner logic, because at the end of the day the intent is just to get the system to boot with the right FS, not to use a specific UUID; what exactly that FS is can’t really be explained by a UUID, so gpt-auto instead imposes rules on the boot disk formatting and uses those rules as the definition of the root FS. It shifts the partition discovery logic into a stateful system instead of a configuration system, which I think is broadly the right move for a lot of cases. If you don’t want to use gpt-auto (its partitioning rules are kind of restrictive), I think using by-label is pretty similar in spirit; the state is “which device has the designated label?”, and the configuration is just the search for that label. I think this is more fraught than gpt-auto, because gpt-auto only requires the boot disk to be formatted a certain way, while by-label requires all disks to avoid using that label unless it’s the intended partition. But it’s a useful approximation, and I’d be interested in exploring more complicated options in this area.

I do also want to mention an interesting hybrid that sits between the “partitions as state” and “partitions as config” ideas, which is dm-verity and the roothash= kernel param. The premise here is somewhat more generic than gpt-auto and has no interest in whether the root FS is on the same disk as the ESP. The roothash= param specifies an immutable, content-addressed partition pair. That hash is the root hash of a dm-verity merkle tree, and instead of generating random UUIDs for the data and verity partitions, the hash is used; the first 128 bits of the hash get stored as the UUID of the data partition, and the last 128 bits get stored as the UUID of the verity partition. At runtime, the OS doesn’t need any stateful information at all; the roothash= is configuration, because it only specifies the cryptographic identity of the OS to boot. That directly tells it what UUIDs to search for, and it doesn’t matter what stateful partitions get found because dm-verity will transparently ensure that all the data that’s accessed matches the roothash=.

Anyway, the big point here is just that generic configs are possible, if you figure out how to separate the state into runtime logic. The most obvious options include gpt-auto and by-label, but that’s only for the FS. For networking, there’s NetworkManager (which has reasonable default behaviors for generic circumstances). For the actual storage drivers that the FS depends on (like nvme and stuff like that, which the initrd needs to have for the FS to even be visible), there’s the hardware.enableAllHardware option that just adds a bunch of them in hopes it covers everything. I think generic configs seem like a pretty good idea a lot of the time, and I think separating your stateful information from your config is good in general. But certainly it’s not really the norm right now.

7 Likes
13

Why would you need the module system for a nix run equivalent? Isn’t a callPackage set more idiomatic?

14

Pretty sure “the latter” is referring to

1 Like
15

I just want to add in addition to what @fricklerhandwerk says:

Pure eval, as implemented with flakes today, is one of the big misfeatures of the flake-guarded features. It’s implemented by copying your repositories into the store wholesale, any time you evaluate them after you make a change, and results in huge performance overheads, alongside security risks and - more relevantly today - SSD wear.

And as @fricklerhandwerk says, most of what it does is just turn off certain built-ins and language constructs that could be prohibited by a linter.

It’s mostly implemented this way because technically you could end up encoding the absolute path of a file during eval. This, too, could probably be prohibited differently. It really pains me that this feature is so terribly implemented, the idea makes a ton of sense, but thanks to how it works it’s most of what is wrong with flakes IMO.

2 Likes
16

Yes, it gives me some “links” to follow, but I would like to know what is your flow.
Can you give me a numbered list of the steps you usually follow? Let’s say you want a new hetzner server, what do you do after purchasing it?

How is the flow with the laptops? Let’s say you get a new one for a new company you will be working for

17

That’s because I don’t expect nix to take care of the specific parts of that host, or at least not my “curated” config.
I want my config to be about users, services, binaries, and user configurations. I don’t want to maintain a repository of hardware configurations. For me that is OK to live in each host as part of the “snowflake”.

So my understanding is that I have to start the basic system with some other tool (like cloud-init for example) then clone my flake and “apply it”.
That simple sentence I have no idea how to do in the most possible automatic fashion.

And also, I was wondering if it is possible to use Nix or NixOs for the entire thing, rather than having to use terraform + ansible + nix or something like that

I’m all questions

18

The big thing about NixOS is that it manages not only binaries and services but also the kernel. (It doesn’t actually manage user configurations in detail - it does a bit, but for a lot of things you need something like Home Manager, which is separate, or else use normal vanilla-linux-type configuration files.) Because it manages the kernel, it needs to know all about your hardware. If you don’t want that then you don’t want NixOS, although you may still want nix, which you can use on top of any linux.

2 Likes
19

With NixOS yes, with just nix no.

20

@ElvishJerricco sketches how you could avoid this. hardware-configuration.nix very rarely contains anything super important (except perhaps the system tuple, but you can solve that by only maintaining one architecture per configuration or somesuch).

It’s similar to the idea of cloud-init, but cloud-init itself is a bit too focused on fhs-compliance, so hand-rolling your own dynamic config tends to be a bit better.

I do recommend starting out with per-host .nix files that isolate the “snowflakey” bits, though, and then just slowly getting rid of each option in them one-by-one.

1 Like