Eventually reproducible nix(os)-builds

Switching between generations requires having both the current system closure and the closure of the generation you’re switching to in the store, so in the worst case you need 2x the storage of a single system closure (this can be mitigated by reducing the difference in the closures, but that’s hard to control since a commit changing glibc will introduce a lot of churn).

This means that on a 10G system you at most have 5G of space for your system closure; NixOS reaches this pretty easily, since the base system is quite big as you say. This leaves very little space for actual storage, let alone getting close to alpine.

Other distros are different, since you don’t have to commit to the full resulting closure in one go, and at most need 2x the size of the biggest package to be updated; presumably hence the idea of a more granular per-package/service switch.

2 Likes

Ha! What a coincidence. I watched your talk about finix yesterday, but that was for the minimalism portion of my “perfect setup”. Liked the ideas. Loved the first demo xD

I would really appreciate that. I’m a somewhat decent software developer, but I’m kind of lost where to even start this endeavour. But with Nix, it seems that my dream setup is actually archievable.

Here and ready :wink: no serious, we can chat about it (maybe in a seperate thread or private, to not annoy everyone here?). Or are you saying that I should open a blog for this? Sry, english is not my first language.

1 Like

Fair enough, until now you’ve just mentioned liking it being distributed. This is true; however, updates operate in large timescales, so I don’t believe deployment server resilience is actually that meaningful. It can also just be any random machine given how nix builds work, so it’s not like the build machine is that valuable or difficult to restore.

Even then, given how your infrastructure works you already won’t be able to update without a separate build server (at least I don’t think rebuilds on nodes with 1GB of memory - a third of which go just to evaluating nixpkgs - and 10GB of disk space will always be feasible), so I don’t really think you’re gaining much by invoking the build on the target.

2 Likes

oh you’re assuming that two generations are entirely different - right, that is a bad worse case scenario

this is heavily mitigated by either frequent updates to frequently updating channels, or regular updates to infrequently updating channels

shout out to ctrl-os which i hope wildly succeeds in LTS channel… that would be helpful here, potentially

1 Like

These are the corner cases which break the neck of my current setup. I am following master eagerly, but sometimes, a lot of packages need updates because of similar commits. Thats why I wanted to implement “eventually reproducible builds” as described on top.

Requirements - Alpine Linux → Alpine needs at least 1GB of Storage, 256 MB of Memory

So I’m inclined to say that a NixOS à la Alpine might need a smaller footprint then regular NixOS because the base system would be smaller. Yes there is some overhead caused by the closure, but it can’t be that bad ?

Exactly, we could have the flexibility of “regular” distros, (here: needing less space), while providing reproducible builds (if everything goes well).

I don’t see a reason, why this has to be the case. Alpine has such a small footprint. And Nix and NixOS are still developing tools. Maybe, some day, the optimization of the build will be so far advanced that this will work without much trouble.

I totally agree, that there are things, that will absolutely impossible on a 1GB/10GB vps, but running NixOS should not be one of them.

CTRL-OS Documentation → Thats probably the one you’re talking about
Codethink Trustable Reproducible Linux (CTRL OS) | Codethink → this one came first in my search.

Naming things is always the hardest part.

It did work until recently (went well for 9 months), i’ve spun up some “worker nodes” (plural for redundancy) with 2GB/60GB for remotebuilds, but they are more expensive. Would be nice if I didn’t have to

You may find Liminix interesting.

The memory overhead in your scenario (on-target eval) is down to using nix to evaluate nixpkgs. nixpkgs evaluation could maybe be made more efficient, as could nix, but this is hard and I expect that the memory overhead of evaluating nixpkgs is just going to grow (that’s certainly been the general trend for the entire lifetime of the project). Perhaps this could be solved if the big split ever happens, but I don’t really see it within the next 5 years (and I won’t try to predict beyond that). The current governance just doesn’t lend itself to the big, committed decisions we’d need for that to happen (or be decided against for good). For the time being, NixOS’ minimum memory target for your use case is probably about 700MB, but that’s just taking alpine’s number and adding the usual nix eval overhead. I don’t think NixOS is hugely more memory hungry than alpine, but systemd & co probably adds some, so a more realistic target is probably ~1GB.

As for disk space, nixpkgs isn’t based on musl, glibc alone is a big deal, let alone things like systemd. Alpine is designed around being small, it’s difficult to get NixOS to that size without building a custom nix-based distro. I can see a world in which things get better (currently my smallest instance is ~6GB), but I don’t think it’ll quite get to alpine sizing, especially since nixpkgs does not focus on this use case - in fact it is probably antithetical to most users’ use cases.

It’s definitely possible - even today - to build an alpine-sized Linux deployment with nix, but nobody has turned that into a popular distro yet. You can indeed probably get quite a bit smaller, but that’s a highly specific use case and you’ll likely have to forego using distros altogether for that goal.

This isn’t impossible, I’d argue not even very hard, like @aanderse says; mostly time consuming. Even if you want to go fully custom, you can still reuse some nixpkgs packages (notably the kernel build infra), and the musl-based stdenv means that a lot of work is done for you, but you’ll have to know what you’re doing with nix (you seem to), nixpkgs (read the repo), and be able to build a Linux system from scratch. Given that you’re saying you don’t have the experience to just sit down and do that today, but that your understanding of nix seems solid enough, I’d say maybe a half year of learning and toying with this, if your spare time looks anything like mine and you invest all of it?

The difficulty after that is to create a NixOS-style deployment system on top of that baseline yourself, since NixOS’ activation script and modules rely very heavily on systemd and a bunch of other stuff which you’d probably eject from a super-minimal Linux. But it can be done, albeit at the cost of some robustness. The activation script is ultimately not that complex, and maybe it can even be reused to some extent. You’d probably also need significantly smaller parts of it for your use cases (at some point it becomes more reasonable just to go with an A/B update scheme instead of having granular generations, but you would still want a module system).

If the Liminix project is anything to go by, a full project, all polished and nice, would probably take a competent developer years. But something to toy around with is probably doable in normal hobby time scales.

As an aside, your goal of an “eventually consistent” deployment strategy comes down to writing a very clever activation script (or maybe an activation daemon since it would need to be stateful?), so if you really want to chase that, learning how the activation script works in detail is where you’d start. It’s very counter to how NixOS is supposed to work, though, so you’ll be facing some technical adversity, and keeping up with changes to NixOS’ activation script will be a continuous maintenance effort.


My attempt at giving you the toys you’re asking for out of the way; the pragmatic answer is to make your updates more granular (a time-based heuristic just doesn’t map well to the backing git model, you should be limiting the number of commits, not the time period), and to use some of the optimizations @bme is suggesting.

2 Likes

Improvements in this are are very much ongoing and drastic savings are on the horizon, so I wouldn’t discount the possibility that nixpkgs eval memory requirements will start decreasing drastically :slight_smile: . Btw memory usage metrics for trunk can be tracked here: Hydra - Job metric ‘nix-env.qa.allocations’.

Nix itself tracks those consistently in Making sure you're not a bot! against a fixed revision of nixpkgs. The most recent drop comes from https://github.com/NixOS/nix/pull/13987.

3 Likes

I mean, I’m sure that’s happening, but… That metric looks like we’re not even back to where we were at start of the year? I’m also pretty sure nixpkgs eval doesn’t usually take 6GB, though, so I’m sure that particular metric doesn’t reflect the practical effect very well.

Well, that’s because that particular metrics is for nix-env over the whole nixpkgs. A more useful metric is Hydra - Job metric ‘nixos.smallContainer.allocations’

2 Likes