Why does the NixOS infrastructure have to be hosted in a centralized way?

Development Infrastructure
31

None of this has anything to do with finance or cryptocurrency.

32

The moonshot is fixing the way we deal with shared libraries. We pay an immense disk-space and bandwidth cost every time we rebuild a library which is depended upon by other packages. It is an insane cost. No other distro (except Guix) pays this cost, and it’s getting increasingly impractical.

I’ve said a bit more about how to fix this elsewhere but it’s still in the oven and not done cooking yet.

So long as its symbol table doesn’t change, modifying glibc shouldn’t trigger rebuilds of things downstream of it. It also shouldn’t change the contents of packages downstream of it, except for a tiny intermediate “indirection table”. The only thing that should change in the downstream outpaths is a pointer to that indirection table.

This goes way beyond CA derivations (but works best in combination with them) and better compression. It’s not the same thing as pkgs.replaceDependency but can be used instead of it if the dependency’s symbol table hasn’t changed.

3 Likes
33

I’ve wanted to apply the same approach to shared libraries, but I’m wondering how you’d intend to accomplish it without CA derivations? You still need a way to turn the new glibc sources into a stubs output, and short‐circuit the rebuild of downstream packages based on the actual hash of that output, right? Dealing with header file changes is also tricky.

LLVM already has a pretty comprehrensive‐looking tool for creating stub shared library files, and of course macOS has been natively doing this kind of thing with .tbd files for years. Though I think you would still need to do a minimal amount of O(n) work across all packages by rewriting the paths in the resulting binaries from the stubs to whatever actual library version you want them to be linked to in the final system. (That’s a lot better than rebuilding the world for a glibc security update, though.)

3 Likes
34

I’ve thus far presumed that the store hash is based on inputs + content of the derivation (damaged by too much git?).
So the hash of a derivation will not change if I modify the build phases (w/o input changes)?

35

Build phases, etc. should be considered inputs here; Nix expressions eventually produce a .drv file with the builder, inputs, and variables, and the hash covers all the information in the .drv file. The key point is that it’s not a hash of the build output.

1 Like
36

Exactly. Levers of power attract unpleasant people and bad behavior. Eliminating those levers is not easy, but the benefits are worth it.

The infra is not the most worrying lever of power, or even a major one. The fact that nixpkgs’ design makes maintaining long-lived forks extremely painful is probably the biggest.[^1] Some of the obstacles to forking nixpkgs include:

  • Constant treewide commits cause horrendous merge conflicts. These treewides are often for trivial reasons, like English grammar nitpicks. Nobody is weighing the benefit of this stuff against the cost of causing merge conflicts.

  • The traditional nixpkgs formatting style was designed to minimize merge conflicts, but the new autoformatter is amplifying them instead. This is not “aesthetics”, and it is a serious enough problem that “mumble mumble RFC process” is not a justification. Making merge conflicts (even) worse will make nixpkgs (even) more centralized, which will – in the long run – make the political battles even more fierce.

  • No static types in the Nix language means we have to rely on expensive build success/failure as the only CI, and we can’t check any meaningful assurances given to out-of-tree uses – which means that moving leaf packages out-of-tree means giving up any CI. [^2]

[^1]: I always thought it was madness to try to maintain a long-lived nixpkgs fork, but apparently there is now a group attempting to do that.

[^2]: With a real type system like Haskell’s, typechecking nixpkgs would provide a guarantee of things like “for all well-typed out-of-tree use of this expression, the resulting use will be well-typed”. Right now none of our CI has the ability to express this kind of “forall” quantification – we’re stuck at the bottom level of the arithmetical hierarchy.

PS, why don’t discourse footnotes work here anymore?

5 Likes
37

[Citation needed]. The traditional style was mostly whatever people were feeling like. Like the current style, everything is a tradeoff between different desirable properties. The RFC text describes the approach taken to reach compromise on these, and why. Also note that one will mainly notice conflicts caused by the new style, and not the many merge conflict situations this style avoids (e.g. last-element insertions due to trailing commas).

[Citation needed]. I think that you are over-emphasizing the (real and non-trivial) effect of merge conflicts, especially regarding centralization. Moreover, I think you are conflating merge conflicts caused by the migration to the new style (which is mostly one-off) with systemic merge conflicts caused by it.

I briefly looked into it back when I was Discourse admin, IIRC it requires some plugin functionality, at least it wasn’t as trivial to enable as flipping an option, unfortunately.

7 Likes
38

Thanks I did not know about this! Yes, llvm-ifs --output-elf is probably 95% of what is needed. We will probably need to do additional normalization (e.g. sorting symbols, maybe additional filtering) since bit-exact linkable stubs are a high priority for us but probably not as important to LLVM.

Yes; that stubs.so gets built by a Fixed Output Derivation.

We’re accustomed to FODs being used for fetching stuff over the network, but they can be used for other things. You can run llvm-ifs --output-elf=$out/stubs.so on .so files in another derivation’s outpath, from within an FOD.

Of course then you have to manually copy the hash into your .nix expression, which is a drag. If you have floating output CA-derivations (FLOCADs?) and use those instead of an FOD you don’t have to do this. So floating output CA-derivations make this much more convenient.

Yes, every derivation which has .so dependencies turns into two separate derivations, which I’ll call compile and relink (this can be automated inside of stdenv.mkDerivation).

  • The compile derivation looks exactly like what we have today, except that it has only the stubs.so of its library dependencies as inputs. So it gets rebuilt only when the symbol table changes in a way that doesn’t get normalized away.

  • The relink derivation takes the compile derivation as an input, and simply uses patchelf to change any references to the stubs derivation so they point to the real dependency.

When you upgrade a library (say, glibc) in a way that doesn’t change the symbol table, the stubs FOD won’t change, so none of the compile derivations will get rebuilt. These derivations are the ones that involve significant build effort. All of the relink derivations will get rebuilt, but those are trivial – they just run patchelf.

These patchelf runs are what you noticed:

To avoid bloating the binary cache, the straightforward approach is to mark all of the relink derivations with allowSubstitutes=false and exclude them from cache.nixos.org. That’s a very crude sledgehammer, but it works today with no changes to the nix interpreter. There are better solutions but they take longer to describe or need new interpreter features.

This two-derivation-step build process would let us use prelink(8) to get faster startup times, like Red Hat and MacOS do.

Also left to future work is dealing with the situation where a new version of a library adds symbols to the symbol table, but doesn’t change or delete any. Ideally we’d like to avoid rebuilding dependencies that don’t need the new symbols, but the mechanism for detecting whether or not those new symbols are needed and making note of that fact in nixpkgs both need to be developed. This is closely connected to the point that you raise:

I have a feeling that the two problems (deciding which versions to take header files from / avoiding rebuilds when unused symbols are added) are related, and will probably both be solved by the same mechanism at some point. But it’s just a hunch. llvm-ifs --output-ifs is probably useful here.

7 Likes
39

Parallel means analogous.

1 Like
40

It sounds like we have had very similar thoughts on this, which is encouraging! I didn’t think of just hacking it up with FODs, though.

I didn’t know about llvm-ifs until recently either! I suspect it may already have bit‐reproducibility as a goal, given the implications of “The linkable shared object stubs can be used to avoid unnecessary relinks when the ABI of shared libraries does not change”. But if not, it should be easy to detour via the text‐based format and do any necessary normalization there.

Re: prelink, @fzakaria’s recent blog posts on optimizing dynamic loading with Nix and “management time” feel very relevant here.

I am not sure the headers problem is quite so simple because, even ignoring the preprocessor or the fact that C++ headers regularly contain actual code, I think there are probably ways that downstream derivations could condition on the availability of an API that would be tricky to deal with. If nothing else, there’s nothing stopping a derivation from textually processing a header to react to pretty much any change to it. My preferred solution that did not risk introducing what are effectively impurities was to write a libclang‐based tool to normalize header files to the greatest extent possible (formatting, comments, order of declarations where irrelevant, etc.), and then just eat the rebuilds when there’s any non‐trivial change to the headers. (Maybe precompiled headers could be useful here.) So far, that’s the best thing I’ve come up with that doesn’t risk jeopardizing the essential properties – i.e., that you should never find yourself in the traditional distro position of “gah, I should have rebuilt with the new library version!”.

But yes, this means mass rebuilds still happen when the API is extended (to some degree; CA derivations mean that we at least benefit from short‐circuiting). I am not sure that can be avoided in a way that doesn’t compromise the essential properties we want, but I would be interested in any attempt to avoid it!

Unfortunately there is also the matter of tests. Tests have to come after relinking executables with the actual libraries, for obvious reasons, and since the whole point is that we’re linking against a new library version that presumably has changed behaviour, you really do have to run all the tests again to stay honest. I suspect that a mass rebuild of all the tests in Nixpkgs is not so much less painful than a mass rebuild of all packages that the wins from this whole arrangement are as big as I’d like, but maybe something clever could be cooked up.

5 Likes
41

You might have heard of a tool that does this on purpose, called “autoconf” :stuck_out_tongue_winking_eye:

Yes I had, but I couldn’t remember where I had read it. Thanks for the reminder; for everybody else: @fzakaria’s article Speeding up ELF relocations for store-based systems shows what’s possible, and the results are much better than prelink(8).

Yeah, in long run the the (install)checkPhase needs to be moved into a separate derivation (like #209870 did for gcc) and nix-build’s scheduler needs to learn validations like Ninja has. Then the checks could run after relinking. It would also let us run the checkPhase for cross-compiled builds, using either qemu or a smaller cluster of native builders. Some more details.

4 Likes
42

Never heard of it! Is it any good? :melting_face:

(But yeah, I was concentrating too much on stuff you could actually do from within C and C++ and missed the elephant in the room…)

Just linking the other two parts here for anyone curious:

It’s very cool that we can still have a meaningful form of “dynamic linking” even in our hermetic hash‐addressed world, and then do the dynamic loading statically. And a little sad that we have hitherto basically not taken advantages of these possibilities at all!

Yes, for sure checkPhase being part of the main derivations is terrible anyway. The idea of validations making sure checks run without getting in the way of the critical path of the builds is brilliant; wish I’d thought of that before!

Running the tests for every package in Nixpkgs still seems like it’d make even a one‐line glibc security patch pretty painful, though. I’m not sure all this machinery is quite enough to free us from the tyranny of staging for that case, as much as I think it’d produce a better system in general. (Though I realize the irony of saying this in a topic about decentralization and multirepos.)

Thanks for sharing your wisdom about this! I am pleased to know that I am not the only one to have independently derived basically the exact system I’d like to see for this, and it makes me more confident that it’s a good path to go down. I’m not sure Nixpkgs could survive the kind of surgery it would involve, but I think there’s clear wins any new system could gain here.

2 Likes
43

Thanks @amjoseph and @emily, reading your conversation was enjoyable and inspiring.

How does this work with pkgsBuildHost/nativeBuildInputs? AFAIU you propose that if a = mkDerivation { ... buildInputs = [ b ]; } then a be implicitly split into a.__stubs and a.__relinked, where a.__stubs is the actual build and only depends on b.__stubs. However, if a = mkDerivation { ... nativeBuildInputs = [ c ]; }, we have to use c.__relinked whose hash is unstable?

Honestly, if we did this on a granular opt-in basis (I don’t see why we couldn’t), this sounds OK? For a maintainer, it’s not much different from having to update the srcs, and as to out-of-tree users they could always “unpin” the derivation by removing outputHash in an overlay if their compute hours are cheaper than their human hours

Taking notes: “FODs in non-leaf nodes to pin down the graph indeed are a thing”

A self-deduplifying back-end at cache.nixos.org doesn’t involve changes to nix either:)

Anytime you realize the build-derivation, you must also realize the test-derivation concurrently with any referrers of the build-derivation.

The word “concurrently” is what distinguishes this from a simple wrapper,

Gosh yes. I wonder if this could fit into tvix’s lazy “we don’t realize the output until we read it” paradigm, but it does seem like these “tests” are their own concept. OTOH, I don’t see how this is better than simply moving all checks to passthru and building better tooling to collect and build them. Without changing anything about the language or about .drvs

This. I’m rather hopeful that having “a” formatter enforced by the CI will, over time, prove to reduce the number of conflicts, even if it makes a few locally suboptimal decisions…

The infra is not the most worrying lever of power, or even a major one.

Yes let’s get rid of these python-updates while at it:) On a serious note, while I think that all decisions that can be made locally should be made locally, there are problems that require making decisions that are global (causing global conflicts), your stub thing and cross-by-default being examples already at hand. It’s not just “power levers” because we’re trying to strike a balance here, between scarce availability/expert time, long-term maintainability, and being still useful in various consumption scenarios. And yes I think occasionally removing cruft can be helpful with the former

1 Like
44

Yeah, I don’t think there’s any getting around changes in build tools resulting in mass compilations. You might still be able to benefit from content‐addressed short‐circuiting behaviour to some extent, but glibc was probably a bad example because it’s unlikely we can come up with any truly satisfying, compromise‐free scheme that doesn’t result in a ton of builds every time glibc changes in any way. There’s still a lot of room for stuff that would need to go to staging now but would result in dramatically less CPU burned under this scheme, though.

I also personally have hopes for a system where we can track dependencies on a more granular level closer to files than entire packages and therefore hopefully achieve even fewer rebuilds, which I think would multiply the potential benefits of a scheme like this. But that, I think, is probably beyond what we could reasonably turn Nix into.

45

Well there is an obvious way out, it’s just slightly embarrassing: we pin a “pre-cached nixpkgs revision” and take pgksBuildHost from there

46

This would be very interesting. I think that Bazel’s approach, where cc_{library,executable} suddenly float up to the same layer as language-agnostic “packages”, is very unsatisfying because it means that if a project uses Bazel it essentially can’t be consumed in any other way. This contrasts with e.g. “a meson project whose authors test and deploy it using nix”, which is normally still just a meson project. I hope we can improve dynamism/granularity without compromising this property

1 Like
47

That falls under “giving up” for me, I’m afraid :slight_smile:

I have a lot of thoughts and ideas about this, but unfortunately this particular margin is much too small to contain them!

2 Likes
48

It may not seem like it, but the tests are just a tiny fraction of the cpu cycles needed in order to do a mass rebuild of staging. Tests seem like a big burden because they often use only one CPU core (enableParallelChecking is awesome when it works though) and they block (run in series with) any downstream dependencies.

Think of it this way: For a fresh rebuild you can’t start building gcc until the (big slow) bison test suite has finished. For a security fix with separate compile/relink derivations you can run both the bison tests and the gcc tests at the same time.

Yeah I’m not sure of that either. And in any case I don’t think I am that surgeon.

49

This is a great question.

And yes, that’s the (start of the) answer!

In the long run something like multiversion packages will be cleaner. Especially if the package meta attrset has a causes-stubs-change field (only necessary for mass-rebuild-triggering libraries, of course). Then the pinning can be automated.

Yes that’s the plan. It’s only needed for packages that produce .so outputs and are widely depended upon.

Sure, but there are a few aspects of the way it formats that aren’t just aesthetics. Minimizing the number of lines that change (because git diffs are line-oriented, not token-oriented) is important. The traditional formatting of ] ++ lib.optionals foo [ was chosen to minimize the number of changed lines, not because it looks asethetically pleasing.

A much older example is the GNU coding standards, which put the { on a separate line. It looks hideous on the screen, but it’s worth putting up with it because in a language with optional curly braces (i.e. not Rust) turning a one-statement body into a two-statement body can cause a conflict avalanche if you don’t do this.

50

Right, I think there is a fundamental difference here where I see the fact that the package set is built as a consistent atomic unit with the same versions for everything (modulo the unfortunate exceptions where we have to carry multiple versions) as one of the core strengths over other distributions that I’d like to maintain. So “just build stuff against the old OpenSSL but link to the new one”, “don’t bother rebuilding just because build tools got bumped”, etc. don’t really appeal to me. (And this of course ties in to what this thread was originally about…)

FWIW I understand that Git diff size/reducing likelihood of conflicts has been one of the top, if not the top priority of the formatter work. Of course whether that’s been fully achieved is another matter, but it’s definitely not been ignored; diff size comes up constantly in discussion from what I’ve seen.

2 Likes