"NixOS is not reproducible"

direct link NixOS is not reproducible

2 Likes

A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.

I feel for the author when someone dismisses their work with “hey, didn’t NixOS fix this a decade ago?”, but if we apply the strict definition of reproducibility then no distribution is reproducibile, and probably never will be.

What I mean is: even if at some point we hit the 100% reproducibility mark it still doesn’t mean much: reproducibile according to what kind of testing? What If I inject randomness in the filesystem timestamps, user ids, build on different machines, is it still reproducible?
Even in the fantastically unlikely event of someone formally proving the entire source-to-build artifacts chain is a mathematical function of its millions(?) of inputs, there’s literally no guarantee that in the next second an update is pushed breaking the reproducibility of a package.

Also, achieving and maintaining reproducibility is often in the hands of the developers of the compilers, build tools and the people using those tools: if a compiler orders outputs in an unpredictable manner, there’s not much Arch or NixOS maintainers can do about it.

6 Likes

The author of this article has been on this hobby horse for years. I thought maybe the article would be interesting based on its edgy title, but less than halfway through it became clear who was writing and what the rest of the article would be. Frankly, it’s boring.

The truth is that Nix environments are reproducible in ways that tools for which the weaker designation ‘repeatable’ is commonly used (e.g., Docker) are not. And in the same way, NixOS systems are reproducible in a way that conventional distros like Arch never have been; running a NixOS system as it existed 5 years ago is trivial and running an Arch system as it existed 5 years ago is virtually impossible. Running a friend’s NixOS system as a VM is a trivial reproduction-with-a-twist and running a friend’s Arch system as a VM is a cumbersome, manual clone-and-repair job at best. Changing the language here effaces more differences than it honors.

Unfortunately for the author, Nix’s form of reproducibility (‘functional reproducibility’, ‘behavioral reproducibility’, or ‘build environment reproducibility’) is just more exciting, interesting, and accessible to people than the narrowly focused ‘I can get the exact same bits if I run this build twice right now (even if I probably can’t do it next month because I may or may not have the resources to reproduce the build environment by then)’ notion of reproducibility that he’s interested in for supply chain security/integrity reasons. The kind of reproducibility that people are going to be excited about at Linux conferences and across the web is, predictably, the one that lets them do cool stuff.

The only distro that has really turned bit-for-bit reproducibility into a thing normal users can easily and conveniently do, afaik, is Guix, with guix challenge. Maybe if that changes, the kind of broad reproducibility Nix and NixOS offer will no longer overshadow the younger, narrower notion of reproducibility meant by the phrase ‘Reproducible Builds’ in the work of reproducible-builds dot org. That would be a cool Nix feature. The website could also probably use a section on kinds of reproducibility, how they work together and reinforce each other, what other projects are doing to advance them, how Nix is different from those projects, and how Nix benefits from other projects’ reproducible builds efforts. There are other potential ‘action items’ for the Nix community in measuring, tracking, and improving our bit-for-bit build reproducibility, too, as zimbatm points out in the Lobsters thread.

But ultimately I don’t think anyone should worry about the clickbait title, or that Foxboron’s feelings about enthusiastic conference-goers mistaking him for a fellow Nixer is the Nix community’s problem to solve.

Props to Foxboron for filing a PR against the NixOS website to get the changes he wanted, though, instead of just griping about it. He’s obviously quite right that reproducible builds is an ongoing project and not a solved problem. That’s a message worth repeating.

17 Likes

Practical reproducibility is probably a term that helps to clearify the situation.

I do care about a certain kind of reproducibility, not about the complete fulfillment of a specificiation, that covers all theoretically imaginable usecases of the universe.

Even though nobody really uses all of that.

Arch is rebuilding distributed packages, and we have reproducible packages across several rebuilders.

It’s interesting to know what exactly does it mean. Do they just make sure they have identical builders that output the same result? Or do they specifically make sure they have different builders that output the same result?

Checkout the section of “Rebuilder Instance” in the ArchWiki, I believe that’s used for the Arch Linux Reproducible Status site.

To me it seems that it is not something widely available. And more importantly awkward to use since pacman is not source-based.

This tool actually looks like it lets you do something similar to guix challenge with local builds. (Last commit is by the author of the OP.) Looks like it may be easier to reproduce a package’s build environment on Arch than I realized. :slight_smile:

@L-as’s work on WASM derivations is neet be cause it points to how we could get correct-by-construction reproducibility in the author’s sense of the word.

1 Like

If I re-watch their talk will that be clear to me? I kinda skimmed it while I was working the other day and while the talk was exciting, that didn’t come through for me. Alternatively, would you mind explaining that a bit?

Thanks for the feedback.

The execution of WASM code is tightly defined by a spec,
and has only one correct interpretation.

This means, modulo bugs, given the same input (derivation), you always get the same output (build result).

This is obviously quite useful, of course it’s also even more work to get it working.

3 Likes

Sounds like I should do more than ‘kinda skim’ it! :flushed: Thanks for your patience here.

1 Like