Determinate Nix 3.0

My understanding, based on my obsession of religiously following the dynamic derivation progress, (and that people who actually contribute to the nix package manager can verify or refute) is that the code is not set up in a way where something like this is easy to do(see @roberth 's comments about technical debt in the post above).

As an example, dynamic derivations were attempted to be enabled twice, and failed both times due the difficulty in working with the scheduler, to the point where the team was basically like ok lets just fix the scheduling code before working on the more fun stuff: Reworking the goals · Issue #12628 · NixOS/nix · GitHub. The blocker to what you’re suggesting is making the code have the right abstractions to make such a system possible, and that’s quite difficult to do based on the technical debt of the code base, but is slowly being worked on.

5 Likes

Addressed this here, it seems like it’s an option: Determinate Nix 3.0 - #210 by numinit

(It was also included in @rhendric’s rollup here, probably best to keep this low noise by avoiding repeating the same observation. But, yeah, the thread is long, so I get it.)

5 Likes

For any other tool, this is relatively simply addressed by pinning. I can choose when to adapt and update my project across a breaking-change boundary, and this helps minimise the extent to which I care about that tool maintaining backwards compatibility.

Many of those tools (say, for example rustc) go to great lengths and efforts to maintain that backwards compatibility, because most of their users aren’t using a system like nix that helps pin the build tools (as well as the libraries), for users who need to be sure that their project from last year still builds. That’s always helpful, but it just helps me delay that adaptation work further when there is some update. They could save at least some of that effort, if tools like nix were more widely used and became the expected norm.

It’s terribly ironic that nix itself can’t take advantage of this, because of course you need nix to evaluate the flake pins. It’s even more ironic that the flake file format itself is not full nix, but a restricted data form that does not support full language evaluation. This would, in theory, allow breaking that coupling, by having a much smaller and simpler mininixÂč that can be responsible for resolving which nix is needed to do the evaluation, but we ironically don’t have that and instead the separation is (apparently) an awkward obstacle in the code without providing this advantage. It’s not really clear to me whether this is an unrealised goal of the design, or an awkward consequence of the implementation, or a bit of both — but it’s clear that people see it as a flaw in both directions. For yet more irony, when building NixOS there is a step essentially the same as this, to first build the nix that nixpkgs depends on, and this mechanism has itself made testing changes to nix harder, hindering improvements to the internal architecture and resulting in unintended breaking changes; some benefits and learnings from the lix overlay have recently addressed this point.

Much like a lot of this conversation, all these ironies mean that we’re stuck in a nest of dependencies and there’s a lot of fear about any stabilisation, any commitment that adds backward-compatibility obligations. The phrase “we have made the boogeyman ourselves” applies long before any mistrust and questions about intentions enter the picture, and of course only makes those worse when they do.

What could we do, and more importantly what could we avoid doing, if there was a mininix that only dealt with finding the right nix to evaluate the rest of the flake, so that nix 4 can worry a bit less about backward compatibility overheads? Will it solve enough? Is it not useful unless we can also have an ecosystem of flakes that can be used as dependencies, evaluated with mixed versions, otherwise we have a giant python2/3 type wall of compatibility, or a giant perl5/6 wall of implementation?

The above are not new questions, of course, but we need a roadmap that has fewer boogeymen and fewer walls for them to hide behind and scare us.


1: I would call this minix but that is trademarked, ironically.

7 Likes

This is a sincere question, not a jab: are you familiar with the idea of user capture? If so, can you repeat it for me to confirm? Your point doesn’t relate to mine, and I don’t want to get derailed.

Verse is the highest form of formality, intentionality, and compassion. There is nothing brighter in the sky than a sun so magnificent. There is nothing so functional as an unbroken ray.

And here we reach the downside of not writing in verse. :point_up: :nerd_face: It makes it easy to misinterpret a single line. A relevant benefit of verse is that one cannot comment without addressing both the minutia and the broader point.

To specify: I wasn’t referring to customers doing the audit. I mean that it would be important for customers to have the code audited.

Now to be extra precise: yes, documentation is more important than the code being open sourced. And yes, compliance is itself is the driving force that makes a manager aware of open source vulnerabilities. I’ve been the manager, I understand the priorities.

But I’m talking about the process of auditing itself. If you look around, you’ll see other companies have echoed my statement regarding the importance of open source for applications with high security contexts. And just to re-iterate, this is not saying every user reviews the entire GitHub before downloading a piece of open source software.

And, if you’ve ever submitted a bug report to another project, you’ll know that the process of checking whether your inputs are correct is significantly quicker if you can trace the application’s behaviour and step it through. It’s like half the reason people love Visual Studio so much, because the debugging suite is superb.

So to reiterate: if DetSys is acting in its own interest, it will want its software to be as secure as possible (including but not limited to audit compliance). And part of making sure that work is done is making it easy for security testers to disclose bugs as they’re found. And it’s easier to do that on an open source project where you can copy the repo, run it locally, step it through, say ‘see here’s where the component fails’, and include it in their disclosure.

Graham it is, then.

1 Like

I have seen people argue that flakes are interdependent across the codebase and that the implementation is fragile in places. This is seemingly the opposite of what Mr. Christensen has claimed, so I ask that there be some kind of open and objective test of which claim is right. I think a 3rd party contractor could serve this role.

when such a positive claim is substantiated we already have our answer, no third parties needed.

1 Like

There’s only so much you can achieve with pinning, because a pin can’t bootstrap itself. You’ll always need some sort of unpinned thing to handle the pin, and/or work with the thing that is pinned.
Branching that responsibility onto other pinning systems works only as long as those pinned things don’t cause conflict and can inherently be trusted (and I wouldn’t exclude other aspects). So when actually pinning Nix or parts of Nix, you’ll run into problems, like incompatibilities between evaluators, and having to trust arbitrary Nix implementations to handle the store in a secure and correct way.

I find that broadly the current architecture of what’s pinned and what’s not makes a lot of sense because it is more effective to get these concepts (mostly) right and do the engineering work of handling migrations and compatibility instead of pushing that complexity onto users in the form of aforementioned issues.

Or to put it differently, change is hard, bootstrapping needs to happen somewhere, and pins by themselves can’t solve that for the thing that interprets the pins.

All systems suffer from bootstrapping issues, whether that’s Nix implementations, lib.fix and specialArgs, or Russell’s Paradox and Gödel’s incompleteness, that kind of thing.
Us mortals are doomed to the complexity of our own abstractions - a prison of our own making, but we’re all in it together, and we can make it fun!

7 Likes

It isn’t clear to me what you mean by “a pin can’t bootstrap itself”. Could you expand on that? What do you mean by “bootstrap” in this context?

1 Like

In my post, mininix would be the bootstrapping thing. Just enough to get the right nix - but not necessarily without help.

Technically, it doesn’t have to be a full bootstrap, in the sense of an installer that can get that nix onto a bare system. That would be handy but is a separate use-case. In my head, this mininix would have an existing nix store and daemon, that it can use to select and possibly install a specific nix that this flake needs. It’s more an early resolver than a bootstrap-from-nothing, more like having nix-direnv load a devshell with the right nix in it. It could also be a full nix, just with a preparation and re-exec stage, roughly analogous to what nixos-rebuild does (without --fast). It’s minimal, in the sense that it uses the smallest parts of a compatibility contract we’re willing to set in harder stone, so that the rest doesn’t have to be quite so hard. I was pointing to the minimised data-only flake file format as potenially having had similar intent, if not necessarily result.

But really, the point wasn’t to advocate a solution (and it’s certainly not a new one), but to use it as an example of the entangled dependencies, bootstrapping included. (The irony framing was also, as a pure coincidental aside, motivated by having read something earlier in the day about a new package manager I’d never heard of, for bash, whose installation instructions suggested using another package manager I’d never heard of.)

On two more specific points:

Yeah, I’m simplifying drastically by only really talking about the top-level flake; issues when multiple flake inputs require different versions of the evaluator are most definitely real, and my pointing at python/perl compatibility/implementation barriers was a nod to that.

Though large, it’s still something of a secondary issue from the context of making sure that my project that worked a year ago still works today, because the pins of my old deps do take care of that. I just may not have a viable solution to update the pins, if not all the deps have mutually compatible updates from there. This can happen for non- nix/flake/syntax reasons as well; from a pragmatic sense it doesn’t make a lot of difference if I want to update an old project, and the abandoned crate it depends on hasn’t been updated to match the rest of the ecosystem, or the flake it’s packaged in hasn’t. I still need to find a workaround or replacement.

This is a good example of something that should be changed in the way it is currently handled. Right now, this can be done either by the nix daemon or by the nix invoked for the build (if suitably privileged), and they could be different versions. The protocol needs work, but really this is an architecture problem that needs to be resolved by isolating those responsibilities: a certain nix should own the store for the system and perform / validate all change operations. That’s presumably the daemon in any common case. Again, there are bootstrappng and upgrade and single-user and probably other corner cases, and again this is all well-covered ground.

5 Likes

Basically just a lead-up to “you’ll always need some sort of unpinned thing to handle the pin [in the end]”.
To make this concrete, it might be fun to consider a historical example. Before flakes (or require.nix) were a thing, we would use nixpkgs = ((import <nixpkgs> {}).fetchgit { ... }; pkgs = import nixpkgs {}; to pin Nixpkgs (builtins.fetchurl was sort of frowned upon, and the github tarballs were sketchier iirc).
It would be impossible to use that pkgs to fetch nixpkgs with pkgs.fetchgit, so you have to rely on something else, like <nixpkgs> to get things going in the first place.
You can swap ((import <nixpkgs> {}).fetchgit for something else, but you can’t remove it completely. For instance flake inputs and locks hide their equivalent (fetchTree) a bit, but still require an implementation of Flakes, even if you had a magic attribute to specify the version of Nix in the flake.

I feel like we’ve discussed this plenty, but if you wish to continue the conversation, it might be a good idea to start a new thread so that it’s discoverable and on-topic. (or move these messages - not sure how that works)

6 Likes

I believe the mods can do that :slight_smile:

1 Like