Determinate Nix 3.0

Links
238

His only post in that entire thread:

image
image1364×747 117 KB

There is a difference between saying “I will revert this thing” (evidently how you interpreted his remarks) and “If it were up to me, I’d rather revert this thing” (a slightly more charitable reading). It’s totally fine to have that opinion, as long as the fellow doesn’t take up a lot of space with argument–which he didn’t–and goes along with the process–which, based on him merging the RFC, he did.

(I’ll also note that at least 6 folks agreed with him, and the other 4 were just confused, to the degree we can trust emojis.)

If the problem is the overreaction to Eelco disagreeing on that or expressing opinions, that’s not on him. If the problem is him actually blocking progress, that’s a technical issue that can be worked around–but at the very least this is not an example of that.

I think a lot of folks here are in the position of having overreacted to Eelco at one time or another–taking him too seriously, fwiw–and then being unable to back down and admit “shit, we made the boogeyman ourselves”. The best example of this is upthread where the exchange went basically:

  • “Eelco and DetSys are holding features back!”
  • “Okay they literally have a PR open though”
  • “But we can’t use that PR, we’re not the authors!”
  • “Bro it’s literally open source, here’s the new PR with the same changes, we have the features”
  • “Well, um, I wouldn’t feel comfortable being rude and taking his work”
  • “Bro your fellow travelers are literally calling for him and DetSys to be banned, after a year of campaigning and smearing (like, has the save-nix-together thing just slipped everybody’s mind?). Why are you worried about being rude to him?”

I know there’s a lot of bad blood, but there’s a lot of own-goals here too. Hence, let’s focus on the boring technical issues–and whenever we do, it seems that the progress is made.

9 Likes
239

I think we both can certainly agree on “we need no more of that” even if the “outright ban” implementation may be a bit much. It’s easy to outright block and bog down PRs, and the RFC process is a decent example.

3 Likes
240

It’s not free work. They say they support Nix and aim to upstream features. They get good PR from the Nix community.

The Nix community is both their potential ticket to enterprise sales and potential workforce.

Being transparent in how much they support open source and upstreaming features to Nix lets the community know how much credit they deserve.

“Oh a company deigns to contribute code to my project, I can’t set expectations on their contributions” is not a very scalable model for an open source project.

Additionally, there is a lack of trust in DetSys so more should be done to verify they are out for more than their own self interests and not something worse such as EEE on the Nix project.

1 Like
241

No, that’s the post I’ve linked. He had more posts in that thread on the same topic. Scroll up.

And it is not up to him, so why mention that? Everyone else has already agreed that the name is a little meh but it’s not a problem at all.

Scroll up to see the full context. He had already wrote multiple messages about naming before. And even if you specifically don’t see a problem - clearly there is one, because multiple people (including yours truly, who held much more favorable opinion of Eelco back then) felt the need to argue the naming question at length with him.

Uhm, it is on him, because informally, until 2022 - the two-person team of Eelco and Graham was the de-facto leadership in the community. And while Graham practically vanished from the community - Eelco never indicated giving up informal power, and kept popping up in discussion he otherwise holds no weight at all, such as RFC in question.

No, that’s a social issue. It’s quite absurd to assert that someone blocking progress with no good reasoning and people being forced to just ignore them and work around it is somehow a normal state of affairs.

Oh, I fully admit this is part of the problem. Issue is - Eelco hasn’t done anything to avoid that, and actually has done things to worsen the issue.

Been doing that for years. I have no impression that any progress has been made. How come?

Apparently we can. But many people disagree.

Part of a reason for a ban is to make the stance clear to the whole community - that behavior is not okay, even if you personally disagree. But many people are still stuck in consensus-minded era. It is impossible to find any consensus now - the community is fractured to bits. We need to take action, not discuss whether we should do anything at all.

6 Likes
242

No, generally it’s a combination of

  • Nix changes for flakes did not benefit its non-flakes behaviors (e.g. pure-eval), while things like operational stability and resource expenditure are affected
  • A pile of serious issues, like the UX regression that is the <system> attributes, performance regressions (copying to the store), etc

We have decided to address the community feedback and only stabilize things when they are ripe for stabilization. Try in serious tech debt instead of whatever you listed there.
I’ve previously asked you to help, but all we get is snarks like these, and large PRs that do not address tech debt. (I appreciate Eelco’s efforts to break down lazy-trees into reviewable parts, but this is unfortunately just a small part of the work to be done)

We’re not talking about a rename that can be handled in a couple lines of code. We find serious bugs, for which retaining backwards compatibility is not feasible without degrading the internal architecture; see my earlier comments.

41 Likes
243

As someone unfamiliar with the codebase; is there a way to make it a “plugin” or some opt-in portion of the code ?

Having finished PlanetNix I really liked Kelsey Hightower’s viewpoint on this.
Add “pressure valves” on how to accept contributions to a project you deem not “ready” ; and ultimately the way to do that is through a plugin system.

all the experimental features can live in separate codebases (or just modules) in Nix and interface via some plugin API…

Of course this presumes there is some sensible API that all these experimental features can leverage but maybe at least for some of them … this is the way ?
(Plugin == hooks into certain parts of the Nix process like eval)

3 Likes
244

My mistake, I didn’t expand the accordions fully when I Ctrl-F’ed. The additional two posts:

These are calm and reasonable technical disagreements–and again, the fellow merged the PR. It’s fine for adults to disagree but commit. He didn’t call anybody names, he didn’t call for bans, and he stayed focused on the technical/architectural issues.

This is–on the face of it–incredibly disrespectful of the efforts put into the nix project by a lot of folks, and I don’t know if you intended to discount their contributions so quickly. Please consider reviewing the Nix values.

7 Likes
245

My understanding, based on my obsession of religiously following the dynamic derivation progress, (and that people who actually contribute to the nix package manager can verify or refute) is that the code is not set up in a way where something like this is easy to do(see @roberth 's comments about technical debt in the post above).

As an example, dynamic derivations were attempted to be enabled twice, and failed both times due the difficulty in working with the scheduler, to the point where the team was basically like ok lets just fix the scheduling code before working on the more fun stuff: Reworking the goals · Issue #12628 · NixOS/nix · GitHub. The blocker to what you’re suggesting is making the code have the right abstractions to make such a system possible, and that’s quite difficult to do based on the technical debt of the code base, but is slowly being worked on.

6 Likes
247

Addressed this here, it seems like it’s an option: Determinate Nix 3.0 - #210 by numinit

(It was also included in @rhendric’s rollup here, probably best to keep this low noise by avoiding repeating the same observation. But, yeah, the thread is long, so I get it.)

5 Likes
248

For any other tool, this is relatively simply addressed by pinning. I can choose when to adapt and update my project across a breaking-change boundary, and this helps minimise the extent to which I care about that tool maintaining backwards compatibility.

Many of those tools (say, for example rustc) go to great lengths and efforts to maintain that backwards compatibility, because most of their users aren’t using a system like nix that helps pin the build tools (as well as the libraries), for users who need to be sure that their project from last year still builds. That’s always helpful, but it just helps me delay that adaptation work further when there is some update. They could save at least some of that effort, if tools like nix were more widely used and became the expected norm.

It’s terribly ironic that nix itself can’t take advantage of this, because of course you need nix to evaluate the flake pins. It’s even more ironic that the flake file format itself is not full nix, but a restricted data form that does not support full language evaluation. This would, in theory, allow breaking that coupling, by having a much smaller and simpler mininix¹ that can be responsible for resolving which nix is needed to do the evaluation, but we ironically don’t have that and instead the separation is (apparently) an awkward obstacle in the code without providing this advantage. It’s not really clear to me whether this is an unrealised goal of the design, or an awkward consequence of the implementation, or a bit of both — but it’s clear that people see it as a flaw in both directions. For yet more irony, when building NixOS there is a step essentially the same as this, to first build the nix that nixpkgs depends on, and this mechanism has itself made testing changes to nix harder, hindering improvements to the internal architecture and resulting in unintended breaking changes; some benefits and learnings from the lix overlay have recently addressed this point.

Much like a lot of this conversation, all these ironies mean that we’re stuck in a nest of dependencies and there’s a lot of fear about any stabilisation, any commitment that adds backward-compatibility obligations. The phrase “we have made the boogeyman ourselves” applies long before any mistrust and questions about intentions enter the picture, and of course only makes those worse when they do.

What could we do, and more importantly what could we avoid doing, if there was a mininix that only dealt with finding the right nix to evaluate the rest of the flake, so that nix 4 can worry a bit less about backward compatibility overheads? Will it solve enough? Is it not useful unless we can also have an ecosystem of flakes that can be used as dependencies, evaluated with mixed versions, otherwise we have a giant python2/3 type wall of compatibility, or a giant perl5/6 wall of implementation?

The above are not new questions, of course, but we need a roadmap that has fewer boogeymen and fewer walls for them to hide behind and scare us.

1: I would call this minix but that is trademarked, ironically.

7 Likes
249

This is a sincere question, not a jab: are you familiar with the idea of user capture? If so, can you repeat it for me to confirm? Your point doesn’t relate to mine, and I don’t want to get derailed.

Verse is the highest form of formality, intentionality, and compassion. There is nothing brighter in the sky than a sun so magnificent. There is nothing so functional as an unbroken ray.

And here we reach the downside of not writing in verse. :point_up: :nerd_face: It makes it easy to misinterpret a single line. A relevant benefit of verse is that one cannot comment without addressing both the minutia and the broader point.

To specify: I wasn’t referring to customers doing the audit. I mean that it would be important for customers to have the code audited.

Now to be extra precise: yes, documentation is more important than the code being open sourced. And yes, compliance is itself is the driving force that makes a manager aware of open source vulnerabilities. I’ve been the manager, I understand the priorities.

But I’m talking about the process of auditing itself. If you look around, you’ll see other companies have echoed my statement regarding the importance of open source for applications with high security contexts. And just to re-iterate, this is not saying every user reviews the entire GitHub before downloading a piece of open source software.

And, if you’ve ever submitted a bug report to another project, you’ll know that the process of checking whether your inputs are correct is significantly quicker if you can trace the application’s behaviour and step it through. It’s like half the reason people love Visual Studio so much, because the debugging suite is superb.

So to reiterate: if DetSys is acting in its own interest, it will want its software to be as secure as possible (including but not limited to audit compliance). And part of making sure that work is done is making it easy for security testers to disclose bugs as they’re found. And it’s easier to do that on an open source project where you can copy the repo, run it locally, step it through, say ‘see here’s where the component fails’, and include it in their disclosure.

Graham it is, then.

1 Like
250

I have seen people argue that flakes are interdependent across the codebase and that the implementation is fragile in places. This is seemingly the opposite of what Mr. Christensen has claimed, so I ask that there be some kind of open and objective test of which claim is right. I think a 3rd party contractor could serve this role.

when such a positive claim is substantiated we already have our answer, no third parties needed.

1 Like
251

There’s only so much you can achieve with pinning, because a pin can’t bootstrap itself. You’ll always need some sort of unpinned thing to handle the pin, and/or work with the thing that is pinned.
Branching that responsibility onto other pinning systems works only as long as those pinned things don’t cause conflict and can inherently be trusted (and I wouldn’t exclude other aspects). So when actually pinning Nix or parts of Nix, you’ll run into problems, like incompatibilities between evaluators, and having to trust arbitrary Nix implementations to handle the store in a secure and correct way.

I find that broadly the current architecture of what’s pinned and what’s not makes a lot of sense because it is more effective to get these concepts (mostly) right and do the engineering work of handling migrations and compatibility instead of pushing that complexity onto users in the form of aforementioned issues.

Or to put it differently, change is hard, bootstrapping needs to happen somewhere, and pins by themselves can’t solve that for the thing that interprets the pins.

All systems suffer from bootstrapping issues, whether that’s Nix implementations, lib.fix and specialArgs, or Russell’s Paradox and Gödel’s incompleteness, that kind of thing.
Us mortals are doomed to the complexity of our own abstractions - a prison of our own making, but we’re all in it together, and we can make it fun!

10 Likes
252

It isn’t clear to me what you mean by “a pin can’t bootstrap itself”. Could you expand on that? What do you mean by “bootstrap” in this context?

1 Like
253

In my post, mininix would be the bootstrapping thing. Just enough to get the right nix - but not necessarily without help.

Technically, it doesn’t have to be a full bootstrap, in the sense of an installer that can get that nix onto a bare system. That would be handy but is a separate use-case. In my head, this mininix would have an existing nix store and daemon, that it can use to select and possibly install a specific nix that this flake needs. It’s more an early resolver than a bootstrap-from-nothing, more like having nix-direnv load a devshell with the right nix in it. It could also be a full nix, just with a preparation and re-exec stage, roughly analogous to what nixos-rebuild does (without --fast). It’s minimal, in the sense that it uses the smallest parts of a compatibility contract we’re willing to set in harder stone, so that the rest doesn’t have to be quite so hard. I was pointing to the minimised data-only flake file format as potenially having had similar intent, if not necessarily result.

But really, the point wasn’t to advocate a solution (and it’s certainly not a new one), but to use it as an example of the entangled dependencies, bootstrapping included. (The irony framing was also, as a pure coincidental aside, motivated by having read something earlier in the day about a new package manager I’d never heard of, for bash, whose installation instructions suggested using another package manager I’d never heard of.)

On two more specific points:

Yeah, I’m simplifying drastically by only really talking about the top-level flake; issues when multiple flake inputs require different versions of the evaluator are most definitely real, and my pointing at python/perl compatibility/implementation barriers was a nod to that.

Though large, it’s still something of a secondary issue from the context of making sure that my project that worked a year ago still works today, because the pins of my old deps do take care of that. I just may not have a viable solution to update the pins, if not all the deps have mutually compatible updates from there. This can happen for non- nix/flake/syntax reasons as well; from a pragmatic sense it doesn’t make a lot of difference if I want to update an old project, and the abandoned crate it depends on hasn’t been updated to match the rest of the ecosystem, or the flake it’s packaged in hasn’t. I still need to find a workaround or replacement.

This is a good example of something that should be changed in the way it is currently handled. Right now, this can be done either by the nix daemon or by the nix invoked for the build (if suitably privileged), and they could be different versions. The protocol needs work, but really this is an architecture problem that needs to be resolved by isolating those responsibilities: a certain nix should own the store for the system and perform / validate all change operations. That’s presumably the daemon in any common case. Again, there are bootstrappng and upgrade and single-user and probably other corner cases, and again this is all well-covered ground.

5 Likes
254

Basically just a lead-up to “you’ll always need some sort of unpinned thing to handle the pin [in the end]”.
To make this concrete, it might be fun to consider a historical example. Before flakes (or require.nix) were a thing, we would use nixpkgs = ((import <nixpkgs> {}).fetchgit { ... }; pkgs = import nixpkgs {}; to pin Nixpkgs (builtins.fetchurl was sort of frowned upon, and the github tarballs were sketchier iirc).
It would be impossible to use that pkgs to fetch nixpkgs with pkgs.fetchgit, so you have to rely on something else, like <nixpkgs> to get things going in the first place.
You can swap ((import <nixpkgs> {}).fetchgit for something else, but you can’t remove it completely. For instance flake inputs and locks hide their equivalent (fetchTree) a bit, but still require an implementation of Flakes, even if you had a magic attribute to specify the version of Nix in the flake.

I feel like we’ve discussed this plenty, but if you wish to continue the conversation, it might be a good idea to start a new thread so that it’s discoverable and on-topic. (or move these messages - not sure how that works)

10 Likes
255

I believe the mods can do that :slight_smile:

1 Like
257

Does anyone have a game plan I sense that this was another convo which we talked a lot but no plan of action was actually proposed :<

4 Likes
258

Quite a few things were proposed; I tried to summarize them here.

Some are things that only leadership (the SC or the Foundation) could do; some are things you could help out with if you want.

3 Likes
259

For a project that’s “a fork and not a downstream distribution”, “upstream first”, etc. you were quite proud yesterday of announcing/previewing a feature on your Twitter feed based on Nix changes that were developed entirely in the DetSys repo with no sign of being upstreamed at all. Oops?

image
image1007×960 95.1 KB

(A few bugfix commits in there are from github:nixos/nix, eveything related to json-log-path and BuildResult logging isn’t.)

18 Likes