Announcing Determinate Nix

[grahamc]

@paperdigits

To go further, since DetSys has a “playbook” and people paid to work on the project, is this going to happen every time there is a vuln in nix or some analogous application that they share with the community? Based on Graham’s response I’d think “Yes this is exactly how it’ll be” and that doesn’t sit well.

Edit: Also seems to go against DetSys’ values of “providing the best vibes”

Thank you for sharing a snippet of our values. Some further values are being calm, rational, and expansive when we talk to each other, and people externally, too.

In this case, declining to communicate public information about a patched vulnerability to our customers fails the values of both “support the user” and vibes. I appreciate that as someone who doesn’t use our software, you may not like how these particular vibes sit with you.

An unfortunate truth is that the vulnerability information was public. I could stop there and allow the the conversation to continue, but I’m happy to share a bit more.

–

Earlier, I posed a couple hypotheticals that I’ll now answer.

[Should we have waited until] the advisory [was] published?

No. The information is public, and users deserve to have the best information they can, and the best security posture they can. In a red/blue team setup, defenders need every advantage they can get. Red teams don’t wait for advisories.

Instead of asking us to wait, another approach is exploring with the Nix team how to ensure future disclosures are accompanied by advisories. Like I said above, we’d be glad to participate in supporting that.

Should we have waited until nixpkgs had accepted a patch?

and further, from @L-as:

It should definitely have been until it was in Nixpkgs’s stable and unstable branches.

Definitely not. If we had, it would have extended the window of a public-but-unaddressed vulnerability significantly more. In this scenario, it would have required:

1. Pushing a PR to nixpkgs master
2. Waiting for evaluation checks to pass
3. Merging
4. Opening up backport PRs
5. Waiting for evaluation checks to pass
6. Triggering evaluations on the unstable and release branches
7. Waiting hours-to-days for the evaluation, build, and release process to complete

…all before being able to share with our users information that was visible, plain as day, on the Nix issue tracker. I don’t find this to be a realistic position to take. And actually, I challenge the premise that Nixpkgs / NixOS has a particularly unique synergy, given that Nix itself is its own project with its own users and goals.

It could be argued that to NixOS’ release process could be improved to support sudden/urgent releases, but that again is not our responsibility as a distributor of Nix. This is also technically very challenging, for various reasons that we explored when I was trying to get NixOS on the embargoed distro list.

Instead of asking us to wait, another option could be publishing the advisory with information about how to incorporate the patched Nix into your NixOS configuration immediately. This would let NixOS users use the patched version, and they can optional remove it after the fact.

–

To be clear, and without the spooky quotes: yes, we do have a playbook (a term of art to describe a documented process) on how to handle scenarios like this.

And yes, as I said above – our customers and security auditors expect us to follow that process. If absolutely nothing changes about how the Nix team handles security incidents: That is exactly how it will be. However, given the recent incidents and retrospectives, I am optimistic the team’s process will improve – for which I am grateful for and glad of. I’d be glad to help.

[rhendric]

@paperdigits

This is bad because curl is a totally separate piece of software, separate team, separate project. There is no synergy between the nix team and the curl team. Nobody is working on nix and on curl.

There doesn’t need to be a connection between Nix and curl. I was constructing a parallel example in which curl took the place of Nix, not in which curl took the place of something with a non-trivial connection to Nix.

To reiterate: the claim being made is that DetSys (party A) should not announce that a security fix for the Nix implementation (party B) is available through DetSys’s channels until it has also been made available through Nixpkgs (party C). I question that claim because we would not expect Debian (party A′) to refrain from announcing that a security fix for curl (party B′) is available though Debian’s channels until it has also been made available through Nixpkgs (party C′). If this is a bad comparison, it’s not because of the lack of a relationship between B and B′. It must be because of some feature of the A-B-C triple that is not present in the A′-B′-C′ triple. I can’t tell whether it’s because of something about the A-B relationship (DetSys and the Nix team), something about the B-C relationship (Nix and Nixpkgs), or something about the A-C relationship (DetSys and Nixpkgs), or whether it’s some emergent property of all three.

@paperdigits

For nix we’re talking about overlapping teams on the same piece of software.

I have done some investigation and the overlap is pretty small! DetSys lists its people, the Nix team is enumerated here, and Nixpkgs is owned by hundreds (or thousands, depending on how you count) but if you look at the set of people who have committed to the pkgs/tools/package-management/nix tree in the last year with:

git log origin/master --since='1 year ago' --format='%an' -- pkgs/tools/package-management/nix | sort -u

you will see that it doesn’t contain any overlap with DetSys.

So what is the overlap that we’re talking about? Is it that members of the Nix team also contribute to Nixpkgs? If so, how is that relevant? If a curl maintainer applied for and received commit bits on Nixpkgs, would you impose the same embargo requirement for already-published security fixes for curl?

Or is it that Eelco is on the Nix team and a founder of DetSys? If so, how is that relevant? If the Nix team and DetSys were literally the same team—if there was one group of people who write and maintain and provide commercial support for a piece of software, and that software is on Nixpkgs, are they obligated not to announce security fixes until the fixes are available on Nixpkgs? And if not, why is it a problem if they have a partial overlap instead of a total overlap?

I’m not a fan of several other choices DetSys has made, but it seems like some of you are holding DetSys to an unusual and (in light of Graham’s post) unrealistic standard in this scenario, and I can’t figure out what the principle is that would lead a reasonable person to justify that standard only in this specific case! I’d appreciate it if you could put that principle into words.

[paperdigits]

@grahamc

I appreciate that as someone who doesn’t use our software, you may not like how these particular vibes sit with you.

I think you’ve hit the point of contention exactly in that your software has a titanic sized overlap with our software. Titanic.

[L-as]

@grahamc

all before being able to share with our users information that was visible, plain as day, on the Nix issue tracker. I don’t find this to be a realistic position to take.

You’re missing my point. Why was it on the Nix issue tracker “plain as day” before being fixed and in Nixpkgs?

[grahamc]

@L-as

You’re missing my point. Why was it on the Nix issue tracker “plain as day” before being fixed and in Nixpkgs?

Oh, I see. I think in a practical sense, it would be very hard to create a process where Nix can be released from Nixpkgs before (or coincident with) Nix itself has a public release.

[jakehamilton]

Despite its difficulty, these things should be handled. Users of Determinate System’s Nix installer had a notification and update available immediately. That’s good! The problem is that all other Nix users didn’t have this same availability and notification. Because of that a large portion of people were behind after an important security update while people with greater reach (eg. DetSys Twitter account) are shouting about the vulnerability from the rooftops (as is normal for an important security fix).

The coordination here is the problem IMO. DetSys should be able to have a patch released at the same time it is announced, but it is important that we don’t leave all other Nix users behind. It should be possible to release a fix simultaneously for these distributions.

[numinit]

@jakehamilton

The coordination here is the problem IMO.

If only there were a group being created to solve these coordination problems… a sort of driving taskforce…

Edit to add: here was my proposed solution when this happened. Coordinate pushing it out on the GH security advisory as a first step, and do so with as few dependencies on the copy as possible so there’s little friction for people copy/pasting the notice.

[grahamc]

@jakehamilton

The coordination here is the problem IMO. DetSys should be able to have a patch released at the same time it is announced, but it is important that we don’t leave all other Nix users behind. It should be possible to release a fix simultaneously for these distributions.

Sure. Maybe it would be useful to start by describing the current process Nix appears to follow:

1. PRs are opened/publicized, or a commit is pushed to the tip of each maintenance branch. This is the moment a minimum set of details become public (specifically: the vuln has been patched, so a careful review can fully understand the scope and impact.) If it was a PR, the PR’s CI needs to pass and then merge.
2. The code is now on the relevant branches. A Hydra evaluation is kicked off for each branch.
3. The evaluation completes, the builds are completed. This usually takes at least a couple hours.
4. The release tags are created. Ideally, an advisory is simultaneously published.
5. (???) somebody opens a PR to Nixpkgs to update nixpkgs.

The easiest thing is combining 4 and 5 into a single step, executed by the Nix team. That’d be a win.

Skipping a PR could and pushing directly to the branches could save a bit of time.

The real time-cost step is #2/#3. I believe this blocks #4 to ensure it did build correctly and pass all its tests, and is also available in the binary cache.

Maybe skipping #2/#3 would be fine, I’m probably not the person to ask. It would certainly cut down significantly on the “TTL”. Unfortunately, Hydra doesn’t really support private projects or have a security model sufficient for handling TLP Red vulnerabilities, and arguably not even TLP Amber. This would likely imply a secondary set of build infrastructure feeding the binary cache.

[Aleksanaa]

Note that Download | Nix & NixOS this official page doesn’t have “nix installer” keyword on it at all, while “Determinate Nix Installer” by Determinate Systems has “nix installer” in the title.

However, if you switch keyword to “install nix”, it becomes much more reasonable

From the perspective of new users, I think more people will search for “install Nix” rather than “nix installer” because most software is considered to be shipped with an installer and obtained from official channels.

This reason cannot be used as a good evidence.

[pxc]

This does point us towards some positive action that can be taken on the community side, though: a bit of good faith ‘search engine optimization’ around this wouldn’t hurt.

[willbush]

installer WG

I’m thinking about helping. Is there a matrix channel or something to see what the next steps are?

[abathur]

There’s a “Nix Installer” room in the NixOS space.

The next wg meeting is Wednesday (afternoon in Europe, morning US). It’s on the “Official NixOS Calendar” linked from Community | Nix & NixOS.

[drakon]

We’ve now graduated to DetSys openly attacking the Nix project and using it as an advertisement for their fork, conveniently citing issues with the Nix project that they are the cause of (Flakes) or just outright lying about the state of the project (“adopting” their installer).

I now have to agree with Domen, this is a declaration of terror against the community version of Nix.

[crertel]

I’m sorry if I can’t make a more constructive criticism here, but you necro’d a two-month-old thread to complain about a post on a different social media platform that rubbed you the wrong way. You also use hyperbolic language (“a declaration of terror”), and conveniently omit any of the fierce opposition and bad actions from some of the community against Detsys.

Like, man, it’s about to be Christmas–can we just all keep it in our pants until next year?

There’ll be plenty of time to be ugly to each other after the holidays.

[allrealmsoflife]

Besides the vulgar first part, I’m also worried by the sarcastic dismissive: “There’ll be plenty of time to be ugly to each other after the holidays.”

Calling this out is the least I can do. Calling out can be difficult at times, and even then, I was bothered by the vulgar comment when I first read it but didn’t address it at the time.

[djacu]

@cafkafk

I assume part of the reason that you’re overly invested in stopping criticism of Determinate Systems is that you prefer using language like this. If you look at some of the people that like your post, we have former NixOS moderators that have resigned after having meltdowns on users, and podcast hosts that have been banned from various social media for their “anti woke” takes.

And before you try to be some moral authority about what is and isn’t acceptable, consider that your own post violates the CoC.

Examples of unacceptable behavior by participants include:

Trolling, insulting/derogatory comments, and personal or political attacks

Calling out and attaching members of the community for liking a post is unacceptable.

[allrealmsoflife]

Sorry, while pointing out a rule violation is valid, adding something like “before you try to be some moral authority” feels unnecessary and comes across as judgmental. It introduces an emotional element and may escalate to a conflict.

Yes, again, calling out to behavior or irregularities, but attaching a tone of moral judgment to it often upsets people, even if the point itself is correct. Just addressing the issue directly without adding that kind of commentary tends to be more effective.

The same message can be rephrased to something like calling members of the community for liking a post such and such could be seen as violation of such…

[cafkafk]

It’s not a personal attack to call out two people in positions of power. And I don’t care about any of their personal attributes, solely their conduct in the community.

Likewise, you’re only defending these posts because you have a history of working for defense contractors, and you’re as far as I’m aware a fairly well off engineer and a former candidate for the steering committee, so I feel equally fine pointing out how your take here has nothing to do with anything more than undermining me, that’s not a personal attack.

Should you be interested in what a personal attack is, I offer the actual definition:

personal attack (plural personal attacks)

1. An abusive remark about a person, without supporting evidence.

If you find it abusive to call out how people behavior is contradictory, or how your own employment history clearly makes you biased in a discussion like this, I’d like you to strongly consider whether that is actually a real belief or just a convenient truth to tell yourself to win arguments online and avoid criticism.

I should remind readers that we’ve all but confirmed that Determinate Systems has worked together with Anduril, the company that Djacu is working for. And further that djacu regularly hides his employment status, although it is possible to discern from e.g. his steering committee application.

---

And re:

Sorry, while pointing out a rule violation is valid, adding something like “before you try to be some moral authority” feels unnecessary and comes across as judgmental. It introduces an emotional element and may escalate to a conflict.

What you’re doing here is called tone policing, which is not only a method of discourse that has histrorically been used to silence women and minority voices, but is also a type of discourse that in no way constitutes a serious position.

And I should add, as someone that considers myself quite effective at bringing about the change I want, I am not personally looking for feedback in regards to the most effective means, at least not without some actual example that you’re worth listening to on this matter, the timid ad hominem being an obvious example against that

[picnoir]

Mod team hat on.

The original announcement was posted two months ago. A lot of valid criticisms have been voiced, we lengthly debated them in this thread together with another one that has been forked off.

At this point, Determinate Nix is not news anymore. We’re not even discussing social or community norms anymore here but just arguing about interpretation of words.

We don’t think anything good but burning out more people will come from pushing further in that direction: we decided to close this thread and ask everybody to disengage from this for now.

[Edit] peer reviewed, the draft has been posted too quick
[Edit 2]: as a remainder, we now have the SC if you feel like escalating anything further.