Announcing Determinate Nix

Announcements
107

This seems kind of circular reasoning to me. “I’m not happy that DetSys is trying to make flakes standard instead of being experimental because right now flakes are experimental.”

It’s like…okay, I guess? Sorry you don’t like them? There’s not a sound argument here that I can see for not doing flakes (I know some of those exist, but again, I haven’t seen any that seem to be true dealbreakers to me and which would preclude other people from using non-flake workflows).

(FWIW, the biggest coup Lix could’ve pulled off at launch would’ve been making flakes non-experimental in their fork, but alas…)

This is an oft-repeated talking point but I never really see any substantiation of it. Open-core means a particular thing, for example with Redis or Canonical or various other examples. Corporate-controlled means something else still, and with the licensing currently in place there’s nothing stopping a fork if it’s deemed needed.

If “worried about open-core” is being used as shorthand for “I don’t like the people involved and they made my friends angry so I don’t want them to make any money or benefit from the ecosystem” that’s a different thing altogether.

I’ll make the grim observation here: the community-controlled model has been bumpy, slow, and not particularly helpful, and has been set upon by various factions with grievances.

Open-source isn’t an aesthetic, it’s an organizational structure that gets picked to solve some set of needs–and if you fail to meet those needs, other structures exist. Open-source projects are not magically exempt from having to display competence and deliver on value, and similarly they are not magically morally better than a closed-source project that ships on time and solves real problems. Claiming otherwise is just fanboyism.

The other thing to note that gets glossed over: the ecosystem as it exists now requires lots of corporate handouts from companies that do not even pretend to otherwise care about it. The S3 bill, because reasons, is absurdly large.

There is no future in which we can both eschew corporate support and continue to be slow getting things done (say, slapfighting about governance and who is being othered by who instead of solving the technical problems that keep us existing on AWS’ largesse) and survive long-term. Pick 2.

25 Likes
108

It seems like you’re missing a lot of detail. Flakes are an annoyance in the community because:

  1. They were added without going through an RFC process, then hidden behind the experimental flag
  2. While the nix team is making progress now, that is relatively recent, it seems like they’ve sat at “experimental” for a long time without direction
  3. DetSys pushed them because it drivers their product, but don’t seem to be doing much to address technical flaws and finalize them, it looks like DetSys has gotten what they need out of it and are on to other things.
13 Likes
109

Flake adoption is so vast that arguably whatever the current technical issues are, it seems most people arent bothered by them. Calling flakes experimental at this point is misleading.

Generally when you say experimental it means “going to change a lot and rapidly”.

Flakes have been stable for a long time now. It stopped being experimental when people stopped changing it and started using it.

This is just my opinion but RFCs are really only useful in so far as creating standards that many different groups need to agree on. Like TCP, UART, DNS, etc.

A single product with a feature being developed does not need an RFC, in fact thats probably why flakes became so widely adopted. It skipped the beaurocratic BS stage and went directly into problem solving stage.

6 Likes
110

An RFC gives people the structure, place, and time to discuss things and RFCs help build community consensus and buy in.

That’s fine for your opinion, but not what the community at large has agreed on.

That’s pure speculation.

And all the features that have gone thru the RFC process, been agreed upon, and have been successful are what, exactly?

11 Likes
111

I’d just like to quote @hraban from A call for the Nix team to present a unified front on the outcome and strategy around Nix flakes - #87 by hraban on this:

I’m in that dataset yet I dislike flakes. I agree with Infinisil’s objections (and more), I wish we would redesign them completely. Yet I add a flake.nix to all my new projects. Why? Because what alternative is there right now? Flakes have sucked all the oxygen out of the room for any alternatives.

25 Likes
112

This cuts to the heart of it right here.

I think useful follow-on questions from here that might help shed light are:

  • How long do various things take to wend their way through approval in the ecosystem?
  • Is there actually a technical benefit in that additional consensus?
  • Who actually is responsible for saying “okay, this is good, fuck it ship it”?
  • To what degree is the community willing to avoid technical iteration in favor of consensus-building?
  • To what degree is the RFC process a good or a bad fit for innovation work? For maintenance work?
  • What RFCs actually have been “widely successful” versus “thoroughly solving particular issues” versus “creating useful discussion” versus “massive dumpsterfire in the guise of discussion”?
  • To what extent does the “bureaucracy” serve the various stakeholders–e.g., hobbyists, academics, industry adopters, industry users, industry vendors of nix?
  • Who actually makes up “the bureaucracy”? What are there backgrounds/tendencies?
  • Where is “the bureaucracy” most clearly felt? Is it evenly applied? When it isn’t applied, does it help or hurt get things shipped?

I think reflecting on those questions–and mods, maybe this would be a good forking point to a new discussion–might yield some interesting information.

4 Likes
113

It sounds like we’re arguing de jure vs. de facto standardization. De jure is when a standards body presents a standard; de facto is when the community adopts one technology over another.

It seems that flakes are a de facto standard in their current form, albeit with competing packages adding structure on top. It’s not unheard of for de facto choices to become a formal standard, just look at how many of the web standards came about. (Concurrent competing proposals and implementations happen all the time to get real-world experience before formalizing standards.)

Personally, I think a standards process that acknowledges flakes as they stand (at least from the DX perspective) and builds on that would benefit us all.

2 Likes
114

Hi @cafkafk,

I thought it might be useful for me to clarify the timeline of this security incident. Especially since this has come up and evolved into other bizarre claims in this thread and across the internet/mastodon/etc. My hope is that it may demonstrate that it was neither an oopsie or impropriety.

The timeline is as follows:

In summary, our communication to our users was over eight hours after the full details of the vulnerability was made public, and over two hours after the upstream Nix release was completed. As far as I can tell, the contents of our advisories did not include any information that wasn’t otherwise completely public.

Why didn’t the Nix team publish the advisory right away? They said weren’t done with it yet: Nix 2.24.8 released fixing builtin:fetchurl credentials leak, severity 5.9 (moderate) - #6 by fricklerhandwerk. I wish it had been, but I also don’t blame them for it. We would be glad to help support the Nix team on this, if that is wanted.

How were we able to act so quickly? We have internal policies and procedures that describes how we handle security incidents. Our procedure documentation includes specific instructions on monitoring for vulnerabilities, our expedited release process, and how and when we communicate to our users. As you have seen, the communication steps include issuing advisories through our general communication channels.

In other words, this is by no means an oopsie, accident, or leak. It is also not part of an effort to delegitimize the Nix project that we are so attached to. It is careful following of procedures.

It isn’t our responsibility to choose to withhold totally public information from our users, especially regarding security issues. Attackers and malicious actors don’t, and communicating actionable information promptly to our customers is part of our responsibility to those customers.

Hopefully this is less scary than it seemed at first blush.

12 Likes
115

We don’t see anything wrong with using our resources to (a) create a completely free and liberally licensed Nix installer and information source that both make it easier for people to get started with Nix and (b) raise awareness of those projects. It’s perfectly legal, of course, and it’s hard for me to think of an emergent norm in the open source world that ads like that—which we ran only for a short while—would violate. At the time, we at DetSys didn’t have any kind of paid product; all we were doing was trying to grow the Nix pie for all of us. And we think we’ve succeeded in bringing many new folks into the Nix fold with the installer, Zero to Nix, our blog posts, FlakeHub, Magic Nix Cache, and many other things, and we’re quite proud of that. As for the search results, it’s not terribly surprising that a “nix installer” query on Google would bring up results like that, even without sponsoring. The official installation script on nixos.org isn’t referred to as an “installer” anywhere in any official Nix docs (I could be wrong, but I can’t find an instance of this).

6 Likes
116

Was the information really all public? I don’t see mention of the attack being specifically about netrc credentials anywhere prior to the tweet, but I could very well be missing something. To some people familiar with the Nix codebase maybe the fact that that is where the attack would happen is obvious, but after that tweet it’s apparent to everyone.

It seems to me that even having been as explicit as mentioning impure derivations in the original PR could have been avoided and would have been a better security practice, but at least it seems to have been worded to downplay the immediate threat and emphasize the fact that it’s just become easier to do proper TLS so why not. But that does bring up the question of getting releases ready before the PR can be seen publicly.

How were we able to act so quickly?

I don’t think this is what anyone is asking. The question is not how you were able to do it (it seems pretty easy, especially when your cofounder wrote the patch) but whether it was in the best interest of the widest group of people, as opposed to your followers on Twitter. This wording almost feels like an ad sneaked in?

(I do also for my own part feel gratitude for the work that went in to the fix, to be clear. And I don’t want to minimize that. Buy sadly other feelings, such as distrust, too.)

5 Likes
117

Yes. Third sentence of the changelog:

However, this can expose authentication data from netrc and URLs to man-in-the-middle attackers.

…

I think it is useful to communicate that we do have a very specific, documented, and semi-automatic process for identifying and responding to security vulnerabilities. That this is specifically part of what our customers and security auditors expect from us. That it is not motivated by wanting to hurt the upstream project.

And, again, I stand by the fact that public information is public, and that it isn’t our responsibility to withhold public information from our customers and users just because we wish the upstream project had a more rigorous process around security vulnerabilities (something that, again, we’re willing, able, and interested in helping improve.)

3 Likes
118

For educational purposes would you be willing to share the runbook for that here? It might be instructive or informative to the community to see how it’s approached by a company in the space; there may be lessons there for us as we formalize and revamp our own security handling procedures.

4 Likes
119

But GitHub - NixOS/nix: Nix, the purely functional package manager is not a Determinate Systems product, and you’re treating it like it is.

Your users here being determinate systems users, and not Nix/NixOS users. That’s the ones I’m talking about. You’re not on the Nix team Nix Team | Nix & NixOS, yet you’re made aware of privileged information (a leak) and tweeting it because your DetSys users are home safe, while security mitigations have yet reached the wider NixOS community, exposing them.

I hope you can see how that can appear as sabotage, someone that isn’t on the Nix team disclosing vulnerabilities they should never have known about, before the Nix team has had time to even publish their security advisor.

And further, seemed that the NixOS contributors where completely unaware of this vulnerability, until around the time determinate systems had rolled out their installer, so it’s equally problematic that you get early security advisors directly from Eelco because you employ him, while nixpkgs is left in the dark and has to react to the public disclosures.

So reiterating:

  • You’re not on the Nix team, but you get priveleged information from your employees.
  • You use this information to patch security issues, and then post it on twitter before nixpkgs has had time to react.
  • You also post priveleged security information you should have never had ahead of the actual security advisor.
  • You and Eelco are aware of security information and will patch your own products without even making nixpkgs aware.

And it’s not surprising you’d be so willing to act in such a irresponsible and frankly hobbyist way, we’ve seen this before with your employee Eelco Dolstra’s undisclosed conflict of interest while being on the board of the foundation. You’ve always put your own business interests over those of the community, while still being in position of power that you have exploited to reach your own financial goals.

At the end of the day, I am a enterprise user, and I feel more safe going with a Nix variant that isn’t compromised by a bunch of opportunists. We are already, somehow, at a point where Lix seems like the more reasonable choice, specially considering how damaging determinate systems influence is to the wider nix projects security efforts, and how you’ve somehow managed to compromise the security of Nix to elevate your own products.

There is also other enterprise users of Nix than determinate systems, and they all gotta exist under similar terms. No one should have an unfair advantage like this.

As I’ve said, terrible open source citizenship all around.

22 Likes
120

This is an astounding amount of word-twisting for someone who was just offered an opportunity to collaborate on these types of issues with someone else in the space who’s directly working on them.

We are here, first and foremost, as individuals working together. Our priority here is to work on Nix projects for the benefit of all their contributors and users.

Also:

And it’s not surprising you’d be so willing to act in such a irresponsible and frankly hobbyist way,

At the end of the day, I am a enterprise user

… to elevate your own mediocre (and mostly non-existing) products.

As a reminder:

We treat each other with respect and civility. No matter one’s individual identity, circumstances, level of contribution to the project, or status, everyone has the right to respect, and everyone has the duty to treat others with respect.

Sure, you can make the argument that individual interests were being prioritized over project interests here, and maybe there is a good argument for that. However, you were also just offered an olive branch and refused it, instead doubling down on the “it was a leak” narrative. This is not a black and white “detsys is exiled or not” problem, and pretending it is would be just as counterproductive as replacing the word “detsys” in the previous with Lix. If we are to get along in all our communities, we can’t just stick our heads in the sand and decide that other people using Nix are personae non gratae. It’s crazy and only breeds resentment.

13 Likes
122

Thanks - and sorry for not having seen that before. Maybe I’m unduly suspicious of motives and attitudes, and that isn’t being fair. I agree now that the information you shared on Twitter was by that time public, and sharing it wasn’t unreasonable. Clearly it was something else that went wrong in that scenario.

5 Likes
123

Can you help me understand specifically what was privileged and when I learned about it? Because the timeline I outlined above is I think fairly clear that the information we had and publicized was not privileged at the time.

I would expect them to be in the same scenario I was in, where I found out about the vulnerability when the PR documenting it was made public.

In your summary, you have some fairly incredible claims:

Do you have evidence of this?

Should we have waited until nixpkgs had accepted a patch? When the advisory is published? Or can we agree that the information was public when the PR was submitted, and 2x so when the release was finished?

Again, I would appreciate evidence or specifics about what we knew that we should not have, what was privileged, when we learned about it, when it stopped being privileged, etc.

It isn’t our responsibility to be the liaison between the Nix team and the Nixpkgs team. They’re equally capable as we and anyone else is of clicking “Watch” on the Nix repository and get attention-getting emails subjected “builtin:fetchurl: Enable TLS verification” directly in their inbox.

10 Likes
125

Why? If curl releases a security patch, is nobody allowed to talk about it until it’s in Nixpkgs stable and unstable branches? Of course not; Nixpkgs isn’t the arbiter of what is and isn’t available to all people. What makes the Nix package different? Many people get Nix through Nixpkgs but there are other ways as well, and DetSys is the steward of one of those other ways. It seems wrong to chastise them for not delaying notifying their users until Nixpkgs catches up, just like it would be wrong to chastise Debian for notifying their users about a fresh fix available in Debian but not in NixOS yet.

5 Likes
126

Come on, this is a terrible example and not even close to the same.

This is bad because curl is a totally separate piece of software, separate team, separate project. There is no synergy between the nix team and the curl team. Nobody is working on nix and on curl.

For nix we’re talking about overlapping teams on the same piece of software.

To go further, since DetSys has a “playbook” and people paid to work on the project, is this going to happen every time there is a vuln in nix or some analogous application that they share with the community? Based on Graham’s response I’d think “Yes this is exactly how it’ll be” and that doesn’t sit well.

Edit: Also seems to go against DetSys’ values of “providing the best vibes” :face_in_clouds:

image
image769×376 77.2 KB

5 Likes
127

This is exactly what I wondered.

I agree with this. If I were part of the nixpkgs security team, or whomever might be concerned about this, and I actually had a problem with the actions of DetSys, I’d reach out and talk to them (or any other company or person). If there was any problem with their actions, I’d see if I could work out a solution with them.

2 Likes
128

Thank you for sharing a snippet of our values. Some further values are being calm, rational, and expansive when we talk to each other, and people externally, too.

In this case, declining to communicate public information about a patched vulnerability to our customers fails the values of both “support the user” and vibes. I appreciate that as someone who doesn’t use our software, you may not like how these particular vibes sit with you.

An unfortunate truth is that the vulnerability information was public. I could stop there and allow the the conversation to continue, but I’m happy to share a bit more.

–

Earlier, I posed a couple hypotheticals that I’ll now answer.

[Should we have waited until] the advisory [was] published?

No. The information is public, and users deserve to have the best information they can, and the best security posture they can. In a red/blue team setup, defenders need every advantage they can get. Red teams don’t wait for advisories.

Instead of asking us to wait, another approach is exploring with the Nix team how to ensure future disclosures are accompanied by advisories. Like I said above, we’d be glad to participate in supporting that.

Should we have waited until nixpkgs had accepted a patch?

and further, from @L-as:

It should definitely have been until it was in Nixpkgs’s stable and unstable branches.

Definitely not. If we had, it would have extended the window of a public-but-unaddressed vulnerability significantly more. In this scenario, it would have required:

  1. Pushing a PR to nixpkgs master
  2. Waiting for evaluation checks to pass
  3. Merging
  4. Opening up backport PRs
  5. Waiting for evaluation checks to pass
  6. Triggering evaluations on the unstable and release branches
  7. Waiting hours-to-days for the evaluation, build, and release process to complete

…all before being able to share with our users information that was visible, plain as day, on the Nix issue tracker. I don’t find this to be a realistic position to take. And actually, I challenge the premise that Nixpkgs / NixOS has a particularly unique synergy, given that Nix itself is its own project with its own users and goals.

It could be argued that to NixOS’ release process could be improved to support sudden/urgent releases, but that again is not our responsibility as a distributor of Nix. This is also technically very challenging, for various reasons that we explored when I was trying to get NixOS on the embargoed distro list.

Instead of asking us to wait, another option could be publishing the advisory with information about how to incorporate the patched Nix into your NixOS configuration immediately. This would let NixOS users use the patched version, and they can optional remove it after the fact.

–

To be clear, and without the spooky quotes: yes, we do have a playbook (a term of art to describe a documented process) on how to handle scenarios like this.

And yes, as I said above – our customers and security auditors expect us to follow that process. If absolutely nothing changes about how the Nix team handles security incidents: That is exactly how it will be. However, given the recent incidents and retrospectives, I am optimistic the team’s process will improve – for which I am grateful for and glad of. I’d be glad to help.

13 Likes