My wishlist for NixOS security in 2024+

I wrote some words about NixOS security. Hopefully some of you like them and find them interesting. Any other ideas / comments on the listed ideas are welcome!

https://delroth.net/posts/nixos-security-wishlist/

21 Likes

“always backport everything” (which we’d use for things like web browsers, chat apps, etc. that have a large attack surface exposed to the internet)

A new meta attribute which would be used by ofborg/action to add the label?

2 Likes

That’s kind of what I had in mind when writing this, but I haven’t thought deeply about the implementation tradeoffs.

1 Like

Two boring reminders that have been repeated to death and back but still not enough.

  1. To be fair to our contributors, even upstreams themselves don’t always have any idea if a bugfix is a security-relevant one. To be fair, anything to auto-extract hints at security fixes would be quite useful, I agree with that part. Also, if it catches upstreams «upgrade immediately» warnings, it will have «security false positives» like data corruption bugs where we want a backport anyway…

  2. Two-person rule does nothing to impede an attacker who has spent at least five minutes on planning. Two-person review with some track record required is a pain to enforce and takes all of one hour work spilled over a month to foil.

3 Likes

Great writeup!

It’s not always obvious that a version bump is security relevant. PR authors don’t necessarily know to mention it in the PR description (nixpkgs-update definitely doesn’t for obvious reasons), reviewers don’t always go and read the changelog, etc.

Might be saying something obvious, but grepping nixpkgs-update for CVE turns this and this up, so it seems some checks for CVE are performed? Moreover, as you probably know repology lists CVEs at least for some projects (although not all, e.g. not chromium, not sure why), so if nixpkgs-update doesn’t already check there, it should be a pretty low hanging fruit indeed to ensure any updates that fix a CVE get a backport- label.

1 Like

TIL, I don’t think I’ve ever actively noticed this in the wild on PRs I’ve reviewed. Does it even still work? Now that I checked, I don’t think the CVE report has happened on any PR since Nov 2020.

1 Like

The CVE feature of nixpkgs-update broke at some point and was turned off. We can probably revive it though!

7 Likes

I knew about it but didn’t look at it recently, and actually it’s a great opportunity to prove one of the points in my post that was unsourced and feelings-based. According to that repology DB:

  • Packages with known CVEs on nixpkgs unstable: 232
  • Packages with known CVEs on nixpkgs 23.05: 312

That’s despite nixpkgs unstable being a +6% larger package set.

(Note: some CVEs are unactionable, so a high base count isn’t necessarily bad. The delta is however notable.)

3 Likes

Those numbers are probably to low and the matching is likely not that great :sweat_smile:

1 Like

To properly track CVEs we would need to stop that hydra stops building things with known vulnerabilities otherwise it is always the decision to properly label data or fuck everyone over with cache misses.

I have been saying this for a long time but to many people are against it for reasons I don’t understand.

7 Likes

so your saying if you want to install software with know vulnerabilities…

you’ll have to compile it yourself.

Seems fair.

I’d be interested to the hear the argument against doing that? Maybe some deep technical details i’m not thinking about.

@delroth , i’d also like to say that is an excellent write up…/

I’d like to wish for a Nixos Bug bounties program…

If you can break nixos apart and tell us, we’ll give you money.

They can be highly useful and also great for marketing, everyone wants to break everything!! LOL!

however, as a bonus we can give REAL bounties (bars)…

image

If we do that then you probably can’t build a NixOS system because the CVE database is full of bogus entries that aren’t real vulnerabilities according to upstream and so don’t get releases (or sometimes, not even official patches). Random example I’ve just looked at: NVD - CVE-2023-45322 should we backport a series of 32 patches because attackers can trigger UAFs in libxml2 when it runs out of memory? If not, should it be marked as knownVulnerabilities = ["CVE-2023-45322"]; and then you can’t build anything without allowing insecure packages, and ~nothing in nixpkgs gets cached anymore?

knownVulnerabilities is a huge hammer currently which doesn’t really fit its name. It’s not knownVulnerabilities, it’s knownRiskyBecauseOf basically.

Meh. Bug bounties programs generally lead to very poor quality reports, and you still need to have people work on fixing it on top of wasting a ton of time on triaging. It’s also very unclear to me how you’d scope such a program, since NixOS is by definition extremely configurable, and we know that some options lead to insecure configurations - in some cases that’s desirable.

9 Likes

Well, from time to time there are 0-day vulnerabilities in Linux kernel. True ones, not random CVE noise. Guess what, nobody marks stuff vulnerable until there is a fix available, for practical considerations.

Then there are vulnerabilities that are real for some use cases, known to take a long time to fix, and instead of recording the best information we have there will be a negotiation about whether the use case is central enough to break the package for everyone.

We have a ton of great ideas about security that are about «security» and not about protecting against specific threats in a specific model, not building things with knownVulnerabilities recorded is one of them that got deployed, in practice it works as well as nearly all «abstract security».

6 Likes

To me a vulnerable configuration is the same as a vulnerable piece of code…

one you can blame the developer, the other you can blame the user. lol!

Interesting take on bug bounties, can you give more details of the reports you got,

I’m talking about big ones, sandbox escapes, nix daemon… these are the big ones… , I’d be interested to find out why insecure configuration desirable ? testing?

So, this is clearly a data quality problem from CVE’s and a labelling problem within nix? , one is easy to fix the other is not so easy to fix. If i understand this right.

just a thought, perhaps i could have a option that prevents me from installing unsafe ‘languages’ on to my machine, like banning all c++ code for a start… that should help a lot with CVE’s in the future.

interesting insights.

Sorry if I repeat other posters but I wanted to summarize it a bit:

Let’s take a browser like chromium or firefox as an example which can barely be compiled on my 4 core 16 GB RAM machine and I think it would take multiple (3+) hours. If we mark any dependency as vulnerable then people can effectively not use the browser anymore and any dependency of a browser is basically excempted from knownVulnerabilities.

7 Likes

It’s actually pretty hard to define sandbox escapes in Nix builds. Pretty sure that anyone’s first attempt at a definition will either be missing some pretty scary hypothetical case, or will be including a limitation already discussed in a forgotten thread at this very forum.

And, of course, Linux kernel zero-days in namespaces (which happen once a year or once a couple of years, if I am not mistaken) would count as the highest level of escape, which is not really productive for improving Nix.

  1. There are very useful toolchains that include running Ghostscript on freshly-generated known-good input. Like generating diagrams for an article from Asymptote code. At the same time, don’t let lack of CVEs fool you: Ghostscript always fixes specific reported bugs in its «safe mode», but the overall track record is concerning and you probably should assume that a determined and experienced attacker can generate a crafted arbitrary-code-execution input for Ghostscipt within a week. There is no good feature-complete replacement for some Postscript rendering tasks, though… (and «reliable» fixes would include a horrible amount of work rewriting a lot of code)

Is a system running Ghostscript on input affected by remote users vulnerable? Depends on the mitigations and your security model. You might consider it worth the risk (or the risk might not even be there). But you might end up running a known-vulnerable system according to some definition not matching your current threat model.

  1. Is a world-appendable world-readable FTP server a security vulnerability? I have seen situations where clearly yes, and I have seen situations where it was clearly no (it’s nice to have physical control over the building where a completely isolated password-protected WiFi network lives)
2 Likes

I see… thanks for the explanation.

how is spectrum OS coming along, at least could have qubes type isolation, using the nix way??

I could run all this dodgyware in isolation…

Right , can someone at the foundation send @Sandro a Ryzen thread ripper 3960X with 128GB of ram if desktops are your thing. :wink: , but point taken

See you again , it was fun last time!!

I think one major aspect about security that is easily forgotten is plain old updates. As @7c6f434c said, you usually only know about actual CVEs after they’ve been fixed.

I think, as a matter of policy, knownVulnerabilities should be reserved for “big” vulns that are (somewhat) publicly known such as log4j, heartbleed and such or known vulns that aren’t or won’t get fixed rather than the dozens of CVEs that quietly get patched every week. The latter category should just be fixed through an update because it’s an equal amount of effort.

So, a large part of vuln tracking IMO is also update tracking; finding out of date packages and (ideally) doing something about them.

4 Likes

its an endless thankless task. So i’m going to take the time to thank the nixos security team, who do the drudge janitor work of security patching.

in 2038, a lot a software is written in memory safe language and languages the enforce correctness and tests. The nixos end to end full integration testing really helped that effort.

These languages have higher performance than C/C++, so in 2037 the nixos security team got to do more interesting due to ‘lack of work’

You’ll be please to know, nix can build them all!

1 Like

This sounds like a very bad idea as it makes it impossible to answer the simple question: Is my system affected by known vulnerabilities.

If you wanna NixOS in a professional setting this type of question becomes important. The Ubuntu Security Team does a pretty good job with their notices. I’m not sure how much human effort goes into this though…

1 Like