Trust model for nixpkgs

Development Security
1

I am implementing reproducible builds using nix packages and the nix package manager. I have received pushback from collegues which my management have taken seriously.

The net of the pushback is that, for example, deb packages, though otherwise unverifiable, have the backing of canonical, so are “trusted” because they are from canonical. Where as nix packages come from an “uncontrolled” nixpkgs github repo, and are essentially unvetted.

Presumably, you have heard this kind of thing before. I need to write a paper on the subject, or find one I can point to, that addresses this kind of thing.

From what I can tell, the idea of trust by organisation isn’t completely wrongheaded, but limited. Essentially, canonical (redhat,etc) don’t vet packages, individuals working for those companies do the vetting. If one of those employees wishes to do something not nice, the reputation of the organisation is irrelevant and perhaps unhelpful.

In the nix case, the package and every tool used in its production is auditable and can be verified (via rebuilding and comparing sums). But reproducibility is orthogonal to “trust” as such, so I’m finding myself arguing “nix vs deb” and “nixpkgs vs deb” and "how do we know the tools being used are OK and “where is the root of trust”.

Any attempt thus far to argue the matter technically (.nix->drv>cache->nixstore is verifiable etc) has been met with an assertion that this is irrelevent detail and a demand to know where is the Trust?

I could use some help addressing these concerns. I expect you have seen this a number of times before over the years and have far more experience and information to address them. If not, I would like to offer to help in that regard.

Thanks in advance

14 Likes
2

A basic answer to kick things off.

Not addressing “uncontrolled” and “unvetted” directly, nixpkgs can be pinned so only the changes you’ve audited as valid will be used, vs debian where you’ll be accepting all of the changes the team has made regardless of the content (unless you maintain a mirror).
https://nixos.wiki/wiki/FAQ/Pinning_Nixpkgs

3 Likes
3

Although a large percentage of the packages in the minimal installation image are reproducible, there is probably still a substantial number in nixpkgs that is not reproducible.

Also, fully reproducible builds (as you were already hinting at) can only improve guarantees that a binary package was not tampered with (and was actually built from a derivation unaltered). It does not protect you against someone from slipping in a backdoor, reverting a package to a version with known CVEs, etc. Most PRs are reviewed, but there is no guarantee that nothing slips through the cracks. Also, committers can commit to master directly, without going through a PR and I am not sure how many people are actively checking all commits to master :wink: .

I think what speaks in favor of nixpkgs is that we have a very open and transparent development process, so the risk of getting caught when introducing a vulnerability is probably higher than in many other distributions. Also, frequently-used derivations tend to be touched and thus also checked by a larger group of people than in the ‘single maintainer owns this package’-model that some distributions use.

At any rate, the problem with such situations are that you are putting yourself in a difficult position. No one ever gets fired for choosing IBM Canonical or Red Hat. So, if it turns out that there is a backdoor in an Ubuntu package, people will shrug and say “how could we know? Canonical was seen as a reputable party”. But if you moved to Nix/nixpkgs and there is a backdoor, you will be blamed choosing a less reputable party despite others warning you about it. Nix/nixpkgs will need to gain a certain amount of reputation before this perception changes. This would, in turn, require that enough people take a chance to show that it is safe to use nixpkgs.

Depending on how many transitive dependencies you need from nixpkgs, you could consider taking a small subset of derivations from nixpkgs and audit them. (E.g. be inheriting just the attributes for which you audited the derivations & derivations of transitive dependencies, plus pinning so that you can review changes.)

9 Likes
4

This isn’t exactly true. Canonical only manages software in main and restricted repositories - which are relatively small, packages in universe and multiverse directly come from Debian, and I believe universe repository is enabled by default.

Also, while strictly speaking this is not deb packages, Snap packages are supported by Ubuntu, and can come from literally anyone. While Snap does have sandboxing, it’s mostly worthless.

2 Likes
5

I think thus far we still haven’t addressed OP’s concern regarding the trust model of Nixpkgs. FWIW I also don’t understand this trust model, and what threats it’s designed to address. I’ve heard this concern come up in the past, so I think it may be worth writing something up and putting it somewhere on the website.

Some basic questions I don’t know the answers to:

  • How many people have push access to the master branch of nixpkgs? Who are they?
  • Is the master branch a protected branch, ie. all changes must come via PRs? If not, why not?
  • Are package maintainers only allowed push to the packages they own? Or do they have push access to arbitrary code anywhere in nixpkgs?
  • Who (person/organization) is responsible for maintaining nixpkgs and is responsible for overseeing and certifying its security and integrity?

The question of whether or not builds are reproducible seems orthogonal to most of these questions IMHO, and I don’t think it’s a practical concern for security-oriented folks: they can always rebuild from scratch. But AFAIU we don’t have an official stance or policy on the security of nixpkgs itself.

7 Likes
6

Warning this post may contain extreme pontification.

There a few trust anchors in a .nix ‘package’

  1. You know where the source code is being fetched from https://mycode.com/

  2. You know it the same piece of code, as it’s pinned by a hash. You can be certain that the code that is fetched when the derivation was written, is the code that will be fetched when you come to use it, hopefully this same code, will create (almost) the same result. (with a few caveats).

  3. The build system can’t access the network, so it stops ‘unpredicatable things’ happening during the build. Think of build system the fetches code dependencies automagically, are you really sure what it’s fetching, from who? Is it the same as when the package ‘built the package’? Will introduce extra features if you compile in on a Friday at 3:33pm? (outside of the original source distribution)

  4. You can turn off the nixos.org cache, which will build the it from source. That really is the holy grail of a distribution as far as i’m concerned. If you don’t trust nixos.org to build it, you can build it yourself (and the rest of your organisation).

  5. If someone ‘slips’ something bad into nixpkgs, then that’s possible, but it’s not without having a audit trail of what exactly was changed, and when (git commit history) and by who… I’m hoping/guessing/predicting that packages that effect security (openssh, remote network services, security subsystems like authentication pam modules, etc etc) get a few more eye’s, than other packages. If someone can get edit the commit history of github, we have bigger problems.

  6. It quite possible to clone nixpkgs to your own repo , or a subset of it, and then cherry pick security changes and upgrades., if you willing to put the work in keeping on top of that. You know what code comes into your organisation. This fine grained control can really stop supply chain attacks in their tracks, or identify their source.

when dealing with binary packages, deb, rpm’s, etc etc, you just have to trust the maintainer in the same manner…, the relationship between the upstream source repo, and the binary files that get delivered to the end user, is not a clear in my IMHO, it’s certainly not as open and transparent.

Maybe nix/nixos needs something like
https://packages.qa.debian.org/o/openssh.html

Rather than tracking everything with git ‘issues’?

but maybe not…

Obviously the nixos system has to build and rebuild itself from itself, and that there might be a trust weakness in this ‘bootstrap’ of the system. If you read

makes you question almost ‘everything’… adjust your tin foil hat… ;-).

The Guix team have made progress on reducing and verifying the ‘system that builds the system’…

https://www.joyofsource.com/guix-reduces-bootstrap-seed-by-50.html

maybe nixos needs to do this as well? Would be an interesting project to attempt… it may be possible…what is the smallest bootstrappable binary code, that could build a full nixos system.

If your running someone else’s code/operating system/network then you have to trust them. If you running a package that some else compiled for you, you have trust a lot more things than if you compile it yourself.

Docker and prebuilt containers, just make this ‘whats in my food’ problem worse. I think the more machines, build things automatically for machines, that this improves trust slightly, anywhere you can remove a human from the loop and their environment…, XcodeGhost - Wikipedia. Less things that can go wrong, however things can still go wrong.

I’m not sure how your going, ‘win’ this argument, because the fact they are ‘trusting’ something, is the problem, but hey… you have to trust something… at some stage , unless you have infinite resources ( you build the hardware, software, compiler, code and everything in between yourself).

Maybe they are replacing the word trust, with ‘they have some one to sue/blame’ if things go wrong… this is very different from ‘trust’…

6 Likes
7

How many people have push access to the master branch of nixpkgs? Who are they?

There are currently 139 people with commit access to nixos/nixpkgs. The list isn’t public, but as a committer, I can see it. I’d personally be in favor of making this list public.

A GitHub issue is used to request new commit access. I don’t know if that’s the only way people are granted access (I’d guess probably not).

There are two RFCs about how commit access is revoked due to inactivity:

nixpkgs committers are added by the NixOS GitHub organization owners. I don’t know who all the owners are but I know @edolstra and @domenkozar are owners.

Is the master branch a protected branch, ie. all changes must come via PRs? If not, why not?

The master branch is not protected. We don’t have the CI set up well enough to protect the branch from all use cases currently. It’s been discussed

Are package maintainers only allowed push to the packages they own? Or do they have push access to arbitrary code anywhere in nixpkgs?

Package maintainers cannot push to nixpkgs, only committers.

Who (person/organization) is responsible for maintaining nixpkgs and is responsible for overseeing and certifying its security and integrity?

I think the answer here is basically all of us as a community are making a best effort. There is a NixOS foundation and a security team.

10 Likes
8

With a little elbow grease, you could also create a list of all sources and patches that went into creating your derivation(s); and then “trust” is gained through known good upstream repositories. It would be a fair amount of work, but sounds like it’s critical to your use case.

As someone else mentioned, if you don’t trust the hydra/binary cache, you can always disable substitutes and build everything in-house.

The sandbox nature of nix builds also works nicely with security concerns, as it’s not possible to have network effects, and the builds are done in their own kernel namespace. Only inputs declared in the .drv will be available during build time.

4 Likes
9

big :+1: for the ref to “reflections on trusting trust” that’s a great paper

1 Like
10

Thanks for such a detailed response!

I’d also be in favor of making this list public, for everyone’s transparency. Even better if committers are required to list their full names, email, GPG key, etc.

This is also something I’d like to advocate for. IIRC GitHub offers branch protection that requires a reviewed PR, but not CI to pass. Could be mistaken though.

4 Likes
11

Requiring listing full names is problematic for privacy reasons. I am also not sure what requiring GPG keys is good for :wink: . I’d rather see that people use hardware tokens for two-factor authentication and their SSH keys. Unfortunately, that would be hard to verify.

GitHub does provide this option. However, enforcing this may also come at a cost – more updates may get stuck. For example, I maintain some derivations for proprietary software (e.g. SoftMaker Office, 1Password GUI). I know from various discussions that these derivations have downstream users, but it is often very hard to get reviews for PRs. Since they are typically simple version bumps, I usually merge them myself after a certain amount of time. If the master branch was protected, these updates would probably get stuck for days/weeks. In some cases, this would be bad (e.g. SoftMaker updates existing archives on updates, invalidating hashes).

I have seen other committers who do this with simple, relatively safe, version bumps that do not get any reviews for several days.

I am not opposed to requiring reviews for merging to master, just wanted to point out that there are negative side-effects as well.

2 Likes
12

I’d also be in favor of making this list public, for everyone’s transparency. Even better if committers are required to list their full names, email, GPG key, etc.

If you do not believe the committers based on the effort spent, you should not believe anything I say about my name. Especially given that every Nixpkgs committer who had a chance to indepedently check if it is correct has just happenned to become inactive.

This is also something I’d like to advocate for. IIRC GitHub offers branch protection that requires a reviewed PR, but not CI to pass. Could be mistaken though.

Requiring two committers to review a change halves the throughput, which will increase, not decrease security risks in the short term. Requiring always a committer and a long-term maintainer would be a separate mess to manage, and I guess only 1.5× of our throughput problems. And if you do not trust the committers based on work invested for leaf packages (core stuff gets more attention as more people know how it impacts them), well, you should also not trust me not to have sockpuppets (I actually have some, but I do not use them for PRs or meaningful comments on Nixpkgs issues — or so I say).

7 Likes
13

For clarification are you saying your company would not allow the use of Debian, but they would allow the use of Ubuntu, using Canonical as the justification for allowing the use of Ubuntu?

Our project is similar to Debian, not Ubuntu, in that we are volunteer based and our trust model for developers with commit/publishing rights is based on gained trust through (time) investment into the project (ie. demonstrating you can be trusted through your various contributions like reviews, PRs, conversations with other community members, solving issues, etc…)

If your company finds Debian acceptable then nixpkgs falls under the same category and should be acceptable. If your company doesn’t find Debian to be acceptable then this project likely won’t pass scrutiny. I won’t get into the details of why I personally disagree with stating “Ubuntu which is based on Debian is fine because Canonical backs it, but Debian is not fine because no one company backs it.”

6 Likes
14

Personally, I would not think making the list public would be a good idea.

Once bad actors understand who they can pressure to commit bad things they might very well do so. I would keep the list confidential, but available to auditors.

3 Likes
15

I personally don’t agree with the argument claiming debian or ubuntu or redhat have some special status. Security by authority strikes me as “good enough” for usual levels of security, but insufficient for high value targets, that said, the argument is there and must be addressed. The hundreds of people massaging the output of a distro strike me to be as much of a threat as a benefit.

To summarise the conversation so far, would it be reasonable to say that

  1. NixOS repo, like debian/ubuntu comes from a known, limited set of committers.
  2. Unlike debian packages the transformation path including tools of any given package are clearly verifiable (audit the source, verify the result by rebuilding the entire thing from source).
  3. Trust is actually invested in both cases in the originating organisation, eg llvm.org.
  4. One can always clone, prune and audit the repo, unlike debian packages.
    ?
4 Likes
16

I would strongly endorse the idea of closing master to all but commiters and doing all contributions via PR. I suspect that would reduce some folks of anxiety.

2 Likes
17

Only one git log away :wink: .

3 Likes
18

To summarise the conversation so far, would it be reasonable to say that

  1. NixOS repo, like debian/ubuntu comes from a known, limited set of committers.

I would call none of these sets of committers limited.

NixOS is above hundred.

  1. Unlike debian packages the transformation path including tools of any given package are clearly verifiable (audit the source, verify the result by rebuilding the entire thing from source).

Yes, although you also need to verify bootstrap tools if you want a high level of security, or build your own set and check convergence (building the bootstrap tools is also a Nix expression).

  1. Trust is actually invested in both cases in the originating organisation, eg llvm.org.

Debian sometimes kind of tries to pick only critical fixes into stable, thus rebalancing the trust (which also means that Debian version typically has a unique subset of bugfixes not observed anywhere else)

  1. One can always clone, prune and audit the repo, unlike debian packages.

Yes, and this has been done before, multiple times by small teams (well, for various values of «audit», of course, but clone/prune/set different policy on versions/reshuffle patches)

19

I would strongly endorse the idea of closing master to all but commiters

This is true anyway.

You mean org owners (single-digit number of people)?

Basically, we have a ton of bottlenecks, so there are plausible claims we need significantly more committers and faster processes in parallel to requests to have fewer committers and more cautious processes.

20

kernel.org has a hierarchy of reviewers. I’ve seen that elsewhere.

By the time it gets to the top any change has been pretty carefully vetted, but at the bottom there are plenty of people to start the process.

That said, I’ve seen kernel.orgs review process reduce grown men to tears. Most of that seems to be style.

2 Likes