My nixpkgs trust worries

#1

Hi,

I would like to share my worries when it comes to trusting nixpkgs. From this GitHub statistics page, we can see over 1 month, 3882 PRs have been merged by 555 people, this is very high and good for nixpkgs to stay up to date.

My problem is that there is so much activity, it’s impossible to review everything going in.

And NixOS modules is double edged here, it’s nice because the community provides you all the work you would normally have to do manually or automate with ansible/salt/custom scripts, but at the same time, any contributor can affect your production systems by modifying the modules.

With that activity, it may even be quite easy if you are skilled, to get merging rights for one account, and then create a second fake account to get authorization for it too, then you can easily create changes, approve and merge then in a single-user managed duo of GitHub accounts.

This could be partially mitigated by having a proper tool for comparing what changed EXACTLY between two NixOS derivations for the modules in use, but AFAIK this doesn’t exist yet, and if so, it would still be tedious to review every change made for daily updates.

I wondered if other people shared these worries. :slight_smile:

Obviously, I’m fully aware that open source relies on trust, but it’s easier to trust something you can actually review as a single person, but nixpkgs is too big to be reviewed almost daily.

14 Likes
#2

I definitely share those worries - though TBH I have the same worries about all other Linux distributions, individual projects, and software (open or closed) created by large organizations/companies. It’s just perhaps more visible with nixpkgs.

I would love to see/contribute to improvements in this area - indeed more tools for tracking/comparing what changed would be interesting, and perhaps encoding the implicit “web of trust” somehow (though that’s easy to get wrong as well…)

12 Likes
#3

Some other project only provide security fixes for stable releases, it’s a lot easier to review the updates, and also more reliable as it’s a lot less likely to break the system.

1 Like
#4

nix-diff does this, you can pass it two system drvs and it will go through the whole dependency closure looking for changes.

3 Likes
#5

It doesn’t tell you much except what derivation changed (mostly, it just tells it’s a different version), it won’t tell you if a module option changed.

1 Like
#6

The derivations are what matter in the end though, e.g. if a firewall port is opened that will be reflected in the difference between the firewall start script derivations’ environments.

2 Likes
#7

I think the people who were pushing for flakes and against monorepo for nixpkgs a few years ago agree with you. The idea, if I recall correctly, was that shrinking nixpkgs down to a core (limited) set of packages and modules turns our currently unsustainable disaster of a github project into an easily maintainable project that has a much smaller team of people with commit rights.

From that point we have 2 possible courses of action that I have spent time thinking about:

  • grow the flake registry like crazy and NixOS becomes a terrible experience to use or contribute to, in my opinion
  • take an Ubuntu or Arch like approach and create separate community maintained repositories that users will opt into for the benefit of convenience but the cost of what you have mentioned with security/quality

Personally I think we should go the Ubuntu route and have several repositories of increasing quality and have mentioned this before at least once or twice on these forums. This solution doesn’t rely on flakes but is easier and better with flakes.

2 Likes
#8

Another way to address this need would be to allow users to easily test their configurations. For instance, we have a NixOS test ensuring a specific port is only available via a VPN: even if some modules are updated, we can be sure they are still doing what we expect. That’s not perfect, but it improves the confidence we have on the server.

However, it is currently hard to reuse existing NixOS tests to run them on your configuration. I don’t know how to do it (maybe via Python test fixtures) but I think it would be a nice improvement. We could even imagine to be able to run some NixOS test parts on the server itself, and not only on a VM sharing the same configuration. That’s Christmas day, so let me dream a bit :wink:

3 Likes
#9

If this is true then someone should tell the kernel. They get more activity than we do and they review it well.

That said, they do have one key thing that we don’t: Subsystem maintainers. Now, I don’t know much about kernel dev, so I might be misrepresenting this, but as I understand it, the kernel has a hierarchy of maintainers who take ownership over specific subsystems. We have some similar ideas in the sense that we do have github codeowners and whatnot, but it’s not nearly as structured. It’s often difficult to know who to ask for review from, because not every change affects an obvious codeowner. So I think there are things that get merged without truly proper review. Adopting something more stringently similar to the kernel’s subsystem process might be a good idea.

10 Likes
#10

Definitely, I am even more worried then I do a pip install, since people can actually build packages on their potentially compromised machines and upload them to PyPI. At least it is clear what is built and who builds it in nixpkgs.

I think one possible solution is to require that PRs are signed off by a smaller group of trusted users. This doesn’t scale currently, but ideally there were a few paid maintainers that could focus significant chunks of time reviewing changes.

An additional issue is that a lot of PRs are accepted when the derivation itself looks ok. But ideally there would also be some vetting of upstream changes on version bumps (check the source tarball with a PGP signature, do a cursory look at the actual changes, etc.).

3 Likes
#11

I know the numbers on there but GitHub usually display the unicorn to me:

image
image678×519 66 KB

You mean nix-diff?

that wouldn’t solve the problem on stable updates. There are always thousands of changes and it is not possible to review all of these.

In practice what would that change? If people require software from universe then they activate it and keep it activated. If we want to separate change permissions then we could also realize this with codeowners. But since nix is a language there could always be other changes sneaked in from other files or repositories. And I am afraid that will happen when people do not have permissions and can’t reach anyone in a short amount of time.

That would give a false sense of security. You could detect if only that port is open but you would miss other things like DNAT. It is not possible to detect all kinds of miss configurations. The software world is very complex and you could always sneak something by.

It is even worse: They can just upload whatever to pypi. Thats why I dislike using sdists or other software downloads then the github archive. We don’t need eg. autoconf generated files and all other included things should be reproducible.

4 Likes
#12

yes, I was referring to nix-diff or nvd, they actually only display the version of the packages, but they output nothing about options you are actually using.

Let’s think about a service from NixOS modules, it’s half configured by NixOS and I add a chunk of configuration on my side, when I update the system, nix-diff can’t help me to know if that service will have a different configuration

#13

Yes it does? If the service is configured differently then the derivations will be different. nix-diff shows you any differences, down to every last detail of the drvs. e.g. If two derivations are identical except one has an environment variable set to a different value, it will show how that variable differs.

2 Likes
#14

Oh, I mixed up with nix store diff-closures which was nix diff in my head

1 Like
#15

I think the base repository would be enough for many people. And when it isn’t maybe you end up importing select modules from the other repositories as needed via flakes. Or maybe you don’t care and you just do what every Ubuntu user does and enable all repositories. :sweat_smile:

#16

Actually, the Linux kernel review process was exploited as part of an academic study, see Linux kernel team rejects University of Minnesota researchers’ apology | Ars Technica

This doesn’t mean we couldn’t benefit from looking at their experience, but we should not do so thinking that they’ve completely solved this problem. There are just better approximations, I guess.

#17

Well, IIRC, that actually demonstrated that the system works at least fairly well. Kernel maintainers ended up catching a number of their commits before the ban. The first one was identified as either incompetent or malicious almost immediately. It wasn’t perfect, but it was better than I think most projects could have handled it.

4 Likes
#18

I could just use the “main” repositories that I know are maintained by trusted people in a server for example. Maybe I will need one or two packages that is not in this repository, but them I can use the “universe” (using the same nomenclature as Ubuntu here) repository, review the two packages that I need there and safely use them.

Or even fork the few derivations I need to a third repository that I control. This is something that is very easy to do with Nix, however nowadays the fact that we have a monolithic nixpkgs makes it very difficult because there is no core or “main” packages, and everyone that is a commiter can commit everything basically.

BTW, yeah, we could have this done with codeowners. It is just that I think with codeowners the chance of error is high (codeowners is not as simple as folks thing, it is easy to create a file that looks correctly but it is not). Also like you said, having a monorepo with everything means it is easy to make a “main” package to depend on another package that is not part of “main”. This is why I would like to see the monorepo being split.

2 Likes
#19

I don’t think this going to happen in the Ubuntu style. It would require us to migrate packages to main as soon as they are used by something. Also treewide changes would require either glue code or coordinated merges. Both means overhead and extra work.

Also this would require us to clasify each package. With the current movement to move all-packages to being autogenerated and remove the directory structure, this is unlikely.
And if we go by maintenance status then even core packages like sed or gawk are rather poorly maintained.

2 Likes
#20

I don’t think this would happen a lot because “main” wouldn’t mean “popular packages” but most packages that are core for the system (like kernel+glibc+coreutils+X11 and a few other packages needed for a minimal server/desktop). We could also have automation to make the changes easier.

If this is still too much, we could maybe have a monolith nixpkgs repository with separate all-packages for each “repo” (so all-packages-main.nix, all-packages-universe.nix, etc.) so we could have a callPackageMainOnly for example to enforce that a package from “main” would never depend on something from “universe”.

This is exactly the point. This classification would be useful for an amount of other reasons, even if it is to show how badly maintained a few core packages are.

Hosted by Flying Circus.