Hi all!
I’ve searched on the main site, on this forum, and on GitHub, but I can’t put together a single explanation of how the Hydra build farm works, as it relates to security.
Could someone point me to any documentation?
In particular, I’m hoping to learn how the machines are administered. Are there parallel builds happening in isolated environments, to double- and triple–check that the hashes are the same? Ideally we’d use something like Triple modular redundancy - Wikipedia
Thanks!
Robert
I don’t have a general answer for you, I just want to note that triple modular redundancy is useful when you need a reliable answer and can’t tolerate errors, but for something like a build farm, if you’re trying to ensure that the hashes are stable you just need to build twice and throw an error if you get different results. If the hash is usually A but sometimes B then the desired behavior is to treat this as failure rather than try and figure out if A or B is “correct”.
I’m not asking about stability of hashes, I’m sure the Nix team has already thought through that.
Instead, I’m asking about security, preventing backdoors, that kind of thing.
Right, but my point is triple modular redundancy is useful for coming up with an answer in a scenario when errors can’t be tolerated, but in something like Hydra, the “triple” part is overkill since any disagreement should be treated as an error, not a fault to work around.
I agree with you – we don’t want to mask failures. I put that link in there for reference, not as a recommendation. 
In particular, I’m hoping to learn how the machines are administered. Are there parallel builds happening in isolated environments, to double- and triple–check that the hashes are the same?
I think there are still packages that are not bit-for-bit reproducible…
Here is the list of packages that are not bit-for-bit reproducible in
the ISO image:
So there’s still quite some work, especially when considering the
packages that are not in the ISO image.
Interesting. Not being bit-for-bit reproducible means it’s even more useful to know what the security situation is of the build farm. This is because we now only have one “source of truth”.
Hi all,
Given the lack of activity on the thread, I’m going to assume that the people who would know the answer haven’t seen it yet.
In the meantime, I’m going to assume that the security posture is best-effort, and that the official build cluster is not audited or hardened. As a result, I’ll be building from source. I still like Nix a lot, but the “operational” side of things could use more documentation.
I’m going to leave this here in the hopes that someone with authority can answer.
Thanks!
I would like to bump this very pertinent request. I’m new to the community, so to be useful I will simply ask: how can I help to improve this situation?
Who, specifically, are the maintainers of the build infrastructure, and can we enumerate what special privileges they have? Once that is established, perhaps a more rigorous process can be created around this grey area.
While I won’t try to stop you from building from source, I would like to point out a much bigger risk: that nixpkgs itself, i.e. the recipes for building from source, contains malicious code, and building yourself won’t help with that. There are over 100 people with commit access to nixpkgs who could easily sneak in something nasty.
There are over 100 people with commit access to nixpkgs who could easily sneak in something nasty.
… or just fail to notice — while reviewing a PR from a first-time contributor — that a patch for a real minor bug has an intentional security problem and is fetched as a a commit from a repository that convincingly pretends to be the upstream repository but isn’t.
Or uses GitHub’s feature where it makes commits of fork repos available through the fork’s upstream: https://github.com/nixos/nixpkgs/commit/7b4c9e9d012
I don’t know what that commit is supposed to demonstrate. After updating my local clone of nixpkgs, git show 7b4c9e9d012 tells me the commit isn’t found. And I can’t fetch that commit either, git fetch origin 7b4c9e9d012 tells me that there’s no such reference.
Imagine I update hello (that hello came from github.com/gnu/hello) and add a patch using:
(fetchpatch { url = "https://github.com/gnu/hello/commit/${id}.patch"; /* ... */ })
This looks like a legit patch from the gnu/hello repo! In reality all forks share the same “commit id name space”. The ID could be a malicious commit for hello that adds a rootkit, while still looking legit from the URL alone.
Oh that’s interesting, I didn’t even notice that the URL you provided was a nixos/nixpkgs URL.
If you haven’t already, I recommend emailing GitHub about this (possibly even using whatever their security contact is, as you’ve demonstrated that this can be used to use GitHub to bypass the normal trust mechanisms people use).
And in addition to that (and maybe even more importantly), archive URLs allow this as well, so fetchFromGitHub is susceptible too.
To say it explicitly, hydra.nixos.org doesn’t do redundancy currently. Some of the machines are owned by the project directly, some are hosted (packet.net), some are “owned” by a long-term contributor (e.g. me). After the build the central machine signs the result, so anything beyond there shouldn’t be relevant for this (S3 and CDN cache). I might be missing some details, but I certainly believe that the nixpkgs repo is a much easier target, but I can’t see how to improve that (you need lots of “trusted manpower”).
First I should say that I love that nixpkgs has so many derivations, often when I discover some new program or library I find that it is packaged in nixpkgs already.
But, hypothetically, I think security could be increased by taking providing a smaller set of core derivations (a fraction of the current nixpkgs), so that long-time contributors do not have to spread themselves to thinly. Other package sets could be maintained by topic-specific SIGs (e.g. machine learning, servers), which have to earn their own trust. This might also scale better, since SIGs could accumulate their own trust + contributors.
But as said, I like it that nixpkgs is such a large, coherent set. But in that model I guess trust + contributors have to grow slowly.
So, shouldn’t fetchFromGitHub always do a git clone, rather than only when submodules are used?