Nixpkgs supply chain security project

Announcements
1

Hi everyone,

a couple of months ago, back in June 2023, multiple groups of Nix ecosystem contributors prepared project proposals for the Sovereign Tech Fund “Contribute Back Challenges”.

Today I’m glad to announce that the Nix ecosystem supply chain security project will be supported by the Sovereign Tech Fund with an investment of 226 000 Euro!

The focus of this project is on reducing our reliance on foreign binaries to compile Nixpkgs from scratch, ensuring we are are indeed running the code we compiled by leveraging existing security components in NixOS, and putting in place mechanisms that allow us to deliver the most up-to-date, secure software whenever it is available in a way that can be sustained given our maintainer capacities.

Specifically, by the end of the year this will provide the Nix community with three major improvements to the ecosystem’s security story:

  1. A state-of-the-art vulnerability tracker for Nixpkgs
  2. TPM2-based UEFI Secure Boot for NixOS
  3. A full-source bootstrap chain for the Standard Environment (stdenv) in Nixpkgs

Check the submitted project description for details on the planned deliverables.

All of these three capabilities combined in one software distribution will be quite a unique feature in the open source world.
Nix is not only particularly suitable to support that with declarative, immutable configurations making highly intricate setups tractable.
The Nix ecosystem also needs them very much due to its immense size, growing popularity, and the resulting attack surface.

I’m quite excited about this happening, and proud to be part of it.
This effort will, with time, greatly improve the security situation for most Nix and NixOS users by default.
And it will also very likely further boost adoption of Nix and NixOS in security-sensitive software development communities.
It’s already an amazing showcase of community collaboration, and a perfect opportunity to show yet another time what’s possible with Nix.

@raitobezarius will work on the boot chain security part, together with @nikstur and @lheckemann. @raitobezarius is a Nixpkgs security contributor and reviewer of security fixes, the and was release manager for NixOS 23.05. @nikstur is one of the maintainers of lanzaboote, and @lheckemann is a prolific long-term contributor to Nixpkgs and NixOS, having managed the releases of NixOS 19.03 and 19.09 as well as being a member of the security team and the RFC Steering Committee.
@thubrecht will set up the web service for the vulnerability tracker and collaborate with the Nixpkgs security team to address their requirements.
The bootstrapping work is led by @emilytrau, who kickstarted the minimal bootstrap project and is involved in the community that delivered the Guix full-source bootstrap.
I will support the team with administrative tasks, external communication, and documentation reviews.

Thank you very much to everyone who made this possible!
Special thanks to the NixOS Foundation board (@edolstra, @domenkozar, @thufschmitt, @ron, @zimbatm) for both financial and moral support with the application process.
Extra special thanks to @proofconstruction for sitting through the arduous last checks of the project descriptions.

The Sovereign Tech Fund (STF) invests in many other open source projects you may be interested in. Check out their announcements for this year’s funding round. STF supports the development, improvement, and maintenance of open digital infrastructure in the public interest. Its goal to strengthen the open source ecosystem sustainably, focusing on security, resilience, technological diversity, and the people behind the code. STF is funded by the German Federal Ministry of Economics and Climate Action (BMWK) and hosted at and supported by the German Federal Agency for Disruptive Innovation GmbH (SPRIND).

Stay tuned for updates, and get in touch if you have any questions.

94 Likes
2

Those are great news! All of those things are sorely needed to make Nix and NixOS a solid choice for businesses and private users alike.

So cool to see that capable contributors will get paid to do good work, I hope that helps with their general workload as well.

The STF announced it would take longer to review the applications, but I didn’t expect them to take two months longer than initially planned.

2 Likes
3

This is really fantastic; thanks to all of those who are helping to achieve these goals, it’s much appreciated.

1 Like
4

I am looking forward to the minimal bootstrap efforts to hopefully one day get rid of the bootstrap archives (at least for the common platforms)! It seems there are still some challenges to overcome there. Is there an issue tracking this, maybe?

The vulnerability tracking sounds interesting, I’d like to see how that turns out.

2 Likes
5

Thanks for the support! Progress on minimal-bootstrap is being tracked in this PR stdenv: begin work on 256b bootstrap i.e. Trusting Trust by emilytrau · Pull Request #227914 · NixOS/nixpkgs · GitHub

14 Likes
6

Just a huge THANK YOU shout out to the folks who did all the incredible work that made this happen! <3

4 Likes
7

Here’s the tracking issue for NixOS boot security: Tracking issue: Boot security in NixOS · Issue #265640 · NixOS/nixpkgs · GitHub

3 Likes
8

About a month left to land the promised deliverables. Here’s a status report.

tl;dr We’re on track.

Since the initial announcement, the working group has grown as the specification became more refined, with every milestone having at least one individual responsible.

NixOS Boot Security

  • @lheckemann got an ISO to build that can be booted with Secure Boot enabled via shim. It currently works by signing the kernel directly. It doesn’t verify the initramfs or store image, so the next step is to build a Unified Kernel Image to get the initrd verified, and do a checksum of the store image to get all the desired security properties. This is not ideal, especially performance-wise, but will unblock kicking off the shim-review process to obtain a signature that allows booting on machines that only ship with Microsoft keys.
  • @ElvishJerricco explored multiple options to verify the store closure
    • The most secure and attainable solution for now is using nix-store --verify on regular systems, despite a significant performance penalty, and dm-crypt on appliances.
    • At the end of the year there will be a an outline for what an optimal solution would loook like: something very much along the lines of APFS Signed System Volume, with ZFS being closest to being able to implement it, despite being far from ideal in many other respects.
  • @nikstur opened a tracking issue: Perlless Activation - Tracking Issue · Issue #267982 · NixOS/nixpkgs · GitHub
    • Booting NixOS without running the Perl interpreter is almost done
    • At the end of the year we’ll have a clear implementation strategy to also get rid of Bash in a possible next round of work
  • @RaitoBezarius
  • @Dmills27 started working on documenting the complete setup for end users

Vulnerability tracker

  • Due to a constellation of unfortunate situations, we had to quickly re-plan some milestones. Special thanks to Ryan Trinkle and Ali Abrar from Obsidian Systems for jumping in to support us with developer time and project management expertise on short notice, with @cidkid being part of the group now.
  • Progress is tracked here: Issues · Nix-Security-WG/nix-security-tracker · GitHub
  • @thubrecht implemented the web service infrastructure and CVE ingestion, and is working on providing APIs to implement clients against
  • @raboof and @cidkid, after an extensive survey of prior art, are developing a tool to match local store path closures and metadata against advisories, which will use the web APIs to improve accuracy.
  • @mightyiam, @jfly, and @modprog work on extending and integrating label-tracker, a tool to track PRs/issues in nixpkgs according to their tags, generously supported with input from @pennae

Nixpkgs full-source bootstrap chain

Phase 2

Overall, the project is on a path to fulfill the envisioned requirements in terms of security, but due to time limits has to strike many compromises between implementation complexity and performance. One important byproduct are findings that inform possible next implementation steps.

The second application phase is coming up, where we can continue to build upon what’s currently being done and incorporate those findings. We invite everyone available to work on security in the first quarter of 2024 to participate in planning. The deadline to hand in a proposal to the STF is Wednesday 2023-12-13.

19 Likes
9

Hi everyone, we just submitted the phase 1 report to the Sovereign Tech Fund. It’s also published on our projects repository and includes the proposal for phase 2.

tl;dr Phase 1:

NixOS now has all pieces of the puzzle to offer security features superior to most other Linux distributions and at least en par with commercial offers, while preserving the ease and freedom of customisation NixOS has always been known for. Completing this project will shift our concern from purely remediation to prevention of security incidents.

tl;dr Phase 2:

Primarily we would like to finish and round off the projects started in Round 1 to make them more immediately useful, and as an addition close a major gap in the software distribution lifecycle in the Nix ecosystem.

Great many thanks to everyone who’s working on the project to make it a success – still in progress until end of the year – or helped writing the report or the proposal (in ASCIIbetical order):

@DMills27
@ElvishJerricco
@Ericson2314
@JulienMalka
@Mic92
@RaitoBezarius
@alejandrosame
@cidkid
@emilytrau
@jeremiahs
@jfly
@lassulus
@lheckemann
@lilyinstarlight
@mightyiam
@modprog
@nikstur
@raboof
@t184256
@thubrecht

18 Likes
10

You guys are heroes.

11

Hi everyone, unfortunately our phase 2 proposal was declined. But we can still apply for general funding.

Regarding the status of the project:

  • The boot security work is fully implemented, but some PRs still need to be merged.
  • The bootstrapping code is in place, but the 32-bit stdenv is not switched over. This may still require some discussion among maintainers.
  • The vulnerability tracker front end work got stuck end of December 2023 for health reasons. Honestly, we were just overworked. The heavy lifting on the back-end is done due to a heroic effort by @raitobezarius, which means that we now have a very faithful, continuously updated database representation of Nixpkgs. @raitobezarius and me will continue making the front-end usable in the coming weeks and months, as time allows. @mightyiam and @jfly continue refining the label-tracker part of the back-end, which also still needs to be integrated.
6 Likes
12

Thank you for information. Are you able to share more details about the reason why it was decided? We could learn from it for the future projects.

1 Like
13

No reason was given.

14

This is really cool! Just, out of curiosity, is there any plan (maybe for another funding?) to absorb some concepts from Qubes, that provides unseen guarantees in term of security, by providing a way to isolate all processes in separated VM? I would love to be able to specify in my configuration.nix something like:

isolatedVMs = {
  vm_browser = {
    color = "red";
    home_folder = "/home/me/foldervms_browser";
    config = {
      environment.systemPackages = with pkgs; [
        firefox
      ];
    }
  };
  vm_email = {
    color = "green";
    home_folder = "/home/me/foldervms_email";
    config = {
      environment.systemPackages = with pkgs; [
        thunderbird
      ];
    }
  };
}

I guess it is not an easy task, but would be really awesome.

I’m aware of https://spectrum-os.org/ but I would prefer to have it as a light layer on top of NixOs to jail some specific processes rather than using a whole different OS.

2 Likes
15

I don’t think it makes that much sense to pursue this sort of lightweight sandboxing, Spectrum is doing a great job here and I don’t believe it makes a lot of sense to divide efforts.

If anything, if I was given the option, I would prefer to offer a funding to Spectrum.

And plenty OOT projects exist: GitHub - jollheef/appvm: Nix-based app VMs.

3 Likes
17

Another update on project status:

We did not manage to make any progress on the security tracker due to other priorities. Concretely, since the last update I only had a minimal amount of time for work that I used to write the documentation team report that is about to be published, and prepared Summer of Nix to make sure it’s on schedule.

@mightyiam reported a release of pr-tracker, which still needs to be reviewed before it can be integrated. That would also require building views to implement the interaction where pull requests are mapped to security issues.

Last week I met @alejandrosame and @RaitoBezarius to plan the next steps. The goal is to make the security tracker useful for the security team’s day-to-day tasks, taking it on carefully since we’re not in any particular rush at the moment. In the coming weeks, @alejandrosame will develop the required views, for now still without styling. @RaitoBezarius will guide the process, checking in regularly to make code reviews and try out the intermediate results. I will step back from the project to focus on other responsibilities, but will probably help out as a volunteer here and there.

Specifically, the agreed-upon roadmap is:

  • Have GitHub login and the record linkage workflow by 2024-05-31
  • Ramp up CSS development in 2024-06
  • Prepare for hand-off to security team on 2024-08-01
    • Prepare org-level documentation in the Nix ecosystem
    • Take care of follow-up funding
10 Likes
18

Despite not getting the funding, @JulienMalka and I have proceeded with the hash collection subproject in our free time. It is early days but getting into a fairly good shape: the basic features work and it can generate basic reports like https://reproducibility.nixos.social/reports/minimal-iso-runtime . There’s still some notable features missing but I think it’s ready for some wider feedback and contributions - feel free to get in touch!

Thanks a lot for your great work here!

9 Likes
19

Speaking of security, I wanted to raise some attention regarding this CVE I just reported https://nvd.nist.gov/vuln/detail/CVE-2024-36050 following my first report here re-fetch source when url changes · Issue #969 · NixOS/nix · GitHub (cf the follow up discussion for details) and to the security team. As of today it is not clear if Nix already got impacted by it or not, but in any case it is, IMHO, quite important to address it in a timely manner.

1 Like
20

What is remaining for the Vulnerability tracker project? Due to upcoming possible grants, it would be good for this effort to come to a conclusion.

Is it front-end work, do we need additional people? Are there blockers that others can help with/

1 Like
21

The security tracker seems like it could achieve its objectives by just leveraging the GitHub dependency submission endpoint:

1 Like