Nixpkgs supply chain security project

Those are great news! All of those things are sorely needed to make Nix and NixOS a solid choice for businesses and private users alike.

So cool to see that capable contributors will get paid to do good work, I hope that helps with their general workload as well.

The STF announced it would take longer to review the applications, but I didn’t expect them to take two months longer than initially planned.

2 Likes

This is really fantastic; thanks to all of those who are helping to achieve these goals, it’s much appreciated.

1 Like

I am looking forward to the minimal bootstrap efforts to hopefully one day get rid of the bootstrap archives (at least for the common platforms)! It seems there are still some challenges to overcome there. Is there an issue tracking this, maybe?

The vulnerability tracking sounds interesting, I’d like to see how that turns out.

2 Likes

Thanks for the support! Progress on minimal-bootstrap is being tracked in this PR stdenv: begin work on 256b bootstrap i.e. Trusting Trust by emilytrau · Pull Request #227914 · NixOS/nixpkgs · GitHub

14 Likes

Just a huge THANK YOU shout out to the folks who did all the incredible work that made this happen! <3

3 Likes

Here’s the tracking issue for NixOS boot security: Tracking issue: Boot security in NixOS · Issue #265640 · NixOS/nixpkgs · GitHub

3 Likes

About a month left to land the promised deliverables. Here’s a status report.

tl;dr We’re on track.

Since the initial announcement, the working group has grown as the specification became more refined, with every milestone having at least one individual responsible.

NixOS Boot Security

  • @lheckemann got an ISO to build that can be booted with Secure Boot enabled via shim. It currently works by signing the kernel directly. It doesn’t verify the initramfs or store image, so the next step is to build a Unified Kernel Image to get the initrd verified, and do a checksum of the store image to get all the desired security properties. This is not ideal, especially performance-wise, but will unblock kicking off the shim-review process to obtain a signature that allows booting on machines that only ship with Microsoft keys.
  • @ElvishJerricco explored multiple options to verify the store closure
    • The most secure and attainable solution for now is using nix-store --verify on regular systems, despite a significant performance penalty, and dm-crypt on appliances.
    • At the end of the year there will be a an outline for what an optimal solution would loook like: something very much along the lines of APFS Signed System Volume, with ZFS being closest to being able to implement it, despite being far from ideal in many other respects.
  • @nikstur opened a tracking issue: Perlless Activation - Tracking Issue · Issue #267982 · NixOS/nixpkgs · GitHub
    • Booting NixOS without running the Perl interpreter is almost done
    • At the end of the year we’ll have a clear implementation strategy to also get rid of Bash in a possible next round of work
  • @RaitoBezarius
  • @Dmills27 started working on documenting the complete setup for end users

Vulnerability tracker

  • Due to a constellation of unfortunate situations, we had to quickly re-plan some milestones. Special thanks to Ryan Trinkle and Ali Abrar from Obsidian Systems for jumping in to support us with developer time and project management expertise on short notice, with @cidkid being part of the group now.
  • Progress is tracked here: Issues · Nix-Security-WG/nix-security-tracker · GitHub
  • @thubrecht implemented the web service infrastructure and CVE ingestion, and is working on providing APIs to implement clients against
  • @raboof and @cidkid, after an extensive survey of prior art, are developing a tool to match local store path closures and metadata against advisories, which will use the web APIs to improve accuracy.
  • @mightyiam, @jfly, and @modprog work on extending and integrating label-tracker, a tool to track PRs/issues in nixpkgs according to their tags, generously supported with input from @pennae

Nixpkgs full-source bootstrap chain

Phase 2

Overall, the project is on a path to fulfill the envisioned requirements in terms of security, but due to time limits has to strike many compromises between implementation complexity and performance. One important byproduct are findings that inform possible next implementation steps.

The second application phase is coming up, where we can continue to build upon what’s currently being done and incorporate those findings. We invite everyone available to work on security in the first quarter of 2024 to participate in planning. The deadline to hand in a proposal to the STF is Wednesday 2023-12-13.

19 Likes

Hi everyone, we just submitted the phase 1 report to the Sovereign Tech Fund. It’s also published on our projects repository and includes the proposal for phase 2.

tl;dr Phase 1:

NixOS now has all pieces of the puzzle to offer security features superior to most other Linux distributions and at least en par with commercial offers, while preserving the ease and freedom of customisation NixOS has always been known for. Completing this project will shift our concern from purely remediation to prevention of security incidents.

tl;dr Phase 2:

Primarily we would like to finish and round off the projects started in Round 1 to make them more immediately useful, and as an addition close a major gap in the software distribution lifecycle in the Nix ecosystem.

Great many thanks to everyone who’s working on the project to make it a success – still in progress until end of the year – or helped writing the report or the proposal (in ASCIIbetical order):

@DMills27
@ElvishJerricco
@Ericson2314
@JulienMalka
@Mic92
@RaitoBezarius
@alejandrosame
@cidkid
@emilytrau
@jeremiahs
@jfly
@lassulus
@lheckemann
@lilyinstarlight
@mightyiam
@modprog
@nikstur
@raboof
@t184256
@thubrecht

18 Likes

You guys are heroes.

Hi everyone, unfortunately our phase 2 proposal was declined. But we can still apply for general funding.

Regarding the status of the project:

  • The boot security work is fully implemented, but some PRs still need to be merged.
  • The bootstrapping code is in place, but the 32-bit stdenv is not switched over. This may still require some discussion among maintainers.
  • The vulnerability tracker front end work got stuck end of December 2023 for health reasons. Honestly, we were just overworked. The heavy lifting on the back-end is done due to a heroic effort by @raitobezarius, which means that we now have a very faithful, continuously updated database representation of Nixpkgs. @raitobezarius and me will continue making the front-end usable in the coming weeks and months, as time allows. @mightyiam and @jfly continue refining the label-tracker part of the back-end, which also still needs to be integrated.
6 Likes

Thank you for information. Are you able to share more details about the reason why it was decided? We could learn from it for the future projects.

1 Like

No reason was given.

This is really cool! Just, out of curiosity, is there any plan (maybe for another funding?) to absorb some concepts from Qubes, that provides unseen guarantees in term of security, by providing a way to isolate all processes in separated VM? I would love to be able to specify in my configuration.nix something like:

isolatedVMs = {
  vm_browser = {
    color = "red";
    home_folder = "/home/me/foldervms_browser";
    config = {
      environment.systemPackages = with pkgs; [
        firefox
      ];
    }
  };
  vm_email = {
    color = "green";
    home_folder = "/home/me/foldervms_email";
    config = {
      environment.systemPackages = with pkgs; [
        thunderbird
      ];
    }
  };
}

I guess it is not an easy task, but would be really awesome.

I’m aware of https://spectrum-os.org/ but I would prefer to have it as a light layer on top of NixOs to jail some specific processes rather than using a whole different OS.

2 Likes

I don’t think it makes that much sense to pursue this sort of lightweight sandboxing, Spectrum is doing a great job here and I don’t believe it makes a lot of sense to divide efforts.

If anything, if I was given the option, I would prefer to offer a funding to Spectrum.

And plenty OOT projects exist: GitHub - jollheef/appvm: Nix-based app VMs.

3 Likes

Another update on project status:

We did not manage to make any progress on the security tracker due to other priorities. Concretely, since the last update I only had a minimal amount of time for work that I used to write the documentation team report that is about to be published, and prepared Summer of Nix to make sure it’s on schedule.

@mightyiam reported a release of pr-tracker, which still needs to be reviewed before it can be integrated. That would also require building views to implement the interaction where pull requests are mapped to security issues.

Last week I met @alejandrosame and @RaitoBezarius to plan the next steps. The goal is to make the security tracker useful for the security team’s day-to-day tasks, taking it on carefully since we’re not in any particular rush at the moment. In the coming weeks, @alejandrosame will develop the required views, for now still without styling. @RaitoBezarius will guide the process, checking in regularly to make code reviews and try out the intermediate results. I will step back from the project to focus on other responsibilities, but will probably help out as a volunteer here and there.

Specifically, the agreed-upon roadmap is:

  • Have GitHub login and the record linkage workflow by 2024-05-31
  • Ramp up CSS development in 2024-06
  • Prepare for hand-off to security team on 2024-08-01
    • Prepare org-level documentation in the Nix ecosystem
    • Take care of follow-up funding
10 Likes

Despite not getting the funding, @JulienMalka and I have proceeded with the hash collection subproject in our free time. It is early days but getting into a fairly good shape: the basic features work and it can generate basic reports like https://reproducibility.nixos.social/reports/minimal-iso-runtime . There’s still some notable features missing but I think it’s ready for some wider feedback and contributions - feel free to get in touch!

Thanks a lot for your great work here!

9 Likes

Speaking of security, I wanted to raise some attention regarding this CVE I just reported https://nvd.nist.gov/vuln/detail/CVE-2024-36050 following my first report here re-fetch source when url changes · Issue #969 · NixOS/nix · GitHub (cf the follow up discussion for details) and to the security team. As of today it is not clear if Nix already got impacted by it or not, but in any case it is, IMHO, quite important to address it in a timely manner.

1 Like

What is remaining for the Vulnerability tracker project? Due to upcoming possible grants, it would be good for this effort to come to a conclusion.

Is it front-end work, do we need additional people? Are there blockers that others can help with/

1 Like

The security tracker seems like it could achieve its objectives by just leveraging the GitHub dependency submission endpoint:

1 Like

Updates on the Nixpkgs security tracker

We’ve reached the Github login and manual record linkage milestone end of August. As of today, delivery of the project is late by another 4 months, after the last plan for being late by 8 months. It’s a disappointing result – many reasons, no excuses.

I’m back to it now and take budget responsibility and full accountability for getting the system into a state that is usable by the security team by Friday 2024-12-13. Which means that if you have questions, ask me directly. I allocated at least 5 hours per week on active planning and direction.

Ping me if you want to join the private Matrix room for beta testing, where we already collected some initial feedback from people involved in Nixpkgs security.

@erictapen will work full time on designing and building the user interface for the planned workflows. @raitobezarius will improve the performance of the data ingestion and processing pipeline, @proofconstruction will support us with backend-related tasks, both a couple of hours per week, with assistance from @alejandrosame as needed. @Erethon will latch onto the NixOS infrastructure team to ensure we have the service deployed to nixos.org for demo day.

This week we have refined the user interactions to a degree where we can break down the implementation requirements into manageable chunks. The general idea is that a vulnerability record goes through three phases:

  1. Initial triaging

    Security team members get presented with a queue of pre-computed suggestions for matching CVEs to Nixpkgs derivations, which they would filter for things to inspect further. We already have most data to do a decent automatic matching, but it needs to be validated by humans, and we expect most of it to be thrown away. The aspirational goal is to make it convenient enough to ultimately reach “inbox zero”.

  2. Draft

    An automatic match will never be perfect, so a record needs to be adjusted. Many CVEs we get have duplicates, the contained information is often coarse or ambiguous, so the security team will have to check and possibly correct which derivations it applies to before publishing the record.

  3. Mitigation

    Once a draft is ready, it should be published as a GitHub issue where all affected maintainers are pinged. The usual workflow continues from there. Once the issue is closed, the vulnerability record is archived automatically.

You can follow progress in the milestones on GitHub.

We have a bunch of obvious refinements for these workflows in the backlog, such as smoother, structured searching and filtering for CVEs and derivations, and periodic notifications for security team members and maintainers, but those are currently not prioritised. We’ll focus on enabling the essential user stories for now.

On using GitHub dependency submissions

@sambacha Thanks for the pointer. I’ve investigated the possibility of offloading advisory handling to GitHub, and doing that can be at best considered future work.

There exist GitHub - tweag/genealogos: Genealogos, a Nix sbom generator (@ErinvanderVeen) and GitHub - tweag/nixtract: A CLI tool to extract the graph of derivations from a Nix flake. (@Arsleust) to generate dependency information that could be consumed by GitHub, but those can’t be used at Nixpkgs scale. The security tracker currently periodically evaluates all tracked channel branches and puts everything in a database. We could construct SBOMs or any required format from there and submit it within rate limits, but that’s essentially a different project altogether.

And then we’d still have an by far inferior UX to address advisories which, most importantly, doesn’t take into account how Nixpkgs is currently developed. This project is to serve the security team and package maintainers, and GitHub simply doesn’t support our current needs.

Last but not least, there are good reasons to keep our dependency to GitHub limited. Both STF and NGI are interested in us (and everyone else) adopting free and open source software for all workflows. Control over the supply chain is part of supply chain security.

16 Likes