Nixpkgs supply chain security project

Announcements
5

Thanks for the support! Progress on minimal-bootstrap is being tracked in this PR stdenv: begin work on 256b bootstrap i.e. Trusting Trust by emilytrau · Pull Request #227914 · NixOS/nixpkgs · GitHub

14 Likes
6

Just a huge THANK YOU shout out to the folks who did all the incredible work that made this happen! <3

3 Likes
7

Here’s the tracking issue for NixOS boot security: Tracking issue: Boot security in NixOS · Issue #265640 · NixOS/nixpkgs · GitHub

3 Likes
8

About a month left to land the promised deliverables. Here’s a status report.

tl;dr We’re on track.

Since the initial announcement, the working group has grown as the specification became more refined, with every milestone having at least one individual responsible.

NixOS Boot Security

  • @lheckemann got an ISO to build that can be booted with Secure Boot enabled via shim. It currently works by signing the kernel directly. It doesn’t verify the initramfs or store image, so the next step is to build a Unified Kernel Image to get the initrd verified, and do a checksum of the store image to get all the desired security properties. This is not ideal, especially performance-wise, but will unblock kicking off the shim-review process to obtain a signature that allows booting on machines that only ship with Microsoft keys.
  • @ElvishJerricco explored multiple options to verify the store closure
    • The most secure and attainable solution for now is using nix-store --verify on regular systems, despite a significant performance penalty, and dm-crypt on appliances.
    • At the end of the year there will be a an outline for what an optimal solution would loook like: something very much along the lines of APFS Signed System Volume, with ZFS being closest to being able to implement it, despite being far from ideal in many other respects.
  • @nikstur opened a tracking issue: Perlless Activation - Tracking Issue · Issue #267982 · NixOS/nixpkgs · GitHub
    • Booting NixOS without running the Perl interpreter is almost done
    • At the end of the year we’ll have a clear implementation strategy to also get rid of Bash in a possible next round of work
  • @RaitoBezarius
  • @Dmills27 started working on documenting the complete setup for end users

Vulnerability tracker

  • Due to a constellation of unfortunate situations, we had to quickly re-plan some milestones. Special thanks to Ryan Trinkle and Ali Abrar from Obsidian Systems for jumping in to support us with developer time and project management expertise on short notice, with @cidkid being part of the group now.
  • Progress is tracked here: Issues · Nix-Security-WG/nix-security-tracker · GitHub
  • @thubrecht implemented the web service infrastructure and CVE ingestion, and is working on providing APIs to implement clients against
  • @raboof and @cidkid, after an extensive survey of prior art, are developing a tool to match local store path closures and metadata against advisories, which will use the web APIs to improve accuracy.
  • @mightyiam, @jfly, and @modprog work on extending and integrating label-tracker, a tool to track PRs/issues in nixpkgs according to their tags, generously supported with input from @pennae

Nixpkgs full-source bootstrap chain

Phase 2

Overall, the project is on a path to fulfill the envisioned requirements in terms of security, but due to time limits has to strike many compromises between implementation complexity and performance. One important byproduct are findings that inform possible next implementation steps.

The second application phase is coming up, where we can continue to build upon what’s currently being done and incorporate those findings. We invite everyone available to work on security in the first quarter of 2024 to participate in planning. The deadline to hand in a proposal to the STF is Wednesday 2023-12-13.

19 Likes
9

Hi everyone, we just submitted the phase 1 report to the Sovereign Tech Fund. It’s also published on our projects repository and includes the proposal for phase 2.

tl;dr Phase 1:

NixOS now has all pieces of the puzzle to offer security features superior to most other Linux distributions and at least en par with commercial offers, while preserving the ease and freedom of customisation NixOS has always been known for. Completing this project will shift our concern from purely remediation to prevention of security incidents.

tl;dr Phase 2:

Primarily we would like to finish and round off the projects started in Round 1 to make them more immediately useful, and as an addition close a major gap in the software distribution lifecycle in the Nix ecosystem.

Great many thanks to everyone who’s working on the project to make it a success – still in progress until end of the year – or helped writing the report or the proposal (in ASCIIbetical order):

@DMills27
@ElvishJerricco
@Ericson2314
@JulienMalka
@Mic92
@RaitoBezarius
@alejandrosame
@cidkid
@emilytrau
@jeremiahs
@jfly
@lassulus
@lheckemann
@lilyinstarlight
@mightyiam
@modprog
@nikstur
@raboof
@t184256
@thubrecht

18 Likes
10

You guys are heroes.

11

Hi everyone, unfortunately our phase 2 proposal was declined. But we can still apply for general funding.

Regarding the status of the project:

  • The boot security work is fully implemented, but some PRs still need to be merged.
  • The bootstrapping code is in place, but the 32-bit stdenv is not switched over. This may still require some discussion among maintainers.
  • The vulnerability tracker front end work got stuck end of December 2023 for health reasons. Honestly, we were just overworked. The heavy lifting on the back-end is done due to a heroic effort by @raitobezarius, which means that we now have a very faithful, continuously updated database representation of Nixpkgs. @raitobezarius and me will continue making the front-end usable in the coming weeks and months, as time allows. @mightyiam and @jfly continue refining the label-tracker part of the back-end, which also still needs to be integrated.
6 Likes
12

Thank you for information. Are you able to share more details about the reason why it was decided? We could learn from it for the future projects.

1 Like
13

No reason was given.

14

This is really cool! Just, out of curiosity, is there any plan (maybe for another funding?) to absorb some concepts from Qubes, that provides unseen guarantees in term of security, by providing a way to isolate all processes in separated VM? I would love to be able to specify in my configuration.nix something like:

isolatedVMs = {
  vm_browser = {
    color = "red";
    home_folder = "/home/me/foldervms_browser";
    config = {
      environment.systemPackages = with pkgs; [
        firefox
      ];
    }
  };
  vm_email = {
    color = "green";
    home_folder = "/home/me/foldervms_email";
    config = {
      environment.systemPackages = with pkgs; [
        thunderbird
      ];
    }
  };
}

I guess it is not an easy task, but would be really awesome.

I’m aware of https://spectrum-os.org/ but I would prefer to have it as a light layer on top of NixOs to jail some specific processes rather than using a whole different OS.

2 Likes
15

I don’t think it makes that much sense to pursue this sort of lightweight sandboxing, Spectrum is doing a great job here and I don’t believe it makes a lot of sense to divide efforts.

If anything, if I was given the option, I would prefer to offer a funding to Spectrum.

And plenty OOT projects exist: GitHub - jollheef/appvm: Nix-based app VMs.

3 Likes
17

Another update on project status:

We did not manage to make any progress on the security tracker due to other priorities. Concretely, since the last update I only had a minimal amount of time for work that I used to write the documentation team report that is about to be published, and prepared Summer of Nix to make sure it’s on schedule.

@mightyiam reported a release of pr-tracker, which still needs to be reviewed before it can be integrated. That would also require building views to implement the interaction where pull requests are mapped to security issues.

Last week I met @alejandrosame and @RaitoBezarius to plan the next steps. The goal is to make the security tracker useful for the security team’s day-to-day tasks, taking it on carefully since we’re not in any particular rush at the moment. In the coming weeks, @alejandrosame will develop the required views, for now still without styling. @RaitoBezarius will guide the process, checking in regularly to make code reviews and try out the intermediate results. I will step back from the project to focus on other responsibilities, but will probably help out as a volunteer here and there.

Specifically, the agreed-upon roadmap is:

  • Have GitHub login and the record linkage workflow by 2024-05-31
  • Ramp up CSS development in 2024-06
  • Prepare for hand-off to security team on 2024-08-01
    • Prepare org-level documentation in the Nix ecosystem
    • Take care of follow-up funding
10 Likes
18

Despite not getting the funding, @JulienMalka and I have proceeded with the hash collection subproject in our free time. It is early days but getting into a fairly good shape: the basic features work and it can generate basic reports like https://reproducibility.nixos.social/reports/minimal-iso-runtime . There’s still some notable features missing but I think it’s ready for some wider feedback and contributions - feel free to get in touch!

Thanks a lot for your great work here!

9 Likes
19

Speaking of security, I wanted to raise some attention regarding this CVE I just reported https://nvd.nist.gov/vuln/detail/CVE-2024-36050 following my first report here re-fetch source when url changes · Issue #969 · NixOS/nix · GitHub (cf the follow up discussion for details) and to the security team. As of today it is not clear if Nix already got impacted by it or not, but in any case it is, IMHO, quite important to address it in a timely manner.

1 Like
20

What is remaining for the Vulnerability tracker project? Due to upcoming possible grants, it would be good for this effort to come to a conclusion.

Is it front-end work, do we need additional people? Are there blockers that others can help with/

1 Like
21

The security tracker seems like it could achieve its objectives by just leveraging the GitHub dependency submission endpoint:

1 Like
22

Updates on the Nixpkgs security tracker

We’ve reached the Github login and manual record linkage milestone end of August. As of today, delivery of the project is late by another 4 months, after the last plan for being late by 8 months. It’s a disappointing result – many reasons, no excuses.

I’m back to it now and take budget responsibility and full accountability for getting the system into a state that is usable by the security team by Friday 2024-12-13. Which means that if you have questions, ask me directly. I allocated at least 5 hours per week on active planning and direction.

Ping me if you want to join the private Matrix room for beta testing, where we already collected some initial feedback from people involved in Nixpkgs security.

@erictapen will work full time on designing and building the user interface for the planned workflows. @raitobezarius will improve the performance of the data ingestion and processing pipeline, @proofconstruction will support us with backend-related tasks, both a couple of hours per week, with assistance from @alejandrosame as needed. @Erethon will latch onto the NixOS infrastructure team to ensure we have the service deployed to nixos.org for demo day.

This week we have refined the user interactions to a degree where we can break down the implementation requirements into manageable chunks. The general idea is that a vulnerability record goes through three phases:

  1. Initial triaging

    Security team members get presented with a queue of pre-computed suggestions for matching CVEs to Nixpkgs derivations, which they would filter for things to inspect further. We already have most data to do a decent automatic matching, but it needs to be validated by humans, and we expect most of it to be thrown away. The aspirational goal is to make it convenient enough to ultimately reach “inbox zero”.

  2. Draft

    An automatic match will never be perfect, so a record needs to be adjusted. Many CVEs we get have duplicates, the contained information is often coarse or ambiguous, so the security team will have to check and possibly correct which derivations it applies to before publishing the record.

  3. Mitigation

    Once a draft is ready, it should be published as a GitHub issue where all affected maintainers are pinged. The usual workflow continues from there. Once the issue is closed, the vulnerability record is archived automatically.

You can follow progress in the milestones on GitHub.

We have a bunch of obvious refinements for these workflows in the backlog, such as smoother, structured searching and filtering for CVEs and derivations, and periodic notifications for security team members and maintainers, but those are currently not prioritised. We’ll focus on enabling the essential user stories for now.

On using GitHub dependency submissions

@sambacha Thanks for the pointer. I’ve investigated the possibility of offloading advisory handling to GitHub, and doing that can be at best considered future work.

There exist GitHub - tweag/genealogos: Genealogos, a Nix sbom generator (@ErinvanderVeen) and GitHub - tweag/nixtract: A CLI tool to extract the graph of derivations from a Nix flake. (@Arsleust) to generate dependency information that could be consumed by GitHub, but those can’t be used at Nixpkgs scale. The security tracker currently periodically evaluates all tracked channel branches and puts everything in a database. We could construct SBOMs or any required format from there and submit it within rate limits, but that’s essentially a different project altogether.

And then we’d still have an by far inferior UX to address advisories which, most importantly, doesn’t take into account how Nixpkgs is currently developed. This project is to serve the security team and package maintainers, and GitHub simply doesn’t support our current needs.

Last but not least, there are good reasons to keep our dependency to GitHub limited. Both STF and NGI are interested in us (and everyone else) adopting free and open source software for all workflows. Control over the supply chain is part of supply chain security.

16 Likes
23

We’re making progress. In the past six weeks we were fighting reproducibility issues, sickness, and various time sinks inside and outside of the project. @proofconstruction had to drop out, @erictapen iterated over a couple of designs, @raitobezarius supplied the data models and a staging deployment, @erethon made the first steps towards a deployment to official infra (thanks @Mic92 and the infra team!).

Two weeks ago we sent our beta testers an interactive demo for clicking around. Now the workflow for dealing with automatic suggestions to match CVEs with Nixpkgs derivations is morally complete, and we all have a much clearer vision of the look and feel we’re moving towards.

The main idea for the triaging is that the queue of automatic suggestions shrinks monotonically. Either a match is irrelevant, or it could be considered in more detail. If a suggestion is selected as a candidate, after removing irrelevant (and later maybe adding missing) affected derivations, it’s supposed to be moved to the next step in the process: a draft security issue, where one can adjust more details and eventually notify maintainers.

image
image707×612 40.4 KB

Note that we’re currently focused on defining workflows and visually structuring the available information. Fine-tuning colors, shapes, and proportions will happen later in the process.

image

We didn’t fully complete our objectives for the first milestone due to some unexpected delays and tricky details that needed to be figured out. What was originally intended to be “quick triaging” was cut down in scope and renamed to basic triaging. We still have to make accessible more data to show for at-a-glance decisions, the query performance is not suitable for real-world use yet, and there’s still a bug in the record linkage algorithm that creates countless duplicates.

The next steps will therefore revolve around making the triage workflow more practical – primarily by reducing the amount of noise and uncertainty – with things such as:

  • showing derivation descriptions in the overview
  • allowing removal of unwanted derivations
  • filtering out dependent derivations
  • sorting suggestions by relevance
  • fixing various bugs and problems with the data
  • displaying state changes in an activity log

@alejandrosame will join us to finish some backend work left over from summer. The plan is to reach the second milestone end of next week or beginning of the week after next, with the goal to have a triaging workflow that feels about right, even if it’s still slow. We’ll then continue with implementing “editing a draft security issue”.

Given all the complications so far, it’s likely we won’t get to “publishing issues” by mid December. It seems there’s still a lot to do for just making the thing run continuously. The development setup is finicky. It’s a lot of data, and many moving parts on the way to presenting it. Ensuring minimal stability always took the largest part of the time, already since the beginning more than a year ago. This is why the user-facing features still appear rather basic so far.

But overall this is finally coming together! By skimming through the data we already found some vulnerable packages. And it promises to become quite satisfying to discover even more, so I hope that more people will pick it up once it’s deployed in production. Even with some comfortable automation lacking, I’m convinced the insights the security tracker will provide to the security team and maintainers will make a big difference for making Nixpkgs and NixOS a reality day to day.

4 Likes
24

I wonder if folks working on this project evaluated GitHub - tiiuae/sbomnix: A suite of utilities to help with software supply chain challenges on nix targets and determined whether the work there might be useful in this project?

1 Like
25

Yes, we did. @raboof even contributed code. That type of local scanner solves a different problem: which packages do I have on my system that may be affected by vulnerabilities? The online security tracker is for distribution maintainers and helps answering: Which vulnerable packages do we have in our distribution channels? The first is useless without the second, because you can’t upgrade if there’s nothing to upgrade to. Maintainers need to know which fixes to apply and which mitigations to prioritise so users have a chance to run secure systems.

Local scanners also deal with different internal problems, mainly an abundance of false-positive matches. The system design for the development program here envisioned improving upon that as well, by providing more accurate data based on the upstream triaging.

2 Likes