2024-06-05 Nix team meeting minutes #150

[fricklerhandwerk]

Attendees: @edolstra @roberth @Ericson2314 @fricklerhandwerk @puckipedia @Mic92
Notes: @fricklerhandwerk

STF sponsoring developer time

• @fricklerhandwerk talked with Neighbourhoodie, the contractor offering developer time as part of the Sovereign Tech Fund Bug Resilience Program
  • the opportunity was mentioned on Matrix on 2024-05-16 to get some maintainer quick feedback, since applications are first-come-first-serve
  • the offer also includes funding for a bug bounty program (asked the security team, but @RaitoBezarius opposed the idea) and a security audit (not clear where it could be meaningfully applied)
  • Neighborhoodie could hire a C++ expert to work with us over a couple of months
    • For us it’s a trade-off, since accepting contributions also costs precious maintainer time, but this particular setup seems to be worth it
  • @fricklerhandwerk’s wishlist, based on last year’s proposal developed with @tomberek and @Ericson2314:
    • primary goal: reduce time required for (incremental) builds and testing, i.e. reduce friction for contributors
    • port the build setup to Meson, one derivation per component (can steal from Lix)
    • upstream the Rust installer as discussed months ago (can steal from Lix)
    • port parser optimizations from Lix
    • test and fix NIX_PATH
    • add tests for the daemon
    • refactor tests to be runnable in the devshell and in NixOS VMs (where possible)
    • run store tests in-memory
    • run the same store tests on all store types
    • maybe automate the Nixpkgs-side testing
      • @Ericson2314: most of the work is probably figuring out what to do, and is barely related to C++
        • communication overhead would not be a good use of contributor’s time
    • test evaluator against all stable Nixpkgs releases
    • get rid of global configuration, pass structs to library components
    • declarative signature specification for builtins
    • allow building documentation independent of the binaries
    • automatic formatting and linting everywhere
    • a bunch of refactorings and clean-ups
      • nix-prefetch-url should use the fetcher cache
      • make the eval cache available to stable Nix expressions
      • separate out built-in fetchers to the level of plugins (and ideally document the plugin mechanism)
      • maybe get rid of most of GitHub actions (those really amount to starting a VM and running nix(-)build])
      • generally simplify the release process
• no additions to the wishlist by the team
• @Ericson2314 will take time over the weekend to help condense this into a statement of work to get approved by STF, and make sure the goals are up to date with current reality of the code base

Improving Nix testing for Nixpkgs releases

• proposal by @edolstra

  • Make a Nix hydra jobset
    • Package depending Nix
    • Important NixOS tests

• proposal by @raitobezarius

  • Build over all supported systems
  • Build static binaries and test them
  • Cross compile to classical supported systems
  • Run i686, misc tests & all installer tests using that specific Nix
  • Run library eval tests (normally, ofborg takes care of this for you, but double check it)
  • Verify that nixos-enter behave as expected (a catastrophic regression for the stable Nix in general)
  • Build Hydra with that Nix and verify it works out of the box
    • @Ericson2314: we do that but I wouldn’t call that a release blocker
  • Verify that the new Nix’s fetchers work with things like npins, with some interesting cases, e.g. private forges, allRefs = true;, etc.
  • Evaluate ALL of nixpkgs with nix-eval-jobs which tests two things: nix-eval-jobs compat and Nixpkgs evaluation ~correctness

• @Mic92: could have a special branch for bumping Nix versions that Hydra observes

  • would run the downstream dependency tests
  • when it succeeds, can be merged

• discussed current state of the Nix release manual

  • needs upload rights to releases.nixos.org
    • @fricklerhandwerk: who has this access currently, who can share permissions?
    • @roberth: we wanted to first document and then automate it, so no one needs to
    • @edolstra: can look at the AWS user IDs, but generally this is a question to the infra team
  • needs access to Hydra to add jobsets
    • need Hydra to build it, because that produces the “fallback paths”
    • the release script used to also automatically update those fallback paths in Nixpkgs, currently done manually (would be great to automate again)
  • requires signing a tag
  • @Mic92: would be great if we could have more automation for that, e.g. through GitHub Actions
    • at least updating nixVersions.git in Nixpkgs

[fricklerhandwerk]

Update on the application to the STF bug resilience program:

Neighborhoodie wasn’t able to support us after all. STF suggested to apply for the general investment program instead. @fricklerhandwerk and @RaitoBezarius had a call with Tara from STF to clarify boundary conditions. Meeting notes:

• Selection criteria:
  • Is it important? Is it vulnerable?
    • Do people depend on it? Will people miss it if it’s gone tomorrow?
    • Is it used by a lot of people? Or if it’s a few, are they somehow critical?
    • Power production, hospitals, academia, gymnjjournalists…
  • Main criterion: Does it concern public interest?
• Not funding new features or prototypes
• Goal is to fund maintenance, or making maintenance of other tools easier
  • Improving CI/CD for the project is in scope, if it would make maintenance easier or the supply chain more secure
  • Improving resilience of the project may leave room for community work
    • For example, Reproducible Builds has some community work
• The first proposal is considered as a whole, and details can be iterated on in the second proposal phase
• It’s not a grant, but more like a government procurement process
  • Propose what you need to do, and we will look at the costs and if the plans are reasonable
  • Too much work for too little time is not realistic
  • It’s not comparable with the “contribute back challenges”, those were supposed to do heavy lifting
• Working with the same budget as last year, but with a lot more applications
  • 1M EUR budget, as for GNOME, is an outlier
    • You have to consider that those bigger investments supports several software projects simultaneously
  • Median so far was 200-300k EUR over 1-1.5 years
  • But it’s contextual. If you make a strong argument and strongly believe in what you’re proposing, try it and it can be discussed
• Things that are upstreamable or work across the ecosystem are particularly strong points
  • The overarching goal is to make open source infrastructure more resilient as a whole
• The timeline can be up to 2 years, but a second year would be tentative
  • One can also have shorter commitments and then apply for something else afterwards
  • Depends on the project’s needs and the concrete proposal