Tweag+Nix dev update #58

[fricklerhandwerk]

Hello esteemed Nixers!

The last developer update from Tweag was two years ago, what a shame. Now we’re back.

First off, big people news:

• @infinisil has taken off his Tweag hat and is moving on to new adventures.

  Infinite thanks from all the team for your incredible work throughout all these years! Nixpkgs – and Tweag – would not be what it is today without your dedication. It was a great joy building with and alongside you. We wish you all the best on your journey, may you bring calm and order wherever you go. And hopefully our build graphs will cross paths again.

• @gilligan and @andir are back at Tweag, after many years.

  @gilligan is now carrying the torch as Head of Tweag, and his programmatic emphasis on our engineering culture makes us Nixers very happy.

  @andir took over many of @infinisil’s responsibilities, now acting as the resident Nixpkgs guru.

Nix stuff. Apart from our usual work to keep increasing developer productivity for clients and community, in the past year we’ve mainly set our mind to improving Nixpkgs supply chain security.

• @balsoft: Working on making Nixpkgs more trustworthy as a software repository

  • Have set up a team that gets pinged on pull requests touching security-critical packages
  • Goal: check packaging standards and look out for potential supply chain attacks
  • Reviewed all of the 231 pull requests at the time of writing, progress tracked on project board (edit: 190 reviewed items are in the archive, which isn’t world-readable for some reason)

• @joaosreis: Getting more tests to run on complex packages that had them turned off for years

  • Examples:
    • gcc: add passthru.tests.withCheck derivation running tests by joaosreis · Pull Request #553621 · NixOS/nixpkgs · GitHub
    • kmod: 31 -> 34.2 by joaosreis · Pull Request #525525 · NixOS/nixpkgs · GitHub
  • Currently working on getting glibc tests to work
  • Half of the packages on the list didn’t have tests running when we started. Now only a few left, work still in progress.

• @andir: Continues @infinisil’s work on the problems infrastructure, originally started in RFC 127

  • problems scans a bunch of fields and also allows maintainers annotating why a given package isn’t built or maybe shouldn’t be used
  • Currently trying to simplify the whole thing (just collect the whole information in one datastructure), and bringing that upstream piece by piece
  • Goal: Obtain a list of everything that’s wrong with packages

• @layus: Working on remote execution between Nix and Bazel

  • At NixCon 2025 presented patches to Nix to improve evaluation times
  • Have been tinkering with a “high availability” version of Nix
    • Motivation: The Nix server is part of CI, a system that keeps the company running, but you can’t really put a load balancer in front of a Nix daemon
    • Still just a sketch, needs more testing to publish if anyone is interested in reusing it: Commits · layus/nix · GitHub

• @fricklerhandwerk: As part of getting some things straight in the Nixpkgs security tracker (recent updates), wrapped the development environment into a virtual machine so the system can be tested locally in its entirety.

  • Getting some of the auto-rebuilding to work was surprisingly challenging!
  • PR still needs cleanup: feat: virtual machine as development environment by fricklerhandwerk · Pull Request #1288 · NixOS/nix-security-tracker · GitHub
  • Main lesson: NixOS VMs are great, but application specifics can be a pain.

Other than that we’re preparing for NixCon 2026 and looking forward to meeting each other and many of you in person!

Keep nixing

[bitbloxhub]

@fricklerhandwerk

Have been tinkering with a “high availability” version of Nix

• Motivation: The Nix server is part of CI, a system that keeps the company running, but you can’t really put a load balancer in front of a Nix daemon
• Still just a sketch, needs more testing to publish if anyone is interested in reusing it: Commits · layus/nix · GitHub

How does this work compare to what’s in, say, rio-build (archived) or Tribuchet, or whatever nixbuild.net does under the hood?

[layus]

AFAICT, rio-build, tribuchet and nixbuild.net all distribute building of derivations to a cluster, while the high-availability part of my branch is to ensure that the main nix server, the part answering the nix protocol, is redundant and can fail gracelully to another instance.
nixbuild.net is of course much more robust than bare nix, but it still needs downtime for maintenance. With a cluster of nix nodes, able to work together, one could upgrade one node without shutting down the whole service. No downtime.
You can tell the difference because my implementation requires the client to try another server in the list when the connection fails.
Ideally, the DNS would provide several IPs for the same name, but I did not go that far yet.

[samiser]

@fricklerhandwerk

wrapped the development environment into a virtual machine so the system can be tested locally in its entirety.

this is a fantastic change btw, running it for testing purposes before was a real pain

[bitbloxhub]

What concrete benefits do you receive by making the Nix daemon (/nix/var/nix/daemon-socket/socket) highly available (as opposed to scheduling derivations on HA/distributed remote builders)? You mention

@layus

With a cluster of nix nodes, able to work together, one could upgrade one node without shutting down the whole service. No downtime.

but AFAIK Tribuchet can survive workers going away and joining with no downtime. Hub changes may be different. I see that Tribuchet requires the hub to be local to the machine initiating the build, @Mic92, are there any plans to make it more HA/distributed (or any other thoughts on this and similar work)?