2022-10-19 Nixpkgs Architecture Team Meeting #14

The NAT is there to solve architectural nixpkgs issues that are too big in scope for any single person to undertake. There are regular 30 minute meetings every week (currently 13:30 to 14:00 UTC) in Jitsi Meet, open for anybody to join, recorded and live-streamed to YouTube, meeting notes are public.

The agenda for the NAT is generally created from discussions in the past week, mostly on Matrix, but you’re also free and encouraged to suggest additional topics by replying to this thread.

Tentative Agenda

Spack: External dependencies and specifications

@tgamblin, the lead of the Spack project talked a bit about how Spack works, especially regarding external packages and its specifications to resolve package versions and dependencies. Spack notably uses answer set programming instead of a SAT solver for this. A new paper by @tgamblin and others goes into this. Spack uses clingo to solve this logic program.

Others involved in the discussion were @toonn, @Growpotkin (who wrote an STM solver in Nix!), @danielbarter, @ronnypfannschmidt:matrix.org
(don’t know if they’re on Discourse)

Deep runtime dependency substitution

Nix currently doesn’t have a good way of substituting deep runtime dependencies, which Spack does and would be useful in a number of use cases. A NAT wiki article on this has been created, and a number of solutions have been proposed after the previous meeting by @Infinisil, @ronnypfannschmidt:matrix.org, @Growpotkin and @Ericson2314. I encourage you to update the wiki article to briefly explain your ideas there.

The Nixpkgs Architecture Team is on nixos.org now

In a PR to the nixos.org homepage, a section about the NAT was added to the website: Nixpkgs Architecture Team | Nix & NixOS. Now duplicated on the website, the information can be removed from the original team page in another PR.

Problems with RFC 127

RFC 127 is in the FCP now, though @Infinisil is not happy with it as it seems to make deprecations throw errors immediately without warning for a release.

JSON schema for NixOS options

@Profpatsch proposed to have a JSON schema for NixOS options, which would allow them to be written with auto-completion in a JSON file. @DavHau, @nbp, @whentze and @piegames were also involved in the discussion

Active RFC drafts

In addition to the agenda, this post can bring attention to the existing RFC drafts of the NAT. Feel free to open issues/PRs in these repositories if you want to bring something to attention relating to that RFC draft. Alternatively feel free to chat on Matrix or reply here on Discourse.

  • Auto-called packages: Make most attribute definitions in pkgs/top-level/all-packages.nix be auto-generated from a single flat (or prefixed) directory instead, where the subdirectory corresponds to the attribute name. The ad-hoc category-based structure of packages will be removed.
5 Likes

Linking: Nix vs Spack why one or the other?
There is also the flatironinstitute/nixpack project (where “nixpack = nix+spack”)

2 Likes

Meeting notes

Protocol

Notes

Spack: External dependencies and specifications

Notes:

  • @tgamblin: Still a SAT solver
    • But specification and optimization is simpler
    • ASP works really well for us
  • @tgamblin: Another idea is Z3, SMT
    • Seems harder to specify
  • @tgamblin: externals aren’t as pure as you’d like
    • Specifying dependencies, build options etc.
    • Could get incompatible upgrade
  • @tgamblin: Main thing about the paper is reusing built binaries
    • If you have a minor tweak, tries to reuse already existing binaries
    • Spack knows about which binaries are available cached
    • Another purity thing we’re planning to do: Currently can’t build on one version, deploy for another. Things like ABI compatibility.
    • Stub symbols sounds similar to something we’ve done. There’s abstract and concrete specs, only give requirements during abstract specs, certain flags. Should only deploy with libraries that fulfil the spec
    • Don’t recommend writing own solver
  • @nbp: Alternative to SAT solver: Nixpkgs doesn’t need one as we’re baking one solution. Same for updating dependencies, security updates PR Shipping Security Updates by nbp · Pull Request #10851 · NixOS/nixpkgs · GitHub
    • @nbp: Not that complicated, but hard to integrate into nixpkgs
    • SOS ([RFC 0003] SOS: Simple Override Strategy. by nbp · Pull Request #3 · NixOS/rfcs · GitHub) would make makes this a lot simpler
    • @tgamblin: Resolver is there because packages typically aren’t written with specific versions, but rather with bounds. Those are preserved and are given to the solver. Can work with existing binaries and external packages. ABI compatibity with the shipping security updates PR
    • @nbp: Up to submitter to ensure compatibility. Generally not our problem.
    • @nbp: Always building latest package. With CV change applied to another, building one latest version for base, one with it applied
    • @tgamblin: Currently struggling with ABI incompatibilities.
    • @infinisil: Nixpkgs can always use the latest version, authorative, single CI. With flakes people want more distributed Nix expressions and dependencies, version requirements come up more.
    • @tgamblin: Environments remind me a bit of flakes

Deep runtime dependency substitution

The Nixpkgs Architecture Team is on nixos.org now

Problems with RFC 127

JSON schema for NixOS options

2 Likes