Tweag+Nix dev update #51

thufschmitt · July 24, 2023, 7:24am

Hello Nixers! For the start of this new week, here are some news from the Tweag Nix team since the previous update.

Nickel-Nix

@balsoft opened a pull-request to simplify the environment handling by leveraging nixpkgs’s stdenv script rather than reimplementing it in Nickel. This gives nickel-nix back most of the features of a native stdenv.mkDerivation call, while keeping the interface we’re trying to build for it.
@vkleen and @radvendii imported GitHub - nickel-lang/json-schema-to-nickel: Convert JSON schemas into Nickel contracts (originally written by the external contributor @mtoohey31) into the Nickel-lang organisation, and started improving it. This tool will be instrumental in allowing nickel-nix to interact with the bazillion tools required for a development environment (following the vision from https://github.com/nickel-lang/nickel-nix/pull/82)

Nix

After several successive delays, funding for [RFC 0143] Nix Store ACLs by balsoft · Pull Request #143 · NixOS/rfcs · GitHub got unlocked, so @balsoft and @ErinvanderVeen will resume their work on it.
The work on [RFC 0152] `local-overlay` store by Ericson2314 · Pull Request #152 · NixOS/rfcs · GitHub is also steadily progressing (thanks to funding from Replit and in partnership with our friends from Obsidian systems). The working branch is already in a fairly usable state, and side improvements are already being upstreamed (#8696, #8342)
@thufschmitt asked for https://github.com/NixOS/rfcs/pull/136 to enter FCP, supported by @infinisil and the rest of the shepherd team. Despite some controversy on its scope and applicability, this RFC should mark another step towards stabilizing these crucial experimental Nix features.

Nixpkgs

After several months of effort by @infinisil and other community members, RFC 140 (Simple package paths) got eventually merged, paving the way for merging the WIP implementation.
After a quiet FCP period, [RFC 0127] Nixpkgs "problem" infrastructure by piegamesde · Pull Request #127 · NixOS/rfcs · GitHub (shepherded amongst others by @infinisil) got accepted and merged, hopefully making it easier to track breakages and various “problems” in the Nixpkgs packages.

CUDA-related efforts

@ConnorBaker kept his tireless work on improving the state of CUDA in Nixpkgs. Do remember to check out CUDA Team Roadmap and Call for Sponsors if you haven’t already – all of the following work is done for or in relation to those efforts. Thank you to PDT Partners for the funding!

python3Packages.torch: statically link against magma by ConnorBaker · Pull Request #238465 · NixOS/nixpkgs · GitHub
- Decreased the size of the torch-lib closure by 3.2 GB, at the cost of a 300 MB increase to the size of the NAR.
python3Packages.torch: drop submodules in favor of Nixpkgs by ConnorBaker · Pull Request #239291 · NixOS/nixpkgs · GitHub
- A from-scratch rewrite of the PyTorch derivation with the goal of enabling:
  - Use of more Nixpkgs-provided dependencies instead of vendored copies.
  - Faster rebuilds through use of Nixpkgs-provided dependencies and fine-grained build targets (i.e., hopefully being able to split the CUDA-enabled libraries out from the CPU-only libraries).
  - A derivation for libtorch, which torch (the Python interface) depends on, instead of having libtorch built within torch.
- This PR involves a number of upstream patches to allow use of Nixpkgs-provided dependencies:
cudaPackages: multiple outputs for redistributables by ConnorBaker · Pull Request #240498 · NixOS/nixpkgs · GitHub
- Creates multiple outputs for CUDA redistributables allowing use of select components (e.g., static libraries).
- Makes the default output, out, the union of all other outputs, effectively allowing downstream derivations to opt-in to using specific components.
- More info on the setup for these numbers is in the PR, but they’re worth putting here:
  - cuDNN closure/NAR size went from 2.4G to 1.1G.
  - Magma closure size went from 2.9G to 1.6G. Mostly as a result of dropping dependencies on static cuBLAS and cuSPARSE.
  - PyTorch closure size went from 9.7G to 8.4G. Much greater gains to be had in migrating away from CUDA Toolkit, since the decrease comes entirely from switch to cudnn.lib instead of cudnn. Recall that Magma does not contribute to the closure size, because we link against it statically (at least, when we’re building with support for CUDA).
cpp_extension: generalize CUDA capability handling by ConnorBaker · Pull Request #104615 · pytorch/pytorch · GitHub
- An upstream patch for PyTorch which refactors the way that they prohibit unrecognized CUDA capabilities, and provides a potential bug-fix for the way they were handling capability versions.
- Instead of blocking them outright, allow unknown capabilities if they are specified by version (e.g., 9.0a) instead of a family name (e.g., Hopper or Thor) and print a warning. This effectively puts the onus of checking capabilities on the CUDA compiler (which, in this author’s opinion, is where it should be).

Teaching and documentation

Except for a short interruption for holidays, @infinisil steadily kept hosting the Nix hour, with help from @yoriksar.
A significant chunk of the team is planning on coming to NixCon, and submit talks on different topics there. Nothing to show yet, but some of our presentations will hopefully be deemed worthwhile
@fricklerhandwerk continued work on Nix reference documentation, specifically on configuration options and builtins.derivation

Community work

@fricklerhandwerk officially launched the 2023 edition of Summer of Nix with help of @mat and community member @mightyiam. Like the previous edition, it will be based on mob programming and focus on packaging for NGI0-supported software.
Thanks to some funding by Antithesis, @infinisil applied as an RFC steering committee member to help with the work of this crucial but overloaded team.

That’s all folks!