Dynamic Derivations Working Group #2

Development
1

Dyn Drvs Working Group #2

infra

  • any progress?

  • @eveeifyeve: will test with small project on a local network

  • @tomberek: talked to Oxide, but nothing yet

  • @tomberek: we may also use the rack used by CUDA team. TODO: ask if that is feasible

  • @eveeifyeve: GHA may be good enough for now

  • @tomberek: nixgg as a stdenv replacement is looking more promising

  • @ericson2314: staging Hydra: https://staging-hydra.nixos.org/

    • can ask infra if we can use
    • might have to hook up special builders or mandatorySystemFeatures

  • How is Darwin support?

    • BSD/Windows is low-priority for now

  • GitHub - nix-dyn-drv/nixpkgs: Nix Packages collection & NixOS ¡ GitHub

    • Having a demo branch of Nixpkgs
    • idea: have a patchset on top of nixpkgs

  • GitHub - nix-dyn-drv/overlay: Overlay of nixpkgs that uses dynamic derivations ¡ GitHub or overlay? (TODO: @omberek)

  • CUDA @SomeoneSerge

    • looking to upgrade Hydra
    • Nix pre-build-hook on remote builders don’t have access to drv for GPU device exposing
      • @ericson2314: we can expose drv in json format
        • input drv → input src rewriting: witness of being able to run a build + schedule
        • individual builds shouldn’t need to know how something got produced
        • only floating-CA cannot be re-written until built
        • currently remote building requires trusted users
          • this can be fixed with rewriting? Planning on a new mode of input-addressing
        • We can provide the resolved derivation in JSON.
    • interested to use the rust-queue-runner
    • @tomberek: can we borrow the Oxide rack?
    • @tomberek : (async) what could we look at building that would be most valuable for CUDA to save rebulid time?

  • @ericson2314:

    • will help @SomeoneSerge
    • update CUDA hydra (on hetnzer)

  • Welcome! @jaen

    • started with larger projects
    • moving down to smaller projects
    • making Nix patches to fix certain pathological situations
      • hash scanning
      • thread + connections
      • finish subprocess/fork overhead fixes
    • Made some progress on building kernel

Packages

  • @tomberek: llvm
    • builder-rpc exposes non-monadic build systems
    • required 3 phase approach because the build needs to run a few intermediate products in the build itself
  • @tomberek: configurePhase
    • idea is to split a derivation into configure / build / check
    • not as fine-grained, a good intermediate step
    • might even be applicable in normal input-addressed Nix
  • @eveeifyeve: gcc
  • @lyndeno: kernel
  • @jaen: chromium
    • overhead is down to 200s
    • currently functions!
  • pytorch
  • tensorflow
  • stdenv

Rust

  • @artemis, Obsidian, and @eveeifyeve (contact @joachin regarding libnix approach?) all interested. There is a proper project coming from Obsidian and a partner org. Everyone is waiting excitedly…

Cache?

  • Cachix compatibility with CA or dyn-drv? Anyone know? (@domenkozar? @sandydoo?)
  • Hydra itself for now can be good enough (esp. with S3)
  • S3? Anyone got credits

Strategy

  • @tomberek: Expose the overlay and keep updating it in order to broaden the audience and usage
    • might need a cache to really end up working
    • @jaen: usually overlay busts a cache. This well demonstrates the CA + dyn-drv benefit
  • @ericson2314: If others can get the overlay, I’ll work on infra.

Async

  • @lyndeno: I’ve been playing around with nixgg and adding functionality on top to make it work with a Linux kernel build. It now builds the Linux 6.18 kernel from nixpkgs and I have a VM test to confirm that it boots. Check GitHub - Lyndeno/nixgg ¡ GitHub on the “kernel-support” branch. On my machine it takes longer to perform the initial build. Incremental builds still take a while as the shims still have to run again, but the kernel is successfully split into over 20,000 derivations during the build process. My plans are to clean up the branch and make it mergable and set up my NixOS machines to run a kernel built this way. This will provide me an excuse to role my own kernel config and see how the dyndrv benefits scale depending on size.

Goals

  • Broader scale testing
  • Need a proof of value
    • What will this be?
    • show a cheaper rebuild

Tasks

Admin

  • Every other Thursdays 16:00 EDT and 20:00 GMT and 22:00 CEST (contact @tomberek for invite)
  • Matrix: nix dynamic derivations #nix-dyn-drv:ggpeti.com (can someone with perms help us move this to NixOS space?)
7 Likes
2

What kind of value are you looking for? Who is the target audience for this proof?
For selling it to devs, maybe adding/updating a dependency of a Rust project? That currently requires keeping a generated file up to date or using IFD if you want incremental builds of dependencies (Crane uses 1 big dependency-compiling drv).

3

@bitbloxhub I think there is no doubt that it will be a killer-feature for devs. The question is that at what point is it a killer feature for Nixpkgs?

Nixpkgs is packaging other people’s software, so low-latency development workflows aren’t really a priority. This is why, historically, Nixpkgs has neglected the type of stuff needed to do this really well, and it has been tackled out of try instead. But outside-Nixpkgs stuff has a tendency to suffer from “many competing standards” problems.

Nixpkgs doesn’t need good dev workflows, but it would, however, benefit from faster mass rebuilds. For example if you change one small thing in a large package like LLVM, Chromium, or the Linux kernel, it would be nice for that to be a correspondingly small rebuilds. Likewise, it would be nice for security updates that do not change headers — only shared libraries — to not cause recompilation downstream.

Dynamic Derivations can, with enough work, solve these problems too. And that would be a really “killer app” for Nixpkgs. So a lot of what we are talking about in terms of Nixpkgs overlays is trying to demonstrate that, not development workflows in general which are fast becoming a solved problem :slight_smile: .

5 Likes
4

I’m not 100% on more granular builds becoming a killer app for all devs. Having tested out crate2nix on a setup containing multiple workers, it was crazy seeing how the naive job distribution algorithm ended up slowing down builds instead of speeding them up.

I think generalizing Hydra‘s scheduling features will become important for usability as dynamic derivations enable more granularity in builds.

2 Likes
5

Not present in the minutes, but after the 1h cut-off we also had a conversation with John and Jaen about why the semantics of “dynamic derivations” has to be, to my taste, so much weirder than that of IFD, and a few observations came up that I found surprising:

  1. Dynamic Derivations are “shaders”.
  2. build-rpc-v0 is not about dynamism, but about relaxing the process communication model of Nix.

Just a teaser, will elaborate later:)

2 Likes