After the long-history (see below) of Dynamic Derivations we are at a point where there are many groups exploring the feature and I am proposing a working group to:
share the experiences of the implementors
refine the user experience based on usage of the various PoCs
determine infrastructure needs
deduplicate work
raise awareness
build momentum for adoption
… anything else relevant
I believe this feature has the potential to be a game-changer. Faster rebuilds, easier development, reduced pressure on CI systems, cacheable and alternative forms of instantiation, less need for “lang2nix”, close the gap with Bazel, and fine-grained incremental builds. Along with the existing work on better schedulers, RBE-style remote builders, bytecode evaluation, parallel eval, memory optimizations (thanks @xokdvium !), and so much more - this is an exciting time for Nix and I can see DynDrvs as a way to fix some of the outstanding UX issues Nix has in day-to-day usage. There is even work to make IFD more performant, there is lots of design space to explore here. This should be a part of what Nix has to offer the world. @Ericson2314 has devoted years to this, and it deserves the payoff of becoming a mature feature.
Dynamic Derivations have been in-progress for a long time. This has much of the history that I can put together. Let me know if this can be more complete. (and if anyone can help move/migrate into the wiki?)
The implementation work was primarily done via manyrefactors and addingin the underlyingmachinery. This was nearly all done by @Ericson2314 who deserves a huge thank-you for his tireless efforts over the years.
A limited form was merged into Nix on Sep 7 2023: https://github.com/NixOS/nix/pull/4628 . So at this point the basic primitives were available, but they were hard to use and required expert knowledge.
2024 consisted of even more work by @Ericson2314 and @roberth to clean up the implementation, and fix bugs. (TODO: with support from Obsidian Systems? PDT?)This time period represents a lot of clean up and bringing more contributors into the Nix team.
Existing! Is the invitation only for people who’ve done some related work, or are people interested in learning more about it (and potentially use it) are also invited?
It’s open to a variety of participants. For a while it took specialized expertise to meaningfully contribute, but now that has changed. We need end-users, documenters, testers, tutorial authors, library authors, developers, integrators, and so forth.
We’re trying to determine whether this is reproducible on more than one Darwin machine, and we’d really appreciate your help if you’re running a Darwin system.
What we’re seeing
When building the Sandstone example derivation, we hit this error:
building '/nix/store/8mdr07snb49axrfkhx88gn771ks69czw-sandstone-dyn-drvs-on-example.drv'...
sandstone-dyn-drvs-on-example> + echo /nix/store/wy51zbjg46lhxlivfqszl1jvmjdkvvac-coreutils-9.5/bin
sandstone-dyn-drvs-on-example> /nix/store/wy51zbjg46lhxlivfqszl1jvmjdkvvac-coreutils-9.5/bin
sandstone-dyn-drvs-on-example> + /nix/store/6hw535bcfaxkncfz6y575nlwg2jcfyv9-ghc-9.6.6/bin/ghc -o /nix/store/ciqp18j8bsg5xhc5a8rfv3q0vgbxkv5i-sandstone-dyn-drvs-on-example /nix/store/y3qjknhz2c45sxs4yms8ngj8784r1007-compile-A-object/A.o /nix/store/70r2jcb3rbk79n57bxkz3858aa1yz9kb-compile-B-object/B.o /nix/store/7cz43zchdmn0fva9g1mnavsy9ykxswp3-compile-C-object/C.o /nix/store/ak0rd83cmji6fxpwb84n7n28g3k68lw6-compile-D-object/D.o /nix/store/shsvkxbkjp4yyzjiqgaj0024pdfw2x9z-compile-Main-object/Main.o /nix/store/g0gm8xbjnkzl42fl83nyfl0dj8zixrzf-compile-Nested.E-object/Nested/E.o /nix/store/5x9fixvbjk4l00v5nzy1b0mnd7qwfwmy-compile-X-object/X.o
error: cannot operate on output 'out' of the unbuilt derivation '/nix/store/j4v89slzf43jm6fdcghqby7gg8lmw0f1-sandstone-dyn-drvs-on-example.drv.drv'
Setup notes
For Sandstone, you’ll need to temporarily comment out the recursive-nix required feature and In WriteDerivation.hs, change x86_64-linux to aarch64-darwin (until @Ericson2314 pushes a fix later).
How you can help
If you have an aarch64-darwin machine and can test this setup, please let us know if you get the same output we are seeing.
This will help us narrow down whether this is a Nix issue.
Thanks! Hope we can fix this issue before 2.36 gets released!
In my aarch64-darwin VM, I’m only able to reproduce your error by using a Nix client older than version 2.35. That is, with Nix daemon version 2.35.0 or master commit 85f1926, and Nix client version 2.34.8:
building '/nix/store/8mdr07snb49axrfkhx88gn771ks69czw-sandstone-dyn-drvs-on-example.drv'...
sandstone-dyn-drvs-on-example> + echo /nix/store/wy51zbjg46lhxlivfqszl1jvmjdkvvac-coreutils-9.5/bin
sandstone-dyn-drvs-on-example> /nix/store/wy51zbjg46lhxlivfqszl1jvmjdkvvac-coreutils-9.5/bin
sandstone-dyn-drvs-on-example> + /nix/store/6hw535bcfaxkncfz6y575nlwg2jcfyv9-ghc-9.6.6/bin/ghc -o /nix/store/ciqp18j8bsg5xhc5a8rfv3q0vgbxkv5i-sandstone-dyn-drvs-on-example /nix/store/y3qjknhz2c45sxs4yms8ngj8784r1007-compile-A-object/A.o /nix/store/70r2jcb3rbk79n57bxkz3858aa1yz9kb-compile-B-object/B.o /nix/store/7cz43zchdmn0fva9g1mnavsy9ykxswp3-compile-C-object/C.o /nix/store/ak0rd83cmji6fxpwb84n7n28g3k68lw6-compile-D-object/D.o /nix/store/shsvkxbkjp4yyzjiqgaj0024pdfw2x9z-compile-Main-object/Main.o /nix/store/g0gm8xbjnkzl42fl83nyfl0dj8zixrzf-compile-Nested.E-object/Nested/E.o /nix/store/5x9fixvbjk4l00v5nzy1b0mnd7qwfwmy-compile-X-object/X.o
error: cannot operate on output 'out' of the unbuilt derivation '/nix/store/j4v89slzf43jm6fdcghqby7gg8lmw0f1-sandstone-dyn-drvs-on-example.drv.drv'
If it’s flipped (daemon version 2.34.8, and client version 2.35.0 or master commit 85f1926), I get a different error:
building '/nix/store/8mdr07snb49axrfkhx88gn771ks69czw-sandstone-dyn-drvs-on-example.drv'...
sandstone-dyn-drvs-on-example> + echo /nix/store/wy51zbjg46lhxlivfqszl1jvmjdkvvac-coreutils-9.5/bin
sandstone-dyn-drvs-on-example> /nix/store/wy51zbjg46lhxlivfqszl1jvmjdkvvac-coreutils-9.5/bin
sandstone-dyn-drvs-on-example> + /nix/store/6hw535bcfaxkncfz6y575nlwg2jcfyv9-ghc-9.6.6/bin/ghc -o /nix/store/ciqp18j8bsg5xhc5a8rfv3q0vgbxkv5i-sandstone-dyn-drvs-on-example /nix/store/y3qjknhz2c45sxs4yms8ngj8784r1007-compile-A-object/A.o /nix/store/70r2jcb3rbk79n57bxkz3858aa1yz9kb-compile-B-object/B.o /nix/store/7cz43zchdmn0fva9g1mnavsy9ykxswp3-compile-C-object/C.o /nix/store/ak0rd83cmji6fxpwb84n7n28g3k68lw6-compile-D-object/D.o /nix/store/shsvkxbkjp4yyzjiqgaj0024pdfw2x9z-compile-Main-object/Main.o /nix/store/g0gm8xbjnkzl42fl83nyfl0dj8zixrzf-compile-Nested.E-object/Nested/E.o /nix/store/5x9fixvbjk4l00v5nzy1b0mnd7qwfwmy-compile-X-object/X.o
warning: the daemon is missing the 'realisation-with-path-not-hash' protocol feature, needed to support content-addressing derivations
error:
… looking up realisation for derivation '/nix/store/j4v89slzf43jm6fdcghqby7gg8lmw0f1-sandstone-dyn-drvs-on-example.drv.drv'
error: cannot operate on output 'out' of the unbuilt derivation '/nix/store/j4v89slzf43jm6fdcghqby7gg8lmw0f1-sandstone-dyn-drvs-on-example.drv.drv'
With both the daemon and the client at either version 2.35.0 or master commit 85f1926, I get no error:
Idea inspired by this, make can also output a build graph and because kbuild ( kernel’s build system ) has a makefile per directly, it may allow very fine grained kernel builds, aka per possibly per config option.
EDIT: thanks for the clarification re: meeting time. I’ll try to make at least part of it, if it’s 1800 GMT then.
I’m focusing mostly on Ninja-based stuff first (so ultimately AOSP, probably with stops at LLVM and Chromium for faster iteration on scale problems), but it’s one of things I wanted to try next after it and I’m trying to write the code so it can be extended with non-Ninja build graph edges without turning everything upside down. So if nobody else beats me to this, then it’s going to happen at some point anyway.
Currently I am trying to either parse the .cmd files the kernel leaves ( which themselves are evaluated gcc commands ) after a compile or look at how something like kninja does it.
