Many thanks to @Ericson2314 for answering my questions
34 Likes
There’s a bright feature ahead of us, finally we could for example write a gradle plugin that writes derivations for all project dependencies, then make nix realise them. That would be significantly better than messing with lang2nix woes.
4 Likes
Just to make sure I understand - does this imply that we must generate store derivations in the aterm format either by hand in another language or via recursive nix to productively use dynamic drvs?
In practice you would probably do it programmatically with
libstoreor at least with separate Nix files
AFAICT libstore is part of the C++ API which I believe was deprecated, do you happen to know if there’s another equivalent API that doesn’t rely on C++?
5 Likes
In the demo he uses hn-nix which is a Haskell Nix library.
There’s also two.
You can also just raw JSON and pass that into ‘nix derivation’ so anything is ok 
2 Likes
This could be used to sandbox Nix evaluation itself, right? You’d have a small outer derivation that runs nix-instantiate (or nix derivation show or something else that does Nix evaluation) in its sandboxed build script, and produces the target derivation with the dynamic derivation mechanism.
That way you could safe-guard against bugs in (or attacks against) the Nix evaluator itself which could allow things from the environment slip into derivations during evaluation.
2 Likes
That’s exactly right; that is possible today already with “recursive nix” but what i understand from @Ericson2314 is that dynamic derivations let’s us accomplish the same thing but with not as much capabilities necessary that recursive Nix requires.
What I see as next steps, let’s use CMake as an example
Nixpkg derivation → includes a CMake project configured to a Nix backend → emits dynamic derivations
So your outer Nix derivation is actually granular to all the CMake code modules and you get “incremental Nix builds”
1 Like
please keep up your excellent blog posts - i am really interested to hear more about this topic
thanks!
1 Like
@fzakaria You might be interested to know that someone/company(?) took your blogpost and re-published it on their own website:
There’s only a Source link to your original blogpost at the end.
I thought this was limit borderline, so I prefer to let you know since you don’t seem to be on the Fediverse.
¯\_(ツ)_/¯
3 Likes
A bit lame… Looks like they are in Hungary.
I doubt I’ll get them to respect my copyright.
Thank you for telling me.
Do you mean this format? Derivation - Nix Reference Manual
And if so, that appears to require nix derivation add, which as I understand would go back to the recursive-nix requirement I mentioned earlier. I’m also unsure of the performance implications of spawning potentially hundreds of recursive nix commands to add the drvs this way, as nlohmann-json (which nix uses) is well-known to be the slowest possible json implementation.
Correct me if I’m wrong, but to sum up, dynamic drvs:
- Require either:
- Familiarity with an obscure, undocumented format (ATerm derivations), or
- Use of recursive nix to work with (currently experimental) JSON derivation format
- Performance may be even worse than IFD, though if multiple nix processes are spawned, parallelisation may reduce this perf hit
- Use of recursive nix to turn nix code into a store drv
- I’ve even less idea about the perf implications here
- Require novel tooling to generate one of the above three formats
- Hence, this does not reduce the amount of code needed to be written, in comparison to IFD/*2nix, as I had originally hoped
The benefit I do see is that they allow splitting off a single derivation’s dependencies into many small, individual derivations (presumably reducing future rebuilds) without blocking eval as IFD would or a separate codegen step as we do now - which is a nice benefit, but when keeping in mind the above, still feels massively unergonomic as of now. It also presumably prevents having to vendor lockfiles, which we could use to shrink nixpkgs somewhat if this feature stabilizes.
(I know you’re not a nix dev, so this criticism isn’t directed at your work. I’m also saying this mostly to temper my own excitement at what this feature could do, since I was imagining things it can’t do.)
EDIT: added third method
My understanding from @Ericson2314 is that the use of nix-instantiate or nix derivation add can be allowed through something not as strong as recursive-nix which has to allow a complete nix build within the sandbox.
So there could be a way to “add to the store” via whatever means without turning on recursive nix; for now though that toggle is necessary otherwise the /nix/store is mounted as read-only.
In @Ericson2314 example; he uses hn-nix (HaskellNix) implementation which writes to the store directly. Given all the “excitement” around twix, using that or the new libstore in CppNix would be also possibilities.
I found nix instantiate easy enough.
You can see how I did it for lang2nix GitHub - fzakaria/NpmNix: A demonstration of how you can leverage Nix dynamic derivations for Node projects felt very easy and natural.
(Almost like IFD)
1 Like
You’re right, I missed that option - generating nix code and evaluating it recursively is a third valid option. I’ll edit my prior comment.
libstore is at least a decade old. And while it’s technically true that one could depend on it, but depending on the C++ interface has been heavily discouraged by nix devs because it’s apparently internal, even though the C interface isn’t stable either. Knowing that, it’s difficult to rely on any other tooling that relies on either the C++ or C interfaces that could break at any time (which iirc includes things like hnix).
I guess it comes down to risk management there, but using an experimental feature that depends on another experimental feature and third-party code that uses yet another experimental interface feels a bit too beyond-the-pale for me personally. I’d probably just stick to generating nix code, as you did.
EDIT: I’m unfamiliar with tvix, so I can’t comment on that.
1 Like
This talk was based on contributions from @Ericson2314 and his talk at PlanetNix; he seems passionate about getting it done (alongside the RFC) similar to CA-derivations /shrug
For Twix, Repl.it is using it in production so I guess it has some level of “formal support” which makes it interesting.
After having seen his talk – I felt the passion and similar inspired; so many idioms flatten with dynamic derivations. My aim in these posts were to inspire others to move the needs slightly from “experimental” 
2 Likes
edit: ignore