Flakes and recursive-nix

enobayram · 2023-08-28T07:59:55.875Z

Our projects have been suffering from an unfortunate Nix ergonomics issue that’s been occupying my mind for a very long time now and flakes + recursive-nix are the ideal solution conceptually, but don’t work together in practice. Let me explain:

We use the excellent haskell.nix infrastructure to wrap our Haskell projects with Nix. You drop the flake into the repo and it just builds your project with the same dependency set and build options as those used by the native Haskell tooling, the dream interface which keeps Nix out of the way of anyone that wants to work with Haskell tooling only.

A haskell.nix-built derivation takes 5-10 seconds to evaluate even if the result is already in the store. Even worse, it involves IFD and the intermediate dependencies of this IFD is almost a GB. Neither of this is too bad for a single Haskell project, since your IFD dependencies get fetched the first time you build, and you nix build occasionally, staying in your nix develop most of the time, building with Haskell tooling incrementally.

The trouble starts when you want to use these haskell.nix-built executables downstream. Say you have 5 Haskell executables from 5 different projects that you want to tie in together in a NixOS configuration. Then you have to wait for 5-10 seconds for each project, making it painful to iterate on your downstream flake. What’s even worse is that the one GB IFD dependencies are often also duplicated for each project (because they tend to depend on different versions of haskell.nix), which means everyone that tries to build your downstream flake ends up fetching a 5GB closure just to evaluate the flake.

You can avoid the IFD in haskell.nix by “materializing”, but that involves bloating your repo with very large generated sources, which spoils the whole “Nix stays out of the way unless you use it” aspect, but worse, it interacts very badly with flakes’ habit of copying the repo to the nix store for every build.

I’ve been thinking about a good solution to this whole thing and every solution (including doing nothing) has very unpleasant trade-offs one way or another. One solution that would have been perfect all-round and is also conceptually very clean is combining Nix flakes with recursive-nix! When you think about it, the two is actually a match made in heaven. Flakes’ purity and reproducibility mean that building a flake output inside a recursive-nix derivation is perfectly well-behaved.

So, imagine that I could write a recursive-nix derivation that essentially builds nix build ${self} (please disregard, for the sake of the argument, the fact that the inputs to my flake can be different from my flake.lock) and expose it as an output of my flake. As long as this derivation is in my binary cache, any downstream users of my flake would just fetch the resulting store path without having to evaluate pretty much anything, completely side-stepping the long evaluation times and the IFD dependencies.

It’s very unfortunate that this theoretically sound application of flakes + recursive-nix doesn’t work in practice today. The build of the recursive-nix derivation fails since it doesn’t have access to the internet (nor even the /nix folder). Flakes’ sandboxing essentially kills recursive-nix while they could’ve been the best buddies in the world.

I think the same perfect solution could be achieved if Nix flake evaluation cache would be extended to the Nix evaluation time. Particularly if we had remote Nix evaluation caches like the current binary caches. But it seems to me like getting recursive-nix to work with flakes is a much simpler undertaking and it reuses the whole existing infrastructure for caching.

enobayram · 2023-08-28T08:07:49.518Z

Oh, I forgot to mention, recursive-nix would also be a great way to flake-ify legacy Nix repositories that are riddled with import <nixpkgs> expressions and builtins.currentSystems, environment variable lookups etc. If only we could integrate recursive-nix with flakes, we could build such projects inside a recursive-nix derivation and sandbox the whole Nix environment, reintroducing reproducibility to these projects, making them well-behaved flake citizens without requiring --impure.

andreabedini · 2023-08-30T07:57:10.143Z

enobayram:

A haskell.nix-built derivation takes 5-10 seconds to evaluate even if the result is already in the store. Even worse, it involves IFD and the intermediate dependencies of this IFD is almost a GB.

FWIW we are hoping to be able to cut down significantly the size of the closure of that IFD (that generates the build plan). In particular we are trying to avoid it depending on GHC itself.

enobayram:

What’s even worse is that the one GB IFD dependencies are often also duplicated for each project (because they tend to depend on different versions of haskell.nix), which means everyone that tries to build your downstream flake ends up fetching a 5GB closure just to evaluate the flake.

I think what you are referring to here is not quite the size of the closure needed for the IFD but the size of the flake inputs which nix eagerly downloads.

enobayram · 2023-08-30T08:38:17.587Z

@andreabedini First of all, thank you for the amazing project that is haskell.nix! As a user, I would of course appreciate any engineering effort that reduces the resource consumption at every stage, but I never intended to frame this as something that should be improved with haskell.nix. My intuition tells me that regardless of how brilliantly you optimize it, a project like haskell.nix will always be resource intensive, because you’re dancing around the native Haskell tooling and trying to construct a pure environment around it. I’m glad this is even possible in the first place.

When I think about it, we probably depend on many other derivations whose build systems are internally much more complicated than our Haskell projects using haskell.nix, but even for a massive dependency like Firefox, the overhead is minimal for a downstream user. The difference is that Firefox isn’t implementing that complexity in Nix. That’s why my opinion is that this shortcoming with haskell.nix for downstream consumers is something that should be solved at the Nix level. Nix shouldn’t punish Nix users for defining their builds in Nix. I find that recursive-nix is the most natural solution since then Nix treats itself as just another build system, but I also understand what sort of complications arise from it, so I understand the desire to handle it at the flake evaluation caching layer at the cost of flexibility.

Either way solving the issue either with recursive-nix and/or flake evaluation caching would be a massive improvement over the status quo.

I think what you are referring to here is not quite the size of the closure needed for the IFD but the size of the flake inputs which nix eagerly downloads.

I didn’t dig too much into why exactly the downloads happen, so you’re probably right. Thank you for the correction!

andreabedini · 2023-08-30T09:08:40.598Z

enobayram:

@andreabedini First of all, thank you for the amazing project that is haskell.nix!

<3 but I am just a contributor

enobayram:

I never intended to frame this as something that should be improved with haskell.nix.

There would be nothing wrong in that. There are plenty of things to be improved TBH.

W.r.t. recursive-nix, I am keeping an eye on it but I am not really up to date and I haven’t tried using it.

enobayram:

I didn’t dig too much into why exactly the downloads happen, so you’re probably right.

Do nix flake metadata github:input-output-hk/haskell.nix and you’ll see what I mean

I can give you one tip: if you don’t need those inputs you can replace them with an empty flake. E.g. if you are not using stack, overriding the stackage input with an empty flake will save you amost a 1GB.

enobayram · 2023-08-30T21:56:27.382Z

if you don’t need those inputs you can replace them with an empty flake.

I love this tip, thank you very much! I’m actually bothered by how much haskell.nix bloats downstream flake.locks, especially when you pull multiple haskell.nix-based flakes, but I can really reduce that noise using this tip.

overriding the stackage input with an empty flake will save you amost a 1GB.

Oh, then I don’t think mine was a case of Nix fetching flake inputs eagerly, I never saw the stackage input getting fetched even though I’m bothered by all the noise in flake.lock. When I mentioned the IFD closure, I was mainly thinking of hackage.nix taking up 600MB after decompression. That and other derivations like nix-tools etc.

enobayram · 2023-11-17T10:48:26.182Z

Hi again, I just wanted to report back here that I have a solution to this problem now. I’ve noticed that the reason why nix build doesn’t work inside recursive-nix is that it tries to fetch its inputs during eval time, similar to how builtins.fetchTarball does. This fails because the recursive-nix derivation’s build environment doesn’t have access to the internet (unless you disable Nix’s sandboxing entirely). However, I’ve discovered since then that it’s possible to make it not fail at that.

One approach is to artificially inject these dependencies into the build closure of the recursive-nix derivation, you can see an example of that here. In this case, I’ve assembled that evalTimeDependencies.nix by attempting to build the recursive-nix derivation and figuring out, based on the error message, what it’s trying to download. Anything that’s in the build-time closure of the recursive-nix derivation will be visible to it in the /nix/store it sees during the build, so when it tries to fetchTarball, it finds the result there and doesn’t try to access the internet.

Another approach that’s actually much more ergonomic for flakes is to instead make the flake fetch its inputs using pkgs.fetchzip instead of fetchTarball. This way, you don’t have to mess with the recursive-nix derivation to inject anything. The way I’ve managed to do this is to build the flake inside recursive-nix with nix-build using flake-compat instead of the usual nix build, but I’ve used a modified version of flake-compat that allowed me to override the fetchTarball with pkgs.fetchzip. The flake builds fine inside recursive-nix, because unlike fetchTarball, pkgs.fetchzip performs the actual download as part of the build of a fixed-output derivation. So, what happens is that the nix-build running inside the recursive-nix offloads the download to an independent build that has access to the internet since it is a fixed-output derivation. I find this approach to be much more ergonomic, because I’ve been able to encapsulate it entirely so that I can expose these recursive-nix derivations easily from existing haskell.nix projects.