I was thinking of having one store as a “datalake”, and then exposing stores on top that workers use. The workers have the shared datalake mounted via say nfs. Similar to what has been discussed in the article.
The “datalake” approach is what nixbuild.net takes, and it works really nicely. We have all nar files available in a central storage location, and then use a fuse-based filesystem to mount specific nar files for each build. We are also thinking about making a fuse-based client for our users that would let them mount their part (or a subset of it) of our “datalake” locally.
This could be useful to use Nix in a Qubes OS template where the template provides /nix, and allow a Qube (template derived system where only /home is persistent or specific directories) to have a persistent overlay on top of the /nix template
In my experience, you get lots of build failures if you try to actually build packages on overlayfs. I’ve heard recently overlayfs has some mode that’s supposed to make it behave more like a normal filesystem, though — I don’t know whether I was using that or not when I tried out Nix-on-overlayfs.
But that was just the beginning, we want to give you more power to configure Repls how you would like and to increase Repl portability with other platforms. We want to give you access to an even larger collection of Nix packages: ones that were published years ago, and the latest ones published today. We also want you to be able to use Nix Flakes on Replit: it shouldn’t require extra work to configure development environments on every platform. Write a Nix Flake once and have the same reproducible environment everywhere.
To achieve this, we’re going to need a way to merge Nix stores!
I am pretty sure I miss something here, sorry if it’s obvious! What is the main difference in the use cases between the overlay fs and Nix binary cache here, is this because of they don’t invoke Nix for setting up dev environments at all?
I’ve wanted to be able to segregate a Nix store into a ‘system’ part and a ‘personal’ part, with the idea that the system store contains off-the-shelf software that wouldn’t need to be particularly protected if my computer were stolen, and the personal store contains software that is a privacy risk, IP that I’m obligated to secure, etc.; but where any paths already in the system store could be used by the personal store as needed. If I could do that by making a personal local overlay store over the system store, that would be pretty great, particularly if the stores could be handled differently in the following ways:
That looks amazing! I have one question: Will your Layered Store support be tied to specific semantics (“upper Nix” having access to entire lower store DB and can instantly substitute) or will it be pluggable? I’m thinking of the following problems:
It may not be feasible to expose the entire lower store as a Nix-compatible SQLIte database. The lower store may be large, dynamic, and even have multi-tenant awareness (e.g., Attic and nixbuild’s datalake mentioned earlier).
Future isolation mechanisms (VMs) and complex setups the may necessitate other zero-copy substitution mechanisms (e.g., virtiofs, NFS, host-initiated bind mount, etc.) that are specific to each provider.
I thought about the general problem earlier, and came up with a different idea that minimizes the awareness of the “upper Nix” regarding the substitution mechanism: Substitution Agents · GitHub
All Nix knows is that the substitution agent will somehow make the store path appear. It might run as a service inside the VM and perform virtiofs mounts, or it can be in a sidecar container and can mount new paths into the CI container’s store.