Super Colliding Nix Stores

Announcements
#1

Replit has teamed up with Obsidian Systems (@Ericson2314) and Tweag (@regnat, @ben) to add a Local Overlay Store to Nix that allows merging of multiple (possibly remote) stores.

You can read more about the motivation and details for this project on our blog:

In the coming weeks, we will be developing these new features, releasing a Nix RFC, and working to upstream it into Nix.

Reply here if you have ideas for how this kind of Store could help you out. We’re interested to learn about more use cases to make it as general as possible.

Ultimately, we’re trying to make all the features of Nix work inside Replit Repls and this will go a long way.

Replit is also hiring for a lot of roles. If you live in North America and want to help bring Nix to millions of developers please reach out to me (@ryantm) and apply.

31 Likes
#2

I think this could help in CI when you build using docker images. The overlay could be used as a cache for everything that isn’t in the docker image’s nix store.

4 Likes
#3

This could help solve remote building in Bazel with Nix-derived tools and toolchains.

2 Likes
#4

I was thinking of having one store as a “datalake”, and then exposing stores on top that workers use. The workers have the shared datalake mounted via say nfs. Similar to what has been discussed in the article.

1 Like
#5

The “datalake” approach is what nixbuild.net takes, and it works really nicely. We have all nar files available in a central storage location, and then use a fuse-based filesystem to mount specific nar files for each build. We are also thinking about making a fuse-based client for our users that would let them mount their part (or a subset of it) of our “datalake” locally.

6 Likes
#6

This could be useful to use Nix in a Qubes OS template where the template provides /nix, and allow a Qube (template derived system where only /home is persistent or specific directories) to have a persistent overlay on top of the /nix template

3 Likes
#7

hmm. Do we really need to change Nix to detect overlays and do a read-through on the nix store database? OverlayFS works fine with nix-store --load-db and is a pattern that is used all over the place.

What’s wrong with just doing nix-store --load-db < /nix/.resgistration-paths at startup to register all the paths that are in the base layer as we do today with our ISOs?

Anyhow. Interested in the RFC to understand more.

4 Likes
#8

In my experience, you get lots of build failures if you try to actually build packages on overlayfs. I’ve heard recently overlayfs has some mode that’s supposed to make it behave more like a normal filesystem, though — I don’t know whether I was using that or not when I tried out Nix-on-overlayfs.

1 Like
#9

But that was just the beginning, we want to give you more power to configure Repls how you would like and to increase Repl portability with other platforms. We want to give you access to an even larger collection of Nix packages: ones that were published years ago, and the latest ones published today. We also want you to be able to use Nix Flakes on Replit: it shouldn’t require extra work to configure development environments on every platform. Write a Nix Flake once and have the same reproducible environment everywhere.

To achieve this, we’re going to need a way to merge Nix stores!

I am pretty sure I miss something here, sorry if it’s obvious! What is the main difference in the use cases between the overlay fs and Nix binary cache here, is this because of they don’t invoke Nix for setting up dev environments at all?

#10

When you use a binary cache, the build products have to be moved to your filesystem which can take time to move and use up disk space.

2 Likes
#11

It may be that overlay stores can help with a use case described in storePath as a settable property of derivation · Issue #5067 · NixOS/nix · GitHub

2 Likes
#12

I’ve wanted to be able to segregate a Nix store into a ‘system’ part and a ‘personal’ part, with the idea that the system store contains off-the-shelf software that wouldn’t need to be particularly protected if my computer were stolen, and the personal store contains software that is a privacy risk, IP that I’m obligated to secure, etc.; but where any paths already in the system store could be used by the personal store as needed. If I could do that by making a personal local overlay store over the system store, that would be pretty great, particularly if the stores could be handled differently in the following ways:

System Personal
Readability world-readable user-readable
Filesystem boring and fast encrypted (with the rest of /home?)
Garbage collection system-defined schedule different user-defined schedule
keep-outputs no yes
4 Likes
#13

That looks amazing! I have one question: Will your Layered Store support be tied to specific semantics (“upper Nix” having access to entire lower store DB and can instantly substitute) or will it be pluggable? I’m thinking of the following problems:

  • It may not be feasible to expose the entire lower store as a Nix-compatible SQLIte database. The lower store may be large, dynamic, and even have multi-tenant awareness (e.g., Attic and nixbuild’s datalake mentioned earlier).
  • Future isolation mechanisms (VMs) and complex setups the may necessitate other zero-copy substitution mechanisms (e.g., virtiofs, NFS, host-initiated bind mount, etc.) that are specific to each provider.

I thought about the general problem earlier, and came up with a different idea that minimizes the awareness of the “upper Nix” regarding the substitution mechanism: Substitution Agents · GitHub

All Nix knows is that the substitution agent will somehow make the store path appear. It might run as a service inside the VM and perform virtiofs mounts, or it can be in a sidecar container and can mount new paths into the CI container’s store.

3 Likes
#14

Yeah, I get it. But isn’t the overlay fs still copy artifacts onto the build host machine anyway?

Hosted by Flying Circus.