In my experience, you get lots of build failures if you try to actually build packages on overlayfs. I’ve heard recently overlayfs has some mode that’s supposed to make it behave more like a normal filesystem, though — I don’t know whether I was using that or not when I tried out Nix-on-overlayfs.
1 Like
But that was just the beginning, we want to give you more power to configure Repls how you would like and to increase Repl portability with other platforms. We want to give you access to an even larger collection of Nix packages: ones that were published years ago, and the latest ones published today. We also want you to be able to use Nix Flakes on Replit: it shouldn’t require extra work to configure development environments on every platform. Write a Nix Flake once and have the same reproducible environment everywhere.
To achieve this, we’re going to need a way to merge Nix stores!
I am pretty sure I miss something here, sorry if it’s obvious! What is the main difference in the use cases between the overlay fs and Nix binary cache here, is this because of they don’t invoke Nix for setting up dev environments at all?
When you use a binary cache, the build products have to be moved to your filesystem which can take time to move and use up disk space.
4 Likes
It may be that overlay stores can help with a use case described in storePath as a settable property of derivation · Issue #5067 · NixOS/nix · GitHub
2 Likes
I’ve wanted to be able to segregate a Nix store into a ‘system’ part and a ‘personal’ part, with the idea that the system store contains off-the-shelf software that wouldn’t need to be particularly protected if my computer were stolen, and the personal store contains software that is a privacy risk, IP that I’m obligated to secure, etc.; but where any paths already in the system store could be used by the personal store as needed. If I could do that by making a personal local overlay store over the system store, that would be pretty great, particularly if the stores could be handled differently in the following ways:
| System | Personal | |
|---|---|---|
| Readability | world-readable | user-readable |
| Filesystem | boring and fast | encrypted (with the rest of /home?) |
| Garbage collection | system-defined schedule | different user-defined schedule |
keep-outputs |
no | yes |
10 Likes
That looks amazing! I have one question: Will your Layered Store support be tied to specific semantics (“upper Nix” having access to entire lower store DB and can instantly substitute) or will it be pluggable? I’m thinking of the following problems:
- It may not be feasible to expose the entire lower store as a Nix-compatible SQLIte database. The lower store may be large, dynamic, and even have multi-tenant awareness (e.g., Attic and nixbuild’s datalake mentioned earlier).
- Future isolation mechanisms (VMs) and complex setups the may necessitate other zero-copy substitution mechanisms (e.g., virtiofs, NFS, host-initiated bind mount, etc.) that are specific to each provider.
I thought about the general problem earlier, and came up with a different idea that minimizes the awareness of the “upper Nix” regarding the substitution mechanism: Substitution Agents · GitHub
All Nix knows is that the substitution agent will somehow make the store path appear. It might run as a service inside the VM and perform virtiofs mounts, or it can be in a sidecar container and can mount new paths into the CI container’s store.
4 Likes
Yeah, I get it. But isn’t the overlay fs still copy artifacts onto the build host machine anyway?
Wouldn’t the upper store break if the lower store gets garbage collected?
2 Likes
Yes, it would. We’re exploring the idea of a store checker that would detect this issue.
3 Likes
The RFC has been open for a bit, and we need more shepherds! Shepherding doesn’t take very much time for most RFCs, so we encourage anyone who would like to use this feature to sign up!
2 Likes
Happy to say that the implementation ([RFC 0152] `local-overlay` store by Ericson2314 · Pull Request #152 · NixOS/rfcs · GitHub) is now merged in upstream Nix as an experimental feature!
19 Likes
First, congrats! I’ve been excited for this since it was announced, given my own weird poor hacky attempts at doing some sort of sqlite syncing with a shared /nix/store, etc. Thanks to you and replit and reviewers, etc.
What’s the best way to get my head around this, strictly from an end-user perspective? Skimming through the RFC is a bit of a doozy, and I’m hoping I can get some working knowledge without needing to consume the whole RFC/impl details.
8 Likes
Yes one should never need to read an RFC or implementation PR as the only option to figure out how something works, of course!
All store types have user-facing documentation, this one included.
It will appear in Store Types - Nix Reference Manual soon once hydra catches up
In the meantime, you can read the source markdown: nix/src/libstore/local-overlay-store.md at master · NixOS/nix · GitHub
6 Likes
The local overlay store is now used in every Replit container, so you could inspect what is going on there to see it in action:
$ cat /etc/nix/nix.conf
experimental-features = nix-command flakes local-overlay-store read-only-local-store
store = local-overlay?lower-store=%2Fmnt%2Fcacache%3Fread-only%3Dtrue&upper-layer=/mnt/nix/store&check-mount=false
gc-reserved-space = 0
$ mount | grep /nix/store
overlay on /nix/store type overlay (rw,relatime,lowerdir=/mnt/cacache/nix/store:/mnt/nixmodules/nix/store:/nix/store,upperdir=/mnt/nix/store,workdir=/mnt/nix/work/store,xino=off,nouserxattr)
/mnt/nix is the upper persistent nix store, and /mnt/cacache/nix/ is the lower store.
6 Likes
Why is only one of the store paths URL-escaped?
I don’t understand from “Lower metadata source:” on the linked documentation how the metadata source is configured. It says:
This is abstract, just some way to read the metadata of lower store store objects. For example it could be a SQLite database […]
It’s not clear from these docs what the options are for providing the lower metadata?
It also says of “Lower store directory”:
Specified with
lower-store.realsetting.
However I didn’t find any examples of this being used?
Thanks for the feedback, @pwaller! I tried to address all your questions in Improve `local-overlay` docs in a few ways by Ericson2314 · Pull Request #10502 · NixOS/nix · GitHub. Do let me know if that makes it clear now.
2 Likes
So, has someone built something neat with this? I still haven’t managed to kick the tires on it yet.
I tried this following the doc here Experimental Local Overlay Store - Nix Reference Manual. I get this errrors:
performing a single-user installation of Nix...
copying Nix to /nix/store....................................................
replacing old 'nix-2.22.1'
installing 'nix-2.22.1'
error:
… while setting up the build environment
error: getting attributes of path '/mnt/wsl/nix/ocean_store/merged_store/nix/store/1rkhjf55x59w6qm1pbhf80ks2wjpg973-libcpuid-0.6.4': No such file or directory
/tmp/nix-binary-tarball-unpack.XxvibFZjgI/unpack/nix-2.22.1-x86_64-linux/install: unable to install Nix into your default profile
It seems the install script still use the /nix/store , but when the path is different from the store option in the ~/.config/nix/nix.conf, which is setted to use the local (overlay) store. I tried these 2 stores both failed.
IS IT POSSIBLE to have a store different to nix/store or real physical storage different to ‘nix/store’?