Yes one should never need to read an RFC or implementation PR as the only option to figure out how something works, of course!
All store types have user-facing documentation, this one included.
It will appear in Store Types - Nix Reference Manual soon once hydra catches up
In the meantime, you can read the source markdown: nix/src/libstore/local-overlay-store.md at master · NixOS/nix · GitHub
6 Likes
ryantm
April 12, 2024, 8:32pm
22
The local overlay store is now used in every Replit container, so you could inspect what is going on there to see it in action:
$ cat /etc/nix/nix.conf
experimental-features = nix-command flakes local-overlay-store read-only-local-store
store = local-overlay?lower-store=%2Fmnt%2Fcacache%3Fread-only%3Dtrue&upper-layer=/mnt/nix/store&check-mount=false
gc-reserved-space = 0
$ mount | grep /nix/store
overlay on /nix/store type overlay (rw,relatime,lowerdir=/mnt/cacache/nix/store:/mnt/nixmodules/nix/store:/nix/store,upperdir=/mnt/nix/store,workdir=/mnt/nix/work/store,xino=off,nouserxattr)
/mnt/nix is the upper persistent nix store, and /mnt/cacache/nix/ is the lower store.
6 Likes
Why is only one of the store paths URL-escaped?
I don’t understand from “Lower metadata source :” on the linked documentation how the metadata source is configured. It says:
This is abstract, just some way to read the metadata of lower store store objects . For example it could be a SQLite database […]
It’s not clear from these docs what the options are for providing the lower metadata?
It also says of “Lower store directory ”:
Specified with lower-store.real setting.
However I didn’t find any examples of this being used?
Thanks for the feedback, @pwaller ! I tried to address all your questions in Improve `local-overlay` docs in a few ways by Ericson2314 · Pull Request #10502 · NixOS/nix · GitHub . Do let me know if that makes it clear now.
2 Likes
So, has someone built something neat with this? I still haven’t managed to kick the tires on it yet.
rxzfn
May 25, 2024, 4:13pm
27
I tried this following the doc here Experimental Local Overlay Store - Nix Reference Manual . I get this errrors:
performing a single-user installation of Nix...
copying Nix to /nix/store....................................................
replacing old 'nix-2.22.1'
installing 'nix-2.22.1'
error:
… while setting up the build environment
error: getting attributes of path '/mnt/wsl/nix/ocean_store/merged_store/nix/store/1rkhjf55x59w6qm1pbhf80ks2wjpg973-libcpuid-0.6.4': No such file or directory
/tmp/nix-binary-tarball-unpack.XxvibFZjgI/unpack/nix-2.22.1-x86_64-linux/install: unable to install Nix into your default profile
It seems the install script still use the /nix/store , but when the path is different from the store option in the ~/.config/nix/nix.conf, which is setted to use the local (overlay) store. I tried these 2 stores both failed.nix/store or real physical storage different to ‘nix/store’?
ryantm
May 26, 2024, 12:33pm
28
Can you share your nix config and mount/findmnt output?
I don’t understand that error because it looks like the installer? But those instructions don’t mention the installer.
Hey team,
opened 03:52PM - 08 Nov 24 UTC
bug
Hello team,
I found interesting article https://blog.replit.com/super-collidi… ng-nix-stores and read [https://nix.dev/manual/nix/2.22/store/types/experimental-local-overlay-store#experimental-local-overlay-store](this).
Our goal is building the system based on local-overlay-store which can work in non-privileged pod (we use OKD and pod should been running under the restricted-scc).
So, I have been inspired your idea to have two stores (big with many packages and local).
My idea is having nfsd server in privileged pod, mount OverlayFS there and export nfs share for merged, upper and lower and mount them into non-privileged nix pod.
Well, but until running this idea I tried to create a privileged pod and run my first overlay-fs store test, but I was failed to start it (or maybe I understood something incorrect (I'm newbie in nix).
Here my steps for reproducing.
1) I have created two empty pvc (for upper and lower) and mounted them into my pod. So it will be something like this:
```
apiVersion: v1
kind: Pod
metadata:
name: nix-test-p
spec:
containers:
- name: volume-test
env:
- name: HOME
value: /nix/home
image: nixos/nix
command:
- tail
- -f
- /dev/null
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
volumeMounts:
- name: pvc-upper
mountPath: /mnt/example/
- name: pvc-lower
mountPath: /mnt/example/store-a
- name: nix
mountPath: /etc/nix/nix.conf
subPath: nix.conf
ports:
- containerPort: 80
volumes:
- name: pvc-lower
persistentVolumeClaim:
claimName: lower-p
- name: pvc-upper
persistentVolumeClaim:
claimName: upper-p
- name: nix
configMap:
name: nix
```
I have mounted nix.conf via the configmap with this content:
```
apiVersion: v1
data:
nix.conf: |+
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
experimental-features = nix-command flakes local-overlay-store read-only-local-store
store = local-overlay://?root=/mnt/example/merged-store&lower-store=/mnt/example/store-a&upper-layer=/mnt/example/store-b&check-mount=false
kind: ConfigMap
metadata:
name: nix
```
So I have taken fully example from the [docs ](https://nix.dev/manual/nix/2.22/store/types/experimental-local-overlay-store#experimental-local-overlay-store) with the path.
First my surprise was the absence of `mount` command :
```
sh-5.2# mount
sh: mount: command not found
```
Okay, I have installed it with `nix-env -iA nixpkgs.mount` and it installed mount into /mnt/example/merged-store/nix/store/ (which is not mounted yet), but okay, I have mount now..
**So the question 1 - how can I install mount in correct way if loading nix container with overlay-fs store for initial mounting?**
btw, I haven't found nothing in `$HOME/.nix-profiles/bin` related to mount and found it with 'find' util:
```
sh-5.2# find / -name mount
/mnt/example/merged-store/nix/store/5lxi14cng9r29nxlbzbszanpbjwvwbi9-util-linux-minimal-2.39.4-mount/bin/mount
/mnt/example/merged-store/nix/store/ibgzqljiski2y3fqvg8fnj2g6bjyhqly-util-linux-2.39.4-mount/bin/mount
/mnt/example/merged-store/nix/store/2wvb4326f069mz8zan43yx6nak6lsjqk-util-linux-2.39.4-bin/share/bash-completion/completions/mount
/mnt/example/merged-store/nix/store/2wvb4326f069mz8zan43yx6nak6lsjqk-util-linux-2.39.4-bin/bin/mount
/mnt/example/merged-store/nix/store/lnk0ky82i3rqvxvi3qknky80qffldyi9-mount-util-linux-2.39.4/bin/mount
```
well, I have mounted overlay fs:
```
sh-5.2# /mnt/example/merged-store/nix/store/5lxi14cng9r29nxlbzbszanpbjwvwbi9-util-linux-minimal-2.39.4-mount/bin/mount -t overlay overlay -o lowerdir=/mnt/example/store-a/nix/store -o upperdir=/mnt/example/store-b -o workdir=/mnt/example/workdir /mnt/example/merged-store/nix/store
sh-5.2# ls /mnt/example/merged-store/nix/store
```
Ok, seems I can install packages.. btw, I mounted overlayfs according the commands from the manual, but it doesn't work without `check-mount=false`
So ok, I have installed packages:
```
sh-5.2# nix-env -iA nixpkgs.vim
installing 'vim-9.1.0377'
these 3 paths will be fetched (8.79 MiB download, 42.94 MiB unpacked):
/nix/store/ddkcg6irdsn0w2q05gphaaw3cblkml69-gawk-5.2.2
/nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377
/nix/store/xbi4fw67yyrqz4mnza64zhw16s4rgybm-vim-9.1.0377-xxd
copying path '/nix/store/ddkcg6irdsn0w2q05gphaaw3cblkml69-gawk-5.2.2' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/xbi4fw67yyrqz4mnza64zhw16s4rgybm-vim-9.1.0377-xxd' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377' from 'https://cache.nixos.org' to 'local-overlay://'...
building '/nix/store/s83pcskzqdz78qwkpydcm33gm40ckfcx-user-environment.drv'...
sh-5.2# nix-env -iA nixpkgs.nano
installing 'nano-8.0'
these 3 paths will be fetched (0.87 MiB download, 10.89 MiB unpacked):
/nix/store/kbfzchcy5x3cwa39ls1dk92hd2q6ppz2-file-5.45
/nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0
/nix/store/z8l5b4hml7fchafzj94zm2pjg7xg5jql-nano-8.0-info
copying path '/nix/store/z8l5b4hml7fchafzj94zm2pjg7xg5jql-nano-8.0-info' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/kbfzchcy5x3cwa39ls1dk92hd2q6ppz2-file-5.45' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0' from 'https://cache.nixos.org' to 'local-overlay://'...
building '/nix/store/rffg19w0n2irxvfmw8paw6g8bx6hzlm0-user-environment.drv'...
```
all of them has been installed into correct path:
```
sh-5.2# find / -name vim
/nix/store/yzqnkb2mfzphmhv3248pw0pqh6f1mn6y-7sprarsdfz9qcd7859phvr9nvhi14mri-source/pkgs/applications/editors/vim
/nix/store/yzqnkb2mfzphmhv3248pw0pqh6f1mn6y-7sprarsdfz9qcd7859phvr9nvhi14mri-source/pkgs/test/vim
/mnt/example/store-b/0x63v0qsqws6iqxi9y151ani984j1n4d-user-environment/share/vim
/mnt/example/store-b/0x63v0qsqws6iqxi9y151ani984j1n4d-user-environment/bin/vim
/mnt/example/store-b/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377/share/vim
/mnt/example/store-b/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377/bin/vim
/mnt/example/merged-store/nix/store/0x63v0qsqws6iqxi9y151ani984j1n4d-user-environment/share/vim
/mnt/example/merged-store/nix/store/0x63v0qsqws6iqxi9y151ani984j1n4d-user-environment/bin/vim
/mnt/example/merged-store/nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377/share/vim
/mnt/example/merged-store/nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377/bin/vim
sh-5.2# find / -name nano
/nix/store/yzqnkb2mfzphmhv3248pw0pqh6f1mn6y-7sprarsdfz9qcd7859phvr9nvhi14mri-source/pkgs/applications/editors/nano
/mnt/example/store-b/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/share/nano
/mnt/example/store-b/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/share/doc/nano
/mnt/example/store-b/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/bin/nano
/mnt/example/merged-store/nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/share/nano
/mnt/example/merged-store/nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/share/doc/nano
/mnt/example/merged-store/nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/bin/nano
```
Well next I want to prepare lower storage. I plan to umount overlayfs and copy all data from upper to lower storage, cleanup database/upper and try to install data from lower-store. So I expect it won't download packages from the Network but will reuse them from lower storage.
So:
install mount again:
```
sh-5.2# nix-env -iA nixpkgs.mount
installing 'mount-util-linux-2.39.4'
....
sh-5.2# /mnt/example/store-b/5lxi14cng9r29nxlbzbszanpbjwvwbi9-util-linux-minimal-2.39.4-mount/bin/umount /mnt/example/merged-store/nix/store
sh-5.2# cp -a /mnt/example/store-b/* /mnt/example/store-a/nix/store/
```
found strange: database on lower layer is updated:
```
sh-5.2# ls -la ./mnt/example/store-a/nix/var/nix/db
total 8280
drwxr-xr-x 2 root root 4096 Nov 8 15:29 .
drwxr-xr-x 6 root root 4096 Nov 8 15:29 ..
-rw------- 1 root root 0 Nov 8 15:29 big-lock
-rw-r--r-- 1 root root 45056 Nov 8 15:29 db.sqlite
-rw-r--r-- 1 root root 32768 Nov 8 15:39 db.sqlite-shm
-rw-r--r-- 1 root root 0 Nov 8 15:29 db.sqlite-wal
-rw------- 1 root root 8388608 Nov 8 15:29 reserved
-rw-r--r-- 1 root root 2 Nov 8 15:29 schema
```
It is not under overlayfs - just folder on dist, but it's a strange, I thought this db should not be updated or I'm thinking incorrect?
But okay, I have removed everything from upper layer
```
sh-5.2# ls -la /mnt/example/store-b/
total 160
drwxrwxr-t 2 root nixbld 159744 Nov 8 15:42 .
drwxr-xr-x 7 root root 4096 Nov 8 15:20 ..
sh-5.2# ls -la /mnt/example/workdir/
total 12
drwxr-xr-x 3 root root 4096 Nov 8 15:20 .
drwxr-xr-x 7 root root 4096 Nov 8 15:20 ..
d--------- 2 root root 4096 Nov 8 15:20 work
sh-5.2# ls -la /mnt/example/workdir/work/
total 8
d--------- 2 root root 4096 Nov 8 15:20 .
drwxr-xr-x 3 root root 4096 Nov 8 15:20 ..
```
Remount overlayfs and trying to reinstall pacakges which stored on lower layer:
```
sh-5.2# /mnt/example/store-a/nix/store/5lxi14cng9r29nxlbzbszanpbjwvwbi9-util-linux-minimal-2.39.4-mount/bin/mount -t overlay overlay -o lowerdir=/mnt/example/store-a/nix/store -o upperdir=/mnt/example/store-b -o workdir=/mnt/example/workdir /mnt/example/merged-store/nix/store
sh-5.2# nix-env -iA nixpkgs.nano
installing 'nano-8.0'
these 9 paths will be fetched (8.13 MiB download, 45.52 MiB unpacked):
/nix/store/kbfzchcy5x3cwa39ls1dk92hd2q6ppz2-file-5.45
/nix/store/87848rvrg5c7jmplpi0iapvbxyj9kfid-glibc-2.39-52
/nix/store/ya4arqpf6vwbq4msi449314kqbhdb4l4-libidn2-2.3.7
/nix/store/503h8rdwzj12dh0fs1b6y643qm26vqsb-libunistring-1.1
/nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0
/nix/store/z8l5b4hml7fchafzj94zm2pjg7xg5jql-nano-8.0-info
/nix/store/qdbfy1ndpz2n868b8y1y3nlfsixgz106-ncurses-6.4
/nix/store/19avl2s0rd4whz3b0n118qdrv6yr3b0g-xgcc-13.2.0-libgcc
/nix/store/2k9k3q1vk8z6w7743k6nb22vnb05xv06-zlib-1.3.1
copying path '/nix/store/z8l5b4hml7fchafzj94zm2pjg7xg5jql-nano-8.0-info' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/19avl2s0rd4whz3b0n118qdrv6yr3b0g-xgcc-13.2.0-libgcc' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/503h8rdwzj12dh0fs1b6y643qm26vqsb-libunistring-1.1' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/ya4arqpf6vwbq4msi449314kqbhdb4l4-libidn2-2.3.7' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/87848rvrg5c7jmplpi0iapvbxyj9kfid-glibc-2.39-52' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/qdbfy1ndpz2n868b8y1y3nlfsixgz106-ncurses-6.4' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/2k9k3q1vk8z6w7743k6nb22vnb05xv06-zlib-1.3.1' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/kbfzchcy5x3cwa39ls1dk92hd2q6ppz2-file-5.45' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0' from 'https://cache.nixos.org' to 'local-overlay://'...
building '/nix/store/rffg19w0n2irxvfmw8paw6g8bx6hzlm0-user-environment.drv'...
sh-5.2# nix-env -iA nixpkgs.vim
installing 'vim-9.1.0377'
these 4 paths will be fetched (9.21 MiB download, 44.48 MiB unpacked):
/nix/store/wckka8fxv4h5hp74cbkhaw3fw7kbvcs1-bash-5.2p26
/nix/store/ddkcg6irdsn0w2q05gphaaw3cblkml69-gawk-5.2.2
/nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377
/nix/store/xbi4fw67yyrqz4mnza64zhw16s4rgybm-vim-9.1.0377-xxd
copying path '/nix/store/wckka8fxv4h5hp74cbkhaw3fw7kbvcs1-bash-5.2p26' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/ddkcg6irdsn0w2q05gphaaw3cblkml69-gawk-5.2.2' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/xbi4fw67yyrqz4mnza64zhw16s4rgybm-vim-9.1.0377-xxd' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377' from 'https://cache.nixos.org' to 'local-overlay://'...
building '/nix/store/s83pcskzqdz78qwkpydcm33gm40ckfcx-user-environment.drv'...
```
And nix downloads them from Internet, but I expected them will be reused from lower-layer.
So , could you please tell me when I did a mistake? Unfortunately there are few docs about overlay-local-store, but this seems to be a killer feature.
Thank you very much.
@ryantm Ryan, it would be great if you can take a look there. Seems you have good experience with local-overlay.
1 Like