I don’t use the nix binary cache since I compile all of my software with CPU optimizations, so instead I have a virtual machine that I use as a substituter with a chroot store. That virtual machine consists of a bootable ISO of my NixOS config and a virtual ext4 drive that I mount at /.disk. The chroot store lives at /.disk/root and it is basically the only thing on that disk.
I also have a couple of client machines that use that virtual machine as a substituter. For simplicity, lets call the virtual machine “Alice” and the client machine “Bob”.
On Bob I will kick off a build with the command:
nix build --extra-experimental-features 'nix-command flakes' /persist/machine_setup/nix/configuration/hosts/bob/../..#bob.whatever --max-jobs 1 --repair -v --keep-going
And I will get the error:
error:
… while setting up the build environment
error: getting attributes of path "/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev": No such file or directory
If I add --option substitute false to the command, then everything will work fine.
But the odd thing is, that path exists on both Alice and Bob (proof below). The substituter itself is definitely working because every build it downloads hundreds of files from it. Seemingly the same files on each build (aside: why is it doing that? Shouldn’t it only need to download the paths the first time? I’m using flakes with no changes in files occurring in between builds, literally running a bash while loop that invokes the build over and over again.)
So right now what I’m doing is running a build in a loop with the substituter enabled and then once it no longer is making progress, I do a final build with the substituter disabled to get past whatever path it mistakenly thinks does not exist.
So does anyone know why this is happening or something I could do to investigate this further? The logs and the error message seem to be no help.
Previous threads / attempts to mitigate
Through searching I have found this thread: Nixos-rebuild switch on unstable gives `error: getting attributes of path '/nix/store/*-gnumake-4.4.1': No such file or directory`
However I have run this numerous times without it solving the issue:
alice$ $ doas -u nixworker env NIX_REMOTE='local?root=/.disk/root' nix-store --verify --check-contents --repair
bob$ doas nix-store --verify --check-contents --repair
I found this mailing list thread but my /etc/nixos is empty on both machines.
I found this 2021 github issue where a bunch of people hit this error from an official nixos release. But allegedly the issue was fixed by rolling back some commits in nix. It’s 2026 and I’m still hitting this issue.
I tried deleting the path but it won’t let me:
bob$ $ nix store delete /nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev
0 store paths deleted, 0.0 KiB freed
error: Cannot delete path '/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev' since it is still alive. To find out why, use: nix-store --query --roots and nix-store --query --referrers
I tried to copy the path but it won’t do anything:
bob$ nix copy -v --from 'ssh-ng://nixworker@alice?remote-store=local?root=/.disk/root' /nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev
copying 0 paths...
Proof the path exists
Alice:
alice$ doas -u nixworker env NIX_REMOTE='local?root=/.disk/root' sqlite3 -header -box -readonly /.disk/root/nix/var/nix/db/db.sqlite "select * from ValidPaths where path='/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev';"
╭──────┬──────────────┬─────────┬────────────┬──────────────┬───────┬────────┬────────────┬──╮
│ id │ path │ hash │registration│ deriver │narSize│ultimate│ sigs │ca│
│ │ │ │ Time │ │ │ │ │ │
╞══════╪══════════════╪═════════╪════════════╪══════════════╪═══════╪════════╪════════════╪══╡
│246205│/nix/store/ │sha256: │ 1790962780│/nix/store/ │ 234464│ 1│alice: │ │
│ │s6srjjjj9h9xa5│9fd205b7d│ │mnwnh3b7gh4mrq│ │ │GPWpb9cPplB1│ │
│ │lhhm0bnc4b1cfa│62d9fa14c│ │m6a80zrzcxr848│ │ │cUauNYtaWYzT│ │
│ │4bp1-xz-5.8.4-│7e2fa2c53│ │49yq-xz-5.8.4.│ │ │NMMkfnTCXAjX│ │
│ │dev │c63bc34e2│ │drv │ │ │wg8LMY/ │ │
│ │ │... │ │ │ │ │... │ │
╰──────┴──────────────┴─────────┴────────────┴──────────────┴───────┴────────┴────────────┴──╯
alice$ doas -u nixworker env NIX_REMOTE='local?root=/.disk/root' find /.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/lib
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/lib/pkgconfig
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/lib/pkgconfig/liblzma.pc
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/nix-support
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/nix-support/propagated-build-inputs
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/stream_flags.h
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/delta.h
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/container.h
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/filter.h
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/block.h
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/check.h
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/version.h
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/hardware.h
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/bcj.h
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/base.h
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/index.h
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/index_hash.h
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/vli.h
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/lzma12.h
/.disk/root/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma.h
Bob:
bob$ sqlite3 -header -box -readonly /nix/var/nix/db/db.sqlite "select * from ValidPaths where path='/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev';"
╭───────┬──────────────┬─────────┬───────────┬──────────────┬───────┬────────┬────────────┬──╮
│ id │ path │ hash │registratio│ deriver │narSize│ultimate│ sigs │ca│
│ │ │ │ nTime │ │ │ │ │ │
╞═══════╪══════════════╪═════════╪═══════════╪══════════════╪═══════╪════════╪════════════╪══╡
│1065735│/nix/store/ │sha256: │ 1790950472│/nix/store/ │ 234464│ │alice: │ │
│ │s6srjjjj9h9xa5│9fd205b7d│ │mnwnh3b7gh4mrq│ │ │GPWpb9cPplB1│ │
│ │lhhm0bnc4b1cfa│62d9fa14c│ │m6a80zrzcxr848│ │ │cUauNYtaWYzT│ │
│ │4bp1-xz-5.8.4-│7e2fa2c53│ │49yq-xz-5.8.4.│ │ │NMMkfnTCXAjX│ │
│ │dev │c63bc34e2│ │drv │ │ │wg8LMY/ │ │
│ │ │... │ │ │ │ │... │ │
╰───────┴──────────────┴─────────┴───────────┴──────────────┴───────┴────────┴────────────┴──╯
bob$ find /nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma.h
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/filter.h
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/block.h
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/bcj.h
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/index_hash.h
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/delta.h
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/index.h
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/vli.h
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/hardware.h
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/stream_flags.h
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/base.h
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/version.h
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/check.h
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/container.h
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/include/lzma/lzma12.h
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/nix-support
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/nix-support/propagated-build-inputs
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/lib
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/lib/pkgconfig
/nix/store/s6srjjjj9h9xa5lhhm0bnc4b1cfa4bp1-xz-5.8.4-dev/lib/pkgconfig/liblzma.pc
Proof it is downloading the same paths over and over again
I ran two builds back-to-back using the above command and wrote their logs to /tmp/log and /tmp/log2
bob$ cat /tmp/log /tmp/log2| grep -i 'copying path' | sort | uniq -c | sort -nr
2 copying path '/nix/store/zr9c7gzzzq5yqcmj5jbfcndmwhgv675v-iptables-1.8.13-man' from 'ssh-ng://nixworker@alice:65122'...
2 copying path '/nix/store/zp5lra854lp4xzz0ipjs6ksc0sr25vgk-gzip-1.14-man' from 'ssh-ng://nixworker@alice:65122'...
2 copying path '/nix/store/zn5mw9f42kn5g15i0xrxvlh3rzywyd1x-iproute2-7.1.0' from 'ssh-ng://nixworker@alice:65122'...
2 copying path '/nix/store/zczyk33m2sclb22898d19pajr0g481hd-iw-6.17-man' from 'ssh-ng://nixworker@alice:65122'...
2 copying path '/nix/store/zas729qbq9s1n2g9j697ddnlmh33yqcn-acl-2.4.0-doc' from 'ssh-ng://nixworker@alice:65122'...
2 copying path '/nix/store/z81a1f0nx8vxh3nqk2ca20ni231ifk66-zstd-1.5.7-dev' from 'ssh-ng://nixworker@alice:65122'...
2 copying path '/nix/store/z5gkgh4fcpmpb6zbb2qn106qqn4k9z5i-getent-glibc-2.44-25' from 'ssh-ng://nixworker@alice:65122'...
2 copying path '/nix/store/yyl3y8b8lki46n4ynax7jx8n5c4dhp9h-acl-2.4.0-bin' from 'ssh-ng://nixworker@alice:65122'...
2 copying path '/nix/store/ysrcy71m02pi7ab7ianxvd3iwy1mc1iz-iptables-1.8.13' from 'ssh-ng://nixworker@alice:65122'...
2 copying path '/nix/store/yr0n24zyx17fdrs6w565nidljc6i41hz-initrd-linux-7.2.8' from 'ssh-ng://nixworker@alice:65122'...
2 copying path '/nix/store/yqjmkk5malhva1482a5hkx6ljg1r14h6-lvm2-2.03.41-man' from 'ssh-ng://nixworker@alice:65122'...
2 copying path '/nix/store/ydk4myz6rmvxcq581k8rghhkd2riq2wn-e2fsprogs-1.47.4-man' from 'ssh-ng://nixworker@alice:65122'...
2 copying path '/nix/store/y7ifc7m8wn1xz1qjs9nxpycvxxz7cf6g-glibc-locales-2.44-25' from 'ssh-ng://nixworker@alice:65122'...
2 copying path '/nix/store/xl2hs646ww061x2g3kcpxx05rc0xwf7a-kmod-debian-aliases.conf-30+20230601-2' from 'ssh-ng://nixworker@alice:65122'...
2 copying path '/nix/store/x6di552jc5bx438x5n4j3vv6p04v6ns7-gnutar-1.35-info' from 'ssh-ng://nixworker@alice:65122'...
(it goes on for much longer)
Nix config (/etc/nix/nix.conf)
Alice
# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix. Do not edit it!
allowed-users = *
auto-optimise-store = true
builders =
connect-timeout = 5
cores = 0
experimental-features = nix-command flakes
fallback = true
fsync-metadata = true
fsync-store-paths = true
keep-derivations = true
keep-outputs = true
max-free = 10737418240
max-jobs = auto
min-free = 5368709120
require-sigs = true
sandbox = true
sandbox-fallback = false
secret-key-files = /persist/manual/nix/nix-cache-key.sec
substitute = false
substituters =
system-features = gccarch-znver4 gccarch-skylake gccarch-kabylake gccarch-x86-64-v3 gccarch-x86-64-v4 benchmark big-parallel kvm nixos-test
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
trusted-substituters =
trusted-users = root @wheel nixworker
warn-dirty = false
extra-sandbox-paths =
Bob
# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix. Do not edit it!
allowed-users = *
auto-optimise-store = true
connect-timeout = 5
cores = 0
experimental-features = nix-command flakes
fallback = true
fsync-metadata = true
fsync-store-paths = true
keep-derivations = true
keep-outputs = true
max-free = 10737418240
max-jobs = auto
min-free = 5368709120
require-sigs = true
sandbox = true
sandbox-fallback = false
secret-key-files = /persist/manual/nix/nix-cache-key.sec
substitute = true
substituters = ssh-ng://nixworker@alice:65122?compress=true&ssh-key=/persist/manual/ssh/root/keys/id_ed25519&remote-store=local?root=/.disk/root
system-features = gccarch-znver4 gccarch-znver5 gccarch-skylake gccarch-kabylake gccarch-x86-64-v3 gccarch-x86-64-v4 benchmark big-parallel kvm nixos-test
trusted-public-keys = alice:<key> bob:<key>
trusted-substituters =
trusted-users = root @wheel nixworker
warn-dirty = false
extra-platforms = aarch64-linux riscv64-linux armv6l-linux i686-linux
extra-sandbox-paths = /run/binfmt /nix/store/7ld8231szz5qjc3596ifckajry4zl1ba-qemu-aarch64-binfmt-P /nix/store/1ni08gvnssi3insx2p4japi90kvx4rx7-qemu-riscv64-binfmt-P /nix/store/s2jz0c5w1hrzi4icbzysix5803lgsim8-qemu-arm-binfmt-P
Why are you using a chroot store
I use a chroot store on that virtual machine because replacing the ISO with a new updated build means a bunch of paths in the nix store will cease to exist, and the overlay store does not like it when paths disappear from the lower store. So I use a chroot store to the ext4 virtual disk whereas the rest of the filesystem is from the ISO.
Is your hardware failing
Both Alice and Bob are running ECC ram and ZFS (Alice’s ext4 virtual disk is a zfs zvol). Neither of them are reporting any issues.