Missing Nix Store Entries After Running Out of Space During Rebuild

I ran out of storage on my drive during a rebuild and after that have been getting the error below when rebuilding, and the rebuild is unable to complete.

error: opening file '/nix/store/1dj4wl6lfhn8ljwhi704izmx6yx532kf-ncurses-abi5-compat-6.4.20221231.drv': No such file or directory

During nix-store --optimise I get a similar error, but for a different entry.

error: getting status of '/nix/store/0wa768y50d3qmifg4k8xm0zlag5x42jv-9157b1282d392b2cc220bafa44b656d6dac311df.patch.drv': No such file or directory

I don’t know what either entry is for, but there definitely are no such files at those paths.

Any help would be appreciated.
Unfortunately I realised AFTER trying to fix it with nix-collect garbage -d (it seems my automatic garbage collect wasn’t working, hence the lack of storage space) that rolling back would have probably worked, although I don’t know why my nixos-rebuild --switch would have switched if it ran out of space and failed.

Have you tried

nix-store --verify --check-contents --repair
1 Like

Looks like it didn’t work, but it might at least reveal every missing entry.

reading the Nix store...
checking path existence...
path '/nix/store/0wa768y50d3qmifg4k8xm0zlag5x42jv-9157b1282d392b2cc220bafa44b656d6dac311df.patch.drv' disappeared, but it still has valid referrers!
warning: cannot repair path '/nix/store/0wa768y50d3qmifg4k8xm0zlag5x42jv-9157b1282d392b2cc220bafa44b656d6dac311df.patch.drv'
path '/nix/store/1dj4wl6lfhn8ljwhi704izmx6yx532kf-ncurses-abi5-compat-6.4.20221231.drv' disappeared, but it still has valid referrers!
warning: cannot repair path '/nix/store/1dj4wl6lfhn8ljwhi704izmx6yx532kf-ncurses-abi5-compat-6.4.20221231.drv'
path '/nix/store/x3ajawy47mng576v28idly5q15rn3z6s-mitm-cache-0.1.1-vendor.tar.gz.drv' disappeared, but it still has valid referrers!
warning: cannot repair path '/nix/store/x3ajawy47mng576v28idly5q15rn3z6s-mitm-cache-0.1.1-vendor.tar.gz.drv'
path '/nix/store/2kl9dp0lkah1cvdcfy6ki7kgvphjp0g3-source.drv' disappeared, but it still has valid referrers!
warning: cannot repair path '/nix/store/2kl9dp0lkah1cvdcfy6ki7kgvphjp0g3-source.drv'
path '/nix/store/44wza69x3nllf4pczv46j5kfwbng700h-pps-tools-1.0.3.drv' disappeared, but it still has valid referrers!
warning: cannot repair path '/nix/store/44wza69x3nllf4pczv46j5kfwbng700h-pps-tools-1.0.3.drv'
path '/nix/store/68lvpry6szp7bwzzy5f1545m9bc9ap5v-baloo-6.5.0.tar.xz.drv' disappeared, but it still has valid referrers!
warning: cannot repair path '/nix/store/68lvpry6szp7bwzzy5f1545m9bc9ap5v-baloo-6.5.0.tar.xz.drv'
path '/nix/store/hcc849ix8df67cagv6xi20g0wv2493nz-lsof-4.99.3.drv' disappeared, but it still has valid referrers!
warning: cannot repair path '/nix/store/hcc849ix8df67cagv6xi20g0wv2493nz-lsof-4.99.3.drv'
path '/nix/store/829pfa72mzfi9j2harbbvl5gaq7bkp79-source.drv' disappeared, removing from database...
error: executing SQLite statement 'delete from ValidPaths where path = '/nix/store/829pfa72mzfi9j2harbbvl5gaq7bkp79-source.drv';': constraint failed, FOREIGN KEY constraint failed (in '/nix/var/nix/db/db.sqlite')

Oh no. I’m surprised it’s possible for sqlite to corrupt itself like this due to out-of-space.

1 Like

I wonder if my backup software Vorta was adding a bunch of cache to my disk at the same time. But I know the rebuild stopped and said it ran out of disk space.

Well there is also a brute-force way to fix things:

  1. Do a backup
  2. Boot from a NixOS live iso
  3. Mount your old /nix/store and wipe it clean
  4. Follow the steps for installing (without the disk-formatting) using your existing config
1 Like

I will definitely do that soon.

More info:

sudo nix-store --delete /nix/store/0wa768y50d3qmifg4k8xm0zlag5x42jv-9157b1282d392b2cc220bafa44b656d6dac311df.patch.drv
finding garbage collector roots...
removing stale link from '/nix/var/nix/gcroots/auto/30n1vd9dqzhpyyg1mr3msf5h4kk5smgx' to '/tmp/nix-build-507963-0/result'
removing stale link from '/nix/var/nix/gcroots/auto/04i8amy0gdbxyij08z3mg3rhnxw319lg' to '/tmp/nixos-rebuild.m0J125/nix.drv'
removing stale link from '/nix/var/nix/gcroots/auto/lazx3vshxmv1ly8zhzb2kb49kwiaksw7' to '/tmp/nixos-rebuild.m0J125/nix'
removing stale link from '/nix/var/nix/gcroots/auto/zp3ps368ik6m61cmf418j0gzhizv8k25' to '/tmp/nix-build-508600-0/result'
removing stale link from '/nix/var/nix/gcroots/auto/7ijsrmk5sdp3khbzwslykhsnl3k0y5zj' to '/tmp/nixos-rebuild.FWLo6t/nix.drv'
removing stale link from '/nix/var/nix/gcroots/auto/8hyvgpkh4w552f7832vjm98lb4jf8gwh' to '/tmp/nixos-rebuild.FWLo6t/nix'
0 store paths deleted, 0.00 MiB freed
error: Cannot delete path '/nix/store/0wa768y50d3qmifg4k8xm0zlag5x42jv-9157b1282d392b2cc220bafa44b656d6dac311df.patch.drv' since it is still alive. To find out why, use: nix-store --query --roots
sudo nix-store --query --roots /nix/store/0wa768y50d3qmifg4k8xm0zlag5x42jv-9157b1282d392b2cc220bafa44b656d6dac311df.patch.drv
/proc/519840/environ -> /nix/store/0wa768y50d3qmifg4k8xm0zlag5x42jv-9157b1282d392b2cc220bafa4>
/proc/519842/environ -> /nix/store/0wa768y50d3qmifg4k8xm0zlag5x42jv-9157b1282d392b2cc220bafa4>

Try actually logging in as root to run nix-store --query-roots. There’s a very frustrating behavior where using sudo nix-store ... puts the path in a couple of PIDs environments and stuff like that, which creates red herrings.

1 Like

It did something:

nix-store --query --roots /nix/store/1dj4wl6lfhn8ljwhi704izmx6yx532kf-ncurses-abi5-compat-6.4.20221231.drv
removing stale link from '/nix/var/nix/gcroots/auto/8l9izzzak2x87y2cq8qclgaf7z2bgkpd' to '/tmp/nix-build-532360-0/result'
removing stale link from '/nix/var/nix/gcroots/auto/gq8n704kl27m68kfyabm6sywjkdsm309' to '/tmp/nixos-rebuild.CJNSzd/nix.drv'
removing stale link from '/nix/var/nix/gcroots/auto/vd73vk5c0ijzf2w93k9pawjsh1c3rphj' to '/tmp/nixos-rebuild.CJNSzd/nix'
removing stale link from '/nix/var/nix/gcroots/auto/2amy59z51vads849lgi0m9kffmwhvpgn' to '/tmp/nix-build-534066-0/result'
removing stale link from '/nix/var/nix/gcroots/auto/xpb2d067qyzsb6c2k1zgl5if8gk43hs7' to '/tmp/nixos-rebuild.wJd4Qs/nix.drv'
removing stale link from '/nix/var/nix/gcroots/auto/fn1jh8zlqg6kch0mh8diz5z7287xacwx' to '/tmp/nixos-rebuild.wJd4Qs/nix'
removing stale link from '/nix/var/nix/gcroots/auto/kn59fdvxz97r6m5mg0m9hya6cvs664h7' to '/tmp/nix-build-534882-0/result'
removing stale link from '/nix/var/nix/gcroots/auto/226k551j3pljjlc4pw7xslk6xxbrn47a' to '/tmp/nixos-rebuild.b7yIJR/nix.drv'
removing stale link from '/nix/var/nix/gcroots/auto/pgnvksjzgl7fsfzxrnspycrbr4cs4aax' to '/tmp/nixos-rebuild.b7yIJR/nix'

But I’m still getting problems with that exact entry

nixos-rebuild boot
building Nix...
building the system configuration...
trace: evaluation warning: The cfg.enablePlasmaBrowserIntegration argument for `firefox.override` is deprecated, please add `pkgs.plasma-browser-integration` to `nativeMessagingHosts` instead
error: opening file '/nix/store/1dj4wl6lfhn8ljwhi704izmx6yx532kf-ncurses-abi5-compat-6.4.20221231.drv': No such file or directory

Well, yea, I didn’t mean to say that would fix anything. I was just saying it would reveal that those supposed roots were actually red herrings. It means the file you’re trying to find the roots of doesn’t have any. It’s probably kept alive by another path referring to it that also doesn’t have a root.

And anyway, like I said, your Nix sqlite DB is corrupted. It has a path registered as valid even though the file doesn’t exist. You might be lucky enough that figuring out how to delete that path will fix things, but I don’t think I’d trust a DB that was once corrupted to no longer be corrupted after that.

2 Likes

I did a terrible, very bad thing when this happened to me due to WSL2 falling over due to a lack of disk space (and bad garbage collection by MS’ disk image format). I remounted the store rw, ran nix-collect-garbage in a loop, parsing out and creating the missing drv file from the error message every time one was missing. Eventually it was able to delete everything and I ran a nixos-rebuild switch --repair to make sure the system GC root was good.

If this was a system I could have “normally” rebooted into I would have just obliterated the store and started over, but it was WSL2…