/nix/store** no such file or directory when using flake from local filesystem

Help
1

Hey guys, since a couple of days I have this strange problem with my NixOS installation:

I cannot run anything that evaluates flakes (that includes stuff like nix flake check nix flake build nixos-rebuild --flake etc. etc…) when the target flake is on my local filesystem. When the same flake is a remote URL it works, see below for more info.

The error I get is always the same (hash changes of course from flake to flake):

error: getting status of ‘/nix/store/bhfv04vznm0hkyjjdidfbjlyid5bfkjv-source’: No such file or directory

I know that similar problems were already posted multiple times, here and on StackOverflow, and contain solutions, in particular this one: Nix flakes /nix/store/***-source no such file or directory.

Suggestions include running garbage collection followed by nix-store --repair-path … and/or nix-store --verify --check-paths --repair. Those suggestions do not help at all in my case, but I used them in the past to fix a similar problem and indeed they helped (concretely the nix-store-one).

So what’s different this time?
A - solutions do not work as mentioned, B - if I build the flake from remote location, it works without problems which confuse the hell out of me. So for example below does not work, fails with the mentioned error:

nixos-rebuild boot --flake .#m3800

While this works without problems (commits on both git repositories are the same):

nixos-rebuild boot --flake github:konradmalik/dotfiles#m3800

Flake that fails is my nix config repo on github, but I don’t think it’s relevant, since the error applies to all flakes I have locally.

Couple of (strange) facts about this whole situation:

  • my flake.lock does not reference local paths like in the linked similar issue
  • the git repo I have locally is of course the same as the remote one, it’s the same commit
  • removing local repo and cloning again did not help
  • cloning to another directory did not help
  • this same flake works without problems on 3 other nixos systems I have, with the exception that this problematic one is a desktop (contains WM, graphical apps etc.) while others are server (just ssh access and terminal apps).
  • this same flake works without problems on my MacBook pro via nix-darwin
  • by “the same flake” I mean mostly the system/nix configuration, because as I said, nix flake check and the rest do not work on all flakes, not just my github repo
  • when I build the flake from the remote path and then try to use the local one it will work (so it will properly create the ‘source’ folder in nix store), but as soon as I change anything, like commit some changes, it stops working until I again build it directly from the remote URL.
  • as it turns out, some flakes fail even when provided with a remote URL. My company’s flake accessed over git+ssh fails with the exact same source-missing error whether it’s being run from local or from remote
  • “in the heat of debugging” the suggestion to try nix-store --verify-path /nix/store/bhfv04vznm0hkyjjdidfbjlyid5bfkjv-source worked once (it downloaded the source from my cachix url) but then the build threw another error about missing store path (‘files’ this time which is just my local package with my dotfiles), this one could not be fixed at all and since then, I’m not able to fix any such error with nix-store --verify-path
  • when I run nix-collect-garbage -d after trying to build the flake from local path, it will remove the lock file for that same source folder that it says is missing (that’s reproducible):
sudo nix-collect-garbage -d
removing old generations of profile /nix/var/nix/profiles/per-user/root/channels
removing old generations of profile /nix/var/nix/profiles/per-user/konrad/home-manager
removing old generations of profile /nix/var/nix/profiles/system
finding garbage collector roots...
deleting garbage...
deleting '/nix/store/bhfv04vznm0hkyjjdidfbjlyid5bfkjv-source.lock'
deleting unused links...
note: currently hard linking saves 2939.27 MiB
1 store paths deleted, 0.00 MiB freed
  • as mentioned, nix-store --verify --check-paths --repair does not fix anything, nor it reports any errors:
reading the Nix store...
checking path existence...
checking link hashes...
checking store hashes...

Any ideas? I’m extremely confused.

1 Like
2

Ok to this is solved I think. The solution… was not the prettiest.

I could not figure out what else could be broken than the /nix/store, so I realized that probably there is something wrong with SQLite db. Then I could not figure out how to fix it (if it’s even possible since it’s the ground truth?) so I just deleted the db folder, then I ran nix-collect-garbage which effectively purged my whole /nix/store. After that, I just inserted nix ISO, mounted my disks, and ran nixos-install from the flake (I needed to add --no-bootloader, otherwise it failed, I think it’s because it tries to install grub, while my flake already has systemd defined).
Then reboot and… so far so good. I’m still wondering what could I’ve done to corrupt SQLite database (if that was indeed the case).

3

What has gone wrong is kind of hard to know, but if the database was indeed corrupted, then there can be lots of reasons for it; fast system shutdown during transactions, perhaps disk drive issues, and I am fairly sure that there at least used to be some issue with having the database on a NFS disk and using wal files.

One thing to note for the future is that you might be able to recover data from the database with the .recover command (if you’re lucky):

> nix run nixpkgs#sqlite <DATABASE> ".recover" > recovered.sqlite

and then swap in recovered.sqlite

2 Likes
4

So… after one day this started happening again, on the exact same machine. Same exact problem.
Any ideas how to debug this?

5

Do you have any way to verify that your file system isn’t in a corrupted state of some kind or running on malfunctioning storage? fsck, zpool scrub, etc? It could be that the way Nix copies local flakes to the store triggers some bad behavior with a malfunctioning file system.

Seems incredibly unlikely, but that’s my only guess at the moment.

Also, what version of Nix is this? Have you tried any others?

1 Like
6

I also don’t have an idea. You might try using the nix store verify command as an additional debugging tool.

https://nixos.org/manual/nix/unstable/command-ref/new-cli/nix3-store-verify.html

I’m not super familiar with the overlap between it and the nix-store --verify command you’re already using.

7

So I remembered one strange thing today that happened after I removed the db: immediately after calling nixos-rebuild, the garbage collector kicked in.
Then I remembered that I set ‘min-free ~ 50GB’ and ‘max-free ~ 100GB’ nix settings.
Then I checked my free space, and it was ~ 44GB (was downloading a running a lot of docker containers recently
Then it all clicked :man_facepalming:

I changed ‘min-free’ to ~ 10 GB instead and it all works as it should be now (fingers crossed).

My fault of course, but… I still don’t know why flakes from remote URLs worked. There could be some better errors as well because from those above not much could be concluded.

8

@konradmalik I would still consider that a bug in Nix. It shouldn’t be GC’ing a flake that the current evaluation is using.