System ran out of inodes, causing nix-collect-garbage to fail. How do I fix this?

Help
1

I’ve got a raspberry pi, where I misconfigured the periodic garbage collect, and no garbage has been taken out for a few months.

Now /nix/store is occupying 100% of the available inodes:

Filesystem      Inodes   IUsed  IFree IUse% Mounted on
...
/dev/mmcblk0p2 1888768 1888768      0  100% /
/dev/mmcblk0p2 1888768 1888768      0  100% /nix/store
...

This is causing all sorts of chaos. Most importantly, nix commands fail, including nix-collect-garbage:

# nix-collect-garbage --delete-old
removing old generations of profile /nix/var/nix/profiles/per-user/root/channels
error: opening lock file '/nix/var/nix/profiles/per-user/root/channels.lock': No space left on device

Rebooting, and, e.g., deleting the content of /tmp changed the symptoms a bit, but did not succeed in fixing the problem.

Apparently I need to delete a few /nix/store paths before I can execute nix-collect-garbage again. /nix/store is mounted read-only though, and I don’t know enough about the inner workings of nix to figure out whether deleting random store paths manually is a good idea anyways.

Can somebody help me figure out what my options are, besides reinstalling the system from scratch?

1 Like
2

You could try pointing nix-store --delete at paths that look like they’d be GC-able (not quite sure how to determine this, worst-case you could write a script to iterate through all of them for you). This should be safe, because that command won’t let you delete anything that’s still live. Either way, if you have a backup it may be worth trying to save time over a reinstall.

1 Like
3

items in /nix/store/trash are usually safe to clean up

3 Likes
4

Apparently there are a few service units which shortly after freeing a few inodes occupy them again. Executing rm -rf /tmp/* && sudo nix-collect-garbage --delete-old worked, and I can now work with the system again. Thanks for all your suggestions!

1 Like