Unable to GC store after DB fail because of full disk

Help
1

Hi folks,

I made a mistake in building something on NixOS that filled the whole disk. After that, I ran nix store gc which was another mistake since I got an output that there was data to remove but I also got an SQLite error due to full disk.

I solved that by removing systemd journal logs and running nix store gc and thankfully I didn’t break anything and the NixOS instance still works just fine (thank you, developers!!!).

I have 2 almost identical NixOS instances. Looking at the one that had this problem it’s clear that some Nix paths were supposed to be removed but they were not deleted. In other words both NixOS instances have almost identical generations, but the other NixOS instance has less data in /nix even after running nix store gc on both of them.

Could you please tell me, if is it possible to properly GC the first one too? Or maybe somehow fix the DB?

Thank you.

2

If you’re not getting actual errors, it’s unlikely for the DB to be corrupt.

You should rather be looking for gcroots keeping some path alive that you don’t actually need anymore.

I have

alias nix-stray-roots='nix-store --gc --print-roots | grep -vE "^(/nix/var|/run/\w+-system|\{memory|\{censored|/proc/maps/)"'

for this purpose.

1 Like
3

Thank you for the answer!

I’m not getting DB errors. I can rebuild the system without problem.

I’ve just run alias nix-stray-roots='nix-store --gc --print-roots | grep -vE "^(/nix/var|/run/\w+-system|\{memory|\{censored|/proc/maps/)"' and I get a large number of paths. 1831 on the broken NixOS and 1821 on the other one. Should I compare the instances and run nix store delete to remove paths that don’t need to be kept?

4

I’m a bit confused what you mean. That command should output a few dozen entries at max, I’m not sure where you are getting these numbers from.

nix store delete has nothing to do with this. You remove the gcroot and then run a regular gc.

5

On my NixOS 24.05 the grep needs a change. For example:
nix-store --gc --print-roots | grep -E '^/proc/[0-9]+/maps' | wc -l
1432

since I’m getting a lot of paths like:
/proc/9990/maps -> /nix/store/xczcfjk4fxqf5svr4nm4w911pzilpvsa-php-fileinfo-8.2.23

6

Maybe like this:
nix-store --gc --print-roots | grep -vE "^(/nix/var|/run/\w+-system|\{memory|\{censored|/proc/)"
and I get only two paths:

/root/.cache/nix/flake-registry.json -> /nix/store/5bs7lbd1fk22w7bzdd8d6fvysnyzgw35-flake-registry.json
/root/.local/state/home-manager/gcroots/current-home -> /nix/store/2sc1bqflwmd49zwnvxggk4syljgwqagz-home-manager-generation
7

Ah, you’re running this as root, that’s why you’re getting more output that my regex doesn’t account for.

Run the command as your regular user on both machines; that’s where gcroots are likely to have been made.

8

Thank you very much @Atemu and appologies for the delay, for some reason I missed your response.

Running as a normal user on the broken VM:

nix-store --gc --print-roots | grep -vE "^(/nix/var|/run/\w+-system|\{memory|\{censored|/proc/maps/)"
/home/user1/.local/state/home-manager/gcroots/current-home -> /nix/store/jl13v25w5j07x0g8i1mf3dlhrbbkkz4s-home-manager-generation
/home/user1/.local/state/nix/profiles/home-manager-1-link -> /nix/store/mzshmbqmz2vyr12w8wg9as9mcv3aq5jn-home-manager-generation
/home/user1/.local/state/nix/profiles/home-manager-2-link -> /nix/store/jl13v25w5j07x0g8i1mf3dlhrbbkkz4s-home-manager-generation
/home/user2/.local/state/home-manager/gcroots/current-home -> /nix/store/zkp01aafxq21fbi47qp34mmpsns072qp-home-manager-generation
/home/user2/.local/state/nix/profiles/home-manager-1-link -> /nix/store/sskp69k42icsxlzbjnyllb7n1n4qvxnp-home-manager-generation
/home/user2/.local/state/nix/profiles/home-manager-2-link -> /nix/store/zkp01aafxq21fbi47qp34mmpsns072qp-home-manager-generation
/home/user3/.local/state/home-manager/gcroots/current-home -> /nix/store/qaaxqvn3z65s31d5pbscharb8f2zgna7-home-manager-generation
/home/user3/.local/state/nix/profiles/home-manager-1-link -> /nix/store/qaaxqvn3z65s31d5pbscharb8f2zgna7-home-manager-generation
/root/.cache/nix/flake-registry.json -> /nix/store/5bs7lbd1fk22w7bzdd8d6fvysnyzgw35-flake-registry.json
/root/.local/state/home-manager/gcroots/current-home -> /nix/store/xpml3777kwmj88yg12zwwlyn03bq9sp8-home-manager-generation

Running as a normal user onn the other VM:

nix-store --gc --print-roots | grep -vE "^(/nix/var|/run/\w+-system|\{memory|\{censored|/proc/maps/)"
/root/.cache/nix/flake-registry.json -> /nix/store/5bs7lbd1fk22w7bzdd8d6fvysnyzgw35-flake-registry.json
/root/.local/state/home-manager/gcroots/current-home -> /nix/store/2sc1bqflwmd49zwnvxggk4syljgwqagz-home-manager-generation

Is it safe to (somehow?) delete the /home/*/.local/state/home-manager/gcroots? And then run nix store gc again?

Thank you!

9

Not at all safe.

Please just delete the old generation(s) by deleting the non-current profile symlink (not the gcroot).

The gcroot will be cleaned up automatically.