How to recover the system after a bad "nix-store --delete --ignore-liveness"

I tried to delete the single package /nix/store/z58hjqhwggw8jvmg90y3ib2lk7vkp2x0-neovim-unwrapped-0.4.4 with the command:

sudo nix-store --delete --ignore-liveness /nix/store/z58hjqhwggw8jvmg90y3ib2lk7vkp2x0-neovim-unwrapped-0.4.4

with the following result:

sudo nix-store --delete --ignore-liveness /nix/store/z58hjqhwggw8jvmg90y3ib2lk7vkp2x0-neovim-unwrapped-0.4.4
finding garbage collector roots...
deleting '/nix/store/b6pqh1wdyzlwf5rilc0b39iphcjfalvj-nixos-system-max-nixos-20.09.4348.86d3781c390'
deleting '/nix/store/l0fwizd2lyza8gp5ckrgb42ngrssw981-etc'
deleting '/nix/store/2bpbvhnhbvhadrrkj9qh3dhkl3s7j7fr-system-units'
deleting '/nix/store/5lrbsp54v8kp6hx1301swgnqpy2bchvv-nixos-system-max-nixos-20.09.4348.86d3781c390'
deleting '/nix/store/6p79rb8hjijn8pwdir6ran133gcahkfy-etc'
deleting '/nix/store/b8ldgvs791c70dm1n10qv4p53mqv792a-system-units'
deleting '/nix/store/l4z38ymcy35vcvmp6qp6kag92z937dp7-nixos-system-max-nixos-20.09.4348.86d3781c390'
deleting '/nix/store/nrqn3dsmax9jk5wlcc14gj7vxc27a55k-etc'
deleting '/nix/store/qp7svs76vbdz443gxd6m4mkwdrlylxfb-system-units'
deleting '/nix/store/1iiqr0dvyk5fnqkpdwabl0070208dwyi-unit-polkit.service'
deleting '/nix/store/dyaag8sss0knqkip7rhmjc4av6chiyvv-unit-systemd-fsck-.service'
deleting '/nix/store/84nswx9xnp7rycjrp0vpa4ynw3jdna5h-unit-dbus.service'
deleting '/nix/store/6yir6w3j505cpwqm36ipicck4zw6c025-user-units'
deleting '/nix/store/xyzc1cx87iw1cgr4r13450si6vwaj4w0-unit-dbus.service'
deleting '/nix/store/g62w17g691drcfgkrgpy0sjh0c9d72xr-dbus-1'
deleting '/nix/store/22b6cmljmsz97w205v4c3cwwgqf17dg8-system-path'
deleting '/nix/store/yg1g6qcc93pdi9vjc87p9wyd6glbzdzp-nixos-system-max-nixos-20.09.4350.05b1f3f0c9d'
deleting '/nix/store/wfynwssr94f8508qkq79ag7b5zk576s8-etc'
deleting '/nix/store/rgy89nr1mi99drx5qamax9gw66v3698x-system-units'
deleting '/nix/store/gk0d9pj0gmb518hzhrf577qgxbvsivm7-unit-dbus.service'
deleting '/nix/store/2bgz6hv8m6ixawbmf7agy6pksd9q2zij-user-units'
deleting '/nix/store/ndc47fh1lz21ghbyxc0splfkr1r58d10-unit-dbus.service'
deleting '/nix/store/37c97jwpalryf61jki117z0grjvsdxxq-dbus-1'
deleting '/nix/store/k82zd4shkrb7jdgj1hqvdmnbil9cmc4a-unit-polkit.service'
deleting '/nix/store/xh6csdfyg85p2x6q0na6q8yv5vc7s0mf-unit-systemd-fsck-.service'
deleting '/nix/store/a9jcirh4jfh46riy71s2z1vj1b1vblgh-system-path'
deleting '/nix/store/q1qgi4af07kxciq7wnzbhjyki05h079b-nixos-system-max-nixos-20.09.4350.05b1f3f0c9d'
deleting '/nix/store/21cr49arfdd238p7izizl8dp9nlvgaxq-etc'
deleting '/nix/store/7r8pbnq5z4rskkda3zafz09ky7svg9i1-system-units'
deleting '/nix/store/fqg86w3jfj32q7syrg7047hzwxwr25gz-unit-polkit.service'
deleting '/nix/store/fvx9wf9r0hgvnj5rynr5gy8g0dpq0g5c-unit-systemd-fsck-.service'
deleting '/nix/store/wwp9r38cwy1p979aphqmwp6z53sdvqgz-user-units'
deleting '/nix/store/4ss17nh5qp5gz61iy0dp13l29hqi63fb-unit-dbus.service'
deleting '/nix/store/b6h7v8gfd9zwzpa1a3zds3xrzxgnj50a-unit-dbus.service'
deleting '/nix/store/nsl08sy8lqsg6c6l5gy7rjsiy86knkqc-dbus-1'
deleting '/nix/store/iiard8zkyhh6hs4rv0r0qyhz08s1v5v2-system-path'
deleting '/nix/store/4mxrzq76541sinx0m7sg50fin98717dq-nixos-system-max-nixos-20.09.4346.d6a5be86b6a'
deleting '/nix/store/nxdb3z14b16nxpdlb7hjfb9hcl49cqzs-etc'
deleting '/nix/store/8m6kjlwygjdg8p9vnbznfyvapfc0afs0-system-units'
deleting '/nix/store/8gmxkzrfdfqnchy5562yq66sq3fm8g6p-unit-systemd-fsck-.service'
deleting '/nix/store/hqqvwqjix26fkwfaii1zmbmndmvlmcaf-unit-polkit.service'
deleting '/nix/store/za8y4cm8dcz42rvjmnhhlib4iwipq6yh-user-units'
deleting '/nix/store/y1fmfzl400mmfvixcbr96p7bsrhn1hh8-unit-dbus.service'
deleting '/nix/store/yazaxfqm36yyhnmqawq7rwy1rx2zkg04-unit-dbus.service'
deleting '/nix/store/kvl9spn3a20f9py5lqx306vmhknk1k4q-dbus-1'
deleting '/nix/store/jqm97aw62k0j15srqslfscffv4adr5yp-system-path'
deleting '/nix/store/nmpglbwvhhi9cx7x4pmf5v5qb54zc3jb-nixos-system-max-nixos-20.09.4350.05b1f3f0c9d'
deleting '/nix/store/wdcwh28cxwfm6czwl12gl4p3qb7hzv6f-etc'
deleting '/nix/store/16d6jwv4a20dm8v92nn8k7yci5g4rgxf-system-units'
deleting '/nix/store/3vms48zs7izcj59svyyk3q837ral7yvn-unit-polkit.service'
deleting '/nix/store/6cfgyg8llrpmzak3hpclz756fzsb1wqs-unit-dbus.service'
deleting '/nix/store/gb2227nrrmfy9w0r1z7lbypn4600bgr1-user-units'
deleting '/nix/store/bqjr0bd9mdciwhk2ybzjfma4mqdckyv7-unit-dbus.service'
deleting '/nix/store/gq6jzh0llmplq8cjyg87af92h19vvcxw-dbus-1'
deleting '/nix/store/xbq49w0ghjgsi6xx0yaq1lph1jh6anan-unit-systemd-fsck-.service'
deleting '/nix/store/mx2yf1ip2lbdx8fpl90j0xycyd454pn7-system-path'
deleting '/nix/store/dzbw6rg40n8dm0pnj952q5n16w3chznc-nixos-system-max-nixos-20.09.4350.05b1f3f0c9d'
deleting '/nix/store/m0iq67z5xqyqz2fvgmsvda7k5bzfvr01-etc'
deleting '/nix/store/10b7i8lwsabwb4j8n4i8xrxrmcq9ixgz-system-units'
deleting '/nix/store/hn5qplw9pj2pqlkxkv1r82pxdyf0qgqf-unit-polkit.service'
deleting '/nix/store/nr29z1a1w8qhwk191s67lm57ksjic1c0-unit-dbus.service'
deleting '/nix/store/9kvpw0gmwzcsldfzfjdycsg70fazgvm2-user-units'
deleting '/nix/store/x3dwxriwr80swcw57kn62yn2ylgxj702-unit-dbus.service'
deleting '/nix/store/nalrpvxvh73g7wfv5p3hqv39r4lkr2gv-dbus-1'
deleting '/nix/store/wq40f1326xyz3jg4gppfxrzpqzc0h4ix-unit-systemd-fsck-.service'
deleting '/nix/store/bzkbdqsr8h7x35zbp9r4j355zw5jr19w-system-path'
deleting '/nix/store/xprqv675sqy4yq7pfdab9851p4j613gc-nixos-system-max-nixos-20.09.4348.86d3781c390'
deleting '/nix/store/mird6gjs3y9wdyxjhs17br19za5c1a6s-etc'
deleting '/nix/store/wh8bxyp1qnj55yys2fd1640ikn4x424m-system-units'
deleting '/nix/store/6jdz92p5r5rcd5j15lczn99217p5k1xc-unit-dbus.service'
deleting '/nix/store/r1dm85677wb9i9y0l70vjpvcljzsnxq3-user-units'
deleting '/nix/store/6vp8rdipqd3p1sll2ml4xnh0vcng1bvp-unit-dbus.service'
deleting '/nix/store/0yz4g1ihbwqq6lzxn91pghryqilbh8m3-dbus-1'
deleting '/nix/store/1sm3j998sgkvjsijl72k7f91an78mspg-unit-systemd-fsck-.service'
deleting '/nix/store/b2niw2plnww5s1dp22bh2qbfwylp0vfw-unit-polkit.service'
deleting '/nix/store/s08qhq290fspyccpl979l0vcmakq0qjp-system-path'
deleting '/nix/store/339qvjgqlnscbamjdy7d0yrdbrx2jjcr-nixos-system-max-nixos-20.09.4350.05b1f3f0c9d'
deleting '/nix/store/hzdyhapw16vfih145vz76p1xsi3b06hi-etc'
deleting '/nix/store/ig24xynp9ajgibk85snv4x9g7c56lmx2-system-units'
deleting '/nix/store/cxsb9cfw779gr7g7h1jr6nnjiidlbxbi-unit-polkit.service'
deleting '/nix/store/d35snh8clkg6ljdb7lixgimc1v2y1xx9-unit-systemd-fsck-.service'
deleting '/nix/store/isfppxl92wbf6cvsbn5qahg0w1mwx9hr-user-units'
deleting '/nix/store/23g8k0vqrfx69fn9wwxyqi2hxvcl98xx-unit-dbus.service'
deleting '/nix/store/l6ajwds1i58xz12z62clwd4wri4v482s-unit-dbus.service'
deleting '/nix/store/xwfkc93w0hz2w4v1cq4v1ikdl2dyv5i8-dbus-1'
deleting '/nix/store/xj664lsm4409sac80rr2kfss62ziqqa5-system-path'
deleting '/nix/store/nwlh9cxv9c6sh7gnvrn9xmzwfcqkc9mf-neovim-qt-0.2.15'
deleting '/nix/store/0f9c06q24slw0inkwc1idi5alh9gi8ks-nixos-system-max-nixos-20.09.4350.05b1f3f0c9d'
deleting '/nix/store/cgsfiz87jda7nn3yklkl1sqq3d999wwm-etc'
deleting '/nix/store/cr996lmj2fl2hxgzdk0fmd2qcjyn50gw-system-units'
deleting '/nix/store/0lg8vhdc6dp5nswrszbp48rng10lcb7c-unit-systemd-fsck-.service'
deleting '/nix/store/3svcbq9z37i8hny2gfmnrnk30b9qaj9v-unit-dbus.service'
deleting '/nix/store/lsih7cs0dqlapxpgx95rpwd9jim6hps2-user-units'
deleting '/nix/store/n8vgrb6kgq07y6mxg6kn3n9b5al1jw8v-unit-dbus.service'
deleting '/nix/store/2ry5205qgv64fl8vc1a556g9wqljifxq-dbus-1'
deleting '/nix/store/bl5g23cwfls7vqiqsnbag9hbpjn3nmx5-unit-polkit.service'
deleting '/nix/store/xhwvcg6w4f7fkk04l63dk873glynlri5-system-path'
deleting '/nix/store/sz9491v96bg971xxalhah64lr5j2iaf9-neovim-0.4.4'
deleting '/nix/store/z58hjqhwggw8jvmg90y3ib2lk7vkp2x0-neovim-unwrapped-0.4.4'
deleting '/nix/store/trash'
deleting unused links...
note: currently hard linking saves -0.00 MiB
99 store paths deleted, 88.83 MiB freed
command not found: date

“Apparently” the whole system is gone:

command not found: whoami
command not found: whoami
command not found: cat

sudo nixos-rebuild switch --upgrade
sudo: unable to initialize PAM: Critical error - immediate abort
command not found: date
  • What can I do to recover the system?
  • I am currently able to use the already opened programs (terminal, browser)
    • I have plenty of old generations
      • but I guess that system is gone for those too (?)

I’d try to run nixos-install from a live-usb with your configuration.nix

1 Like

Well, that is not how you delete packages. Why did you do that?

To salvage the situation you could try to rollback, either using

sudo nix-env --rollback --profile /nix/var/nix/profiles/system

or if that does not work, reboot and select the previous generation at the bootloader.

You said, you are not sure whether you nuked all the old generations as well. You can check which ones still exist by only using bash builtins like this:

$ for profile in /nix/var/nix/profiles/system-*-link; do
>     for entry in "$profile/"*; do
>         echo "$entry"
>     done
> done
2 Likes

Thank you for your reply.

  • I read the manual
    • for what I read ( --delete is a more targeted version of --gc) should be relatively safe ( With the option --ignore-liveness , reachability from the roots is ignored. However, the path still won’t be deleted if there are other paths in the store that refer to it (i.e., depend on it).) command to use
      • most likely I misunderstood its content
  • it was a last try to be sure that I did not have the neovim-unwrapped-0.4.4 package in the system

unfortunately everything I tried from terminal, failed (also, every program I tried to launch, failed).

This basically helped to confirm what should be the first corrupted generation:

zsh: no matches found: /nix/var/nix/profiles/system-310-link/*

The boot menu seems to be intact (all generations are there), I tried a few recent ones and they failed at init stage (not sure what the exact message was). I tried older generations until I have been able to start the system normally.

Summarizing, it seems that now I built a new, working,
generation.

Still, when building, logs accuse:

skipping corrupt system profile entry ‘/nix/var/nix/profiles/system-324-link’
skipping corrupt system profile entry ‘/nix/var/nix/profiles/system-323-link’
skipping corrupt system profile entry ‘/nix/var/nix/profiles/system-321-link’
skipping corrupt system profile entry ‘/nix/var/nix/profiles/system-320-link’
skipping corrupt system profile entry ‘/nix/var/nix/profiles/system-319-link’
skipping corrupt system profile entry ‘/nix/var/nix/profiles/system-318-link’
skipping corrupt system profile entry ‘/nix/var/nix/profiles/system-317-link’
skipping corrupt system profile entry ‘/nix/var/nix/profiles/system-316-link’
skipping corrupt system profile entry ‘/nix/var/nix/profiles/system-315-link’
skipping corrupt system profile entry ‘/nix/var/nix/profiles/system-314-link’
skipping corrupt system profile entry ‘/nix/var/nix/profiles/system-313-link’
skipping corrupt system profile entry ‘/nix/var/nix/profiles/system-312-link’
skipping corrupt system profile entry ‘/nix/var/nix/profiles/system-311-link’
skipping corrupt system profile entry ‘/nix/var/nix/profiles/system-310-link’

basically these seems to be the nuked generations (still, all those subsequent to 303 failed at boot time). How can I get rid of those corrupted profiles the very safe way?

Currently available generations:

 290   2021-05-22 16:36:53   
 291   2021-05-22 18:41:41   
 292   2021-05-23 00:27:55   
 293   2021-05-23 00:37:48   
 294   2021-05-23 03:27:26   
 295   2021-05-24 03:11:26   
 296   2021-05-24 03:32:36   
 297   2021-05-26 19:27:06   
 298   2021-05-26 19:46:54   
 299   2021-05-28 16:22:24   
 300   2021-05-29 04:42:03   
 301   2021-05-30 04:41:31   
 302   2021-06-13 00:30:08   
 303   2021-06-14 04:42:13   
 304   2021-06-14 16:23:03   
 305   2021-06-16 03:42:06   
 306   2021-06-17 02:57:56   
 307   2021-06-17 03:19:14   
 308   2021-06-19 17:37:09   
 309   2021-06-22 01:12:36   
 310   2021-06-24 03:42:44   
 311   2021-06-24 23:59:18   
 312   2021-06-25 00:23:19   
 313   2021-06-25 01:03:58   
 314   2021-06-25 01:08:12   
 315   2021-06-25 01:29:53   
 316   2021-06-27 02:54:08   
 317   2021-06-27 03:02:58   
 318   2021-06-27 03:05:50   
 319   2021-06-27 03:07:25   
 320   2021-06-27 03:08:11   
 321   2021-06-27 03:10:22   
 322   2021-06-27 03:11:40   
 323   2021-06-27 03:22:40   
 324   2021-06-27 03:24:46   
 325   2021-06-27 03:31:16   
 326   2021-06-27 03:34:37   
 327   2021-06-27 08:35:38   
 328   2021-06-27 08:49:45   (current)

Actually curious to know why this happened.

Thank you for your reply.

Fortunately, it seems that I have been able to avoid the live-usb chore.

To remove packages, simply remove it from environment.systemPackages (or run nix-env -e if you installed via nix-env), rebuild the system and then run nix-collect-garbage -d. Be aware that this deletes all generations but the current one, so if the current one is broken you can nuke your way to rollback. Generally it’s better to run nix-collect-garbage --delete-older-than 10d or so to only delete generations older than 10 days.

You can delete old generations using

nix-env -p /nix/var/nix/profiles/system --delete-generations 310 311 312 ...

If that does not work for some reason, the directories /nix/var/nix/profiles/system-*-link are just symlinks. Delete the ones that are broken and run nix-collect-garbage to clean up the dangling store paths.

1 Like

First of all, the good news.

thanks for your painstaking help. This is what I did (fingers crossed way) and seems that everything went well.

thankfully I did not do either one of these two things (including nix-collect-garbage --delete-older-than 10d or too reduced time period):

the first generation that I could successfully log into was more than one month old, 30 or so generations back.

I can say that the generation features saved the day, still worked in this case only because I had plenty and relatively old alternatives.

I’d like to further investigate this event, even if at the moment I don’t know where to start .

Especially how a command described in the manual as

--delete deletes the store paths paths from the Nix store, but only if it is safe to do so…

Thus, --delete is a more targeted version of --gc .

(adding the --ignore-liveness detail, of course) could wipe tens of generations instead of the single targeted package.

Yes, indeed. --ignore-liveness is extremely unsafe. The manpage reads

With the option --ignore-liveness, reachability from the roots is ignored. However, the path still won’t be deleted if there are other paths in the store that refer to it (i.e., depend on it).

So even after nix-store --delete --ignore-liveness the path should still exist on disk. After checking this myself, I found that this is not the case. Instead Nix also deletes every path that depends on the path to be deleted.

I’ve opened an issue regarding the inconsistent manpage description:

https://github.com/NixOS/nix/issues/4964

1 Like