So I was doing a nixos-rebuild switch --upgrade and in the middle of that operation, my laptop decided that 15% battery level was too low to go on, and promptly turned itself off.
I figured no problem, this is NixOS, I’ll just run the command again.
However, I am getting the following error:
{23:17}~ ➭ nixos-rebuild switch --upgrade
unpacking channels...
error: syntax error, unexpected $end, at /nix/store/scd85qqipci9mf80vi42xg547x7wqr4z-env-manifest.nix:1:1
error: program '/nix/store/1hxgccnj93j33xap44v1dzzb24ly1m2r-nix-2.3.7/bin/nix-env' failed with exit code 1
I think this should rollback your env to point to previous generation so the corrupted files won’t be in the current generation. Then you can garbage collect them and rebuild with upgrade again preferably with power adapter attached
Thanks! So when I tried rolling back, that was fine. And when I tried nixos-rebuild switch, that was fine as well. But when I try to do nixos-rebuild switch --upgrade, it fails with the expected string Derive([ error again.
Thanks! So nix-store --verify --repair just resulted in reading the Nix store... checking path existence... and nothing else. I added --check-contents and I got several errors:
{9:16}~ ➭ nix-store --verify --repair --check-contents
reading the Nix store...
checking path existence...
checking hashes...
path '/nix/store/10094j0xxzad05jpgi94xsg3ncz8mbx9-aws-checksums-0.1.7' was modified! expected hash 'sha256:19xjrnna383sgkd6r5nldx5zy5xf3k0cly5gzz67k9p9caca94r8', got 'sha256:1whfy9v9i2q8ihjlcknlibgfg9wbhs9kn5m3dbpm8zn3myzkzq88'
copying path '/nix/store/10094j0xxzad05jpgi94xsg3ncz8mbx9-aws-checksums-0.1.7' from 'https://cache.nixos.org'...
path '/nix/store/3cmqs83m64hm51vcdj697bbyx1v53cz8-aws-c-event-stream-0.1.1.drv' was modified! expected hash 'sha256:0m3f7073jwivhjky5hk94xwcw4m411avy4sl1ndm902n8k84wkn0', got 'sha256:0ip26j2h11n1kgkz36rl4akv694yz65hr72q4kv4b3lxcbi65b3p'
error: cannot repair path '/nix/store/3cmqs83m64hm51vcdj697bbyx1v53cz8-aws-c-event-stream-0.1.1.drv'
path '/nix/store/581m1wg98b7wcyccv0413fadklvfjmjx-aws-sdk-cpp-1.7.90.drv' was modified! expected hash 'sha256:0459b0hzgngzgjsfh2kjyld5r6dk9qplx4gji3l237ydg9k8wkkh', got 'sha256:0ip26j2h11n1kgkz36rl4akv694yz65hr72q4kv4b3lxcbi65b3p'
error: cannot repair path '/nix/store/581m1wg98b7wcyccv0413fadklvfjmjx-aws-sdk-cpp-1.7.90.drv'
...
path '/nix/store/yi5b872jznfvmnxxbljfkr8f39z6g52p-libarchive-3.4.3.drv' was modified! expected hash 'sha256:1sziwix8qgpvjd5c6pzq7lq957kbg6nmfk3ym5vi25v4czxdc3jn', got 'sha256:0ip26j2h11n1kgkz36rl4akv694yz65hr72q4kv4b3lxcbi65b3p'
error: cannot repair path '/nix/store/yi5b872jznfvmnxxbljfkr8f39z6g52p-libarchive-3.4.3.drv'
warning: not all errors were fixed
And when I try nixos-rebuild switch --upgrade I now get what I got before without --upgrade, namely:
I’m really not sure since this sort of behavior depends on some implementation details I’m not familiar with but as I said I would try to garbage collect the broken entries in your nix store. That means the complete list of things to try is something like:
Thanks @turboMaCk, I’m still getting the same result, I’m afraid. The rollback works fine, but the nixos-rebuild switch still results in the error as described above. I’ve tried running nix-collect-garbage -d and that succeeds just fine. But it doesn’t seem to change the result of either nixos-rebuild switch or nixos-rebuild switch --upgrade. Those still result in the errors described above.
You can also try to run garbage collection by nix-store --gc --delete. Also check the man pages for these command to make sure they won’t keep the generation that is corrupted around. Currently it might be deleting just a generations that are older than 30 days or so while the broken one might be younger than that. Anyway I think the simplest way to resolve this is to rollback and rebuild the system and then figure out how to get rid of that broken generation.
To add an additional detail, if it helps, the file /nix/store/sgpnpfyq30bl37kjv4hh70a0cwxqzrrz-nix-2.3.9.drv that nixos-rebuild complains about, is actually a completely empty file.
I assume nix is an important derivation. Should I try to re-build it individually? If so, how might I accomplish that?
I try to rebuild again but I’m getting an Exec format error, which I thought was a problem with 32 vs 64 bit. Not sure how that could be an issue here.
{20:21}~ ➭ nixos-rebuild switch
these derivations will be built:
/nix/store/2pfwxc4g4axpi1zxprjaxmq9ww7kz050-nixos-rebuild.drv
these paths will be fetched (0.07 MiB download, 0.23 MiB unpacked):
/nix/store/18ggindri5xm8zbdpzy7x2927z1lg5gc-stdenv-linux
/nix/store/5gzrrvski9kpjsf4f7gl9xcgk99cgifv-patchelf-0.12
copying path '/nix/store/5gzrrvski9kpjsf4f7gl9xcgk99cgifv-patchelf-0.12' from 'https://cache.nixos.org'...
copying path '/nix/store/18ggindri5xm8zbdpzy7x2927z1lg5gc-stdenv-linux' from 'https://cache.nixos.org'...
building '/nix/store/2pfwxc4g4axpi1zxprjaxmq9ww7kz050-nixos-rebuild.drv'...
building Nix...
/nix/store/8zm1fl21zvd0224l9di91gmrmhcmkiaf-nixos-rebuild/bin/nixos-rebuild: line 400: /tmp/nixos-rebuild.tCMUQ1/nix/bin/nix-instantiate: cannot execute binary file: Exec format error
building the system configuration...
/nix/store/8zm1fl21zvd0224l9di91gmrmhcmkiaf-nixos-rebuild/bin/nixos-rebuild: line 171: /tmp/nixos-rebuild.tCMUQ1/nix/bin/nix-build: cannot execute binary file: Exec format error
Does anyone have any concerns about running nixos-install at this late a stage? I am not at all a NixOS expert, so it is very possible I am misunderstanding something, but since my system is bootable, but my nix store is broken, can I just re-run nixos-install and it will look at my configuration.nix and it will fix my nix store “from scratch”?
You’d also get an exec format error if a file is truncated, I bet the version of Nix used here is corrupted. You could trace through the dependencies of /nix/store/8zm1fl21zvd0224l9di91gmrmhcmkiaf-nixos-rebuild and delete the Nix referenced there.
@shlevy That was very promising! When I ran that, I got a lot of checking paths, finding corrupted or missing paths, copying paths, repairing outputs, but at the very end (and this is the only part that seems to error) is this:
...
repairing outputs of '/nix/store/xxka7iim7mch3ddj28srbphp3lk3y7qz-system-path.drv'...
./createPPD.sh: line 82: ppdc: command not found
gzip: ppd/hpcups/*.ppd: No such file or directory
./createPPD.sh: line 88: ppdc: command not found
gzip: ppd/hpijs/*.ppd: No such file or directory
sh: -std=c++11: unknown operand
checking for scanner build... yes
checking for gui build... yes
checking for fax build... yes
checking for apparmor profile... no
checking for dbus build... yes
checking for cups 1.1.x build... no
checking for udev sysfs enable rules... no
checking for shadow build... no
checking for libusb-0.1 build... no
checking for foomatic ppd install... no
checking for foomatic drv install... no
checking for cups drv install... yes
checking for cups ppd install... no
checking for foomatic-rip-hplip install... no
checking for qt5... yes
checking for qt4... no
checking for qt3... no
checking for policykit... yes
checking for host machine platform... x86_64
Checking that Nix can read nix.conf...
/build/.attr-0: line 26: /nix/store/xhz3ybvfn5s1qiza4fijd2l05cfd6082-nix-2.3.9/bin/nix: cannot execute binary file: Exec format error
repairing outputs of '/nix/store/zkqmqn6v8gxr9ls4cb47hj9acq60f5rf-system-path.drv'...
checking for CRYPTO_free in -lcrypto... yes
builder for '/nix/store/ngv5630ljg74s84dy911fhb8d350ilac-nix.conf.drv' failed with exit code 126
cannot build derivation '/nix/store/3qrly9q35q11dh95f8yv1cwqgmqcnibx-etc.drv': 1 dependencies couldn't be built
repairing outputs of '/nix/store/lcbvlvq38pxwgvnb1fligx39sbp7wcvv-systemd-boot-builder.py.drv'...
repairing outputs of '/nix/store/ygpv14kwcqznpybzrvyhlb0syk0hkywn-vim-doc.drv'...
repairing outputs of '/nix/store/68rk2dlc55h3459ysndi17j41baa41qk-vim.drv'...
repairing outputs of '/nix/store/nv22hw24905b6k44qjzhbza4pb634khg-zoom_x86_64.tar.xz.drv'...
cannot build derivation '/nix/store/6dq1a024scqa6vbsc9j2i64dzm11j0cf-nixos-system-nixos-yoga-20.09.2090.e111e9d4c05.drv': 1 dependencies couldn't be built
error: build of '/nix/store/6dq1a024scqa6vbsc9j2i64dzm11j0cf-nixos-system-nixos-yoga-20.09.2090.e111e9d4c05.drv' failed
