Nixos-rebuild switch fails due to broken glibc

Help
1

I recently tried to update a aarch64-linux host after updating my flakes with nix flake update. However, it seems like something broke with the switch to glicb 2.34:

/nix/store/3fhxwd5iw6yzky55qhjv5z6zs4niikff-bash-5.1-p16/bin/bash: symbol lookup error: /nix/store/qyd4c78dl0kbpbc3lc2klp5bzy6585g7-glibc-2.33-108/lib/libpthread.so.0: undefined symbol: __pthread_cond_destroy, version GLIBC_PRIVATE
warning: error(s) occurred while switching to the new configuration

This seems to happen when trying to execute switch-to-configuration in the new profile (/nix/var/nix/profiles/system-243-link/bin/switch-to-configuration in my case). This uses /nix/store/g0cykbwxxiqvsp7nnsj3dbr7i0z23vgf-perl-5.34.0-env/bin/perl, which is a Bash script; it uses /nix/store/3fhxwd5iw6yzky55qhjv5z6zs4niikff-bash-5.1-p16/bin/bash, which fails with the previous error. Trying to manually run perl with another version of Bash also fails:

# bash /nix/store/g0cykbwxxiqvsp7nnsj3dbr7i0z23vgf-perl-5.34.0-env/bin/perl /nix/var/nix/profiles/system-243-link/bin/switch-to-configuration
/nix/store/dl014barwv2ayl46chrn8v9pb241p6n2-perl-5.34.0/bin/perl: /nix/store/9r9igk3kgfykppdzbaq5cn86qvz3fyb0-glibc-2.34-115/lib/libpthread.so.0: version `GLIBC_PRIVATE' not found (required by /nix/store/qyd4c78dl0kbpbc3lc2klp5bzy6585g7-glibc-2.33-108/lib/librt.so.1)

I’ve tried manually rolling back the configuration on the host itself with nixos-rebuild switch --rollback, but this doesn’t seem to fix anything - rerunning nixos-rebuild build --target-host root@aarch64-host --build-host root@aarch64-host --flake '.#aarch64-host' gives the same error.

1 Like
2

I’m seeing the same issue on 21.11 with pretty much every Nix binary:

 > sudo nixos-rebuild switch
building the system configuration...
nix: /nix/store/bvy2z17rzlvkx2sj7fy99ajm853yv898-glibc-2.34-210/lib/libpthread.so.0: version `GLIBC_PRIVATE' not found (required by /nix/store/qjgj2642srlbr59wwdihnn66sw97ming-glibc-2.33-123/lib/librt.so.1)

 > sudo nix-channel --list
nix-channel: /nix/store/bvy2z17rzlvkx2sj7fy99ajm853yv898-glibc-2.34-210/lib/libpthread.so.0: version `GLIBC_PRIVATE' not found (required by /nix/store/qjgj2642srlbr59wwdihnn66sw97ming-glibc-2.33-123/lib/librt.so.1)

No clue what to do about that other than reinstall the system.

3

I’ve managed to find that older versions of nix seem to work fine:

 > which nix
/run/current-system/sw/bin/nix

 > ls -l /run/current-system/sw/bin/nix                                   
lrwxrwxrwx 1 root root 61 Jan  1  1970 /run/current-system/sw/bin/nix -> /nix/store/y6daxy2r6l2w8afpqw8dmxks71p9v8va-nix-2.7.0/bin/nix

 > /nix/store/y6daxy2r6l2w8afpqw8dmxks71p9v8va-nix-2.7.0/bin/nix --version
/nix/store/y6daxy2r6l2w8afpqw8dmxks71p9v8va-nix-2.7.0/bin/nix: /nix/store/bvy2z17rzlvkx2sj7fy99ajm853yv898-glibc-2.34-210/lib/libpthread.so.0: version `GLIBC_PRIVATE' not found (required by /nix/store/qjgj2642srlbr59wwdihnn66sw97ming-glibc-2.33-123/lib/librt.so.1)

 > find /nix/store -maxdepth 1 -iname '*nix-2.7.0'                        
/nix/store/7c4cp4pj389j8rmrc40x5sq6csf0m8ks-nix-2.7.0
/nix/store/h538v75x5xgwzy31s5ia3ah7k6cdyzmy-nix-2.7.0
/nix/store/v2f9abhmmvk2slns100c2h696d5xj4gm-nix-2.7.0
/nix/store/rvwxh34igy78fzgjwr3z1biygpbxn66h-nix-2.7.0
/nix/store/y6daxy2r6l2w8afpqw8dmxks71p9v8va-nix-2.7.0

 > /nix/store/7c4cp4pj389j8rmrc40x5sq6csf0m8ks-nix-2.7.0/bin/nix --version
nix (Nix) 2.7.0

So I’m able to find a working version of nixos-rebuild and use that older version to rebuild the system.

But this is a really weird issue.

4

Nope, despite upgrading to newest Nix 2.8.1 the issue still persists. I have no idea what to do about it:

 > nix --verison
nix: /nix/store/bvy2z17rzlvkx2sj7fy99ajm853yv898-glibc-2.34-210/lib/libpthread.so.0:
  version `GLIBC_PRIVATE' not found (required by /nix/store/qjgj2642srlbr59wwdihnn66sw97ming-glibc-2.33-123/lib/librt.so.1)

I also ran a store verification and repair with a working version, but nothing came up:

 > sudo /nix/store/7c4cp4pj389j8rmrc40x5sq6csf0m8ks-nix-2.7.0/bin/nix-store \
      --verify --check-contents --repair
reading the Nix store...
checking path existence...
checking link hashes...
checking store hashes...

And nothing changed.

5

I’d highly recommend reporting this at Issues · NixOS/nix · GitHub instead.

1 Like
6

Sure, wanted to get some feedback here first to not spam the issues there with something stupid.

But it doesn’t appear like this is something I broke myself. Here’s the issue: