Nixos-rebuild switch|boot segfault

Help
1

Today I wanted to upgrade my nixos server to 21.05 (from 20.09) and hit a pretty big snag.
When I try to do a rebuild switch or a rebuild boot, this command segfaults:

% sudo nixos-rebuild switch
[1]    15570 segmentation fault  sudo nixos-rebuild switch

The only additional information I can find is in dmesg:

[  352.645196] nixos-rebuild[15576]: segfault at 668f00000000 ip 0000668f7abf09ec sp 0000750d0c3dd1e0 error 4 in ld-2.32.so[668f7abe2000+20000]
[  352.645206] Code: 03 48 03 41 08 48 c1 ea 03 85 d2 74 27 83 ea 01 48 8d 58 08 4c 8d 34 d3 eb 08 0f 1f 40 00 48 83 c3 08 4c 89 ea 4c 89 e6 89 ef <ff> 10 48 89 d8 49 39 de 75 ea 5b 5d 41 5c 41 5d 41 5e c3 90 a8 03

Any tips how I can rescue this system? Everything else appears to work fine.
(nixos-rebuild build works without problems.

2

I have tracked this issue down a bit further, and am noticing the bash used as interpreter by the activate script (/nix/store/kxj6cblcsd1qcbbxlmbswwrn89zcmgd6-bash-4.4-p23/bin/bash) is actually segfaulting.

Any hints would be appreciated.

3

do you mind doing: nix-shell -p nix-info --run "nix-info -m"?

4 
% nix-shell -p nix-info --run "nix-info -m"
these paths will be fetched (0.05 MiB download, 0.28 MiB unpacked):
  /nix/store/lp90qhw19dcp79cnlicjg09llgh7dzsz-bash-interactive-4.4-p23-dev
copying path '/nix/store/lp90qhw19dcp79cnlicjg09llgh7dzsz-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
[1]    8388 segmentation fault (core dumped)  nix-shell -p nix-info --run "nix-info -m"

I also found something from the linking of the activate bash that looks out of place to me:

% ldd /nix/store/kxj6cblcsd1qcbbxlmbswwrn89zcmgd6-bash-4.4-p23/bin/bash
        linux-vdso.so.1 (0x00006a1d6b072000)
        /nix/store/qrjm2j7sxa5vxivv73jz3l5g37saizjd-malloc-provider-scudo/lib/libclang_rt.scudo-x86_64.so (0x00006a1d6a798000)
        libdl.so.2 => /nix/store/ikl21vjfq900ccbqg1xasp83kadw6q8y-glibc-2.32-46/lib/libdl.so.2 (0x00006a1d6a793000)
        libc.so.6 => /nix/store/ikl21vjfq900ccbqg1xasp83kadw6q8y-glibc-2.32-46/lib/libc.so.6 (0x00006a1d6a5d2000)
        libgcc_s.so.1 => /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libgcc_s.so.1 (0x00006a1d6a5b8000)
        **/nix/store/ikl21vjfq900ccbqg1xasp83kadw6q8y-glibc-2.32-46/lib/ld-linux-x86-64.so.2 => /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib64/ld-linux-x86-64.so.2 (0x00006a1d6b073000)**
        librt.so.1 => /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/librt.so.1 (0x00006a1d6a5ae000)
        libpthread.so.0 => /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libpthread.so.0 (0x00006a1d6a58b000)
        libstdc++.so.6 => /nix/store/c10296m7xgm3ksibcklb2xf48jr635x3-gcc-9.3.0-lib/lib/libstdc++.so.6 (0x00006a1d6a3aa000)
        libm.so.6 => /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libm.so.6 (0x00006a1d6a269000)

I can’t really understand that link from new glibc to old glibc.

5

nix-shell is segfaulting too, ouch!

if you rollback to 20.09 , do your nix commands start working again?

6

hmm, only thing I can think of is adding GC_DONT_GC=1 nix... to the commands. If your derivation is doing a lot of nix trickier, it may be exceeding the default heapsize

7

Yes, that seems to work fine:

% sudo nix-channel --list
[sudo] password for jeroen:
nixos https://nixos.org/channels/nixos-21.05-small
jeroen@vb ~ % sudo nix-channel --remove nixos
uninstalling 'nixos-21.05.1002.8db24dec536'
jeroen@vb ~ % sudo nix-channel --add https://nixos.org/channels/nixos-20.09-small nixos
jeroen@vb ~ % sudo nix-channel --update
unpacking channels...
jeroen@vb ~ % nix-shell -p nix-info --run "nix-info -m"
these paths will be fetched (0.39 MiB download, 1.89 MiB unpacked):
  /nix/store/7fqqkrz6m11793cw7by8wbn2q87za8ny-gnumake-4.3
  /nix/store/a9g7p6fwanw66j5djzila7ql1hky759z-bash-interactive-4.4-p23-dev
  /nix/store/m40wl6v6ayalys7l5x5iqif6qwjj1dsx-patchelf-0.12
  /nix/store/wgap303sj9zqz63gw7nqxvf4dqz2hgai-stdenv-linux
copying path '/nix/store/a9g7p6fwanw66j5djzila7ql1hky759z-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/7fqqkrz6m11793cw7by8wbn2q87za8ny-gnumake-4.3' from 'https://cache.nixos.org'...
copying path '/nix/store/m40wl6v6ayalys7l5x5iqif6qwjj1dsx-patchelf-0.12' from 'https://cache.nixos.org'...
copying path '/nix/store/wgap303sj9zqz63gw7nqxvf4dqz2hgai-stdenv-linux' from 'https://cache.nixos.org'...
 - system: `"x86_64-linux"`
 - host os: `Linux 5.4.122-hardened1, NixOS, 20.09.4325.cd1febccec5 (Nightingale)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.11`
 - channels(jeroen): `""`
 - channels(root): `"nixos-20.09.4325.cd1febccec5"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

jeroen@vb ~ %

My config really isn’t that complicated and I am definately not good at nix trickery, so don’t use any, just default options.

8

It is something in the channel that’s making stuff segfault and apears to be reproducable:

% sudo nix-channel --remove nixos
uninstalling 'nixos-20.09.4325.cd1febccec5'
jeroen@vb ~ % sudo nix-channel --add https://nixos.org/channels/nixos-21.05-small nixos
jeroen@vb ~ % sudo nix-channel --update
unpacking channels...
jeroen@vb ~ % nix-shell -p nix-info --run "nix-info -m"
[1]    20540 segmentation fault (core dumped)  nix-shell -p nix-info --run "nix-info -m"
9

So, I tracked down my issue.
I was running nixos with the <nixpkgs/nixos/modules/profiles/hardened.nix> imported.
Something in that module seems to break things. I rebuild my current 20.09 version without that hardened profile, rebooted and now I have succesfuly upgraded to 21.05.

1 Like
10

can you open a issue on github please, with a reproducible test case.

11

I’ll see what I can come up with. Any pointers to how I should start building the case?
Should I send in a git repo or a config? Or are other things required?

12

just enough so that someone else can reproduce it…and probably ping the maintainer of the hardend profile. This may be a classic case of locking things down so much, user space applications can’t actually function, or something more silly.

either way, thanks for using nixos.

13

For reference:
https://github.com/NixOS/nixpkgs/issues/127070

14

LGTM, thanks for taking the time to do the report. extensive.

Why are you running a hardened profile anyhows, do you have security worries?

15

Not so much worries, but it is a machine that is connected directly to the internet (in a datacenter) and is running quite a few public services, including some that contain some private information. So I wanted to be on the safe side :slight_smile: