### Author:  [@jsimonetti](https://discourse.nixos.org/u/jsimonetti)
#### Post date: [June 15, 2021, 4:19pm UTC](https://discourse.nixos.org/t/nixos-rebuild-switch-boot-segfault/13643/1 "2021-06-15T16:19:47Z")
Today I wanted to upgrade my nixos server to 21.05 (from 20.09) and hit a pretty big snag.
When I try to do a `rebuild switch` or a `rebuild boot`, this command segfaults:
```auto
% sudo nixos-rebuild switch
[1] 15570 segmentation fault sudo nixos-rebuild switch
```
The only additional information I can find is in `dmesg`:
```auto
[352.645196] nixos-rebuild[15576]: segfault at 668f00000000 ip 0000668f7abf09ec sp 0000750d0c3dd1e0 error 4 in ld-2.32.so[668f7abe2000+20000]
[352.645206] Code: 03 48 03 41 08 48 c1 ea 03 85 d2 74 27 83 ea 01 48 8d 58 08 4c 8d 34 d3 eb 08 0f 1f 40 00 48 83 c3 08 4c 89 ea 4c 89 e6 89 ef 10 48 89 d8 49 39 de 75 ea 5b 5d 41 5c 41 5d 41 5e c3 90 a8 03
```
Any tips how I can rescue this system? Everything else appears to work fine.
(`nixos-rebuild build` works without problems.
---
### Author:  [@jsimonetti](https://discourse.nixos.org/u/jsimonetti)
#### Post date: [June 15, 2021, 5:41pm UTC](https://discourse.nixos.org/t/nixos-rebuild-switch-boot-segfault/13643/2 "2021-06-15T17:41:17Z")
> [@jsimonetti](#):
>
> `error 4 in ld-2.32.so`
I have tracked this issue down a bit further, and am noticing the bash used as interpreter by the `activate` script (/nix/store/kxj6cblcsd1qcbbxlmbswwrn89zcmgd6-bash-4.4-p23/bin/bash) is actually segfaulting.
Any hints would be appreciated.
---
### Author:  [@jonringer](https://discourse.nixos.org/u/jonringer)
#### Post date: [June 15, 2021, 5:48pm UTC](https://discourse.nixos.org/t/nixos-rebuild-switch-boot-segfault/13643/3 "2021-06-15T17:48:21Z")
do you mind doing: `nix-shell -p nix-info --run "nix-info -m"`?
---
### Author:  [@jsimonetti](https://discourse.nixos.org/u/jsimonetti)
#### Post date: [June 15, 2021, 5:54pm UTC](https://discourse.nixos.org/t/nixos-rebuild-switch-boot-segfault/13643/4 "2021-06-15T17:54:08Z")
```auto
% nix-shell -p nix-info --run "nix-info -m"
these paths will be fetched (0.05 MiB download, 0.28 MiB unpacked):
/nix/store/lp90qhw19dcp79cnlicjg09llgh7dzsz-bash-interactive-4.4-p23-dev
copying path '/nix/store/lp90qhw19dcp79cnlicjg09llgh7dzsz-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
[1] 8388 segmentation fault (core dumped) nix-shell -p nix-info --run "nix-info -m"
```
I also found something from the linking of the activate bash that looks out of place to me:
```auto
% ldd /nix/store/kxj6cblcsd1qcbbxlmbswwrn89zcmgd6-bash-4.4-p23/bin/bash
linux-vdso.so.1 (0x00006a1d6b072000)
/nix/store/qrjm2j7sxa5vxivv73jz3l5g37saizjd-malloc-provider-scudo/lib/libclang_rt.scudo-x86_64.so (0x00006a1d6a798000)
libdl.so.2 => /nix/store/ikl21vjfq900ccbqg1xasp83kadw6q8y-glibc-2.32-46/lib/libdl.so.2 (0x00006a1d6a793000)
libc.so.6 => /nix/store/ikl21vjfq900ccbqg1xasp83kadw6q8y-glibc-2.32-46/lib/libc.so.6 (0x00006a1d6a5d2000)
libgcc_s.so.1 => /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libgcc_s.so.1 (0x00006a1d6a5b8000)
**/nix/store/ikl21vjfq900ccbqg1xasp83kadw6q8y-glibc-2.32-46/lib/ld-linux-x86-64.so.2 => /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib64/ld-linux-x86-64.so.2 (0x00006a1d6b073000)**
librt.so.1 => /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/librt.so.1 (0x00006a1d6a5ae000)
libpthread.so.0 => /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libpthread.so.0 (0x00006a1d6a58b000)
libstdc++.so.6 => /nix/store/c10296m7xgm3ksibcklb2xf48jr635x3-gcc-9.3.0-lib/lib/libstdc++.so.6 (0x00006a1d6a3aa000)
libm.so.6 => /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libm.so.6 (0x00006a1d6a269000)
```
I can’t really understand that link from new glibc to old glibc.
---
### Author:  [@nixinator](https://discourse.nixos.org/u/nixinator)
#### Post date: [June 15, 2021, 6:39pm UTC](https://discourse.nixos.org/t/nixos-rebuild-switch-boot-segfault/13643/5 "2021-06-15T18:39:23Z")
> [@jsimonetti](#):
>
> `nix-shell`
nix-shell is segfaulting too, ouch!
if you rollback to 20.09 , do your nix commands start working again?
---
### Author:  [@jonringer](https://discourse.nixos.org/u/jonringer)
#### Post date: [June 15, 2021, 8:02pm UTC](https://discourse.nixos.org/t/nixos-rebuild-switch-boot-segfault/13643/6 "2021-06-15T20:02:57Z")
hmm, only thing I can think of is adding `GC_DONT_GC=1 nix...` to the commands. If your derivation is doing a lot of nix trickier, it may be exceeding the default heapsize
---
### Author:  [@jsimonetti](https://discourse.nixos.org/u/jsimonetti)
#### Post date: [June 15, 2021, 8:23pm UTC](https://discourse.nixos.org/t/nixos-rebuild-switch-boot-segfault/13643/7 "2021-06-15T20:23:19Z")
> [@nixinator](#):
>
> if you rollback to 20.09 , do your nix commands start working again?
Yes, that seems to work fine:
```auto
% sudo nix-channel --list
[sudo] password for jeroen:
nixos https://nixos.org/channels/nixos-21.05-small
jeroen@vb ~ % sudo nix-channel --remove nixos
uninstalling 'nixos-21.05.1002.8db24dec536'
jeroen@vb ~ % sudo nix-channel --add https://nixos.org/channels/nixos-20.09-small nixos
jeroen@vb ~ % sudo nix-channel --update
unpacking channels...
jeroen@vb ~ % nix-shell -p nix-info --run "nix-info -m"
these paths will be fetched (0.39 MiB download, 1.89 MiB unpacked):
/nix/store/7fqqkrz6m11793cw7by8wbn2q87za8ny-gnumake-4.3
/nix/store/a9g7p6fwanw66j5djzila7ql1hky759z-bash-interactive-4.4-p23-dev
/nix/store/m40wl6v6ayalys7l5x5iqif6qwjj1dsx-patchelf-0.12
/nix/store/wgap303sj9zqz63gw7nqxvf4dqz2hgai-stdenv-linux
copying path '/nix/store/a9g7p6fwanw66j5djzila7ql1hky759z-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/7fqqkrz6m11793cw7by8wbn2q87za8ny-gnumake-4.3' from 'https://cache.nixos.org'...
copying path '/nix/store/m40wl6v6ayalys7l5x5iqif6qwjj1dsx-patchelf-0.12' from 'https://cache.nixos.org'...
copying path '/nix/store/wgap303sj9zqz63gw7nqxvf4dqz2hgai-stdenv-linux' from 'https://cache.nixos.org'...
- system: `"x86_64-linux"`
- host os: `Linux 5.4.122-hardened1, NixOS, 20.09.4325.cd1febccec5 (Nightingale)`
- multi-user?: `yes`
- sandbox: `yes`
- version: `nix-env (Nix) 2.3.11`
- channels(jeroen): `""`
- channels(root): `"nixos-20.09.4325.cd1febccec5"`
- nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`
jeroen@vb ~ %
```
My config really isn’t that complicated and I am definately not good at nix trickery, so don’t use any, just default options.
---
### Author:  [@jsimonetti](https://discourse.nixos.org/u/jsimonetti)
#### Post date: [June 15, 2021, 8:33pm UTC](https://discourse.nixos.org/t/nixos-rebuild-switch-boot-segfault/13643/8 "2021-06-15T20:33:16Z")
It is something in the channel that’s making stuff segfault and apears to be reproducable:
```auto
% sudo nix-channel --remove nixos
uninstalling 'nixos-20.09.4325.cd1febccec5'
jeroen@vb ~ % sudo nix-channel --add https://nixos.org/channels/nixos-21.05-small nixos
jeroen@vb ~ % sudo nix-channel --update
unpacking channels...
jeroen@vb ~ % nix-shell -p nix-info --run "nix-info -m"
[1] 20540 segmentation fault (core dumped) nix-shell -p nix-info --run "nix-info -m"
```
---
### Author:  [@jsimonetti](https://discourse.nixos.org/u/jsimonetti)
#### Post date: [June 15, 2021, 9:13pm UTC](https://discourse.nixos.org/t/nixos-rebuild-switch-boot-segfault/13643/9 "2021-06-15T21:13:20Z")
So, I tracked down my issue.
I was running nixos with the `` imported.
Something in that module seems to break things. I rebuild my current 20.09 version _without_ that hardened profile, rebooted and now I have succesfuly upgraded to 21.05.
---
### Author:  [@nixinator](https://discourse.nixos.org/u/nixinator)
#### Post date: [June 16, 2021, 4:15am UTC](https://discourse.nixos.org/t/nixos-rebuild-switch-boot-segfault/13643/10 "2021-06-16T04:15:01Z")
can you open a issue on github please, with a reproducible test case.
---
### Author:  [@jsimonetti](https://discourse.nixos.org/u/jsimonetti)
#### Post date: [June 16, 2021, 7:56am UTC](https://discourse.nixos.org/t/nixos-rebuild-switch-boot-segfault/13643/11 "2021-06-16T07:56:56Z")
I’ll see what I can come up with. Any pointers to how I should start building the case?
Should I send in a git repo or a config? Or are other things required?
---
### Author:  [@nixinator](https://discourse.nixos.org/u/nixinator)
#### Post date: [June 16, 2021, 11:02am UTC](https://discourse.nixos.org/t/nixos-rebuild-switch-boot-segfault/13643/12 "2021-06-16T11:02:38Z")
just enough so that someone else can reproduce it…and probably ping the maintainer of the hardend profile. This may be a classic case of locking things down so much, user space applications can’t actually function, or something more silly.
either way, thanks for using nixos.
---
### Author:  [@jsimonetti](https://discourse.nixos.org/u/jsimonetti)
#### Post date: [June 16, 2021, 11:13am UTC](https://discourse.nixos.org/t/nixos-rebuild-switch-boot-segfault/13643/13 "2021-06-16T11:13:20Z")
> [@nixinator](#):
>
> can you open a issue on github please, with a reproducible test case.
For reference:
>
>
> \*\*Describe the bug\*\*
> When running the \`\\` profile, my system was unable to upgrade to 21.05.
>
> Yesterday I wanted to upgrade my nixos server to 21.05 (from 20.09) and hit a pretty big snag.
> When I tried to do a \`nixos-rebuild switch\` or a \`nixos-rebuild boot\`, the command segfaults:
>
> \`\`\`
> % sudo nixos-rebuild switch
> \[1\] 15570 segmentation fault sudo nixos-rebuild switch
> \`\`\`
>
> \`dmesg\` showed:
>
> \`\`\`
> \[352.645196\] nixos-rebuild\[15576\]: segfault at 668f00000000 ip 0000668f7abf09ec sp 0000750d0c3dd1e0 error 4 in ld-2.32.so\[668f7abe2000+20000\]
> \[352.645206\] Code: 03 48 03 41 08 48 c1 ea 03 85 d2 74 27 83 ea 01 48 8d 58 08 4c 8d 34 d3 eb 08 0f 1f 40 00 48 83 c3 08 4c 89 ea 4c 89 e6 89 ef \ 10 48 89 d8 49 39 de 75 ea 5b 5d 41 5c 41 5d 41 5e c3 90 a8 03
> \`\`\`
>
> When I rolled back to the 20.09 channel, the tools work again:
> \`\`\`
> % sudo nix-channel --list
> \[sudo\] password for jeroen:
> nixos https://nixos.org/channels/nixos-21.05-small
> jeroen@vb ~ % sudo nix-channel --remove nixos
> uninstalling 'nixos-21.05.1002.8db24dec536'
> jeroen@vb ~ % sudo nix-channel --add https://nixos.org/channels/nixos-20.09-small nixos
> jeroen@vb ~ % sudo nix-channel --update
> unpacking channels...
> jeroen@vb ~ % nix-shell -p nix-info --run "nix-info -m"
> these paths will be fetched (0.39 MiB download, 1.89 MiB unpacked):
> /nix/store/7fqqkrz6m11793cw7by8wbn2q87za8ny-gnumake-4.3
> /nix/store/a9g7p6fwanw66j5djzila7ql1hky759z-bash-interactive-4.4-p23-dev
> /nix/store/m40wl6v6ayalys7l5x5iqif6qwjj1dsx-patchelf-0.12
> /nix/store/wgap303sj9zqz63gw7nqxvf4dqz2hgai-stdenv-linux
> copying path '/nix/store/a9g7p6fwanw66j5djzila7ql1hky759z-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
> copying path '/nix/store/7fqqkrz6m11793cw7by8wbn2q87za8ny-gnumake-4.3' from 'https://cache.nixos.org'...
> copying path '/nix/store/m40wl6v6ayalys7l5x5iqif6qwjj1dsx-patchelf-0.12' from 'https://cache.nixos.org'...
> copying path '/nix/store/wgap303sj9zqz63gw7nqxvf4dqz2hgai-stdenv-linux' from 'https://cache.nixos.org'...
> - system: \`"x86\_64-linux"\`
> - host os: \`Linux 5.4.122-hardened1, NixOS, 20.09.4325.cd1febccec5 (Nightingale)\`
> - multi-user?: \`yes\`
> - sandbox: \`yes\`
> - version: \`nix-env (Nix) 2.3.11\`
> - channels(jeroen): \`""\`
> - channels(root): \`"nixos-20.09.4325.cd1febccec5"\`
> - nixpkgs: \`/nix/var/nix/profiles/per-user/root/channels/nixos\`
> \`\`\`
>
> Rolling forward, many nix tools seem to fail:
> \`\`\`
> % sudo nix-channel --remove nixos
> uninstalling 'nixos-20.09.4325.cd1febccec5'
> jeroen@vb ~ % sudo nix-channel --add https://nixos.org/channels/nixos-21.05-small nixos
> jeroen@vb ~ % sudo nix-channel --update
> unpacking channels...
> jeroen@vb ~ % nix-shell -p nix-info --run "nix-info -m"
> \[1\] 20540 segmentation fault (core dumped) nix-shell -p nix-info --run "nix-info -m"
> \`\`\`
>
> I managed to pinpoint this to the used memory allocator \`environment.memoryAllocator.provider = "scudo";\` from \`\\`
>
> \*\*To Reproduce\*\*
> I have tried to build a \`pkgs.nixosTest\` to reproduce the behaviour and it appears to work. I copied in the relevant parts from \`hardening.nix\` manually into the test vm configuration.
>
> \`\`\`
> let
> #build a 20.09 vm
> nixpkgs = fetchTarball
> "https://github.com/NixOS/nixpkgs/archive/cd1febccec5fd39336475c98cd9fd088715adb57.tar.gz";
> pkgs = import nixpkgs { };
> in pkgs.nixosTest ({
> system = "x86\_64-linux";
> virtualisation.writableStore = true;
> nodes = {
> server = { config, pkgs, lib, ... }:
> let
> # use nixpkgs from 21.05 to install nix-info
> newnixpkgs = fetchTarball
> "https://github.com/NixOS/nixpkgs/archive/8db24dec5368ef35657bbca14f0a3baaeefb0f88.tar.gz";
> newpkgs = import newnixpkgs { };
> in {
> environment.systemPackages = \[newpkgs.nix-info \];
> boot.kernelPackages = pkgs.linuxPackages\_hardened;
> environment.memoryAllocator.provider = "scudo";
> };
> };
> testScript = ''
> start\_all()
> server.wait\_for\_unit("multi-user.target")
> print(server.succeed("uname -a"))
> server.succeed("nix-info -m")
> '';
> })
> \`\`\`
>
> Results in:
> \`\`\`
> server: must succeed: uname -a
> (0.06 seconds)
> Linux server 5.4.122-hardened1 #1-NixOS SMP Wed May 26 10:05:21 UTC 2021 x86\_64 GNU/Linux
>
> server: must succeed: nix-info -m
> server # \[35.949319\] nix-info\[829\]: segfault at 6d0c00000000 ip 00006d0c1cf769ec sp 00007cdf8dc951a0 error 4 in ld-2.32.so\[6d0c1cf68000+20000\]
> server # \[35.954566\] Code: 03 48 03 41 08 48 c1 ea 03 85 d2 74 27 83 ea 01 48 8d 58 08 4c 8d 34 d3 eb 08 0f 1f 40 00 48 83 c3 08 4c 89 ea 4c 89 e6 89 ef \ 10 48 89 d8 49 39 de 75 ea 5b 5d 41 5c 41 5d 41 5e c3 90 a8 03
> \`\`\`
>
> \*\*Expected behavior\*\*
> No segfault to occur.
>
> \*\*Notify maintainers\*\*
> @joachifm
> @emilazy
>
> \*\*Metadata\*\*
> \`\`\`
> nix-shell -p nix-info --run "nix-info -m"
> these paths will be fetched (0.39 MiB download, 1.89 MiB unpacked):
> /nix/store/7fqqkrz6m11793cw7by8wbn2q87za8ny-gnumake-4.3
> /nix/store/a9g7p6fwanw66j5djzila7ql1hky759z-bash-interactive-4.4-p23-dev
> /nix/store/m40wl6v6ayalys7l5x5iqif6qwjj1dsx-patchelf-0.12
> /nix/store/wgap303sj9zqz63gw7nqxvf4dqz2hgai-stdenv-linux
> copying path '/nix/store/a9g7p6fwanw66j5djzila7ql1hky759z-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
> copying path '/nix/store/7fqqkrz6m11793cw7by8wbn2q87za8ny-gnumake-4.3' from 'https://cache.nixos.org'...
> copying path '/nix/store/m40wl6v6ayalys7l5x5iqif6qwjj1dsx-patchelf-0.12' from 'https://cache.nixos.org'...
> copying path '/nix/store/wgap303sj9zqz63gw7nqxvf4dqz2hgai-stdenv-linux' from 'https://cache.nixos.org'...
> - system: \`"x86\_64-linux"\`
> - host os: \`Linux 5.4.122-hardened1, NixOS, 20.09.4325.cd1febccec5 (Nightingale)\`
> - multi-user?: \`yes\`
> - sandbox: \`yes\`
> - version: \`nix-env (Nix) 2.3.11\`
> - channels(jeroen): \`""\`
> - channels(root): \`"nixos-20.09.4325.cd1febccec5"\`
> - nixpkgs: \`/nix/var/nix/profiles/per-user/root/channels/nixos\`
> \`\`\`
>
> Maintainer information:
> \`\`\`yaml
> \# a list of nixpkgs attributes affected by the problem
> attribute:
> \# a list of nixos modules affected by the problem
> module:
> \`\`\`
---
### Author:  [@nixinator](https://discourse.nixos.org/u/nixinator)
#### Post date: [June 16, 2021, 2:07pm UTC](https://discourse.nixos.org/t/nixos-rebuild-switch-boot-segfault/13643/14 "2021-06-16T14:07:51Z")
LGTM, thanks for taking the time to do the report. extensive.
Why are you running a hardened profile anyhows, do you have security worries?
---
### Author:  [@jsimonetti](https://discourse.nixos.org/u/jsimonetti)
#### Post date: [June 16, 2021, 3:27pm UTC](https://discourse.nixos.org/t/nixos-rebuild-switch-boot-segfault/13643/15 "2021-06-16T15:27:53Z")
Not so much worries, but it is a machine that is connected directly to the internet (in a datacenter) and is running quite a few public services, including some that contain some private information. So I wanted to be on the safe side 🙂