Segmentation fault when running any nix program (SIGSEGV, exit code 139)

I first noticed an issue with my nix installation when running darwin-rebuild switch --verbose, the rebuild just stopped dead at this point:

...
querying info about '/nix/store/xwwq90k84aljwh4xvbbm8kc213y2p20v-options.json' on 'https://cache.nixos.org'...
downloading 'https://cache.nixos.org/xwwq90k84aljwh4xvbbm8kc213y2p20v.narinfo'...
these 27 derivations will be built:
  /nix/store/dhl3yyz0ljhlcf4c57az28dy2clakhb7-builder.pl.drv
  /nix/store/2j91vwjq1mx9kb3h0w7lygriglq4c64w-options.json.drv
  /nix/store/m7rmwvpksff81ga2mq46b27h9f82a7rb-home-configuration-reference-manpage.drv
  /nix/store/0dzwfxvmny0njwgwy6g8xj7bly0jh61x-home-manager-fonts.drv
  /nix/store/6kcj8qykvqapdnwfgqbax5zqfcc5j1g5-darwin-version.json.drv
  /nix/store/zqiqrk0k29jm33zkas4a3ajb8yc5himr-options.json.drv
  /nix/store/4njv5zdjb4wcgf81hkpniv2w35r8ig7k-options.json.drv
  /nix/store/j2v9afrlyppqjd5p968f2zpvsgykx85v-darwin-manpages.drv
  /nix/store/zb5r27lzvdklvxdvvc7qwcc1rza839hw-manpage-urls.json.drv
  /nix/store/plbq8ahhbdgv73xwbjbxpcywxsxk91yn-darwin-manual-html.drv
  /nix/store/v5krgcg9rq9bglga7fi6gffj1i3lmrjc-darwin-help.drv
  /nix/store/j9a39fhvv6049m9k0fp9p07rnrjfimar-system-path.drv
  /nix/store/xv42yj5jh1zw8kmcr7kf0jvmarjh6y3n-system-applications.drv
  /nix/store/rsa8pwqwvg6zg06mns8mivjfy8dxcll6-darwin-system-24.05pre556102.09ec6a0881e1+darwin4.drv
  /nix/store/c4kzqjfrrijk682s73331v9q54p5cknv-darwin-uninstaller.drv
  /nix/store/a66vf6ch29q15xsg3jrab9ya0hsfkb08-system-path.drv
  /nix/store/lrndp1aplgkawmp1d53adg1khcrzl8w1-home-manager-applications.drv
  /nix/store/gn039gyz2v735fjgccqx1nlp9pgibdas-home-manager-files.drv
  /nix/store/k9pk3lhb7x8rrhbpjhcdglp6nnw3s3xn-home-manager-path.drv
  /nix/store/ismcmfmx1mqmlxdn7syihac204m7klm9-hm-modules-messages.drv
  /nix/store/dq6rwf7v7ya1kcldzdb02pifjc9maw11-check.drv
  /nix/store/q60y0raiv5zh693y4bdp3ch841ffva7z-cleanup.drv
  /nix/store/p1h1l5qac08sb2jglz1mkjh128vbrhkb-activation-script.drv
  /nix/store/3dskw54jas00pi3q8hsm9vk0wrk843zz-home-manager-generation.drv
  /nix/store/mzw2mwyyx5hgfhfdzsapjdcmcqyk25sc-activation-aaronkelly.drv
  /nix/store/yp75pql9qfjr174q2x8fdgzjz6syn664-system-applications.drv
  /nix/store/2w8mw73jpc75y5cjnd90k20vkpq9fhvn-darwin-system-24.05pre556102.09ec6a0881e1+darwin4.drv
$ 

It looked like a segmentation fault, I could see the exit code 139 with this command:

$ echo $?
139

I then tried to run the standard nix-info command, but it also has the same issue:

$ nix-shell -p nix-info --run "nix-info -m" --verbose
...
evaluating file '/nix/store/h2l8kfyq4zca3k0nw2ygvz1h2rc0iy3k-nixpkgs/nixpkgs/pkgs/development/python-modules/future/default.nix'
evaluating file '/nix/store/h2l8kfyq4zca3k0nw2ygvz1h2rc0iy3k-nixpkgs/nixpkgs/pkgs/tools/nix/info/default.nix'
evaluating file '/nix/store/h2l8kfyq4zca3k0nw2ygvz1h2rc0iy3k-nixpkgs/nixpkgs/pkgs/stdenv/generic/common-path.nix'
[1]    43172 segmentation fault  nix-shell -p nix-info --run "nix-info -m" --verbose
$ echo $?
139

Even poor cowsay cannot be run in a shell:

$ nix-shell -p cowsay --verbose
...
valuating file '/nix/store/h2l8kfyq4zca3k0nw2ygvz1h2rc0iy3k-nixpkgs/nixpkgs/pkgs/development/python-modules/future/default.nix'
evaluating file '/nix/store/h2l8kfyq4zca3k0nw2ygvz1h2rc0iy3k-nixpkgs/nixpkgs/pkgs/tools/misc/cowsay/default.nix'
evaluating file '/nix/store/h2l8kfyq4zca3k0nw2ygvz1h2rc0iy3k-nixpkgs/nixpkgs/pkgs/stdenv/generic/common-path.nix'
[1]    43294 segmentation fault  nix-shell -p cowsay --verbose

How can I investigate/debug this issue? I don’t know where to start looking

If it’s segfaulting there should be a crash report in console.app.

Does nix run nixpkgs#nix-info -- -m work? (You might have to specify the --extra-experimental-features 'nix-command flakes'.)

Thanks for getting back to me @abathur, here’s the output of the command, it seems to work fine:

$ nix run --extra-experimental-features 'nix-command flakes' nixpkgs#nix-info -- -m
 - system: `"x86_64-darwin"`
 - host os: `Darwin 19.6.0, macOS 10.15.7`
 - multi-user?: `no`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

Glad that worked.

I think I’ve been seeing (and trying to figure out) the same thing for the last few days. We’ve been talking about it a little in the “Nix on macOS” room on Matrix.

It seems like it’s only segfaulting when I do something with a traditional CLI tool that needs to realize a store path. (The new/experimental CLI doesn’t segfault, and the traditional CLI is fine as long as it doesn’t need to realize something.)

Do you see roughly what I see if you try a sequence like below?

$ nix-shell -p cowsay --run "cowsay heh"
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
Segmentation fault: 11

$ nix run --extra-experimental-features 'nix-command flakes' nixpkgs#cowsay heh
 _____ 
< heh >
 ----- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

$ nix-shell -p cowsay --run "cowsay heh"
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
 _____ 
< heh >
 ----- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

@abathur here’s what I get:

nix-shell -p cowsay --run "cowsay heh"
[1]    54984 segmentation fault  nix-shell -p cowsay --run "cowsay heh"
$ nix run --extra-experimental-features 'nix-command flakes' nixpkgs#cowsay heh
 _____ 
< heh >
 ----- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
$ nix-shell -p cowsay --run "cowsay heh"
[1]    55105 segmentation fault  nix-shell -p cowsay --run "cowsay heh" 
$ echo $?
139

Thanks for mentioning the “Nix on macOS” room, I didn’t know about it - I’ll join it!

Not certain why we aren’t seeing quite the same behavior. I have flaked my system and removed my channels, so I guess maybe both of my invocations are pointing to the same nixpkgs rev while yours may be pointing to different ones.

I do still think it smells like the ~same segfault.

Here’s what I see when I use --verbose (though I’m changing up the package since I now have both cowsay and nix-info in my store and don’t want to gc atm…):

$ nix-shell -p hello --run "hello heh" --verbose
... stuff in nixpkgs ...
evaluating file '/Users/abathur/.config/nixpkgs/overlays/overrides.nix'
evaluating file '/Users/abathur/.config/nixpkgs/overlays/packages.nix'
... stuff in nixpkgs ...
evaluating file '/Users/abathur/.config/nixpkgs/config.nix'
... stuff in nixpkgs ...
evaluating file '/nix/store/1dklh99pdzxcxmrxz3i63rb3pa832yik-source/pkgs/by-name/he/hello/package.nix'
evaluating file '/nix/store/1dklh99pdzxcxmrxz3i63rb3pa832yik-source/pkgs/stdenv/generic/common-path.nix'
querying info about '/nix/store/67bnnxkgnqcf3id74d4n8ijizxifcjvl-hello-2.12.1' on 'https://cache.nixos.org'...
downloading 'https://cache.nixos.org/67bnnxkgnqcf3id74d4n8ijizxifcjvl.narinfo'...
Segmentation fault: 11

That said, poking at this one disproved my own thesis about this only affecting the traditional commands:

$ nix run --verbose --print-build-logs nixpkgs#hello heh
Segmentation fault: 11

$ nix run -vvvvv  nixpkgs#hello heh
acquiring write lock on '/nix/var/nix/temproots/95971'
using cache entry '{"name":"flake-registry.json","type":"file","url":"https://channels.nixos.org/flake-registry.json"}' -> '{"etag":"W/\"5af996fe37b5c4ba47ef9b3d545ba977b527e2dcdc6e926dda7787b7db6bcf6e\"","url":"https://channels.nixos.org/flake-registry.json"}', '/nix/store/5fjljy6g6l4lfdxcb1hjkg4bwrs2kqpx-flake-registry.json'
looked up 'flake:nixpkgs' -> 'path:/nix/store/1dklh99pdzxcxmrxz3i63rb3pa832yik-source?lastModified=1701336116&narHash=sha256-kEmpezCR/FpITc6yMbAh4WrOCiT2zg5pSjnKrq51h5Y%3D&rev=f5c27c6136db4d76c30e533c20517df6864c46ee'
using substituted/cached input 'path:/nix/store/1dklh99pdzxcxmrxz3i63rb3pa832yik-source?lastModified=1701336116&narHash=sha256-kEmpezCR/FpITc6yMbAh4WrOCiT2zg5pSjnKrq51h5Y%3D&rev=f5c27c6136db4d76c30e533c20517df6864c46ee' in '/nix/store/1dklh99pdzxcxmrxz3i63rb3pa832yik-source'
got tree '/nix/store/1dklh99pdzxcxmrxz3i63rb3pa832yik-source' from 'path:/nix/store/1dklh99pdzxcxmrxz3i63rb3pa832yik-source?lastModified=1701336116&narHash=sha256-kEmpezCR/FpITc6yMbAh4WrOCiT2zg5pSjnKrq51h5Y%3D&rev=f5c27c6136db4d76c30e533c20517df6864c46ee'
checking access to '/nix/store/1dklh99pdzxcxmrxz3i63rb3pa832yik-source/flake.nix'
evaluating file '/nix/store/1dklh99pdzxcxmrxz3i63rb3pa832yik-source/flake.nix'
old lock file: {
  "nodes": {
    "root": {}
  },
  "root": "root",
  "version": 7
}
computing lock file node ''
new lock file: {
  "nodes": {
    "root": {}
  },
  "root": "root",
  "version": 7
}
trying flake output attribute 'apps.x86_64-darwin.hello'
using cached attrset attribute ''
trying flake output attribute 'packages.x86_64-darwin.hello'
using cached attrset attribute ''
trying flake output attribute 'legacyPackages.x86_64-darwin.hello'
trying flake output attribute 'hello'
using cached attrset attribute ''
using cached string attribute 'legacyPackages.x86_64-darwin.hello.type'
using cached string attribute 'legacyPackages.x86_64-darwin.hello.drvPath'
using cached string attribute 'legacyPackages.x86_64-darwin.hello.outPath'
using cached string attribute 'legacyPackages.x86_64-darwin.hello.outputName'
using cached string attribute 'legacyPackages.x86_64-darwin.hello.name'
using cached string attribute 'legacyPackages.x86_64-darwin.hello.meta.mainProgram'
querying info about missing paths...
starting pool of 24 threads
checking substituter 'https://cache.nixos.org' for path '/nix/store/67bnnxkgnqcf3id74d4n8ijizxifcjvl-hello-2.12.1'
checking substituter 'https://cache.nixos.org' for path '/nix/store/67bnnxkgnqcf3id74d4n8ijizxifcjvl-hello-2.12.1'
Segmentation fault: 11

Discussion on Matrix (mostly others reporting not having this problem) is making me think this might be hinging on macOS version. We’re both using 10.15.7, and I’ve also been able to confirm similar behavior on an old 10.14.6 system.

This may also have something to do with the recent stdenv bump to LLVM 16.

I don’t see the segfault with a pre-2.19 build of Nix that uses libcxx-11.1.0, while I do see segfaults with a more recent build of Nix 2.18.1 that uses libcxx-16.0.6.

Edit: libcxx-{15,16} segfault when dlopened on macOS 11.7.10 x64 · Issue #269548 · NixOS/nixpkgs · GitHub may also be related

I poked at this a bit this weekend and have a better sense of where it’s failing and even a workaround for at least one case.

The segfaults appear to happen when printMissing tries to printMsg the float document/NAR sizes here:

We only end up on this code path when print-missing is true (though this is the default):

So, if you invoke with --option print-missing false, it should at least work for now. This segfault occurs before the --dry-run flag takes effect, so you can use it to help confirm the segfault/no-segfault behavior here without realizing whatever test package you’re using.

2 Likes

@abathur the --option print-missing false workaround you gave is working great for me, thanks for sharing it!

my darwin-rebuild now runs to completion

There truly is no better christmas gift than a working nix system, thank you :slight_smile:

1 Like

Hi I have a similar problem, as stated in Nix-shell by signal SIGSEGV (Address boundary error) (in my case in Linux/NixOS and only with nix-shell) and this flag made the command working again :tada: Thanks!

Without the flag, nix-shell is working in my system only if the derivation is not available locally, and it has to be downloaded (so the first time). At the second run, I have a segfault. Adding the option, nix-shell continues working after the first invocation.

@aaronpkelly I’m not sure if you can un-mark the solution, but it might be useful to leave this ~open so that it’s easier for others to find and interact with until we see if we can get a more proper fix here (whether in Nix or Nixpkgs).

To that end, I reported this up in segfault during substitution on x86-64_darwin · Issue #9640 · NixOS/nix · GitHub