Most of my programs segfault

Help
1

The following programs segfault after a few seconds/immediately on my system:

  • spotify – stdout/stderr outputs only Segmentation fault (core dumped)
  • discord – the same, sometimes never opens
  • steam – after a few seconds, goes into a death spiral of crashing/reopening, stdout unhelpful
  • chromium – freezes after ~10 secs

All these happen under Hyprland, Gnome, and Gnome under Xorg

  • FreeCAD – crashed yesterday under Hyprland with info that this happens in glibc somewhere, but now doesn’t seem to crash under Gnome/Xorg

programs that don’t crash:

  • firefox
  • waybar
  • thunderbird
  • kitty
  • nautilus
  • etc.

my current system info is:

 OS: NixOS 24.11pre690827.5633bcff0c61 (Vicuna)
 Kernel: x86_64 Linux 6.6.54
 Uptime: 2h 53m
 Packages: 11664
 Shell: bash 5.2.32
 Resolution: 3840x1080
 DE: GNOME 46.4
 WM: Mutter
 WM Theme: 
 GTK Theme: Adwaita [GTK2/3]
 Icon Theme: Adwaita
 Font: Cantarell 11
 Disk: 838G / 1,8T (49%)
 CPU: Intel Core i7-9750H @ 12x 4.5GHz [86.0°C]
 GPU: NVIDIA GeForce GTX 1660 Ti
 RAM: 5929082880-

nix-info:

  • system: "x86_64-linux"
  • host os: Linux 6.6.54, NixOS, 24.11 (Vicuna), 24.11pre690827.5633bcff0c61
  • multi-user?: yes
  • sandbox: yes
  • version: nix-env (Nix) 2.18.8
  • channels(root): "nixos"
  • nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

I am currently on nixos-unstable, but rolling back to nixos-24.05 didn’t work while now having every single graphical program (inside the gnome-shell) crashes. I also tried switching from the nvidia beta drivers to the stable drivers, and also switched to the open drivers – nothing.

My system used to work flawlessly for months, I don’t know what changed. After nixos-rebuild switch something broke and I don’t have a working version anymore (because first my browser broke fsr and then I had to nix-collect-garbage because boot was full, and then firefox worked again but after reboot my other apps broke). I don’t know if it’s electron breaking upstream or the nvidia drivers crashing or something completely different.
Any ideas?

2

Check coredumpctl’s backtrace output to try and identify the common functions that are failing

3

Unfortunately, all crashing apps are complicated, with many threads. So I’m ignoring threads that are in epoll_wait, _poll, _futex_*, or other obvious waiting

Otherwise, chromium gave me the following stack traces (added demangling in post)
Stack trace of thread 6005:

#0  0x00007f3d48dabac5 realpath@@GLIBC_2.3 (libc.so.6 + 0x41ac5)
#1  0x000055b88396371a base::MakeAbsoluteFilePath(base::FilePath const&) (chromium + 0x8b0271a)
#2  0x000055b885133bef ui::(anonymous namespace)::ReadCursorFromTheme(std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> > const&, std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> > const&) (chromium + 0xa2d2bef)
... (many more copies of the same function)

Stack trace of thread 6007:

#0  0x00007f3d48e6c45b readlink (libc.so.6 + 0x10245b)
#1  0x00007f3d48dabdfa realpath@@GLIBC_2.3 (libc.so.6 + 0x41dfa)
#2  0x000055b88396371a base::MakeAbsoluteFilePath(base::FilePath const&) (chromium + 0x8b0271a)
#3  0x000055b885133bef ui::(anonymous namespace)::ReadCursorFromTheme(std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> > const&, std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> > const&) (chromium + 0xa2d2bef)
... (many more copies of the same function)

Stack trace of thread 6006:

#0  0x00007f3d48e6c45b readlink (libc.so.6 + 0x10245b)
#1  0x00007f3d48dabdfa realpath@@GLIBC_2.3 (libc.so.6 + 0x41dfa)
#2  0x000055b88396371a base::MakeAbsoluteFilePath(base::FilePath const&) (chromium + 0x8b0271a)
#3  0x000055b885133bef ui::(anonymous namespace)::ReadCursorFromTheme(std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> > const&, std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> > const&) (chromium + 0xa2d2bef)
... (many more copies of the same function)

Stack trace of thread 4018:

#0  0x000055b883880af3 base::FilePath::DirName() const (chromium + 0x8a1faf3)
#1  0x000055b88388074f base::FilePath::GetComponents() const (chromium + 0x8a1f74f)
#2  0x000055b885133b58 ui::(anonymous namespace)::ReadCursorFromTheme(std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> > const&, std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> > const&) (chromium + 0xa2d2b58)
... (many more copies of the same function)

Stack trace of thread 6022:

#0  0x000055b8839c0840 allocator_shim::internal::PartitionMalloc(unsigned long, void*) [clone .cfi] (chromium + 0x8b5f840)
#1  0x000055b88318f8cc base::allocator::dispatcher::internal::DispatcherImpl<base::PoissonAllocationSampler>::AllocFn(unsigned long, void*) [clone .cfi] (chromium + 0x832e8cc)
#2  0x000055b8839bfb9e operator new(unsigned long) (chromium + 0x8b5eb9e)
#3  0x000055b8831bb248 std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> >::basic_string(std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> > const&, unsigned long, unsigned long, std::__Cr::allocator<char> const&) (chromium + 0x835a248)
#4  0x000055b883880b47 base::FilePath::DirName() const (chromium + 0x8a1fb47)
#5  0x000055b88388074f base::FilePath::GetComponents() const (chromium + 0x8a1f74f)
#6  0x000055b885133b58 ui::(anonymous namespace)::ReadCursorFromTheme(std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> > const&, std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> > const&) (chromium + 0xa2d2b58)
... (many more copies of the same function)

Stack trace of thread 6016:

#0  0x00007f3d48e6c3dc read (libc.so.6 + 0x1023dc)
#1  0x000055b8836f92fe (anonymous namespace)::ShutdownDetector::ThreadMain() (chromium + 0x88982fe)
#2  0x000055b88394de07 base::(anonymous namespace)::ThreadFunc(void*) [clone .a67435033112019129ad5c28ddc47327] [clone .cfi] (chromium + 0x8aece07)
#3  0x00007f3d48dfaa42 start_thread (libc.so.6 + 0x90a42)
#4  0x00007f3d48e7a05c __clone3 (libc.so.6 + 0x11005c)

For Discord, it was these not currently polling/futex/suspended:
Stack trace of thread 7291:

#0  0x00007f409362eba5 realpath@@GLIBC_2.3 (libc.so.6 + 0x41ba5)
#1  0x000055fb77554528 n/a (.Discord-wrapped + 0x60bc528)
#2  0x000055fb786784ac n/a (.Discord-wrapped + 0x71e04ac)
#3  0x000055fb7867821b n/a (.Discord-wrapped + 0x71e021b)
... (many more copies of the same function)

Stack trace of thread 7290:

#0  0x00007f40936ef45b readlink (libc.so.6 + 0x10245b)
#1  0x00007f409362edfa realpath@@GLIBC_2.3 (libc.so.6 + 0x41dfa)
#2  0x000055fb77554528 n/a (.Discord-wrapped + 0x60bc528)
#3  0x000055fb786784ac n/a (.Discord-wrapped + 0x71e04ac)
#4  0x000055fb7867821b n/a (.Discord-wrapped + 0x71e021b)
... (many more copies of the same function)

Stack trace of thread 7313:

#0  0x00007f40936ef3dc read (libc.so.6 + 0x1023dc)
#1  0x000055fb73ad7633 n/a (.Discord-wrapped + 0x263f633)
#2  0x000055fb77543251 n/a (.Discord-wrapped + 0x60ab251)
#3  0x00007f409367da42 start_thread (libc.so.6 + 0x90a42)
#4  0x00007f40936fd05c __clone3 (libc.so.6 + 0x11005c)

Stack trace of thread 7702:

#0  0x00007f40936ef3dc read (libc.so.6 + 0x1023dc)
#1  0x00007f4093674753 _IO_file_underflow@@GLIBC_2.2.5 (libc.so.6 + 0x87753)
#2  0x00007f4093669000 __getdelim (libc.so.6 + 0x7c000)
#3  0x00007f3ff0f547ca get_reply (libspeechd.so.2 + 0x47ca)
#4  0x00007f3ff0f54bf0 spd_events_handler (libspeechd.so.2 + 0x4bf0)
#5  0x00007f409367da42 start_thread (libc.so.6 + 0x90a42)
#6  0x00007f40936fd05c __clone3 (libc.so.6 + 0x11005c)

and Spotify gave me similar results.

Doing coredumpctl debug -1 on the Discord crash and then a bt in gdb only shows me the realpath@@GLIBC_2.3 () thread, so I would guess that’s the origin of the segfault? Or just the first thread and that is coincidence?

4

I’m now finding crashes in what I believe to be __printf_buffer (libc.so.6). Could it be that glibc is loaded incorrectly and calling it leads to jumping to wrong addresses? So the chromium engine (which seems to be the thing present in all crashing programs) has the wrong glibc as a linking input?

5

What I can say is that (after rerunning) all threads for spotify and all except one for discord & chromium (unrelated functions in each) are inside libc.so.6 in the core dump.

6

Had a similar issue. Turned out I had a broken RAM module which resulted in a bunch of files in the nix store to be corrupted.

I tested my RAM modules with tools like memtest86 (assuming you’re running an X86 platform).

After removing the broken RAM I booted from an USB stick (since I mistrusted nix to be unaffected).
AFAICR I fixed my nix store with nix store repair --all --repair which took quite a bit.

1 Like
7

So after minimizing my installation and deleting all garbage, nix store repair --all --repair fails with
error: cannot repair path '/nix/store/bdgm65yp8kw52hxw7k6f8al97qmc9msv-glibc-locales-2.39-52.drv',
which from investigating the crashes I already assumed to be broken, and that particular derivation seems to be missing. What would I do to fix this?

8

Okay so I took the nuclear option of reinstalling nixos. Just hope I backed up everything correctly…