Build-id, `separateDebugInfo` and nixseparatedebuginfod2

Help
1

So today on Nixos 25.11 I tried adding a task via merkuro-calendar and the app crashed. I tried opening drkonki to gather the crashdump and… it crashed.

I have enabled the nixseparatedebuginfo2 service.

Here is the log:

Starting debugger /nix/store/9zp694qlqvalxa72fj83k1r9jhq97gbf-gdb-host-cpu-only-16.3/bin/gdb --init-eval-command=set auto-solib-add off --nw --nx --batch --init-eval-command=set debuginfod enabled on --command=/tmp/drkonqi.UIrXQy --command=/tmp/drkonqi.fZZsab --core=/tmp/drkonqi-core.jRpfRp/core /nix/store/3a7m9rbyifqwzwjazdcznlwyk7g6sjjp-merkuro-25.08.3/bin/.merkuro-calendar-wrapped

warning: Can't open file /memfd:JSGCHeap:QtQml (deleted) during file-backed mapping note processing

warning: Can't open file /memfd:sljit (deleted) during file-backed mapping note processing

warning: Can't open file /memfd:JSVMStack:QtQml (deleted) during file-backed mapping note processing

warning: Can't open file /memfd:unknown-usage:QtQml (deleted) during file-backed mapping note processing

warning: Can't open file anon_inode:i915.gem during file-backed mapping note processing

warning: Can't open file /memfd:kwin-dmabuf-feedback-table (deleted) during file-backed mapping note processing

warning: Can't open file /memfd:JITCode:QtQml (deleted) during file-backed mapping note processing
[New LWP 149206]
[New LWP 149207]
[New LWP 149217]
[New LWP 149242]
[New LWP 149240]
[New LWP 149246]
[New LWP 149245]
[New LWP 149250]
[New LWP 149208]
[New LWP 149238]
[New LWP 149210]
[New LWP 149243]
[New LWP 149223]
[New LWP 149222]
[New LWP 149211]
[New LWP 149224]
[New LWP 149221]
[New LWP 149209]
[New LWP 149219]
[New LWP 149216]
[New LWP 149239]
[New LWP 149247]
[New LWP 149249]
[New LWP 149244]
[New LWP 149248]
[New LWP 149241]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/xx7cm72qy2c0643cm1ipngd87aqwkcdp-glibc-2.40-66/lib/libthread_db.so.1".
Core was generated by `/nix/store/3a7m9rbyifqwzwjazdcznlwyk7g6sjjp-merkuro-25.08.3/bin/.merkuro-calendar-wrapped'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f12efe9caac in __pthread_kill_implementation () from /nix/store/xx7cm72qy2c0643cm1ipngd87aqwkcdp-glibc-2.40-66/lib/libc.so.6
[Current thread is 1 (Thread 0x7f12d1c3f540 (LWP 149206))]
Traceback (most recent call last):
  File "/nix/store/fzlkp2s7751zq1kgi86hd2hijz9mjjl8-drkonqi-6.5.3/share/drkonqi/gdb/python/gdb_preamble/preamble.py", line 608, in print_preamble
    print_preamble_internal()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/nix/store/fzlkp2s7751zq1kgi86hd2hijz9mjjl8-drkonqi-6.5.3/share/drkonqi/gdb/python/gdb_preamble/preamble.py", line 582, in print_preamble_internal
    resolve_modules()
    ~~~~~~~~~~~~~~~^^
  File "/nix/store/fzlkp2s7751zq1kgi86hd2hijz9mjjl8-drkonqi-6.5.3/share/drkonqi/gdb/python/gdb_preamble/preamble.py", line 577, in resolve_modules
    image = CoreImage(line)
  File "/nix/store/fzlkp2s7751zq1kgi86hd2hijz9mjjl8-drkonqi-6.5.3/share/drkonqi/gdb/python/gdb_preamble/preamble.py", line 545, in __init__
    raise NoBuildIdException(f'No build id found in core image: {eu_unstrip_line}')
NoBuildIdException: No build id found in core image: 0x7f1233f84000+0x7bcb8 - /nix/store/yijhn548p2589pkybgvbhll09bqsxy0q-ncurses-6.5/lib/libncursesw.so.6 - libncursesw.so.6
No build id found in core image: 0x7f1233f84000+0x7bcb8 - /nix/store/yijhn548p2589pkybgvbhll09bqsxy0q-ncurses-6.5/lib/libncursesw.so.6 - libncursesw.so.6
Sentry is attempting to send 2 pending events
Waiting up to 30 seconds
Press Ctrl-C to quit
Debugging ended with exit code '1' and exit status 'NormalExit'
Debugging ended with exit code '1' and exit status 'NormalExit'

So it seems it crashed because libncurses does not have the .note.gnu.build-id metadata in the ELF. I checked out and some programs and libraries from cached nixpkgs have it and others don’t. For example, GNU hello does not provide it even when using the dontStrip option, but it does if you use the separateDebugInfo flag. But wrapped firefox does without using the latter flag.

Now, I was under the impression that even without separateDebugInfo, one could get debug symbols on demand with nixseparatedebuginfo2 or dwarffs. However, if one uses nixseparatedebuginfo2 to fetch those symbols, but the binary is stripped from the build-id, how could it work?
If instead the binary already comes with both build-id and symbols, why use nixseparatedebuginfod2 in the first place?

2

When a derivation is built with separateDebugInfo = true, the final libraries will have a build-id, and the debug symbols will be separated from the libraries into a separate output called debug. So on a typical install, one only has the stripped executables/libraries, for example /nix/store/jlv0a6iyh8vb7pfyhjj93xy245xlmgh5-gnumake-4.4.1/bin/make with build-id 5ba4a279aeaa0f717a07b1b5298cbdef3210ca4e, but not the debug symbols (which can be quite large, and are typically not used). The debug symbols are in /nix/store/jyd69jw95fcsgnl9p5r6k3iycls32vsy-gnumake-4.4.1-debug/lib/debug/.build-id/5b/a4a279aeaa0f717a07b1b5298cbdef3210ca4e.debug which must be downloaded when gdb needs them. That’s the job of nixseparatedebuginfod2 or dwarffs.

When a derivation is not built with separateDebugInfo = true, debug symbols are stripped or not even produced in the first place. They also don’t have build-id, as the purpose of the build-id is to be a link between the original executable and its debug symbols.

3

I see. I must have confused enableDebugging and separateDebugInfo.
Still, I am not sure whether stripping/not generating a build-id by default is a good idea. As I showed above, KDE’s drkonqi crashes if ncurses does not have a build-id, even though it does not actually use its debug symbols. Why would ncurses (a CLI library) matter for dr konqi anyway?
I think some KDE maintainer should step into the discussion.

EDIT: it seems this bug was solved upstream 2 years ago, but it never made it to any KDE release for some reason (edit2: apparently the author of that commit backtracked and stated this is expected behaviour). I guess this can be closed.

4

I don’t really know what exactly drkonki is supposed to do, but if the fix was reverted in drkonki it might make sense to carry a revert of the revert downstream, especially if it makes drkonki completely useless on NixOS. I expect very few if any executables on a NixOS system have build ids on all their libraries, and this will not change any time soon. Before we can make separateDebugInfo the default I suspect we would need a good solution for downsizing the binary cache, as debug info can be larger than the original software.

1 Like