Announcing nixseparatedebuginfod, a tool to fetch sources+debug symbols of nix packages in gdb

Announcements
1

Using gdb to debug executables built with nix is a frustrating experience because it is hard to get debug symbols. Most nix packages are just stripped, so there is no hope for them, but a number are built with separateDebugInfo set to true, meaning that debug symbols are in a separate -debug store path. Sounds good, right? Unfortunately you don’t download them from hydra, and even if you had them in your local store, gdb would not know how to find them. To actually use those you need to do one of the following:

  • download the -debug output store path yourself, and put it in the NIX_DEBUG_INFO_DIRS env var, and repeat for every shared lib you need
  • use the environment.enableDebugInfo = true; NixOS option (or the hm equivalent), but this does not work for libraries, unless you add libraries to environment.systemPackages, which you shouldn’t do. Besides, waiting the long minutes of downloading 1GB of qemu debug symbols on most nixos-rebuild ends up frustrating in the long run.
  • use dwarffs, which downloads debug symbols from hydra on the fly and magically gives them to gdb. Great until you end up wanting to debug a library which does not have separateDebugInfo set to true, and recompile it locally with the flag: dwarffs works only for debug symbols on hydra, and not those in your own store.

And if you manage to overcome all those hurdles, here is the last straw: gdb does not have access to the source files. It will tell you the file name, but it’s up to you to download the tarball, find the file, and go the indicated line to follow.

To help will all this frustrating process, I am happy to announce nixseparatedebuginfod, a debuginfo server that provides debug symbols and sources to gdb for your separateDebuginfo nix packages, whether built locally, on hydra, or any other substituter.

It is based on the debuginfod protocol, which gdb can understand, and presumably other tools (although I know of no others, tell me if you know). And with nixseparatedebuginfod I can actually see where gdb is stopped when debugging nix, for example:

https://asciinema.org/a/556944

There are a few limitations, notably you still get debug symbols for packages built with separateDebugInfo set to true only, and source fetching not 100% reliable, but overall I hope this makes your debugging experience better :slight_smile:

24 Likes
2

I just tried this in NixOS and it worked like a charm :D. Unfortunately, this does not work with nix-darwin. It errors out at a missing pacakge pkgs.nixseparatedebuginfod

3

separatedebuginfo is not supported on darwin. See Darwin doesn't support separateDebugInfo (a fact which is also undocumented) · Issue #203380 · NixOS/nixpkgs · GitHub

Apparently a solution is not very far away bintools-wrapper: Set ZERO_AR_DATE and re-enable LC_UUID on Darwin by zhaofengli · Pull Request #188347 · NixOS/nixpkgs · GitHub but someone motivated and with access to darwin hardware should try to push this through the finish line.

2 Likes