One thing which is surprisingly hard for me in nixpkgs is reliably getting debug symbols. I’d like to start a discussion on whether this could be improved.
First, I’ll explain where the difficulties seem to lie to me as a relative newcomer (1.5 years) to the Nix ecosystem:
-
The nixpkgs configuration side is confusing, I’d like to say arcane:
separateDebugInfo,enableDebugging,dontStrip, etc. I think I mostly understand it now, but I must be in the minority. -
Too many packages built by Hydra don’t get built with debug symbols (like libstdc++)—I assume mostly because of storage space reasons instead of build time reasons? This means that if I want to debug anything in nixpkgs that depends on libstdc++ and have the debug symbols for libstdc++, I need to rebuild almost the world.
-
Writing derivations to reliably get debug info is just hard because upstream build scripts can do arbitrary things.
What I’ve done to tackle it for my own use
I’ve wasted too many days of my life figuring out how to make things have debug info, so I wrote a helper that can be used to assert at build time that there is debug info. It works like this:
verifyDebugInfo {
pkg = pkgs.hello; # a derivation that should have debug info (build fails if it doesn't)
globs = [ "$out/lib/*.so" ];
files = [ "$out/bin/hello" ];
}
Here’s an actual snippet from my config to get sway and some dependencies to have debug symbols:
sway =
let
wlroots-debug = pkgs.mylib.verifyDebugInfo {
pkg = pkgs.wlroots.overrideAttrs {
mesonBuildType = "debugoptimized";
separateDebugInfo = true;
};
globs = [ "$out/lib/*.so" ];
};
sway-debug = pkgs.mylib.verifyDebugInfo {
pkg = (
(pkgs.sway-unwrapped.overrideAttrs {
mesonBuildType = "debugoptimized";
separateDebugInfo = true;
}).override
{
wlroots = wlroots-debug;
}
);
globs = [ "$out/bin/sway*" ];
};
Implementation
# A Nix expression that adds a postFixup hook to verify that specified files
# contain debug information. It supports both explicit file paths and glob
# patterns.
# Usage:
#
# Call this function with a derivation and an attrset containing:
# - pkg: the derivation to verify
# - files: a list of explicit file paths to check
# - globs: a list of glob patterns to expand and check
#
# Returns: the derivation with an added postFixup hook that verifies debug info.
#
# Example:
# let
# verifyDebugInfo = import ./lib/verifyDebugInfo.nix { lib = lib; elfutils = pkgs.elfutils; };
# in
# verifyDebugInfo pkg { files = [ "$out/bin/mybinary" ]; globs = [ "$out/lib/*.so" ]; }
{ lib, elfutils }:
{
pkg,
files ? [ ],
globs ? [ ],
}:
pkg.overrideAttrs (oldAttrs: {
postFixup = (oldAttrs.postFixup or "") + ''
(
echo "verifyDebugInfo: Checking ${pkg.name}"
# Determine the debug output path (if it exists)
# Nix sets $debugOutput variable if separateDebugInfo = true,
# or we can try to guess via the $debug environment variable.
DEBUG_OUT="''${debug:-}"
check_debug() {
local f="$1"
[ -f "$f" ] || return 0
# Skip non-ELF files
if ! head -c 4 "$f" | grep -q 'ELF'; then
echo "verifyDebugInfo: SKIP (not ELF): $f"
return 0
fi
# 1. Check for inline debug info (fat binary)
if ${elfutils}/bin/eu-readelf -S "$f" | grep -q '\.debug_info'; then
echo "verifyDebugInfo: OK (Inline): $f"
return 0
fi
# Check for split debug info via build ID
local build_id
build_id=$(${elfutils}/bin/eu-readelf -n "$f" \
| grep "Build ID:" \
| sed 's/.*Build ID: \([0-9a-f]*\).*/\1/')
if [ -n "$build_id" ] && [ -n "$DEBUG_OUT" ]; then
# Nixpkgs stores debug info at: <debug-out>/lib/debug/.build-id/xx/yyyy.debug
local c1="''${build_id:0:2}"
local c2="''${build_id:2}"
local debug_file="$DEBUG_OUT/lib/debug/.build-id/$c1/$c2.debug"
if [ -f "$debug_file" ]; then
echo "verifyDebugInfo: OK (Split/BuildID): $f -> .../$c1/$c2.debug"
return 0
else
echo "verifyDebugInfo: ERROR: $f has Build ID $build_id, but debug file is missing!" >&2
echo " Expected: $debug_file" >&2
return 1
fi
fi
echo "verifyDebugInfo: ERROR: $f lacks debug info!" >&2
echo " - No inline .debug_info" >&2
echo " - No separate debug file found in $DEBUG_OUT (via build ID)" >&2
return 1
}
# we only exit at end
failed=0
# 1. Check explicit files
for raw_file in ${lib.escapeShellArgs files}; do
eval "file=\"$raw_file\""
if [ ! -e "$file" ]; then
echo "verifyDebugInfo: ERROR: Explicit path '$file' missing!" >&2; exit 1
fi
check_debug "$file" || failed=1
done
# 2. Check globs
for raw_pattern in ${lib.escapeShellArgs globs}; do
eval "pattern=\"$raw_pattern\""
shopt -s nullglob
expanded_files=( $pattern )
shopt -u nullglob
if [ ''${#expanded_files[@]} -eq 0 ]; then
echo "verifyDebugInfo: ERROR: Glob '$raw_pattern' matched nothing!" >&2; exit 1
fi
for file in "''${expanded_files[@]}"; do
check_debug "$file" || failed=1
done
done
if [ $failed -ne 0 ]; then
echo "verifyDebugInfo: FAILED"
exit 1
fi
echo "verifyDebugInfo: OK"
)
'';
})
Thoughts/proposals
There’s a few things on my wishlist.
1. nixpkgs configuration side confusing
This point is tricky; a lot of that is legacy, and it probably can mostly be improved by documentation.
2. Hydra building too little with debug symbols
This, I suspect (but do not know) is a conscious saving decision. Having said that, I wonder if we could at least build some more of the very basic libraries like libstdc++ with debug symbols.
Besides that, I have one radical proposal. If I guess right and the major issue is storage cost, could we have Hydra build the derivations with split debug symbols and then just throw the debug symbol part away? Rationale: Due to how nix works, anything that changes the derivation hash is a massive pain. This way it would at least be possible to build the split debug symbols yourself and just use them with whatever is in nixpkgs—i.e. you’d need to rebuild libstdc++, but not every derivation that depends on libstdc++.
Now, I’m not sure if there’s some reason why that doesn’t work because builds are not reproducible (but I understand nix tries hard to use same build ids).
# On the reliability of getting debug symbols
This is where I’d actually love something like my verifyDebugInfo get wider use. I have a couple of very initial ideas of how this could look like:
- Derivations can opt into being checked by either setting
verifyDebugInfo = true→ all ELFs in $out should have debug info when built with debug infoverifyDebugInfo = { files = ...; globs = ... }→ opt in for specific files in $outverifyDebugInfo = { exclude = ...; }→ general opt in with exclusions
- What this does depends on whether the derivation is built with some mechanism that asserts it wants debug info (it could be an env variable set by things like
enableDebugging, although I dislike how that conflates debug info and no optimizations):- If built with debug info: fail the build if the debug infos are not there
- If built without: Help maintain the fields by checking that the named files/globs are present and match something, but do not require debug info.
In any case for backwards compatibility this needs to clearly be opt in for derivations. And, I don’t really know if people care enough; sometimes I get the feeling I’m one of approximately three people in the world using debug symbols in nixpkgs, as evidenced by the state of libstdc++? 
Any thoughts?