I gazed into the gold abyss today while asking about updating binutils to backport some CVE patches in #staging in Matrix and since I don’t have other nixpkgs responsibilities, I thought I’d initiate a discussion here to eventually move onto a GitHub issue about removing gold (which is deprecated) from nixpkgs.
For context, gold is (apparently) marginally faster than some other upstream GNU linkers, but Google’s original development team has largely moved on from its development. In a recent binutils release, the maintainers acknowledged this by beginning to sunset gold by removing it from source distributions entirely in even-numbered release tarballs.
This puts nixpkgs on somewhat of a timer to clean it up from the repo for other, more-maintained options.
The remainder of this post is seat-of-my-pants planning from me, a non-commit-bit-holding contributor to nixpkgs, so take it all with a grain of salt. If someone wants to handle this work instead, please feel free to do so (I’m assuming there’s a good chance that someone with more history maintaining binutils might have started this work already).
Nixpkgs Context
The list of dependent packages on gold is large but not excessively daunting; many packages like mongo likely now better support other linkers than when their gold reliance was originally written. Others like ghc should support alternative linkers with some edits; the comments I’ve read seem to point at gold mostly as a “this seems like a good idea for performance reasons” and not other extenuating circumstances.
I am emphatically not a toolchain expert but if there are performance considerations for linker choice I would wager that falling back on something like lld would be sufficient. Given Rust 1.90.0’s recent move to lld it should be a stable target and it’s specifically used over other linkers in many spots in nixpkgs to crib from.
Proposal
Start whittling away at gold dependent packages
A naive rg gives us a place to start with packages like mongo, ghc, etc.
Once gold reliance is gone, remove support for gold entirely
Get this in for 26.05
And track it in a GitHub issue.
Help
@K900 and I talked about this in Matrix, they invoked @fabianh as someone who has worked on binutils as well. Again, anybody feel free to grab this baton from me and run with it if they’d prefer. But I’ve got nothing else on my plate and don’t mind doing the toil since others probably have more important maintenance work to do.
This makes sense to me. It use to make sense to “root for gold” because it was the only other game in town (only other FOSS alternative to the GNU BFD ld), but now we have LLD, Mold, Wild, and many other options. GNU BFD ld will not be the only game in town either way.
Not quite so marginally, unfortunately. We investigated switching from ld.gold to ld.bfd for Haskell this summer. It does indeed look like the ld.bfd bugs that necessitated using ld.gold in the past, have been fixed, but performance is still terrible for linking Haskell executables. @wolfgangwalthercame up with 3-4x increase in total build time (!) for some packages.
It does look like using lld in the future is a good option, but I don’t really see the story for mixing different bintools derivations being very good at the moment. It would be great to get rid of the ld.gold logic, but ideally not at the cost of wiring up lld in an even yankier way.
Another question on my mind is how reliably lld is able to link C++ libs compiled by GCC, but maybe someone here can answer that.
@trofi thanks for the lead, I’m probably most-familiar with GHC/the Haskell ecosystem among the list of gold removal candidates, so I’ll likely start there.
@Ericson2314 thanks for the +1, I also was mostly looking at lld or mold if linking performance is a priority after dropping gold from the dependent packages.
@reckenrode I came across that thread actually; I’ll probably reference that when I reach Swift in the migration. If it defaults to the Clang linker then maybe lld would be okay; but regardless, I’ll ensure to read through the Swift repo to ensure it’s not going to have silent failures with whatever I replace gold with.
@sternenseemann thank you for the context. Based on that PR it really seems like we shouldn’t just rely on older/suboptimal linkers for packages like that. At the very least, ld.bfd would be a significant downgrade in terms of performance.
I’m giving this thread time to bake before starting work on anything, but one conclusion I arrived at in the ~24h thinking about it is that I’m definitely not going to swap out linkers blindly at the risk of inflating link times for the impacted packages. My intent is to swap out gold for something more performant than vanilla ld like lld or mold and also ensure to benchmark them so we’re not regressing build times. Basically my intent is to take something like ghc and build with hyperfine, then benchmark build times with lld and any other candidate linkers that might work to gauge impact (maybe not hyperfine, but you get the idea).
The primarily concern I’m still thinking about is derivations that build successfully but have silent, broken linking failures (I ran into this myself doing some unrelated Rust 1.90.0 lld work). I’m not sure how to ensure I can reliably catch that, but I’m consciously thinking about it.
My personal take is that I’ve usually heard positive reviews for mold among the linker pantheon and intend to at least try it for the impacted packages; maybe we end up with faster linking in many cases without breakage. But please rest assured the primary goal here is functional build artifacts and we certainly won’t be swapping in anything that causes instability. Fortunately lots of projects like Swift and GHC have lots of prior discussion with alternative linker findings to help guide a lot of this.
Since Mold/Wild are just linkers, and not entire suites of bintools, do we already have ways of using them with GNU binutils for the other tools? (I.e. it wouldn’t be a new hack that is just created for the purpose of retiring Gold?)
Absolutely agreed the packages that are specifically using Gold to avoid BFD should not go back to BFD.
Note that BFD is not an abandoned linker. It gets it’s speedups time to time and certainly is the primary target to support new features like new relocation types or interaction with gcc compiler plugins. For example https://www.youtube.com/watch?v=h5pXt_YCwkU describes some recent speedup progress.
I would consider any major speed slowdown in BFD a bug worth reporting to Making sure you're not a bot!. Sometimes the fix is trivial. Sometimes it’s not.
I can have a look at the performance regressions on ghc packages.
I think some packages just have special logic to use mold if available (yikes!), other than that I see just the stdenv adapter, but I’m not sure whether using a modified stdenv for haskellPackages is that nice.
Since we just pass absolute paths to tools to GHC, in theory it is very easy to change to e.g. lld. The story for letting the user control this is not great, though. Also, I do not know whether this going to continue to be so easy with whatever changes ghc-toolchain imposes in 9.14 (?).
My recommendation would be to save Swift for last. When I call it a rewrite, I mean the whole hierarchy under development/compilers/swift is getting replaced, so you could end up doing wasted work. Ideally, when I change it to use the existing LLVM derivation to build Swift’s fork, it will use whatever LLVM in nixpkgs defaults to using.
@reckenrode I appreciate that. Without targeting 25.11 here we should have some breathing room so when I draft up a tracking issue I’ll defer any Swift changes to wait on any work ongoing with Swift and wait until a green light from you/Swift maintainers.
@trofi thanks, I’ll also denote that you’re looking at GHC. Like I mentioned, I’ve dabbled more with Haskell than Swift but I’m happy to let you own at least a portion of work there.
@RossComputerGuy just note here if you want to take a portion of the migration and I’m happy to track it in the subsequent GitHub issue. Thanks
I’ll leave this topic up a bit longer to solicit input before formalizing a list in GitHub, after which I’ll mark this as closed and carry on the work closer to the repo.
I don’t plan to work on ghc linker re-plumbing in nixpkgs. I planned to look at performance regression of ld.bfd compared to ld.gold on ghc inputs. Here is the resulting upstream bug report: Making sure you're not a bot!
I think it has a chance of a simple fix as it seems to be triggered primarily by -fsplit-sections / -Wl,--gc-sections on ghc side and seems to redo the same scanning work.
That’s too bad since those things are pretty important to Nixpkgs since it is the principal way GHC is able to do dead code elimination. Without link time elimination of code, we incur a lot of incorrect references in real world Haskell packages that prevent using multiple outputs without horrible hacks.
I think the aim of ghc-toolchain is to make these sorts of things easier and to avoid having to reinstall GHC when you want a different toolchain. Although changes like this often lead to teething difficulties.
@TeofilC I came across your shelved PR and yes, I think we should probably revive the effort. gold is on life support now so if we do indeed want to remove it somewhere along the timeline for 26.05 then GHC will have to migrate eventually, I guess. Consensus here seems to indicate that it’s time to do so
@trofi noted looking into performance changes when swapping to ld.bfd, so feel free to own the GHC/${other linker} side. You know the problem better than I so do what seems best; I assume that lld may be the next-best-supported option and we probably want at least some form of benchmark comparison against gold to look at. I’m sort of curious about the delta between lld and mold but the latter seems like it may be a more weighted risk/perf tradeoff, but who knows Thanks for the help!
@sternenseemann I think we could raise the issue of Gold’s looming deprecation with the Haskell community / Foundation, and they could perhaps self-fund some investigation into either using/endorsing another non-BFD linker, or investigation and fixing the issue with ld.bfd if it is indeed a simple fix.
Ultimately it is in everyone’s interest (GHC official bindists, GHCup, all downstream packagers, etc.) not to rely on deprecated tools for performance.
That sounds like a good idea. FWIW GHC already prefers lld to bfd if the ld-override is enabled (the linker choice is made at bindist install time and isn’t hardcoded), but GHCup recently disabled this to use the system linker instead, ie, whatever $LD corresponds to.
Thanks for the help!