Do you have this in a public repo by any chance? It’s the evening my time but I’ll come back to this tomorrow.
Hi Harris,
The repo is here: GitHub - DruidNx/geonix-gdal-compile-test: Trying to compile gdal with non-standard libc
It was basically generated using the Geospatial Nix today configurator, with your suggested changes overlaid.
Thanks a lot!
I’ve just looked at my code. If I build a binary using a custom stdenv, I can check which glibc it uses via ldd --version ./result/bin/my_binary. So:
$ ldd --version result/bin/my_binary
ldd (GNU libc) 2.38
Copyright (C) 2023 Free Software Foundation, Inc.
(where glibc 2.38 was the one I specified).
However, this doesn’t mean that all of the dependencies of that binary use the same glibc:
ldd result/bin/my_binary | grep glibc
libdl.so.2 => /nix/store/d4452z0kdlib9g36j926n91936jhra9i-glibc-2.28/lib/libdl.so.2 (0x00007f943e82c000)
libpthread.so.0 => /nix/store/d4452z0kdlib9g36j926n91936jhra9i-glibc-2.28/lib/libpthread.so.0 (0x00007f943e2ab000)
librt.so.1 => /nix/store/d4452z0kdlib9g36j926n91936jhra9i-glibc-2.28/lib/librt.so.1 (0x00007f943e2a1000)
That outcome (that the binary plus all dependencies would be built using the specified stdenv) was the one that I initially wanted, but it is much harder to achieve - I spent about a week on it before giving up.
Hi Harry, thanks a lot for trying. I didn’t know the binary itself had a different glibc version. I will try to just hardcode the stdenv in the dependencies.
I found this thread helpful while attempting to build nixpkgs-2411 using older glibc 2.34 from nixpkgs-22.05, and wanted to contribute my working solution in case others find it useful.
@harris-chris’s functions were a great starting point, and I could define custom stdenvs easily enough, but there were a few more hoops to jump through in order to replace the default stdenv for the entire nixpkgs closure.
Using an Older glibc (2.34) from nixpkgs-22.05 with nixpkgs-2411
Overriding glibc in the New nixpkgs Context
There are several tricks going on here. First, referring to the older instantiated package but overriding a few important args so that it actually builds in the context of the newer nixpkgs.
glibc = (nixpkgs-2205.glibc.override {
inherit (prev) lib stdenv callPackage buildPackages;
})
Normally I prefer to refer to the older src file directly and callPackage it in the new context, like this
glibc = prev.callPackage "${nixpkgs-2205-tarball}/pkgs/development/libraries/glibc/default.nix" {};
which typically works for simpler packages, but does not work for glibc at this time. Maybe because it itself does callPackage ./common.nix? I didn’t quite finish understanding why.
Using Both an Overlay and config.replaceStdenv
A major lesson I learned: you cannot simply set stdenv = inside an overlay—at least not using the native bootstrap path. This approach does seem to work in cross-compilation, but not in a native build.
The issue with setting stdenv directly in an overlay is that it triggers assertion failures in pkgs/stdenv/linux/default.nix. If I set both glibc and stdenv in the overlay, I run into:
assertion '(isFromBootstrapFiles (prevStage).binutils.bintools)' failed
Conversely when I do everything in config.replaceStdenv and then try to assign glibc = stdenv.cc.libc, I hit the opposite error
assertion '(isBuiltByNixpkgsCompiler (prevStage)."${(localSystem).libc}")' failed
I’m not sure if disabling those assertions would make it work. Maybe they are there for my own protection and sanity. The commit that introduced them explains that these assertions were added to prevent stdenv bootstrap comments from becoming outdated.
Those assertions are absent from pkgs/stdenv/cross/default.nix, could explain why it works in a cross-compile context.
Note that using config.replaceStdenv works by appending one extra stage to the bootstrap chain. The new final stage has a stdenv which refers to glibc built from the N-1 stage.
Adjustments Needed to Build glibc in the New Context
The third trick is not exactly a trick – merely all the shenanigans required to get glibc to build and function in the newer nixpkgs context
- Disabling
-werrorsince newer gcc is more strict - Using an older make
- Ensuring
libgccvisibility - Exposing
getentas a separate output
Sidecar packages
Lastly, glibcLocales and glibcIconv ought to come along for the ride. These are simple enough that I did not even bother rebuilding them in the newer nixpkgs context; I simply took their instantiations straight from nixpkgs-22.05 and they seem to work.
1 Like
Sorry to bump, but I’m back with an addendum and it’s too late to edit the previous post. I got too excited that my previous post successfully eval’ed and I had not checked that it builds successfully. Two minor changes were needed to build:
ld cannot find appropriate glibc symbols
When building, I got this error while trying to build expand-response-params:
/nix/store/yfssjrhmk8gwm7igbphx2kgjss3qrir7-binutils-patchelfed-ld-2.43.1/bin/ld: /nix/store/85zzxby4jkvd8wqglvj94rbj58643d3a-glibc-2.34-210/lib/libc.so.6: version 'GLIBC_2.38' not found (required by /nix/store/yfssjrhmk8gwm7igbphx2kgjss3qrir7-binutils-patchelfed-ld-2.43.1/bin/ld
I guess the problem here is you’re really not ever supposed to downgrade glibc. The binutils at fault is from stage2 in the stdenv bootstrap. That stage doesn’t rebuild binutils, it just uses patchelf to make it link against the “new” glibc. This works if you’re upgrading, and breaks if you’re downgrading.
The solution is: avoid downgrading glibc. That means our bootstrap packages have to use a glibc at least as old as what we want to end up with. The new replaceBootstrapFiles config option is handy for this.
config.replaceBootstrapFiles = prevFiles:
(nixpkgs-2205.callPackage "${nixpkgs-2205-tarball}/pkgs/stdenv/linux/make-bootstrap-tools.nix" { })
.bootstrapFiles;
ld: cannot find -lgcc_s: No such file or directory
This turned out to be a pretty simple mistake on my part. pkgs.wrapCCWith needs to accept an unwrapped cc argument, or else it gets double-wrapped and cannot find the compiler libs.
pkgs.wrapCCWith {
- cc = pkgs.gcc;
+ cc = pkgs.gcc-unwrapped;
I updated the gist with these changes and am now able to build successfully.