Linking issue with libpthread from glibc 2.17

midchildan · October 25, 2023, 9:12pm

I’m trying to rebuild some packages in nixpkgs-unstable with glibc 2.17. So I made a package for glibc 2.17 and replaced stdenv’s libc with it. It got to the point where many packages now build, but some packages still fails to build due to a linking error.

After some investigation, I found out that the linker was unable to link a specific function called pthread_atfork in libpthread. At first, I suspected it was an issue of a missing -pthread compiler flag. But this wasn’t the case.

I created minimal example below. The same issue occurs despite having called gcc with the -pthread flag.

github.com

midchildan/glibc-2-17-flake/blob/45db8760381a0ab069d30998e93c5116373a62f6/flake.nix#L65-L82


      
          default = pkgs.runCommandCC "link-pthread-atfork"
            {
              code = ''
                void pthread_atfork(void);
                int main(void) {
                  pthread_atfork();
                  return 0;
                }
              '';
              executable = true;
              passAsFile = [ "code" ];
            }
            ''
              n=$out/bin/$name
              mkdir -p "$(dirname "$n")"
              mv "$codePath" code.c
              $CC -pthread code.c -o "$n"
            '';

❯ nix build .#
error: builder for '/nix/store/4fxz9q13pdvm033h7alzhii7nip0q87j-link-pthread-atfork.drv' failed with exit code 1;
       last 3 log lines:
       > /nix/store/74y3751gsixaz9797ib0hp7c658sp1y5-binutils-2.40/bin/ld: /build/ccqHWMtb.o: in function `main':
       > code.c:(.text.startup+0x5): undefined reference to `pthread_atfork'
       > collect2: error: ld returned 1 exit status
       For full logs, run 'nix log /nix/store/4fxz9q13pdvm033h7alzhii7nip0q87j-link-pthread-atfork.drv'.

Next, I also checked whether libpthread was in fact missing the pthread_atfork symbol. But again, all was normal.

github.com

midchildan/glibc-2-17-flake/blob/45db8760381a0ab069d30998e93c5116373a62f6/flake.nix#L86-L90


      
          default = mkApp (pkgs.writeShellScript "show-symbol" ''
            set -euxo pipefail
            export PATH=${lib.makeBinPath (with pkgs; [ binutils gnugrep ])}
            nm ${pkgs.stdenv.cc.libc}/lib/libpthread.so.0 | grep pthread_atfork
          '');

❯ nix run .#
+ export PATH=/nix/store/mc6q3cdz5s0p1aj4y586bglsfsnsf2k8-binutils-wrapper-2.40/bin:/nix/store/xafzciap7acqhfx84dvqkp18bg4lrai3-gnugrep-3.11/bin
+ PATH=/nix/store/mc6q3cdz5s0p1aj4y586bglsfsnsf2k8-binutils-wrapper-2.40/bin:/nix/store/xafzciap7acqhfx84dvqkp18bg4lrai3-gnugrep-3.11/bin
+ nm /nix/store/ajyzs6yvw923wa3phimy56cid5y62szi-glibc-2.17/lib/libpthread.so.0
+ grep pthread_atfork
000000000000e370 t __dyn_pthread_atfork
000000000000e370 T pthread_atfork@GLIBC_2.2.5

Does anyone have any ideas on how to fix this issue? Thanks in advance.

trofi · October 26, 2023, 7:01am

Nice example! Here is the reason why linkage fails:

$ nm -D /nix/store/ajyzs6yvw923wa3phimy56cid5y62szi-glibc-2.17/lib/libpthread.so | fgrep pthread_atfork
000000000000e370 T pthread_atfork@GLIBC_2.2.5

$ nm -D /nix/store/ajyzs6yvw923wa3phimy56cid5y62szi-glibc-2.17/lib/libpthread.so | fgrep sem_init
000000000000e4b0 T sem_init@@GLIBC_2.2.5

Note the presence and lack of @@ which signifies the default version of a symbol (the one should be used when no version is specified). This means default version of pthread_atfork should come from somewhere else. In glibc-2.17 case it is libpthread_nonshared.a:

$ nm /nix/store/ajyzs6yvw923wa3phimy56cid5y62szi-glibc-2.17/lib/libpthread_nonshared.a

pthread_atfork.oS:
0000000000000000 T pthread_atfork
...

If you add a -Wl,-t to your linker command you will see that libpthread_nonshared.a does not get pulled in:

--- a/flake.nix
+++ b/flake.nix
@@ -78,7 +78,7 @@
             n=$out/bin/$name
             mkdir -p "$(dirname "$n")"
             mv "$codePath" code.c
-            $CC -pthread code.c -o "$n"
+            $CC -pthread code.c -o "$n" -Wl,-t
           '';
       });

link-pthread-atfork> /nix/store/ajyzs6yvw923wa3phimy56cid5y62szi-glibc-2.17/lib/crt1.o
link-pthread-atfork> /nix/store/ajyzs6yvw923wa3phimy56cid5y62szi-glibc-2.17/lib/crti.o
link-pthread-atfork> /nix/store/h5kvfrjmpw792v8jg7nrzfkffmn0iyy8-gcc-12.3.0/lib/gcc/x86_64-unknown-linux-gnu/12.3.0/crtbegin.o
link-pthread-atfork> /build/ccDarPNX.o
link-pthread-atfork> /nix/store/h5kvfrjmpw792v8jg7nrzfkffmn0iyy8-gcc-12.3.0/lib/gcc/x86_64-unknown-linux-gnu/12.3.0/libgcc.a
link-pthread-atfork> /nix/store/xq05361kqwzcdamcsxr4gzg8ksxrb8sg-gcc-12.3.0-lib/lib/libgcc_s.so
link-pthread-atfork> /nix/store/xq05361kqwzcdamcsxr4gzg8ksxrb8sg-gcc-12.3.0-lib/lib/libgcc_s.so.1
link-pthread-atfork> /nix/store/h5kvfrjmpw792v8jg7nrzfkffmn0iyy8-gcc-12.3.0/lib/gcc/x86_64-unknown-linux-gnu/12.3.0/libgcc.a
link-pthread-atfork> /nix/store/ajyzs6yvw923wa3phimy56cid5y62szi-glibc-2.17/lib/libpthread.so
link-pthread-atfork> /nix/store/ajyzs6yvw923wa3phimy56cid5y62szi-glibc-2.17/lib/libc.so
link-pthread-atfork> /nix/store/ajyzs6yvw923wa3phimy56cid5y62szi-glibc-2.17/lib/libc.so.6
link-pthread-atfork> /nix/store/ajyzs6yvw923wa3phimy56cid5y62szi-glibc-2.17/lib/libc_nonshared.a
link-pthread-atfork> /nix/store/ajyzs6yvw923wa3phimy56cid5y62szi-glibc-2.17/lib/ld-linux-x86-64.so.2
link-pthread-atfork> /nix/store/ajyzs6yvw923wa3phimy56cid5y62szi-glibc-2.17/lib/libc_nonshared.a
link-pthread-atfork> /nix/store/ajyzs6yvw923wa3phimy56cid5y62szi-glibc-2.17/lib/ld-linux-x86-64.so.2
link-pthread-atfork> /nix/store/h5kvfrjmpw792v8jg7nrzfkffmn0iyy8-gcc-12.3.0/lib/gcc/x86_64-unknown-linux-gnu/12.3.0/libgcc.a
link-pthread-atfork> /nix/store/xq05361kqwzcdamcsxr4gzg8ksxrb8sg-gcc-12.3.0-lib/lib/libgcc_s.so
link-pthread-atfork> /nix/store/xq05361kqwzcdamcsxr4gzg8ksxrb8sg-gcc-12.3.0-lib/lib/libgcc_s.so.1
link-pthread-atfork> /nix/store/h5kvfrjmpw792v8jg7nrzfkffmn0iyy8-gcc-12.3.0/lib/gcc/x86_64-unknown-linux-gnu/12.3.0/libgcc.a
link-pthread-atfork> /nix/store/h5kvfrjmpw792v8jg7nrzfkffmn0iyy8-gcc-12.3.0/lib/gcc/x86_64-unknown-linux-gnu/12.3.0/crtend.o
link-pthread-atfork> /nix/store/ajyzs6yvw923wa3phimy56cid5y62szi-glibc-2.17/lib/crtn.o
link-pthread-atfork> /nix/store/74y3751gsixaz9797ib0hp7c658sp1y5-binutils-2.40/bin/ld: /build/ccDarPNX.o: in function `main':
link-pthread-atfork> code.c:(.text.startup+0x5): undefined reference to `pthread_atfork'
link-pthread-atfork> collect2: error: ld returned 1 exit status

Normally libpthread_nonshared.a is expected to be supplied via libpthread.so (it should be a linker script). Just like libc_nonshared.a is provided via libc.so linker script.

But glibc-2.38 derivation in nixpkgs clobbers it for you with a backwards compatibility symlink:

# Backwards-compatibility to fix e.g.
# "configure: error: Pthreads are required to build libgomp" during `gcc`-build
# because it's not actually needed anymore to link against `pthreads` since
# it's now part of `libc.so.6` itself, but the gcc build breaks if
# this doesn't work.
ln -sf $out/lib/libpthread.so.0 $out/lib/libpthread.so
ln -sf $out/lib/librt.so.1 $out/lib/librt.so
ln -sf $out/lib/libdl.so.2 $out/lib/libdl.so
ln -sf $out/lib/libutil.so.1 $out/lib/libutil.so
touch $out/lib/libpthread.a

You need to remove all of that block. I’ll remove just the symlink to illustrate the difference. Example hack:

--- a/packages/glibc/default.nix
+++ b/packages/glibc/default.nix
@@ -18,9 +18,11 @@ let
     ./fix-configure.patch
   ];
 in
-glibc.overrideAttrs (_: {
+glibc.overrideAttrs (oa: {
   inherit (glibc-2_17) name src configureFlags postPatch;
   version = "2.17";

   patches = oldPatches ++ backportPatches ++ localPatches;
+  # remove harmful compat symlink hacks
+  postInstall = lib.replaceStrings ["ln -sf $out/lib/libpthread.so.0 $out/lib/libpthread.so"] [""] oa.postInstall;
 })

That way the linker script is restored and you get the linkage succeeded:

link-pthread-atfork> /nix/store/ppny4fbad8rsq2dywl7cgs092j58h7fx-glibc-2.17/lib/crt1.o
link-pthread-atfork> /nix/store/ppny4fbad8rsq2dywl7cgs092j58h7fx-glibc-2.17/lib/crti.o
link-pthread-atfork> /nix/store/h5kvfrjmpw792v8jg7nrzfkffmn0iyy8-gcc-12.3.0/lib/gcc/x86_64-unknown-linux-gnu/12.3.0/crtbegin.o
link-pthread-atfork> /build/ccPZwXYG.o
link-pthread-atfork> /nix/store/h5kvfrjmpw792v8jg7nrzfkffmn0iyy8-gcc-12.3.0/lib/gcc/x86_64-unknown-linux-gnu/12.3.0/libgcc.a
link-pthread-atfork> /nix/store/xq05361kqwzcdamcsxr4gzg8ksxrb8sg-gcc-12.3.0-lib/lib/libgcc_s.so
link-pthread-atfork> /nix/store/xq05361kqwzcdamcsxr4gzg8ksxrb8sg-gcc-12.3.0-lib/lib/libgcc_s.so.1
link-pthread-atfork> /nix/store/h5kvfrjmpw792v8jg7nrzfkffmn0iyy8-gcc-12.3.0/lib/gcc/x86_64-unknown-linux-gnu/12.3.0/libgcc.a
link-pthread-atfork> /nix/store/ppny4fbad8rsq2dywl7cgs092j58h7fx-glibc-2.17/lib/libpthread.so
link-pthread-atfork> /nix/store/ppny4fbad8rsq2dywl7cgs092j58h7fx-glibc-2.17/lib/libpthread.so.0
link-pthread-atfork> /nix/store/ppny4fbad8rsq2dywl7cgs092j58h7fx-glibc-2.17/lib/libpthread_nonshared.a
link-pthread-atfork> /nix/store/ppny4fbad8rsq2dywl7cgs092j58h7fx-glibc-2.17/lib/libpthread_nonshared.a
link-pthread-atfork> /nix/store/ppny4fbad8rsq2dywl7cgs092j58h7fx-glibc-2.17/lib/libc.so
link-pthread-atfork> /nix/store/ppny4fbad8rsq2dywl7cgs092j58h7fx-glibc-2.17/lib/libc.so.6
link-pthread-atfork> /nix/store/ppny4fbad8rsq2dywl7cgs092j58h7fx-glibc-2.17/lib/libc_nonshared.a
link-pthread-atfork> /nix/store/ppny4fbad8rsq2dywl7cgs092j58h7fx-glibc-2.17/lib/ld-linux-x86-64.so.2
link-pthread-atfork> /nix/store/ppny4fbad8rsq2dywl7cgs092j58h7fx-glibc-2.17/lib/libc_nonshared.a
link-pthread-atfork> /nix/store/ppny4fbad8rsq2dywl7cgs092j58h7fx-glibc-2.17/lib/ld-linux-x86-64.so.2
link-pthread-atfork> /nix/store/h5kvfrjmpw792v8jg7nrzfkffmn0iyy8-gcc-12.3.0/lib/gcc/x86_64-unknown-linux-gnu/12.3.0/libgcc.a
link-pthread-atfork> /nix/store/xq05361kqwzcdamcsxr4gzg8ksxrb8sg-gcc-12.3.0-lib/lib/libgcc_s.so
link-pthread-atfork> /nix/store/xq05361kqwzcdamcsxr4gzg8ksxrb8sg-gcc-12.3.0-lib/lib/libgcc_s.so.1
link-pthread-atfork> /nix/store/h5kvfrjmpw792v8jg7nrzfkffmn0iyy8-gcc-12.3.0/lib/gcc/x86_64-unknown-linux-gnu/12.3.0/libgcc.a
link-pthread-atfork> /nix/store/h5kvfrjmpw792v8jg7nrzfkffmn0iyy8-gcc-12.3.0/lib/gcc/x86_64-unknown-linux-gnu/12.3.0/crtend.o
link-pthread-atfork> /nix/store/ppny4fbad8rsq2dywl7cgs092j58h7fx-glibc-2.17/lib/crtn.o

Note the presence of libpthread_nonshared.a in the trace now.

Hope that helps a bit.

midchildan · October 30, 2023, 6:17pm

Thanks for the detailed answer! It worked like a charm.

To future proof it a bit, I wrapped the ln command in postInstall to stop it from overwriting existing files.

github.com

midchildan/glibc-2-17-flake/blob/5800da76f5e8b79327703d9c0c39d84c72b4c887/packages/glibc/common.nix#L33-L48


      
          # Prevent compatiblity symlinks for deleted files overwriting the actual files
          # in older packages.
          # https://discourse.nixos.org/t/linking-issue-with-libpthread-from-glibc-2-17/34601
          postInstall = ''
            ln() {
              local dst="''${@: -1}"
              if [[ "$dst" == *.so && -f "$dst" ]]; then
                return
              fi
              command ln "$@"
            }
          
          
  ${drv.postInstall}
          
          
  unset -f ln
          '';