Including an older glibc in stdenv

[harris-chris]

Hello all,

I need to build some binaries, in nix, using an older glibc version (glibc 2.17). The approach I’ve taken is to create a local copy of the nixpkgs repo, and (at least, to get me off the ground) to replace the default glibc with glibc 2.17.

After some patching and trial and error, I’ve got things to the stage where I can do nix build .#glibc (I’m in the directory of my local nixpkgs) and see that glibc 2.17 is being built, and, therefore, the default stdenv uses glibc 2.17.

This unfortunately doesn’t fully solve my problem, because it seems that some of the utilities I need are still being built with glibc 2.27 - for example, if I try to build a test package using this stdenv, I am told:

/nix/store/d18ll2slr04w93pkzhn96748ssadldmj-binutils-patchelfed-ld-2.39/bin/ld: /nix/store/iqbzava2rqy6m0lq1j2mld74ll49j52b-glibc-2.17-224/lib/libc.so.6: version `GLIBC_2.27' not found (required by /nix/store/d18ll2slr04w93pkzhn96748ssadldmj-binutils-patchelfed-ld-2.39/bin/ld)

If I look at the bootstrap process that stdenv uses - in pkgs/stdenv/linux/default.nix of the current nixpkgs - it uses a tarball to kick off the build process:

bootstrapTools = import <nix/fetchurl.nix> {
  url = "http://tarballs.nixos.org/stdenv-linux/x86_64/c5aabb0d603e2c1ea05f5a93b3be82437f5ebf31/bootstrap-tools.tar.xz";
  sha256 = "a5ce9c155ed09397614646c9717fc7cd94b1023d7b76b618d409e4fefd6e9d39";
};

and this tarball contains glibc 2.27, amongst other related files:

include-glibc/bits/libc-header-start.h
include-glibc/gnu/libc-version.h
include-glibc/linux/libc-compat.h
lib/libc-2.27.so
lib/libc.so
lib/libc.so.6

So I assume that this glibc is being injected early into the bootstrap process, and that some tools that are required for the bootstrap process - automake, patchelf etc - are built against this glibc, rather than my glibc 2.17, which gets inserted later into the bootstrap process.

Would anyone have a rough idea what might be the best way to work around this? It seems to me that I could:

• replace the libc-related binaries in the bootstrap-tools tarball with binaries built from glibc 2.17, or
• attempt to tweak the bootstrap process so that my glibc 2.17 gets used earlier on, or
• attempt to rebuild the affected tools (patchelf and whatever else) with glibc 2.17 further downstream
  or some mysterious fourth thing I’ve not considered.

Thanks in advance, and also after the event

Chris

[7c6f434c]

Stupid question: how much work would it be to fork 2.17-era Nixpkgs and add whatever you need there?

(You can install stuff from multiple checkouts from Nixpkgs into the same profile, if necessary)

[harris-chris]

Not a stupid question, that was the first thing I tried. But nixpkgs from that era is lacking a lot of modern syntax (ie, overlays and overrideAttrs), as well as a modern GCC , so it turned out not to be practical.

[7c6f434c]

Hmm, and if you try to port just the bootstrap tools and glibc from there?

(I just hope the glibc you need and the gcc you need will play nicely together, which is also not 100% obvious…)

[Sandro]

if you are going to replace the bootstrap glibc with a very old version you will encounter compile errors. You probably want to rebuild the relevant tools against that older glibc version but they are probably not 100% isolated and dirty references will happen which could be hard to cleanup.

[kotborealis]

@harris-chris Sorry to bother you half-a-year later, but did you have any success with this? I have exactly the same task, so i was hoping you could share your results

[harris-chris]

Hello, I didn’t find a generalisable solution to this, but I did manage to set up a few build environments, using older gccs and glibcs. What version are you looking for? I may be able to help.

[kotborealis]

I’ve actually already found your github repo (harris-chris/nix-development-configs/cpp-specific-toolchain and it actually almost works for my case, thank you so much.

But, I’ve encountered another problem: my goal is to use older glibc to compile a whole tree of C projects, a piece of software and all of it’s dependencies. I’ve managed to apply custom stdenv to the nixpkgs tree, but it messes with the bootstrap process and outright refuses to work.

To be specific, I’m trying to compile Qemu to be as portable as possible, so ideally I need to recompile all of it’s dependencies - SDL2, x11, alsa, pixman, so on.

I think I might be able to pull it off by applying stdenv overlay to a specific list of packages, not a whole nixpkgs tree. But, right now, I could not get it to work as I have zero experience with nix and the learning process is a bit painfull

[7c6f434c]

@kotborealis

I’ve managed to apply custom stdenv to the nixpkgs tree, but it messes with the bootstrap process and outright refuses to work.

Is the problem arising via «gcc as we build it needs a fresher glibc» mechanism?

@kotborealis

To be specific, I’m trying to compile Qemu to be as portable as possible, so ideally I need to recompile all of it’s dependencies - SDL2, x11, alsa, pixman, so on.

And which of those things you need to be fresher than the Nixpkgs revision contemporary with the glibc version you want?

[kotborealis]

@7c6f434c

Is the problem arising via «gcc as we build it needs a fresher glibc» mechanism?

No, it was triggering something like assertion isFromBootstrapFiles failed.

@7c6f434c

And which of those things you need to be fresher than the Nixpkgs revision contemporary with the glibc version you want?

Good catch, I probably don’t need anything to be really really fresh, but it would be nice. I should try it and see how it goes. I suspect that there might be a lot of work to patch CVEs in old nixpgks packages.

[7c6f434c]

I mean, you can start with an older checkout and override things forward (possibly using the packages from a newer checkout…)

As for CVEs, I guess it depends on your threat model — and you have some upper bounds on success rate; glibc has had recent CVEs of its own, and some of the new versions of other stuff wlll probably want fresh glibc, so… it’s a non-negligible project to get a good overview…

[harris-chris]

I think that that personal repo stuff is an earlier stage of my experimenting with this issue, I have the final thing on the work system. It’s a set of stdenvs, each representing a GCC and glibc version, so possibly what you need. I’ll put it up here in the next couple of days.

[harris-chris]

The way I did this is actually quite simple:

1. I got some older glibcs and gccs from older nixpkgs revs:

glibc_2_18 = (import (builtins.fetchGit {
    name = "glibc_2_18";
    url = "https://github.com/NixOS/nixpkgs/";
    ref = "refs/heads/nixpkgs-unstable";
    rev = "ab6453c483e406b07c63503bca5038838c187ecf";
}) { inherit system; }).glibc;

gcc_8_3_0 = (import (builtins.fetchGit {
   name = "gcc_8_3_0";
   url = "https://github.com/NixOS/nixpkgs/";
   ref = "refs/heads/nixpkgs-unstable";
   rev = "a9eb3eed170fa916e0a8364e5227ee661af76fde";
}) { inherit system; }).gcc-unwrapped;

Nix Package Versions is a good resource for finding older package versions within the nixpgks history.

2. I created a function which puts together one of these older gccs with an older glibc:

getCustomGccStdenv = customGcc: customGlibc: origStdenv: { pkgs, ... }:
  with pkgs; let
    compilerWrapped = wrapCCWith {
      cc = customGcc;
      bintools = wrapBintoolsWith {
        bintools = binutils-unwrapped;
        libc = customGlibc;
      };
    };
  in
    overrideCC origStdenv compilerWrapped;

3. Then I applied function to the glibcs/gccs to create some gcc/glibc combinations:

gcc_8_3_0_glibc_2_18 = getCustomGccStdenv
        gcc_8_3_0 glibc_2_18 pkgs.stdenv pkgs;

4. … then passed that into my derivations. If you’re re-building existing nixpkgs derivations, then you can use override to pass the custom stdenv in, like pkgs.examplePkg.override { stdenv = gcc_8_3_0_glibc_2_18; };
   (see Overriding | nixpkgs) for more info on this. If you need all the dependencies of your target package to be re-built with the older gcc/glibc, then you might need to override the stdenv in pkgs with the custom stdenv.

I’m not sure if that precisely addresses your problem because the whole thing is quite fiddly, older nixpkgs revs don’t necessarily have the same API as newer ones so it requires tweaking on a case-by-case basis.

If any of that isn’t clear then please let me know.

[tejx]

These functions look like exactly what I need, wondering if you pushed them to a git repo somewhere?

[harris-chris]

I haven’t, or at least not to a public one. They’re pretty much as I’ve written them above, I’d try it by copying and pasting from there. If you have any trouble feel free to ask.

[tejx]

It worked perfectly for the hello example like this

# default.nix
let
  nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/tarball/nixos-22.11";
  pkgs = import nixpkgs { config = {}; overlays = []; };
system = "x86_64-linux";
  glibc_2_27 = (import (builtins.fetchGit {
    name = "glibc_2_18";
         url = "https://github.com/NixOS/nixpkgs/";
         ref = "refs/heads/nixos-19.09";
         rev = "b79f64b5eb5fa8ca2f844ddb4d7c186b6c69a293";
}) { inherit system; }).glibc;

gcc_8_3_0 = (import (builtins.fetchGit {
   name = "gcc_8_3_0";
   url = "https://github.com/NixOS/nixpkgs/";
   ref = "refs/heads/nixpkgs-unstable";
   rev = "a9eb3eed170fa916e0a8364e5227ee661af76fde";
}) { inherit system; }).gcc-unwrapped;

getCustomGccStdenv = customGcc: customGlibc: origStdenv: { pkgs, ... }:
  with pkgs; let
    compilerWrapped = wrapCCWith {
      cc = customGcc;
      bintools = wrapBintoolsWith {
        bintools = binutils-unwrapped;
        libc = customGlibc;
      };
    };
  in
  overrideCC origStdenv compilerWrapped;

gcc_8_3_0_glibc_2_27 = getCustomGccStdenv gcc_8_3_0 glibc_2_27 pkgs.stdenv pkgs;


in
{
  hello = pkgs.callPackage ./hello.nix {stdenv = gcc_8_3_0_glibc_2_27; };
}

Now im trying to rebuild gdal from the geospatial nix packages in a flake with custom glibc and gcc but not sure how to do it.
Here is the flake:

{
  description = "Geospatial NIX";

  nixConfig = {
    extra-substituters = [
      "https://geonix.cachix.org"
      "https://devenv.cachix.org"
    ];
    extra-trusted-public-keys = [
      "geonix.cachix.org-1:iyhIXkDLYLXbMhL3X3qOLBtRF8HEyAbhPXjjPeYsCl0="
      "devenv.cachix.org-1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw="
    ];
    bash-prompt = "\\[\\033[1m\\][geonix]\\[\\033\[m\\]\\040\\w >\\040";
  };

  inputs = {
    geonix.url = "github:imincik/geospatial-nix";
    nixpkgs.follows = "geonix/nixpkgs";
    devenv = {
      url = "github:cachix/devenv";
      inputs.nixpkgs.follows = "geonix/nixpkgs";
    };
    nix2container = {
      url = "github:nlewo/nix2container";
      inputs.nixpkgs.follows = "geonix/nixpkgs";
    };
    mk-shell-bin.url = "github:rrbutani/nix-mk-shell-bin";
  };

  outputs = inputs@{ flake-parts, ... }:
    flake-parts.lib.mkFlake { inherit inputs; } {

      imports = [
        inputs.devenv.flakeModule
      ];

      systems = [ "x86_64-linux" ];

      perSystem = { config, self', inputs', pkgs, system, ... }: {

        devenv.shells.default = {
          imports = [
              inputs.geonix.modules
            ./geonix.nix
          ];
        };

        packages.geonixcli = inputs.geonix.packages.${system}.geonixcli;
      };

      flake = { };
    };
}

I know gdal is huge, if you could give a simple example with a flake that would be great.

[harris-chris]

OK, it should be as simple as:

• move the code which creates the gcc_8_3_0_glibc_2_27 variable into the perSystem = { config, self', inputs', pkgs, system, ... }: {...} part of your flake, so that you have a gcc_8_3_0_glibc_2_27 variable per system, and
• use overrideAttrs on your original geonixcli, so:

packages.geonixcli = inputs.geonix.packages.${system}.geonixcli.overrideAttrs (finalAttrs: previousAttrs: {
  stdenv = gcc_8_3_0_glibc_2_27;
});

Let me know if that doesn’t work!

[tejx]

I got an error saying undefined getCustomGccStdenv with that approach, so i modified it a bit like so

{
  description = "Geospatial NIX";

  nixConfig = {
    extra-substituters = [
      "https://geonix.cachix.org"
      "https://devenv.cachix.org"
    ];
    extra-trusted-public-keys = [
      "geonix.cachix.org-1:iyhIXkDLYLXbMhL3X3qOLBtRF8HEyAbhPXjjPeYsCl0="
      "devenv.cachix.org-1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw="
    ];
    bash-prompt = "\\[\\033[1m\\][geonix]\\[\\033\[m\\]\\040\\w >\\040";
  };

  inputs = {
    geonix.url = "github:imincik/geospatial-nix";
    nixpkgs.follows = "geonix/nixpkgs";
    devenv = {
      url = "github:cachix/devenv";
      inputs.nixpkgs.follows = "geonix/nixpkgs";
    };
    nix2container = {
      url = "github:nlewo/nix2container";
      inputs.nixpkgs.follows = "geonix/nixpkgs";
    };
    mk-shell-bin.url = "github:rrbutani/nix-mk-shell-bin";
  };

  outputs = inputs@{ flake-parts, ... }:
    flake-parts.lib.mkFlake { inherit inputs; } {

      imports = [
        inputs.devenv.flakeModule
      ];

      systems = [ "x86_64-linux" ];

      perSystem = { config, self', inputs', pkgs, system, ... }: 
	  
	  let 
	 
	 glibc_2_27 = (import (builtins.fetchGit {
    name = "glibc_2_18";
         url = "https://github.com/NixOS/nixpkgs/";
         ref = "refs/heads/nixos-19.09";
         rev = "b79f64b5eb5fa8ca2f844ddb4d7c186b6c69a293";
}) { inherit system; }).glibc;

	  gcc_8_3_0 = (import (builtins.fetchGit {
   name = "gcc_8_3_0";
   url = "https://github.com/NixOS/nixpkgs/";
   ref = "refs/heads/nixpkgs-unstable";
   rev = "a9eb3eed170fa916e0a8364e5227ee661af76fde";
}) { inherit system; }).gcc-unwrapped;

	  getCustomGccStdenv = customGcc: customGlibc: origStdenv: { pkgs, ... }:
  with pkgs; let
    compilerWrapped = wrapCCWith {
      cc = customGcc;
      bintools = wrapBintoolsWith {
        bintools = binutils-unwrapped;
        libc = customGlibc;
      };
    };
  in
  overrideCC origStdenv compilerWrapped;
  gcc_8_3_0_glibc_2_27 = getCustomGccStdenv gcc_8_3_0 glibc_2_27 pkgs.stdenv pkgs;
  in
	  
	  
	  {

        packages.geonixcli = inputs.geonix.packages.${system}.geonixcli;
		
		packages.gdal = inputs.geonix.packages.${system}.gdal.overrideAttrs (finalAttrs: previousAttrs: {
		  stdenv = gcc_8_3_0_glibc_2_27;
		});
		
      };

      flake = { };
    };
}

Now when i run rm flake.lock && nix build .#gdal it seems to work, but the result bins are still with a newer glibc and not the 2.27 as expected.

[harris-chris]

Hmm. How are you checking the glibc version?

[tejx]

ldd result/bin/ogrinfo | grep glibc
        libm.so.6 => /nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib/libm.so.6 (0x00007f3f2c26f000)
        libc.so.6 => /nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib/libc.so.6 (0x00007f3f2c065000)
        libdl.so.2 => /nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib/libdl.so.2 (0x00007f3f2c03c000)
        /nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f3f2e124000)
        libpthread.so.0 => /nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib/libpthread.so.0 (0x00007f3f250d3000)
        librt.so.1 => /nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib/librt.so.1 (0x00007f3f1e325000)
        libresolv.so.2 => /nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib/libresolv.so.2 (0x00007f3f1dcc4000)
        libmvec.so.1 => /nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib/libmvec.so.1 (0x00007f3f1d8a8000)

Also with the ‘file’ command. Do i have to clear my cache or something?