Argument list too long due to inflated NIX_CFLAGS_COMPILE

Help
1

I am currently encountering “Argument list too long” when attempting to use a dev shell that uses clang with nix-ros-overlay when I have a bunch of transitive dependencies:

$ nix build -L .#devShells.x86_64-linux.default
argument-list-too-long> Running phase: buildPhase
argument-list-too-long> /nix/store/93nj5v35ldxpmjsl6rhl9i394kaf54ni-stdenv-linux/setup: line 1752: /nix/store/8ksax0a2mxglr5hlkj2dzl556jx7xqn5-coreutils-9.7/bin/date: Argument list too long
error: Cannot build '/nix/store/m217xc3z50h50c6hbr9v4sf051dv8vv1-argument-list-too-long.drv'.
       Reason: builder failed with exit code 126.
       Output paths:
         /nix/store/7pcjypww7a37z6pbpjhmsk3722b7mvfw-argument-list-too-long
       Last 2 log lines:
       > Running phase: buildPhase
       > /nix/store/93nj5v35ldxpmjsl6rhl9i394kaf54ni-stdenv-linux/setup: line 1752: /nix/store/8ksax0a2mxglr5hlkj2dzl556jx7xqn5-coreutils-9.7/bin/date: Argument list too long
       For full logs, run:
         nix-store -l /nix/store/m217xc3z50h50c6hbr9v4sf051dv8vv1-argument-list-too-long.drv

See:

My understanding is that this is a long-standing issue in nix:

Here is a flake to reproduce the issue:

{
    inputs = {
        nix-ros-overlay.url = "github:lopsided98/nix-ros-overlay/master";
        nixpkgs.follows = "nix-ros-overlay/nixpkgs";    # IMPORTANT!!!
    };
    outputs = { self, nix-ros-overlay, nixpkgs }:
        nix-ros-overlay.inputs.flake-utils.lib.eachDefaultSystem (system:
            let
                pkgs = import nixpkgs {
                    inherit system;
                    overlays = [ nix-ros-overlay.overlays.default ];
                };
                llvm = pkgs.llvmPackages_latest;
                clangStdenv = llvm.stdenv;
                jazzy = pkgs.rosPackages.jazzy.overrideScope (rosSelf: rosSuper:
                let
                    buildRosPackageClang = rosSelf.buildRosPackage.override {
                        stdenv = clangStdenv;
                    };
                in pkgs.lib.mapAttrs (_: package: (package.override {
                        buildRosPackage = buildRosPackageClang;
                    }).overrideAttrs ({
                        propagatedBuildInputs ? [],
                        ...
                    } : {
                        propagatedBuildInputs = propagatedBuildInputs ++ [ llvm.openmp ];
                    })) {
                        inherit (rosSuper)
                        nav2-bringup
                        ;
                });
            in {
                devShells.default = pkgs.mkShell.override {
                    stdenv = clangStdenv;
                } {
                    name = "argument-list-too-long";
                    packages = with pkgs; [
                        (with jazzy; buildEnv {
                            paths = [
                                nav2-bringup
                            ];
                        })
                    ];
                };
            });

    nixConfig = {
        extra-substituters = [
            "https://cache.nixos.org"
            "https://ros.cachix.org"
        ];
        extra-trusted-public-keys = [
            "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
            "ros.cachix.org-1:dSyZxI8geDCJrwgvCOHDoAfOm5sV1wCPjBkKL+38Rvo="
        ];
    };
}

note that the issue does not happen if stdenv = clangStdenv is not included.

This is because of the following:

This causes -fmacro-prefix-map= to be spammed a ton, massively inflating the size of the NIX_CFLAGS_COMPILE environment variable. In my case, I’m seeing a NIX_CFLAGS_COMPILE that is 182667 characters long, which is ABSURDLY huge.

This is in addition to the fact that every added include directory is duplicated 4 times. (NixOS/nixpkgs#364984, lopsided98/nix-ros-overlay#544)

Another environment variable significantly contributing to this is NIX_LDFLAGS. For me, that environment variable is 74514 characters log, with everything duplicated 6x.

For the NIX_CFLAGS_COMPILE environment variable, from what I can tell, the issue essentially boils down to the ccWrapper_addCVars hook being called 4 times, twice for envHostHostHook and twice for envHostTargetHook.
And for NIX_LDFLAGS, it gets called 6 times, three times for envHostHostHook and three times for envHostTargetHook.
Not sure why they’re being called multiple times however. I’d guess it has something to do with in _addToEnv:

Is my understanding of the issue here correct, or is there something I’m missing? I’m not too familiar with nix, so I could very well be wrong about what it’s doing.
And if this is the cause, I could PR a ‘hack’ to fix this by just scanning the existing env variable for any copies of it and if they exist. would such a PR be accepted?

however, IMO, the real fix for this would be to

  1. store NIX_CFLAGS_COMPILE and NIX_LDFLAGS in a file rather than an environment variable, and then read from this file in the cc/ld wrappers. (though, to a degree this is also just kinda pushing it down the road for GCC a bit)
  2. figure out why the hooks are getting called multiple times, and have them not do that.
2

I think your analysis is right: -fmacro-prefix-map= can’t be used in nixpkgswithout measures to reduce environment size increase..

I tried -fmacro-prefix-map=hack a few years ago at Maximum argument count on linux and in gcc and instantly got qemufailing. Long command lines also have negative performance implications). I ended up patching gccin nixpkgsat https://github.com/NixOS/nixpkgs/pull/255192 instead.

As some packages still fail to build due to command line length (SDL in haskell) I ended up patching linuxkernel locally to allow a few more parameters:

  boot.kernelPatches = [
    {
      # https://lkml.org/lkml/2023/9/24/381
      name = "6M-args.patch";
      patch = ./0001-uapi-increase-MAX_ARG_STRLEN-from-128K-to-6M.patch;
    }
  ];
3

This problem in my opinion requires rewriting the various hook code so that it uses sets instead of directly appending to the env vars blindly. This however would potentially create performance issues.

Longer term I think nixpkgs needs to start using the native tooling for injection a bit like Conan does with its generator functionality. It would keep a build model defined and from the model it would then extract all needed flags as needed on demand.

For the ros overlay, this is also related to the blind propagation of inputs because of transitive headers needing to be available in leaf libraries, I am convinced this could be done in a more fine grained manner.