Why cross-compilation nixpkgs require LLVM rebuild?

Help
1

I’m trying to use nix in a project which involves compiling C++ to wasm with clang.
Here’s my first attempt at defining the environment for doing it:

{
  description = "mypackage";

  inputs = {
    nixpkgs.url = "nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { nixpkgs, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgsCross = import nixpkgs {
          localSystem = system;
          crossSystem = {
            config = "wasm32-wasi";
            useLLVM = true;
          };
        };
        stdenvCross = pkgsCross.clangStdenv;
      in
      {
        devShells.default = pkgsCross.mkShell.override { stdenv = stdenvCross; } {
          packages = [
          ];
        };
      });
}

Can someone please help me with the following problems:

  1. This triggers a full build of LLVM from source (unlike clangStdenv from regular, non-cross nixpkgs). Why? How can I use a regular, cached LLVM for cross-compilation?
  2. It uses clang 18. I’d rather have clang 17. How can I specify another version?

I’m working on Mac in case if it matters.

2

Darwin to wasm simply isn’t a combination of platforms that hydra builds. I’m half surprised that even works tbh.

Darwin to Darwin or Linux to Linux using clang OTOH is quite common and happens within regular Nixpkgs.

We provide multiple versions of LLVM in Nixpkgs following the pattern of llvmPackages_xy where xy is the major version. In your case simply use llvmPackages_17.stdenv.

3

Darwin to wasm simply isn’t a combination of platforms that hydra builds.

But isn’t clang (unlike gcc) is a universal cross-compiler by itself, so same clang binary can be used for any target, so there is just no need to rebuild it regardless of the target?

We provide multiple versions of LLVM in Nixpkgs following the pattern of llvmPackages_xy where xy is the major version. In your case simply use llvmPackages_17.stdenv .

I tried to change stdenvCross = pkgsCross.clangStdenv; to stdenvCross = pkgsCross.llvmPackages_17.stdenv;. Now when I run nix develop, it starts building both llvm-17 and llvm-18. That really confuses me

4

I don’t understand even where does clang-18 come from. I didn’t ask for it anywhere. My regular (non-cross) stdenv and clangStdenv give me clang-16

5

In theory yes, in practice because of how Nixpkgs cross is structured we can’t share those binaries. That’s an unfortunate limitation that hopefully we will one day fix.

LLVM 18 is the default for all platforms except Darwin, including WASM. (In Nixpkgs 25.05 it’ll be LLVM 19 on all platforms.)

FWIW your config is not quite right. I would suggest trying pkgsCross.wasi32 instead.

2 Likes
6

That’s an unfortunate limitation that hopefully we will one day fix.

Got it. Is there anything that can be done about it now? Maybe there’s some workaround I could apply? Maybe is makes sense to ask somewhere for adding pkgsCross.wasi32 to public cache, at least stdenv, since it’s quite a popular platform nowadays?

FWIW your config is not quite right. I would suggest trying pkgsCross.wasi32 instead.

Alright, now the derivation looks like this:

{
  description = "mypackage";

  inputs = {
    nixpkgs.url = "nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { nixpkgs, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          system = system;
        };
        pkgsWasm = pkgs.pkgsCross.wasi32;
        stdenvWasm = pkgsWasm.clang17Stdenv;
      in
      {
        devShells.default = pkgsWasm.mkShell.override { stdenv = stdenvWasm; } {
          packages = [
          ];
        };
      });
}

It still tries to build both llvm 17 and 18; it’s unclear to me why and is it avoidable?

7

You can maybe try some rewrapping stuff, but it’s kind of a pain and will be problematic with library dependencies. I think your best option at present is just to deal with the LLVM build.

I think it’s possible (though I haven’t verified) that the only reason it’s cached on Linux is because of Firefox, so it’s possible that will happen for Darwin too soon.

1 Like
8

Why use stdenvWasm = pkgsWasm.clang17Stdenv? You can just use pkgsWasm.mkShell, etc.; the WASI standard environment is already LLVM.

9

Best is to pass in an overlay that modifies the default llvmPackages version, e.g.

self: super: { llvmPackages = self.llvmPackages_17; }

This is correct and clang-unwrapped as well as llvm would be shard if they were the same. However we currently have some minor differences depending on the platform we are targeting which should eventually be eliminated. You can inspect differences yourself using nix-diff, e.g. nix-shell -p nix-diff --run "nix-instantiate -A llvmPackages_18.clang-unwrapped -A pkgsCross.wasi32.stdenv.cc.cc | xargs nix-diff.

Current things preventing clang-unwrapped from being shared:

10

Great, changing compiler version using the overlays seems to work. I’ll leave the full flake code here.

Flake 
{
  description = "mypackage";

  inputs = {
    nixpkgs.url = "nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { nixpkgs, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          system = system;
        };
        pkgsWasi32 = (pkgs.pkgsCross.wasi32.appendOverlays
          [
            (self: super: { llvmPackages = self.llvmPackages_17; })
          ]
        );
      in
      {
        devShells.default = pkgsWasi32.mkShell {
          packages = [
          ];
        };
      });
}