Cross compiling a package that needs to link a library for both build and host platforms

1

I am at a bit of an impasse for cross compilation with Nix.

Let’s use openssl as an example but this could go for any library and can be a
particular pain with Rust build scripts (build.rs) because they are actually
native programs that may want to dynamically link against a library instead of
shelling out to a program on the $PATH.

I have a package that requires linking a native version of openssl (x86_64)
for a “build script” (think a build.rs for instance that may make use of
something like GitHub - coralogix/protofetch: A source dependency management tool for Protobuf modules. and it’s not behind a
feature flag.)

However, the package also requires linking against openssl (aarch64) for a
build output program with the host architecture.

I can’t quite figure out a way to make this work with pkg-config and a funny
detail shows up here because the order of buildInputs matters for
PKG_CONFIG_PATH and the first one it finds will be used, without any way to do
this architecture-dependent.

    let
      pkgs = import nixpkgs { system = "x86_64-linux"; };

      test =
        {
          stdenv,
          openssl,
          openssl-native ? openssl,
          pkg-config,
        }:
        stdenv.mkDerivation {
          nativeBuildInputs = [
            pkg-config
            openssl-native # if we put this here, pkg-config will do nothing with it
          ];

          buildInputs = [
            openssl-native # if we put this here, linking the aarch64 binary would fail
            openssl
            openssl-native # if we put this here, linking the x86 binary would fail
          ];
        };
    in
    {
      packages.test = pkgs.callPackage test { };
      packages.test-aarch64 = pkgs.pkgsCross.aarch64-multiplatform.callPackage test {
        openssl-native = pkgs.openssl; # we can do this to bypass the callPackage cross arch
      };
    };

The entire approach doesn’t feel great. Is there any way here that would actually work and maybe even be sane?

2

If you need to compile during the build for the buildPlatform, you should need openssl from pkgsBuildBuild in depsBuildBuild and then also a buildPlatform → buildPlatform rust compiler to build the build script for the buildPlatform. I don’t know how the rust mkDerivation wrapper works here though w.r.t. providing a second buildBuild rust compiler and using that to build the build script.

1 Like
3

It would help if you posted a link to a concrete example that you’re trying to get working.

The sane approach is called crate2nix. Unfortunately buildRustPackage smashes all the crates – the packaged crate and all its dependencies – together into one big derivation.

With crate2nix you can express the fact that protofetch needs pkgsBuildBuild.openssl but t0ms-mystery-crate needs pkgsBuildHost.openssl.

Check out default-crate-overrides.nix for examples.

2 Likes
4

Hmm that’s annoying. Would it perhaps be possible to split this drv into two: First compile the build script using pkgsBuildBuild.buildRustPackage and pkgsBuildBuild.openssl and then run it inside of the pkgsBuildHost.buildRustPackage of the actual drv?

5

I think the problem would be with this part:

buildRustPackage lets cargo control the build process. I don’t think there’s any way to tell cargo “hey, I already compiled build.rs; here’s the binary, please use this instead of compiling build.rs”. I’m sure you could trick cargo into using your prebuilt build.rs, but to do that you’d need to rely on things that cargo explicitly says aren’t stable and can change at any time, like it’s directory-naming scheme and the build artifact fingerprint algorithm.

I think this is why the Nix ecosystem has stabilized around basically only two approaches for building Rust packages:

  1. Let cargo control everything (buildRustPackage)
  2. Don’t use cargo for anything except querying https://crates.io (i.e. buildRustCrate / crate2nix)

These two extremes (“cargo everything” and “cargo nothing”) are the only ones that really work well. Unfortunately cargo isn’t like the UNIXy build processes where you have a bunch of small simple tools (make, grep, ld.so, etc) that each do one thing well and don’t know about each other.

6

Thanks for the help!

I actually got it to work in a minimal reproducer like this.

[package]
name = "cross-thing"
version = "0.1.0"
edition = "2021"

[dependencies]
openssl = "0.10.68"

[build-dependencies]
openssl = "0.10.68"

build.rs and main.rs both call
dbg!(openssl::base64::encode_block(b"hello world!")) just to make sure linking
actually has to happen.

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    flake-parts.url = "github:hercules-ci/flake-parts";
  };

  outputs =
    inputs@{
      self,
      nixpkgs,
      flake-parts,
    }:
    let
      systems = [
        "aarch64-linux"
        "x86_64-linux"
      ];
    in
    flake-parts.lib.mkFlake { inherit inputs; } {
      inherit systems;

      perSystem =
        {
          self',
          config,
          pkgs,
          system,
          ...
        }:
        let
          cargoToml = builtins.fromTOML (builtins.readFile ./Cargo.toml);

          package =
            {
              lib,
              rustPlatform,
              pkgsBuildBuild,
              openssl,
            }:
            rustPlatform.buildRustPackage {
              pname = cargoToml.package.name;
              version = cargoToml.package.version;
              src = lib.cleanSource ./.;

              cargoLock.lockFile = ./Cargo.lock;

              depsBuildBuild = [
                pkgsBuildBuild.pkg-config
              ];
              nativeBuildInputs = [
                pkgsBuildBuild.openssl
                pkgsBuildBuild.pkg-config
              ];
              buildInputs = [
                openssl
              ];
            };
        in
        {
          packages =
            {
              default = config.packages.${cargoToml.package.name};
            }
            // builtins.listToAttrs (
              builtins.map (
                s:
                let
                  pkgs' = import nixpkgs {
                    localSystem = system;
                    crossSystem = s;
                  };
                in
                {
                  name = if s == system then cargoToml.package.name else "${cargoToml.package.name}-${s}";
                  value = pkgs'.callPackage package { };
                }
              ) systems
            );

          devShells.default = pkgs.mkShell {
            inputsFrom = [ config.packages.default ];
          };
        };
    };
}

With this I can now build both of these successfully:
nix build .#packages.aarch64-linux.cross-thing-x86_64-linux
nix build .#packages.x86_64-linux.cross-thing-aarch64-linux

I will have to check that out, I have used naersk and crane before but not this one.

Interesting, but this is also used by buildRustPackage since it’s part of nixpkgs, right? I will have to wrap my head around what’s happening here exactly.

1 Like
7

No; see the Rust section; there are two separate flows for building Rust packages:

  • buildRustPackage: Compiling Rust applications with Cargo
  • buildRustCrate: Compiling Rust crates using Nix instead of Cargo