Cross-compile Rust from M1

I’m following along with this guide on my Macbook M1: Generating a docker image with nix

So at the point where I build a Docker container, the container I natively create does not run in Docker.

Now I’m trying to cross-compile it with some guides I scrounged together from crane and elsewhere but it busts out with an error.

Is this fixable? Should I even try doing this?

I foresee one cross-compile to get it to run on the Docker on my M1 machine but then I’d probably have to cross compile it again to deploy it to a cloud somewhere.

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";

    crane = {
      url = "github:ipetkov/crane";
      inputs = {
        nixpkgs.follows = "nixpkgs";
      };
    };
    flake-utils.url = "github:numtide/flake-utils";

    rust-overlay = {
      url = "github:oxalica/rust-overlay";
      inputs = {
        nixpkgs.follows = "nixpkgs";
        flake-utils.follows = "flake-utils";
      };
    };
  };
  outputs = { self, nixpkgs, flake-utils, rust-overlay, crane }:
    flake-utils.lib.eachDefaultSystem (localSystem:
        let
          crossSystem = "aarch64-linux";
          pkgs = import nixpkgs {
            inherit crossSystem localSystem;
            overlays = [ (import rust-overlay) ];
          };

          # rustToolchain = pkgs.pkgsBuildHost.rust-bin.fromRustupToolchainFile ./rust-toolchain.toml;

          rustToolchain = pkgs.pkgsBuildHost.rust-bin.stable.latest.default.override {
            targets = [ "aarch64-unknown-linux-gnu" ];
          };

          craneLib = (crane.mkLib pkgs).overrideToolchain rustToolchain;

          crateExpression = {
            darwin
          , openssl
          , libiconv
          , lib
          , pkg-config
          , stdenv
          }: craneLib.buildPackage {
            src = craneLib.path ./.;

            nativeBuildInputs = with pkgs; [pkg-config] ++
              lib.optionals stdenv.buildPlatform.isDarwin
              [libiconv darwin.apple_sdk.frameworks.Security darwin.apple_sdk.frameworks.CoreFoundation];
            buildInputs = with pkgs; [ openssl ];

            cargoExtraArgs = "--target aarch64-unknown-linux-gnu";
          };

          bin = pkgs.callPackage crateExpression {};

          dockerImage = pkgs.dockerTools.buildImage {
            name = "cuppings";
            tag = "latest";
            copyToRoot = [ bin ];
            config = {
              Cmd = [ "${bin}/bin/cuppings" ];
            };
          };
        in
        with pkgs;
        {
          packages =
            {
              inherit bin dockerImage;
              default = bin;
            };
          devShells.default = mkShell {
            inputsFrom = [ bin ];
          };
        }
      );
}

The error:

  error occurred: Command "aarch64-unknown-linux-gnu-gcc" "-O0" 
  "-ffunction-sections" "-fdata-sections" "-fPIC" "-arch" "arm64" 
  "-DSQLITE_CORE" "-DSQLITE_DEFAULT_FOREIGN_KEYS=1" 
  "-DSQLITE_ENABLE_API_ARMOR" "-DSQLITE_ENABLE_COLUMN_METADATA" 
  "-DSQLITE_ENABLE_DBSTAT_VTAB" "-DSQLITE_ENABLE_FTS3" 
  "-DSQLITE_ENABLE_FTS3_PARENTHESIS" "-DSQLITE_ENABLE_FTS5" 
  "-DSQLITE_ENABLE_JSON1" "-DSQLITE_ENABLE_LOAD_EXTENSION=1" 
  "-DSQLITE_ENABLE_MEMORY_MANAGEMENT" "-DSQLITE_ENABLE_RTREE" 
  "-DSQLITE_ENABLE_STAT2" "-DSQLITE_ENABLE_STAT4" "-DSQLITE_SOUNDEX"
   "-DSQLITE_THREADSAFE=1" "-DSQLITE_USE_URI" "-DHAVE_USLEEP=1" 
   "-D_POSIX_THREAD_SAFE_FUNCTIONS" "-DHAVE_ISNAN" 
   "-DHAVE_LOCALTIME_R" "-DSQLITE_ENABLE_UNLOCK_NOTIFY" "-o" 
   "/private/tmp/nix-build-cuppings-deps-aarch64-unknown-linux-gnu-0.1.0.drv-0/cuppings/target/release/build/libsqlite3-sys-e7e9592c5dbb57a8/out/sqlite3/sqlite3.o" 
   "-c" "sqlite3/sqlite3.c" with args "aarch64-unknown-linux-gnu-gcc" 
   did not execute successfully (status code exit status: 1).

It looks like it’s trying to build its own copy of the sqlite3 library. Is it possible to tell whatever crate is trying to do this that it should use a system-provided sqlite3, and then provide it with sqlite3 from Nixpkgs?

If you are using rusqlite with the bundled feature, try disabling that feature

I’m going more closely with what the crane example says here, and that still does not work.

Full error message here: Cannot find CoreFoundation · Issue #320 · ipetkov/crane · GitHub

Not that I know of.

I mean that’s what I thought I was doing but anyway now it breaks on CoreFoundation.

Just to explain why you weren’t (unless crane does some magic I don’t know about):

sqlite is not included as a build input, so the sqlite from Nixpkgs won’t be available in your build, won’t be found by whatever build.rs is running, and so it’ll try to build its own.


Anyway, to restate what I said on GitHub just in case you look here first, you’re trying to do something quite advanced (although it definitely should be possible). If it’s your first time cross-compiling with Nix, you’ll probably have an easier time doing it with just Nixpkgs first than bringing in two other third-party dependencies and trying to get them all to work together first try.

1 Like

OK. I’m ditching this approach and trying it with fenix which seems a bit more straight-forward.

They have this example: “cross-compiling with Naersk”

Which seems to build a lot of things (it takes ages) for me bugs out with an openssl problem.

See the ticket here: Compiling from macOS to Linux failing with openssl · Issue #106 · nix-community/fenix · GitHub

Like realistically, Cargo won’t do this cross-compile without nix because lots of things are missing on macOS. My hope was that nix provides a hermetic bunch of Linux tools that are working on macOS to be able to cross-compile for Linux.

I still think you’d have an easier time getting started with Nixpkgs’s own Rust stuff (buildRustPackage etc.) rather than fenix or crane. It doesn’t do incremental builds, so your builds will be slow, but it means you don’t have to troubleshoot which component is at fault if something doesn’t work, you can find lots of examples to work with in Nixpkgs, and more people will be familiar with it and able to help you, and then you can look into the more specialised tools once you have the basics working.

(BTW: please post full logs when reporting issues — both here and in your fenix issue there’s a lot of missing context which makes it hard to speculate about what’s going on.)

1 Like

Thanks, but I’m searching and I can’t find a coherent example that uses buildRustPackage that I can crib from.

This seems outdated and is not complete:
https://www.tomhoule.com/2021/building-rust-wasm-with-nix-flakes/

I can’t piece the bits and pieces here together to get to a working flake.nix:

Searching does not turn anything else up. Any examples would be great.

OK. I have the barest thing here:

flake.nix

{
  description = "another fetch tool written in rust";

  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  outputs = {
    nixpkgs,
    self,
    ...
  }: let
    lib = nixpkgs.lib;
    withSystem = f:
      lib.foldAttrs lib.mergeAttrs {}
      (map (s: lib.mapAttrs (_: v: {${s} = v;}) (f s))
        ["x86_64-linux" "x86_64-darwin" "aarch64-linux" "aarch64-darwin"]);
  in
    {
      overlay = _: final: {
        fetch-rs = self.packages.${final.system}.default;
      };
    }
    // withSystem (
      system: let
        pkgs = nixpkgs.legacyPackages.${system};
      in {
        # formatter = pkgs.alejandra;

        packages.default = pkgs.callPackage ./. {};

        devShells.default = pkgs.mkShell {
          packages = [
            pkgs.rustfmt
            pkgs.cargo
            pkgs.rustc
            pkgs.rust-analyzer
          ];
        };
      }
    );
}

default.nix

{
  lib,
  rustPlatform,
}:
rustPlatform.buildRustPackage rec {
  pname = "cuppings";
  version = "1.0.0";

  src = ./.;

  nativeBuildInputs = [
    # pkgs.darwin.apple_sdk.frameworks.Security
  ];

  buildInputs = [
  ];

  cargoLock = {
    lockFile = ./Cargo.lock;
  };

  meta = with lib; {
    homepage = "https://github.com/ISnortPennies/fetch-rs";
    description = "";
    license = licenses.unlicense;
    maintainers = with maintainers; [];
    # platforms = platforms.linux;
  };
}

This does not build on darwin because it can’t find the Security framework (and I have no idea how to pass it in above as per the comment).

And I don’t know how to get this to build for linux but that might actually work if I can figure out what command to use to kick that off.

the Security framework should be in buildInputs

I don’t really know how to get it there. Do I leaf pkgs through to the other file?

I asked for an example with everything in a single file, but that wasn’t possible.

  buildInputs = [
    pkgs.darwin.apple_sdk.frameworks.Security
  ];

And this will have to be conditional, so that it doesn’t try to use it on Linux, where that dependency doesn’t exist.

I don’t have the first clue how to get a pkgs into the file.

(Nor really where flake.nix reads in default.nix.)

you can put darwin in the lambda inputs and use darwin.apple_sdk.frameworks.Security

Yeah, I’ve seen the conditional code in the more complicated examples. I can lift that as soon as I understand how to get the right bits in the right spots.

OK. That seems to work though I don’t understand who is passing darwin in.

if you used callPackage to load default.nix, that’s what passed darwin in

I found some documentation on callPackage, but it’s very complicated it seems.

Anyway, it passes stdenv in too it seems so with this, the thing builds on mac:

  buildInputs = [
  ] ++ lib.optionals stdenv.buildPlatform.isDarwin [
    darwin.apple_sdk.frameworks.Security
  ];

Now how do I force it to build for Linux to see if that works?