Speeding up Rust app packaging

I have a Rust app that I’m ready to get packaged up. I have a derivation that builds it, but the final product isn’t quite right. I’ve been working on the post-install script to try to find the magic combination of things that need to be wrapped around the executable…

but it takes 20-ish minutes with every one of my cores to build this app using rustPlatform.buildRustPackage (which is far more than it takes with just Cargo), plus it does a fresh rebuild on every install, even though my only change is to the postInstall script.

So, I’m thinking to speed things up a lot by having one derivation for the package itself, and then another derivation that wraps the final executable in whatever scripts that I need to make it run right.

With that, I have crate.nix:

{ pkgs,
  fetchFromGitHub ? pkgs.fetchFromGitHub,
  stdenv ? pkgs.stdenv,
  rustPlatform ? pkgs.rustPlatform,
}:
let
in rustPlatform.buildRustPackage rec {
  ...
}

And then I have default.nix:

{ pkgs,
  fetchFromGitHub ? pkgs.fetchFromGitHub,
  stdenv ? pkgs.stdenv,
  rustPlatform ? pkgs.rustPlatform,
}:
let crate = import ./crate.nix { inherit pkgs fetchFromGitHub stdenv rustPlatform; };
in stdenv.mkDerivation rec {
  name = "fitnesstrax-${version}";
  version = "0.0.5-pre3";

  buildInputs = [
    crate
    pkgs.makeWrapper
    pkgs.gsettings-desktop-schemas
    ];

  postInstall = ''
    makeWrapper $fitnesstrax/bin/fitnesstrax-gtk $out/bin/fitnesstrax --set GSETTINGS_SCHEMA_DIR $out/share --set XDG_DATA_DIRS=$GSETTINGS_SCHEMAS_PATH:$XDG_DATA_DIRS
  '';
}

But… mkDerivation requires a src path, which isn’t relevant in this case.

  • Should I be using mkDerivation, or something like symlinkJoin?
  • Is postInstall the right way to go here, or should I instead be creating a launcher script, like… launcher = 'makeWrapper $fitnesstrax/bin/fitnesstrax-gtk $out/bin/fitnesstrax --set ...'?
1 Like

This is something I’d like to have generic solution for. Your approach is fine

  • you can set src = crate;
  • but you also can also use dedicated pkgs.runCommand
  • symlinkJoin is fine if ln -s approach can’t be applied

Two community projects that attempt to behave better with regards to incremental builds:

YMMV but I find the former gels slightly more with my mindset and workflows, and I’m definitely feeling the pain of upstream Nix support while trying to work in my first non-toy Rust project. I’m using the kube, k8s-openapi, prometheus, and actix-web crates in practice. Crate2nix handles that very satisfactorily, up to and including the ability to generate Docker images out of it using the nixpkgs facilities for this.

There are a few more options cited in the two READMEs that I don’t have any first-hand experience with.

I actually got some advice that .nix files generate from the likes of Carnix (and it looks like Crate2Nix) generally won’t be accepted into nixpkgs because they’re too large to be properly reviewed. That’s why I’m using buildRustCrate.

naersk looks interesting, though.

That’s interesting to hear, because there is a small number of packages in Nix that use buildRustCrate. (Maybe 2 or 3.)

One reason why it takes longer than a clean cargo build is that Nix is currently building/installing the release binary target, but it’s running the tests in debug mode, so it ends up compiling all the libraries twice.

There’s work underway to try to address this, which you might be interested in trying out:

I wonder if this is actually a good idea, debug doesn’t just compile with fewer optimizations, it also turns on integer overflow checks and debug assertions (which are used throughout the standard library and crates to do more expensive checks).

Edit: maybe I should raise my concern in the issue?

Edit: maybe I should raise my concern in the issue?

Yes, may be better (I’m not an active discourse reader).

I wonder if this is actually a good idea, debug doesn’t just compile with fewer optimizations, it also turns on integer overflow checks and debug assertions (which are used throughout the standard library and crates to do more expensive checks).

Hmm, I see. But isn’t this an issue we have with every rust-package with doCheck = false?

Also, those sort of issues should be something catched by upstream before publishing a release, right? I mean during development, you’d usually have debug enabled I guess.

My understanding is that the goal of running the test suite as part of a derivation is to check that the patching we do with our uncommon toolchain (hardcoded paths, patchelf, etc.) does not break the software too much. I think this specific type of problems will surface as easily with a debug build or a release build. Problems with overflows should be caught upstream.

2 Likes

Thanks for this. Not trying it out on this derivation, at least not right now, since I’ve been running as hard as I can to finish a release and get this app out and available to people.

But, in starting to look at PRs (I had no idea how many open PRs there were), I actually came across this, so it will be one of the things that I attempt to review and test.