Rust-overlay: a better rust toolchain provider and drop-in replacement of nixpkgs-mozilla

Announcements
1

https://github.com/oxalica/rust-overlay

rust-overlay is a drop-in replacement of nixpkgs-mozilla’s rust overlay, supporting almost ARBITRARY version of rust toolchains from ALL three channels with FULLY customization support for components and targets. And of course, with Flake support.

Features:

  • Fully customizable binary distributed rust with all optional components and targets provided by upstream, from stable, beta and nightly channels. All versions later than 1.29.0 or 2018-09-13 are available. (Just currently. May include more early toolchains in the future)
  • Pure and no eval-time downloading. Flake support.
  • Toolchain selection by rust-toolchain file (Both TOML and single-line format are supported).
  • Custom rust-bin.distRoot if you prefer a rustup mirror, without triggering any rebuild.
  • Fully compatible with nixpkgs-mozilla’s rust overlay interface. latest.rustChannels, rustChannelOf and etc. are all available and functional as nixpkgs-mozilla. Though the new rust-bin interface is recommended. (Helper functions which require network access are still removed.)
  • Daily auto update with GitHub Action.

Related works:

36 Likes
2

Thank you, could you show an example on how to use it with a wasm example ?

currently, wasm-bindgen=cli is built from rustPlatform, it there a way to build such a thing using a rust package coming from the overlay?

1 Like
3

@freezeboy

could you show an example on how to use it with a wasm example ?

I’m not familiar with wasm. But I found a guide https://gist.github.com/oxalica/310d9a1ba69fd10123f2d70dc6e00f0b and it just works.
You can use flake devShell for setup of stable rust from overlay, wasm-pack and wasm-bindgen-cli from nixpkgs. I wrote an example dev flake here: https://gist.github.com/oxalica/310d9a1ba69fd10123f2d70dc6e00f0b
After setting flake.nix in the example project according to the guide. Type nix develop and run wasm-pack build --target web. Then it is built successfully for me. (Since wasm-pack is already provided by devShell, no need to cargo install again)

currently, wasm-bindgen=cli is built from rustPlatform, it there a way to build such a thing using a rust package coming from the overlay?

I’m not sure what do you want. wasm-bindgen-cli is already provided by nixpkgs. I think the compiler of wasm-bindgen-cli itself doesn’t matter here. The compiler in your development environment, which compiles your project, matters.

4

Thank you, your flake looks close to mine, I restarted the devShell and launched the build process. I got this exception:

Error: wasm-bindgen call returned a bad status error: 

it looks like the Rust project used to create this wasm file was linked against
a different version of wasm-bindgen than this binary:

  rust wasm file: 0.2.68
     this binary: 0.2.67

Currently the bindgen format is unstable enough that these two version must
exactly match, so it's required that these two version are kept in sync by
either updating the wasm-bindgen dependency or this binary. You should be able
to update the wasm-bindgen dependency with:

    cargo update -p wasm-bindgen

or you can update the binary with

    cargo install -f wasm-bindgen-cli

That is what made me think aof a toolchain version problem

5

@freezeboy It’s a version mismatch between your crate dependency wasm-bindgen and installed package wasm-bindgen-cli. Your CLI version is 0.2.67 as the error message (this may change if you update nixpkgs). I think you should write wasm-bindgen = "=0.2.67" in Cargo.toml to make the dependency match your CLI version.

2 Likes
6

Oh thank you very much, with all the troubles I had to get a working compiler, I didn’t think of this possibility.

Now it builds :tada:

7

This is overlay absolutely great! I’ve been using it together with a flake definition as a development shell for my project. I’ve been trying to expose my project as a package via the flake now as well, do you have an example of how I’d do that?

I’ve been looking at either rustPlatform.buildRustPackage or naersk.buildPackage I guess. Something like this?

let
  rustPlatform = makeRustPlatform { inherit (pkgs.rust-bin.stable) rustc cargo };
in rustPlatform.buildRustPackage { … }
8

You simply replace you devShell attribute with a defaultPackage one.

e.g.:

devShell.${system} = pkgs.mkShell {
  nativeBuildInputs = [ ... ];
  buildInputs = [ ... ];
  ...
};

becomes

defaultPackage.${system} = whatever.buildPackage {
  nativeBuildInputs = [ ... ];
  buildInputs = [ ... ];
  ...
};

You only need to keep the devShell attribute if some devtools are not dependencies for the build process

9

This was so easy to change out from nixpkgs-mozilla and keep using rustChannelOf:

{ sources ? import ./sources.nix }:

let
  pkgs = import sources.nixpkgs {
    overlays = [
      (import sources.rust-overlay)
    ];
  };

  chan = (pkgs.rustChannelOf {
    date = "2021-01-15";
    channel = "nightly";
  }).rust.override {
    targets = [ ];
    extensions = [
      "clippy-preview"
      "rust-src"
      "rustc-dev"
      "rustfmt-preview"
    ];
  };

in chan

Thanks for the work!

3 Likes
10

This sounds pretty great! I have to wonder if maybe Mozilla might accept this as a replacement for their own overlay.

1 Like
11

I would like to cross compile from Linux to Windows.

Would something like this works?

{ sources ? import ./nix/sources.nix }:

let
  pkgs = import sources.nixpkgs {
    overlays = [
      (import sources.rust-overlay)
    ];
  };

  chan = (pkgs.rustChannelOf {
    date = "2021-01-15";
    channel = "nightly";
  }).rust.override {
    targets = [ "x86_64-pc-windows-gnu" ];
    extensions = [
      "clippy-preview"
      "rust-src"
      "rustc-dev"
      "rustfmt-preview"
    ];
  };

in chan

I got:

❯ cargo build --target x86_64-pc-windows-gnu
   Compiling squad-clean v0.1.0 (/home/bbigras/dev/squad-clean)
error[E0463]: can't find crate for `std`
  |
  = note: the `x86_64-pc-windows-gnu` target may not be installed

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: could not compile `squad-clean`

To learn more, run the command again with --verbose

EDIT:

it seems I needed:

in
pkgs.mkShell {
  buildInputs = with pkgs; [
    chan
  ];
}

instead of just chan.

12

I guess I need a bit more hand-holding. I’m trying to build veloren which uses the legacy rust-toolchain format. How do I get a rustPlatform? I can’t figure out how to use the helper, or rustChannelOf. Someone else requested it too, can we just get an example of how to construct a rustPlatform? That seems highly useful since it would let us write derivations as if they were to be upstream but can still work when them while they’re in the “needs to be built with nightly” stage.

edit: or just buildRustPackage I guess?

13

Also, are there plans for more automation? It seems like it has 1.48.0 as stable, whereas it seems like 1.49.0 was actually stable as of end of December 2020.

(I don’t want to sound ungrateful, I appreciate this, thank you for it! Just evaluating it and fenix while trying to build veloren. Cheers, and thanks again for working on it!)

I was able to get this working:

          base = (pkgs.rustChannelOf { channel = "nightly"; date = "2021-01-01"; });

          rustPlatform = pkgs.recurseIntoAttrs (pkgs.makeRustPlatform {
            rustc = base.rust;
            cargo = base.cargo;
          });

          velorenPkg = pkgs.callPackage ./pkgs/veloren/default.nix {
            rustPlatform = rustPlatform;
            velorenSrc = (velorenSrc system);
          };

but I can’t get it to work when I change:

  base = (pkgs.rust-bin.fromRustupToolchainFile "${velorenSrc system}/rust-toolchain");

I’m not sure if there’s something I need to do with the output frm fromRustupToolchainFile before it’s more usable?

edit2: extra confusing, I could presumably parse the toolchain file and call it the way I showed was wokring, but I’d assume that’s what fromRustupToolchainFile is meant to do?

1 Like
14

Also, are there plans for more automation? It seems like it has 1.48.0 as stable, whereas it seems like 1.49.0 was actually stable as of end of December 2020.

Of course it already has. You can reference it by rust-bin.stable.latest.rust or rust-bin.stable."1.49.0".rust.

Currently, fromRustupToolchainFile produce the aggregated derivation instead of a set of derivation, since you can customize components and targets in rust-toolchain file.
For the usage with makeRustPlatform, I’m not sure how to fix it ideally. Maybe you can open an issue for it? I’ll look into it.

1 Like
15

If anybody needed help getting this to work as a nix flake, here’s my flake.nix file in a project I just got to work where I pull in mdbook for one purpose and separately have a Rust app that will be a custom RSS feed & static site generator.

{
  description = "ramda.guide";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/master";
    naersk = {
      url = "github:nmattia/naersk";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    rust-overlay = {
      url = "github:oxalica/rust-overlay";
      flake = false;
      inputs.nixpkgs.follows = "nixpkgs";
    };
    utils = {
      url = "github:numtide/flake-utils";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { naersk, nixpkgs, rust-overlay, self, utils }:
    utils.lib.eachDefaultSystem (system:
      let
        rust-overlay' = import rust-overlay;
        pkgs = import nixpkgs {
          inherit system;
          overlays = [ rust-overlay' ];
        };
        rust = (pkgs.rustChannelOf {
          date = "2021-01-28";
          channel = "nightly";
        }).rust;
        naersk-lib = naersk.lib."${system}".override {
          cargo = rust;
          rustc = rust;
        };
      in rec {
        packages = utils.lib.flattenTree {
          mdbook = pkgs.mdbook;
          news = naersk-lib.buildPackage {
            pname = "news";
            root = ./src/news;
          };
        };

        defaultPackage = pkgs.mdbook;

        apps.book = utils.lib.mkApp {
          drv = pkgs.mdbook;
        };

        apps.news = utils.lib.mkApp {
          drv = packages.news;
        };

        defaultApp = apps.book;

        devShell = pkgs.mkShell {
          buildInputs = [ pkgs.mdbook ];
          nativeBuildInputs = [ rust ];
        };
      }
    );
}

I can then run nix run .#news to build the separate Rust app, and I can run nix run .#book build to build the mdbook project.

If anyone also has advice on simplifying this flake, I’m all ears!

2 Likes
16

Just wanted to say thanks for this, we switched from the Mozilla overlay to this one and it is just great! Thanks!

17

Any plans for a cachix cache?

18

As far as I understand the project it pulls the pre-built rust binaries and patchelfs them, downloading the binary is usually the bottleneck, and providing a binary cache wouldn’t remove it.

1 Like