It’s very similar to Naersk in the regard that it will build the entire workspace in one go. The unit of caching is the entire derivation itself, just like in Naersk.
The advantage over Naersk is that the nix API makes it much easier to compose different cargo invocations as completely separate derivations (you could do this with Naersk but you really have to understand all the internals to wire it all up right).
Here’s a quick flake example illustrating what you could do:
{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
crane.url = "github:ipetkov/crane";
crane.inputs.nixpkgs.follows = "nixpkgs";
flake-utils.url = "github:numtide/flake-utils";
flake-utils.inputs.nixpkgs.follows = "nixpkgs";
};
outputs = { self, nixpkgs, crane, flake-utils, ... }:
flake-utils.lib.eachDefaultSystem (system:
let
pkgs = import nixpkgs {
inherit system;
};
craneLib = crane.lib.${system};
src = ./.;
# Build *just* the cargo dependencies, so we can reuse
# all of that work (e.g. via cachix) when running in CI
cargoArtifacts = craneLib.buildDepsOnly {
inherit src;
};
# Run clippy (and deny all warnings) on the crate source,
# resuing the dependency artifacts (e.g. from build scripts or
# proc-macros) from above.
#
# Note that this is done as a separate derivation so it
# does not impact building just the crate by itself.
my-crate-clippy = craneLib.cargoClippy {
inherit cargoArtifacts src;
cargoClippyExtraArgs = "-- --deny warnings";
};
# Build the actual crate itself, reusing the dependency
# artifacts from above.
my-crate = craneLib.buildPackage {
inherit cargoArtifacts src;
};
# Also run the crate tests under cargo-tarpaulin so that we can keep
# track of code coverage
my-crate-coverage = craneLib.cargoTarpaulin {
inherit cargoArtifacts src;
};
in
{
defaultPackage = my-crate;
checks = {
inherit
# Build the crate as part of `nix flake check` for convenience
my-crate
my-crate-clippy
my-crate-coverage;
};
});
}
It’s possible to add a devshell hook which will copy the artifacts from the store to the target directory, yes, and then keep developing locally with cargo build outside.
Note that by default Crane will build everything in release mode, so if you copy the artifacts over but invoke cargo build, cargo may decide to rebuild everything with debug flags on, etc.
Crate sources do get put in the store at some point in order to prepare a vendored directory so cargo can use it during sandboxed builds.
If you really wanted to you could copy them over to cargo’s home directory and avoid having cargo touch the network. I don’t think cargo’s internal file layout is documented as stable, so YMMV if any breakage happens there
(FWIW sources are usually cheap to download from crates.io, but the artifact caching that nix can provide is the real winner over the default cargo experience in my view)
❯ nix build
error: access to canonical path '/nix/store/5mvgi56jp7gjm88whxp0h22i8yx4065s-nix-shell' is forbidden in restricted mode
(use '--show-trace' to show detailed location information)
Do I need to use --impure? Maybe I did something wrong. I used the “custom toolchain” example to use oxalica/rust-overlay.
EDIT: I think I did something wrong, I have the same problem with nix flake check.
Nice work! It’s always exciting to see folk aiming at improving the Rust dev workflow, especially w.r.t. improving caching.
Q: What’s the advantage over naersk, does it cache each crate build individually?
A: It’s very similar to Naersk in the regard that it will build the entire workspace in one go. The unit of caching is the entire derivation itself, just like in Naersk.
The fact that you’ve managed to separate the dependencies from the src of the crate itself already sounds like a big win for the common caching needs!
That said, I’d love to hear your thoughts on the possibility of creating derivations per crate. Is this something that you considered but turned down due to difficulty of implementation? Or perhaps the state of cargo, its arbitrary-rust-code build scripts and macros make this simply not worth it?
One of my most major issues with the Rust dev process is the need to build every crate from scratch when starting new projects or updating the Rust version, even if you’ve just built the same list of dependencies for 100 other similar projects. The lack of user/system-wide caching of build artifacts is painful, results in huge initial compilation times and bloated target directories.
It seems like Nix is uniquely positioned to have a chance at solving this with its flexible-yet-hermetic build system. I think the cargo2nix project is having a go at this:
Design
This Nixpkgs overlay builds your Rust crates and binaries by first pulling the dependencies apart, building them individually as separate Nix derivations and linking them together. This is achieved by passing custom linker flags to the cargo invocations and the underlying rustc and rustdoc invocations.
Last time I tried I remember running into a lot of odd build errors, but that must have been a year ago now - perhaps it’s worth trying again…
I will happily give credit to Naersk for this technique before me, it really goes a long way!
My approach with Crane was not in eschewing cargo but embracing it completely, only doing the necessary work to make cargo work within Nix’s sandbox, and then let it take over the hard work.
The crate2nix project takes a very interesting approach to effectively removing cargo altogether and therefore gaining better caching at the crate level itself (but more on this later). There’s two main reasons I did not want to take a similar path:
although I find what crate2nix is doing as laudable and inspiring, I could not hope to be able to re-implement all of cargo’s behavior within Nix itself (nor do I want to find myself playing catch up with cargo’s features and behavior)
removing cargo means losing the ability to use cargo specific tools (clippy and wasm-pack are some which come to mind, others may be needed for things like generating models/types from schemas, etc.)
On the topic of per-crate caching: changing which feature flags are enabled requires rebuilding the crate, so the best possible global caching you could get is one-copy-per-crate-per-feature-flag-combination. My hope is that the broader Rust community can solve this problem in a more universal way and then find ways to integrate that with Nix!
My view is cargo2nix is very important to show upstream Cargo what we would like to do. Then we can figure out together what is missing in Cargo itself — namely the “planning” vs “execution” isn’t separated enough.
Without going the full derivation-per-crate route, they will not be able to understand where Cargo fall short as well, because there is no demonstration of what the missing functionality can do.
My view is cargo2nix is very important to show upstream Cargo what we would like to do.
I agree with this - long term we want language build tools to make available their build plan so we can have Nix execute it.
But short term the approach of trying to emulate the build tool’s behavior is of limited viability, as it creates only a second rate experience with truly incredible amounts of effort. @nmattia tried to do this for Haskell with Snack (I tried helping out a tiny bit) and we both came to that conclusion.
Crane v0.3.0 now supports automatically handling git dependencies!
There is no example this time because there is nothing to configure, it should Just Work
It’s also worth noting that cargo retains the flexibility to only use these dependencies when they are actually needed, without forcing an override for the entire workspace (for example, a pinned git dependency only used in a testing crate doesn’t end up being used for building the actual production binaries).
I think I’ve covered most of the edge cases, but if something doesn’t seem to work please open an issue!
