Using `crate2nix` to create a `cargo` alternative or wrapper that transparently uses `/nix/store` as a cache?

Help
1

Hi folks!

I’m a frequent Rust user, and recently have been diving deeper into the Nix rabbit hole.

I recently came across crate2nix, an awesome tool for generating a nix expression that allows for crate-by-crate caching.

A common pattern I’m noticing is:

  1. Use cargo to generate a lock file if one doesn’t yet exist.
  2. Use crate2nix to generate the nix expression.
  3. Use the generated expr to build a crate in the workspace, i.e.
    nix-build -E "with import <nixpkgs> {}; let c = callPackage ./Cargo.nix {}; in c.workspaceMembers.\"my-crate\".build"

I imagine it might be useful to have a tool in day-to-day Rust dev that automated each of these steps (generating temporary “Cargo.nix” and “crate-hashes.json” files on each invocation) so that the build process was as easy as just calling cargo build. I’m thinking that this would provide the benefit of transparently using the /nix/store as a system-wide cache for each crate without the need for polluting crate repositories with extra nix-specific files.

While I don’t mind adding nix-specific files to personal projects and repos that I govern, I frequently contribute to a lot of other open source Rust projects that involve a majority of devs and maintainers on other operating systems who may not know what nix is and prefer not to include package-manager specific files in the main git repository. One solution could be to use a personal .gitignore to disclude the Nix-specific files in my contributions, but it would be nice to not have to do this for every different Rust project I contribute to.

Have you come across any nix/Rust build tools along these lines? Or perhaps you have a nicer solution for taking advantage of the nix cache while hacking on open source crate repos? I’m considering throwing together a tool along these lines using crate2nix but am keen to continue exploring what’s already out there first.

2

You do not have to create a Cargo.nix file, since crate2nix support IFD (import from derivation). Here is an example:

You could create a default.nix file that uses IFD and works across projects.

1 Like
3

Thanks for the tip! This certainly seems nicer than using the crate2nix cli in a custom derivation.

I noticed that you’re importing crate2nix via fetchFromGitHub - do you happen to know if it’s possible to use this tools.nix file via the crate2nix that exists in nixpkgs? If it’s not possible, what do you think the best way of packaging generatedCargoNix for nixpkgs would be? I guess a crate2nixTools could be added. It would be cool if we could expose something like crate2nix.tools, though I’m unsure how to achieve this.

4

Not in the built derivation:

❯  find $(nix-build -A crate2nix --no-out-link) -name tools.nix
# Vast emptiness ;).

However crate2nix.src would normally work:

❯  find $(nix-build -A crate2nix.src) -name 'tools.nix'
/nix/store/qhgj6xjj44y53bw81slscp5w1xlddw59-source/tools.nix

However, due to how src is used in the crate2nix derivation, it is not a derivation. Also, it points to a subdirectory of the crate2nix sources. I have made a PR to change this. After that, you should be able to use ${crate2nix.src}/tools.nix in your own Nix code.

2 Likes
5

Nice one @danieldk, looks like "{$crate2nix.src}/tools.nix" is working as of the latest nixos-unstable :slight_smile:

Before integrating this into a standalone tool, I first wanted to try using "${crate2nix.src}/tools.nix" to build a single crate. Here’s the small .nix expression I’m using to do so:

{
  callPackage,
  fetchFromGitHub,
  crate2nix,
}:
let
  crateTools = callPackage "${crate2nix.src}/tools.nix" { };
  generatedCargoNix = crateTools.generatedCargoNix {
    name = "nannou";
    src = builtins.toPath (builtins.getEnv "PWD");
  };
  cargoNix = callPackage generatedCargoNix { };
in cargoNix.workspaceMembers.nannou.build

However, when I attempt to build the expression with:

nix-build -E "with import <nixos> {}; callPackage ./nannou.nix {}"

I run into the following error:

error: unsupported argument 'submodules' to 'fetchGit', at /nix/store/qhgj6xjj44y53bw81slscp5w1xlddw59-source/tools.nix:235:15

In case you’re interested, here’s the full trace of the error:

Full Trace 
error: while evaluating 'callPackageWith' at /nix/store/03pwhxb4wgf8pj91aq5kz090rs8p7h19-nixos-21.05pre281538.9e377a6ce42/nixos/lib/customisation.nix:117:35, called from (string):1:25:
while evaluating 'makeOverridable' at /nix/store/03pwhxb4wgf8pj91aq5kz090rs8p7h19-nixos-21.05pre281538.9e377a6ce42/nixos/lib/customisation.nix:67:24, called from /nix/store/03pwhxb4wgf8pj91aq5kz090rs8p7h19-nixos-21.05pre281538.9e377a6ce42/nixos/lib/customisation.nix:121:8:
while evaluating anonymous function at /home/mindtree/programming/rust/nannou/nannou.nix:1:1, called from /nix/store/03pwhxb4wgf8pj91aq5kz090rs8p7h19-nixos-21.05pre281538.9e377a6ce42/nixos/lib/customisation.nix:69:16:
while evaluating 'callPackageWith' at /nix/store/03pwhxb4wgf8pj91aq5kz090rs8p7h19-nixos-21.05pre281538.9e377a6ce42/nixos/lib/customisation.nix:117:35, called from /home/mindtree/programming/rust/nannou/nannou.nix:12:14:
while evaluating 'makeOverridable' at /nix/store/03pwhxb4wgf8pj91aq5kz090rs8p7h19-nixos-21.05pre281538.9e377a6ce42/nixos/lib/customisation.nix:67:24, called from /nix/store/03pwhxb4wgf8pj91aq5kz090rs8p7h19-nixos-21.05pre281538.9e377a6ce42/nixos/lib/customisation.nix:121:8:
while evaluating the attribute 'buildPhase' of the derivation 'nannou-crate2nix' at /nix/store/qhgj6xjj44y53bw81slscp5w1xlddw59-source/tools.nix:39:7:
while evaluating the attribute 'text' of the derivation 'vendor-config' at /nix/store/03pwhxb4wgf8pj91aq5kz090rs8p7h19-nixos-21.05pre281538.9e377a6ce42/nixos/pkgs/build-support/trivial-builders.nix:76:7:
while evaluating the attribute 'buildCommand' of the derivation 'deps' at /nix/store/03pwhxb4wgf8pj91aq5kz090rs8p7h19-nixos-21.05pre281538.9e377a6ce42/nixos/pkgs/build-support/trivial-builders.nix:76:7:
while evaluating 'concatMapStrings' at /nix/store/03pwhxb4wgf8pj91aq5kz090rs8p7h19-nixos-21.05pre281538.9e377a6ce42/nixos/lib/strings.nix:53:25, called from /nix/store/03pwhxb4wgf8pj91aq5kz090rs8p7h19-nixos-21.05pre281538.9e377a6ce42/nixos/pkgs/build-support/trivial-builders.nix:366:9:
while evaluating anonymous function at /nix/store/03pwhxb4wgf8pj91aq5kz090rs8p7h19-nixos-21.05pre281538.9e377a6ce42/nixos/pkgs/build-support/trivial-builders.nix:366:31, called from undefined position:
while evaluating 'escapeShellArg' at /nix/store/03pwhxb4wgf8pj91aq5kz090rs8p7h19-nixos-21.05pre281538.9e377a6ce42/nixos/lib/strings.nix:318:20, called from /nix/store/03pwhxb4wgf8pj91aq5kz090rs8p7h19-nixos-21.05pre281538.9e377a6ce42/nixos/pkgs/build-support/trivial-builders.nix:367:33:
while evaluating the attribute 'name' at /nix/store/qhgj6xjj44y53bw81slscp5w1xlddw59-source/tools.nix:277:21:
while evaluating the attribute 'buildCommand' of the derivation 'hotglsl-0.1.0' at /nix/store/03pwhxb4wgf8pj91aq5kz090rs8p7h19-nixos-21.05pre281538.9e377a6ce42/nixos/pkgs/build-support/trivial-builders.nix:76:7:
while evaluating the attribute 'outputHash' of the derivation 'hotglsl-0.1.0' at /nix/store/03pwhxb4wgf8pj91aq5kz090rs8p7h19-nixos-21.05pre281538.9e377a6ce42/nixos/pkgs/build-support/fetchgit/default.nix:54:10:
while evaluating the attribute '"hotglsl 0.1.0 (git+https://github.com/nannou-org/hotglsl?branch=master#0066deaf2d428ffe73e697c7ec7f2e5af74571e4)"' at /nix/store/qhgj6xjj44y53bw81slscp5w1xlddw59-source/tools.nix:245:13:
while evaluating the attribute 'buildCommand' of the derivation 'hash-of-hotglsl' at /nix/store/03pwhxb4wgf8pj91aq5kz090rs8p7h19-nixos-21.05pre281538.9e377a6ce42/nixos/pkgs/build-support/trivial-builders.nix:76:7:
unsupported argument 'submodules' to 'fetchGit', at /nix/store/qhgj6xjj44y53bw81slscp5w1xlddw59-source/tools.nix:235:15

It looks like support for submodules in fetchGit was only added last year in this PR:

https://github.com/NixOS/nix/pull/3166

This leads me to suspect that in order to use the crate2nix/tools, the unstable version of nix is required. Does that sound right?

If so, I might put this tool on the backburner for a while. While I’m excited about Nix 3, flakes, etc, I think using nixos-unstable is already pushing my mental instability budget :sweat_smile:

1 Like