rustPlatform.buildRustPackage and caching?

Help
1

The rust builds using rustPlatform.buildRustPackage always re-build all dependencies, which makes builds rather slow. I’ve been trying to use sccache to speed it up, but my possibly quite naive approach doesn’t work:

app = rustPlatform.buildRustPackage {
  name = "TheApp;
  version = "0.1";

  src = lib.cleanSource ./.;

  cargoSha256 = sha256:04s31gvr1c0m5ldhnmw105c8a86cwrh3724nbdna5kmmasffl569;

  buildInputs = [sccache];
  nativeBuildInputs = [sccache];

  RUSTC_WRAPPER = "sccache";
  SCCACHE_DIR = toString ./. + "/.sccache";
};

When building a shell.nix on the above it works as expected, i.e. envvars are set and the cache is used. When building using nix-build however, the cache isn’t created where I hoped it would be and a second build takes just as long as the first. I’m guessing there’s some sort of “sandboxing” by nix-build. Maybe that completely prevents the use of sccache.

Is there some other way of speeding up builds of rust packages?

1 Like
2

I’m guessing there’s some sort of “sandboxing” by nix-build . Maybe that completely prevents the use of sccache .

Thats right and it is a major feature of Nix! As you don’t have to worry about cache invalidation anymore. Ever.

For my own Rust projects I don’t use buildRustPackage for precisely the problem you encountered. There are a couple of alternatives, but my absolute favorite is crate2nix.

3

@erictapen Do you use that on a CI system?

I had a quick look and I can’t find any documentation on where the cache ends up. I need to know since I want TravisCI to cache it for the next build. (Asking before I dig deeper to find the answer for myself.)

4

No I don’t use CI atm. I used to use Hydra though.

There is no dedicated cache directory with Nix builds. Everything that gets built ends up in the Nix store. I never used Travis, but as long as you can keep /nix/store/ through CI invocations, your individual crates will be cached.

You might want to read a bit into the concept of the Nix store and the Nix binary cache. Nix has a very different model compared to traditional build systems.

5

I use crate2nix/buildRustCrate. Caching /nix with GitHub Actions or Travis CI always seemed a bit fragile to me, so I use Cachix or a self-hosted cache across some projects (S3-compatible object storage). Both work very nicely, though uploads to the self-hosted cache are a bit slow, because Nix uses LZMA for the NAR files.

For private caches, I use something along the lines of:

name: "Test"
on:
  pull_request:
  push:
jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - uses: cachix/install-nix-action@v11
      with:
        extra_nix_config: |
          substituters = https://cache.nixos.org s3://nix-cache?scheme=https&endpoint=<s3 cache>
          trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= <s3 cache>:<publickey>
    - name: Storing pre-build store paths
      run: nix path-info --all | grep -v '\.drv$' | sort > /tmp/pre-build-paths
    - name: Build CI tests
      run: nix-build ci.nix
    - name: Upload new store paths
      if: ${{ always() }}
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        NIX_SIGNING_KEY: ${{ secrets.NIX_SIGNING_KEY }}
      run: |
        comm -13 /tmp/pre-build-paths <(nix path-info --all | grep -v '\.drv$' | sort) > /tmp/new-paths
        echo "$NIX_SIGNING_KEY" > /tmp/sign-key
        nix sign-paths --key-file /tmp/sign-key $(cat /tmp/new-paths)
        nix copy -L --to "s3://nix-cache?scheme=https&endpoint=<s3 cache>" $(cat /tmp/new-paths)

Which seems to work well enough.