Fetching git submodules broken when using fetchFromGitHub in buildRustPackage

Help
1

Trying to update the wasmtime package, I’m hitting an issue with the build script complaining that a submodule’s Cargo.toml doesn’t exist:

❯ nix-build -A wasmtime -K
these derivations will be built:
  /nix/store/xf18l83v6aff4q1zq77brk3qb8cnmk17-wasmtime-20190521-vendor.drv
  /nix/store/wf5czc954zpw68vsijw69fyrd62bflcn-wasmtime-20190521.drv
building '/nix/store/xf18l83v6aff4q1zq77brk3qb8cnmk17-wasmtime-20190521-vendor.drv'...
unpacking sources
unpacking source archive /nix/store/yq7pldkm5422f70d0704zqmx77nvha1a-source
source root is source
patching sources
applying patch /nix/store/vbsca57gwcjamcjwbbxncp0f44b7159c-cargo-lock.patch
patching file Cargo.lock
installing
error: failed to read `/build/source/lightbeam/Cargo.toml`

Caused by:
  No such file or directory (os error 2)
Traceback (most recent call last):
  File "/nix/store/bmv13g410xz281bcclp5n5mphpgd5xgz-cargo-vendor-normalise/bin/.cargo-vendor-normalise-wrapped", line 42, in <module>
    main()
  File "/nix/store/bmv13g410xz281bcclp5n5mphpgd5xgz-cargo-vendor-normalise/bin/.cargo-vendor-normalise-wrapped", line 17, in main
    assert list(data.keys()) == ["source"]
AssertionError
note: keeping build directory '/tmp/nix-build-wasmtime-20190521-vendor.drv-6'
builder for '/nix/store/xf18l83v6aff4q1zq77brk3qb8cnmk17-wasmtime-20190521-vendor.drv' failed with exit code 1
cannot build derivation '/nix/store/wf5czc954zpw68vsijw69fyrd62bflcn-wasmtime-20190521.drv': 1 dependencies couldn't be built
error: build of '/nix/store/wf5czc954zpw68vsijw69fyrd62bflcn-wasmtime-20190521.drv' failed

If I change the package to use fetchgit instead, things work as expected. There seems to be a weird interaction between rustPlatform.buildRustPackage and fetchFromGitHub - as if I try to use mkDerivation directly, fetchFromGitHub works as expected.

Is this a known issue?

2

Why is it trying to read from the absolute path /build/…?

3

@lilyball AFAIK that’s just the build process being run inside a chroot, i.e. that’s a “simulated” view of the filesystem.

4

I think I have a bit more context: the git repo has submodules, and fetchFromGitHub accepts two different hashes: the one produced by nix-prefetch-git --submodules and the one produced by nix-prefetch-git (without --submodules).

If I paste in the one without submodules, the build does not fail on a mismatch hash (although fetchSubmodules = true!), and instead breaks at build-time when the submodule directory is empty.

You can reproduce this with two hashes:

  • 0bagdgi9mbrifdnaz267khphnkfwmx02pwggnnbw9514bfmh22ij (no submodules)
  • 13lqf9dp1cnw7ms7hcgirmlfkr0v7nrn3p5g7yacfasrqgnwsyl8 (with submodules)

Basically, the first hash should simply not match, since we’re explicitly requesting submodules.

5

Looking at the definition of fetchGitHub and stripping out the stuff to deal with private repos, it’s ultimately a pretty thin wrapper around fetchgit (if submodules are included, otherwise fetchzip). In particular, the actual fetching of submodules and the hash handling are delegated to fetchgit. So it seems pretty strange to me that you’re saying fetchgit works while fetchGitHub doesn’t.

6

The point is that fetchgit handles fetchSubmodules = true correctly, and demands the second hash from my post above.

But you can verify for yourself if you want with the link in my top post :slight_smile:

7

I understand what you’re saying, I’m just saying that ultimately fetchGitHub with fetchSubmodules = true turns into a call to fetchgit, passing through the sha and fetchSubmodules attributes.

Unfortunately I can’t actually test your package because I’m running on darwin.