Nixpkgs 22.11 -> 23.05: Rust hdf5 crate stops compiling

Help
1

I’m using a flake (with crate2nix) to manage a Rust project which needs the HDF5 libray. When trying to upgrade the flake from nixos 22.11 to 23.05 I get the following error, when compiling the Rust code:

error: failed to run custom build command for `hdf5-sys v0.8.1`

Caused by:
  process didn't exit successfully: `/tmp/bump-nix/target/debug/build/hdf5-sys-af0ab5be5860baa3/build-script-build` (exit status: 101)
  --- stdout
  Setting HDF5 root from environment variable:
      HDF5_DIR = "/nix/store/lixvnqz0s5cylwl8grxlas3gxl7hb6s9-hdf5"
  Custom HDF5_DIR provided; rpath can be set via:
      RUSTFLAGS="-C link-args=-Wl,-rpath,/nix/store/lixvnqz0s5cylwl8grxlas3gxl7hb6s9-hdf5/lib"
  Parsing HDF5 config from:
      "/nix/store/lixvnqz0s5cylwl8grxlas3gxl7hb6s9-hdf5/include/H5pubconf.h"

  --- stderr
  thread 'main' panicked at 'Invalid H5_VERSION: "1.14.0"', /home/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hdf5-sys-0.8.1/build.rs:200:21
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: command `/nix/store/3zgxqy978lcdgxldfpz0gggivlamhmrv-cargo-1.70.0-x86_64-unknown-linux-gnu/bin/cargo test --no-run --message-format json-render-diagnostics --workspace --exclude bindings` exited with code 101
error: Recipe `test-rust-pure` failed on line 9 with exit code 101
error: Recipe `test-rust` failed on line 6 with exit code 101

The part about ‘Setting HDF5 root from environment variable’ relates to this

devShell = pkgs.mkShell {

  ...

  HDF5_DIR = pkgs.symlinkJoin { name = "hdf5"; paths = [ pkgs.hdf5 pkgs.hdf5.dev ]; };
};

  • Naively following the advice about setting rpath doesn’t seem to improve the situation.

  • I don’t know whether the former is just a warning and the only fatal part is the ‘Invalid H5_VERSION: “1.14.0”’ panic. I see that nixpkgs 23.05 has hdf5 version 1.14.0, while 22.11 has 1.12.2. Does this mean that there is an incompatibility between what the hdf5 crate expects and nixpkgs 23.05 provides?

Do you have any hints about how to get past this problem?

2

The hdf5 crate indeed doesn’t support 1.14.0 yet, see this PR.

1 Like
3

Having two flake inputs pointing at two different versions of nixos:

nixpkgs.url = "github:nixos/nixpkgs/nixos-23.05";
oldpkgs.url = "github:nixos/nixpkgs/nixos-22.11";

and grabbing everything from the first, except for HDF5 which I grab from the second, seems to be working.

Am I setting myself up for some subtle problems by mixing these two versions in the same flake, or is everything sufficiently isolated?

4

Should be sufficiently isolated, but be aware that this probably results in additional disk space used for some recursive dependencies.

Alternatively, it doesn’t look like the build process for the package has changed much, you can override its source to what it used to be instead:

devShell = pkgs.mkShell {

  ...

  HDF5_DIR = let
    hdf5 = pkgs.hdf5.overrideAttrs(old: {
      version = "1.12.2";

      src = fetchurl {
        url = "https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-1.12.2/src/hdf5-1.12.2.tar.bz2";
        sha256 = "sha256-5OeUM0UO2uKGWkxjKBiLtFORsp10+MU47mmfCxFsK6A=";
      };

      postFixup = ""; # The 1.14.0 derivation needs to patch some additional things
    });
  in
    pkgs.symlinkJoin { name = "hdf5"; paths = [ hdf5 hdf5.dev ]; };
};

This will obviously break if the build process does eventually change, but it should be fine while we wait for the crate to update.

1 Like
5

Thank you so much for the code sample. This is the sort of thing that I understand in principle, but would take me a prohibitive amount of time to sort out the details in practice.

One thing that has me stymied: I get the error

error: hash mismatch in file downloaded from 'https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-1.12.2/src/hdf5-1.12.2.tar.bz2':
         specified: sha256:181bdh8hp7v9xqwcby3lknr92lxlicc2hqscba3f5nhf8lrr9rz4
         got:       sha256:1zlawdzb0gsvcxif14fwr5ap2gk4b6j02wirr2hcx8hkcbivp20s

while the code that I copy-pasted from your example contains

url = "https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-1.12.2/src/hdf5-1.12.2.tar.bz2";
sha256 = "sha256-5OeUM0UO2uKGWkxjKBiLtFORsp10+MU47mmfCxFsK6A=";

The URLs match, but the specified (or even the got) SHA in the error message does not match the one specified in the code.

Oh … your code did not qualify fetchurl and I guessed that it comes from builtins, which is what resulted in the above error message. But there is also a fetchurl in nixpkgs. If I use that one instead, I now get

specified: sha256-5OeUM0UO2uKGWkxjKBiLtFORsp10+MU47mmfCxFsK6A=
   got:    sha256-Goi742ITos6gyDlyAaRZZD5xVcnckeBiZ1s/sH7jiv4=

that is to say, the specified SHA in the error message now does match the one specified in the source.

This leaves me with two puzzles:

  1. WTF is the difference between builtins.fetchurl and nixpkgs.fetchurl? OK, I’ve found this, but I don’t have time to ruminate on it and to try to digest its meaning. It’s tempting to conclude that it’s an evil practical joke. These sorts of things certainly don’t help wider adoption of Nix.

  2. If the URLs match exactly, but the SHAs differ, I infer that they update the tarball at that location (or a miscreant is up to something nefarious), and if they do that then this derivation is going to be running into this problem repeatedly. If so, is there a workaround?

6

That, or I copied the wrong sha from the commit that updated it: hdf5: 1.12.2 -> 1.14.0 · NixOS/nixpkgs@bd33397 · GitHub

Absolutely my bad, sorry for the confusion.

That’s a good question! I did not realize that they will create different hashes either.

The main difference is that nixpkgs.fetchurl uses curl in a derivation to download files, while builtins.fetchurl is a nix builtin.

Is the difference whether it decompresses the downloaded tarball by default? I thought neither of them did, but it looks like builtins.fetchurl does?

7

Thought I’d share this, it seems the crate has merged 1.14 support: Support hdf5 1.14.0 by mulimoen · Pull Request #227 · aldanor/hdf5-rust · GitHub

Just need to wait for dependants to update to this version now.

1 Like