Hash mismatch building `jre` on release-20.09

I’m upgrading a project to nixpkgs-20.09 and I’m running into a problem building the jre package on x86_64-linux.

Note that I disable the upstream Nix cache (cache.nixos.org) to ensure our own cache will contain all required dependencies to build my project.

Note the failure when I run the following in nixpkgs on branch release-20.09 on revision dcc62359cd517eed9e201dafb076a452367a326b:

$ nix-build -A jre
these derivations will be built:
  /nix/store/55qlj9cq22i7qkkw19xcmcw2y1scc093-nashorn-jdk8u272-b10.tar.gz.drv
  /nix/store/c82w44q9al45alcy8sgrvsxlqi925ws0-langtools-jdk8u272-b10.tar.gz.drv
  /nix/store/d5z62h83yp3mkk7m9fm9bfs87yzzlcpg-jdk8-jdk8u272-b10.tar.gz.drv
  /nix/store/glfrj2i1qhvyfp304143fdw3nsbsnv34-hotspot-jdk8u272-b10.tar.gz.drv
  /nix/store/j2vkxj2cahyzp59f3ncrfcgb2jn0ilw3-jaxp-jdk8u272-b10.tar.gz.drv
  /nix/store/l6gzcwsyfyf3ifldwwp39lbkcbcb46gl-jdk-jdk8u272-b10.tar.gz.drv
  /nix/store/x524b90plb2xb53x2dhybx544kwlksw1-corba-jdk8u272-b10.tar.gz.drv
  /nix/store/zp3yg0zh23xyd5812bcg99r60ir6vq1i-jaxws-jdk8u272-b10.tar.gz.drv
  /nix/store/y4dzxwpbvrpxmha8c5s410bfcdxy4d38-openjdk-8u272-b10.drv
these paths will be fetched (12.81 MiB download, 65.81 MiB unpacked):
  /nix/store/0akp8l3g38y6fzsqxxx9jqqvclv8hfns-polkit-0.116
  /nix/store/0y4msdbr0jvh80mkqkzizh5x0bgxr9cv-gconf-3.2.6
  /nix/store/127mdqmhbk6vfwifv9d39qhmzqs6v67m-gconf-3.2.6-dev
  /nix/store/2yn3nirw4m5x64lffy3pym0hi46srjxy-gnome-vfs-2.24.4
  /nix/store/6mh6c9rpgwkdcn5rd5jhjyr6r32amqns-dbus-glib-0.110
  /nix/store/am74srim458p8sda7n9xv7jxkpi7xra8-avahi-0.7
  /nix/store/f7q34r7m00xpiq297h9sinzpxcmmz16k-gtk+-2.24.32
  /nix/store/f7xpkxhkn06yi09zd8a3wj1k915i4nmd-dbus-1.12.20
  /nix/store/ijc9zwh7sssj2ilyspb58dhis8l5fzns-dbus-glib-0.110-dev
  /nix/store/l69c28a72h6cbb8w6g59z0kwsy44qdx8-libusb-1.0.23
  /nix/store/mym96ssf5sfyxk9pi55g5mka5j9yb62j-cups-2.3.3-lib
  /nix/store/mzgf336rfc63nczc8kgzvzgwgvzykj4i-gnome-vfs-2.24.4-dev
  /nix/store/pl1fpshm5mprh74266s1wnakah7kv695-cups-2.3.3
  /nix/store/rpf6fxqpiqciiiwr0ih18bammbzc48a6-cups-2.3.3-dev
  /nix/store/rxapwgz2hrj5y7d8wrqkhw6l7446f23c-dbus-1.12.20-lib
  /nix/store/wmr1qk9dbmjp0pwrys6gx5fn32mgjgid-dbus-1.12.20-dev
  /nix/store/wpaz3f5h69k7yai6i9kqcljqkgnfsdv9-gtk+-2.24.32-dev
building '/nix/store/x524b90plb2xb53x2dhybx544kwlksw1-corba-jdk8u272-b10.tar.gz.drv'...
building '/nix/store/glfrj2i1qhvyfp304143fdw3nsbsnv34-hotspot-jdk8u272-b10.tar.gz.drv'...
building '/nix/store/j2vkxj2cahyzp59f3ncrfcgb2jn0ilw3-jaxp-jdk8u272-b10.tar.gz.drv'...
building '/nix/store/zp3yg0zh23xyd5812bcg99r60ir6vq1i-jaxws-jdk8u272-b10.tar.gz.drv'...

trying https://hg.openjdk.java.net/jdk8u/jdk8u/corba/archive/jdk8u272-b10.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

trying https://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/archive/jdk8u272-b10.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

trying https://hg.openjdk.java.net/jdk8u/jdk8u/jaxp/archive/jdk8u272-b10.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

trying https://hg.openjdk.java.net/jdk8u/jdk8u/jaxws/archive/jdk8u272-b10.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1356k    0 1356k    0     0   263k      0 --:--:--  0:00:05 --:--:--  274k
hash mismatch in fixed-output derivation '/nix/store/b6kgljakabikdc1br0ya5dh5q7qksc0w-corba-jdk8u272-b10.tar.gz':
  wanted: sha256:163dganpd2g67ia3lcf05dvq2jpvz4brhmv2mvr1yg449kb9h01j
  got:    sha256:1jfj8gf2yfkj2dwbvnx5wj100yl36w984dhva3vqsfp53qfj0q6g
cannot build derivation '/nix/store/y4dzxwpbvrpxmha8c5s410bfcdxy4d38-openjdk-8u272-b10.drv': 1 dependencies couldn't be built
error: build of '/nix/store/y4dzxwpbvrpxmha8c5s410bfcdxy4d38-openjdk-8u272-b10.drv' failed

What’s strange about this is that the hash 163dganpd2g67ia3lcf05dvq2jpvz4brhmv2mvr1yg449kb9h01j doesn’t occur anywhere in nixpkgs and doesn’t match the specified hash of corba:

What’s even stranger is that the derivation corresponds to what’s specified in its corresponding Nix expression:

$ nix show-derivation /nix/store/x524b90plb2xb53x2dhybx544kwlksw1-corba-jdk8u272-b10.tar.gz.drv
{
  "/nix/store/x524b90plb2xb53x2dhybx544kwlksw1-corba-jdk8u272-b10.tar.gz.drv": {
    "outputs": {
      "out": {
        "path": "/nix/store/pfa0baxqn3hi8ddwfh2bwjhm1qni1zn3-corba-jdk8u272-b10.tar.gz",
        "hashAlgo": "sha256",
        "hash": "320098d64c843c1ff2ae62579817f9fb4a81772bc0313a543ce68976ad7a6d98"
      }
    },
    ...
    "env": {
      ...
      "out": "/nix/store/pfa0baxqn3hi8ddwfh2bwjhm1qni1zn3-corba-jdk8u272-b10.tar.gz",
      "outputHash": "320098d64c843c1ff2ae62579817f9fb4a81772bc0313a543ce68976ad7a6d98",
      "outputHashAlgo": "sha256",
      "outputHashMode": "flat",
      ...
    }
  }
}

But building it fails as before:

$ nix-build /nix/store/x524b90plb2xb53x2dhybx544kwlksw1-corba-jdk8u272-b10.tar.gz.drv
these derivations will be built:
  /nix/store/x524b90plb2xb53x2dhybx544kwlksw1-corba-jdk8u272-b10.tar.gz.drv
building '/nix/store/x524b90plb2xb53x2dhybx544kwlksw1-corba-jdk8u272-b10.tar.gz.drv'...

trying https://hg.openjdk.java.net/jdk8u/jdk8u/corba/archive/jdk8u272-b10.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1356k    0 1356k    0     0   318k      0 --:--:--  0:00:04 --:--:--  318k
hash mismatch in fixed-output derivation '/nix/store/b6kgljakabikdc1br0ya5dh5q7qksc0w-corba-jdk8u272-b10.tar.gz':
  wanted: sha256:163dganpd2g67ia3lcf05dvq2jpvz4brhmv2mvr1yg449kb9h01j
  got:    sha256:1jfj8gf2yfkj2dwbvnx5wj100yl36w984dhva3vqsfp53qfj0q6g
error: build of '/nix/store/x524b90plb2xb53x2dhybx544kwlksw1-corba-jdk8u272-b10.tar.gz.drv' failed

What’s going on here?

The PR is for this version bump is here

@asbachb

Not sure whats going , or how those hashes are being generated?

I’d be interested to know what going on here.

I’d use that script to generate the hashes.

URL_CISC="https://hg.openjdk.java.net/jdk8u/jdk8u"
URL_RISC="https://hg.openjdk.java.net/aarch64-port/jdk8u-shenandoah"

FILE_SUFFIX_CISC="jdk8u272-b10"
FILE_SUFFIX_RISC="aarch64-shenandoah-$FILE_SUFFIX_CISC"
 
modules=(
	"langtools"
	"hotspot"
	"corba"
	"jdk"
	"jaxws"
	"jaxp"
	"nashorn"
)

URL="${URL_CISC}/archive/${FILE_SUFFIX_CISC}.tar.gz"
echo -e "jdk8: \n\tURL: $URL\n\tSHA256: `curl -s $URL | sha256sum`"
for module in "${modules[@]}"; do
	URL="${URL_CISC}/$module/archive/${FILE_SUFFIX_CISC}.tar.gz"
	echo -e "$module: \n\tURL: $URL\n\tSHA256: `curl -s $URL | sha256sum`"
done

URL="${URL_RISC}/archive/${FILE_SUFFIX_RISC}.tar.gz"
echo -e "jdk8: \n\tURL: $URL\n\tSHA256: `curl -s $URL | sha256sum`"
for module in "${modules[@]}"; do
	URL="${URL_RISC}/$module/archive/${FILE_SUFFIX_RISC}.tar.gz"
	echo -e "$module: \n\tURL: $URL\n\tSHA256: `curl -s $URL | sha256sum`"
done% 
1 Like

Maybe that mercurial repository is generating the tarballs on the fly since I’m also not able to reproduce that hashes with my script.

@asbachb what about: https://github.com/NixOS/nixpkgs/pull/103171?

Are you sure this helps in case the tarball changes in different point of times?

those hashes in the commit still don’t match what the script is returning?! a mystery.

It helps a bit since we now have a script in there that somebody can run to generate the new hashes. Better than doing it manually.

I’d love to get answer to the original question…that the hashes that nix was searching for were not in nixpkgs… is this some sort of magic i don’t understand yet?

Answer to that Q: there’s no magic. Nix supports either base16, base32, or base64 encoding of hashes, which all represent the same value.

$ nix-hash --type sha256 --to-base16 163dganpd2g67ia3lcf05dvq2jpvz4brhmv2mvr1yg449kb9h01j 
320098d64c843c1ff2ae62579817f9fb4a81772bc0313a543ce68976ad7a6d98

$ git grep 320098d64c843c1ff2ae62579817f9fb4a81772bc0313a543ce68976ad7a6d98
pkgs/development/compilers/openjdk/8.nix:51:                      else "320098d64c843c1ff2ae62579817f9fb4a81772bc0313a543ce68976ad7a6d98";

In the src the author put the base16 hash, presumably because they’re matching upstream’s published hash. When it fails, nix is printing the base32 hash by default.

It appears this mercurial server is generating non-reproducible tarballs on the fly. Is there a stable alternative we could use for JDKs instead?

Most OpenJDK projects moved to GitHub (beside Java 8 Updates). So maybe switching to GitHub as source solves this problem. (see https://github.com/NixOS/nixpkgs/pull/102418)

DOH! … i should of spotted that, slow brain day.

okay, this makes sense… and now i know , i know.

It maybe a good idea for nix to support the IPFS multi-hashes as some point, rather than depend on the bit length of the hash to infer the encoding.

The great thing about standard encoding, there’s so many to choose from.

maybe nixpkgs has to standardise that all nix fetchers should be encoded with the format that

sha265 myfile

just to avoid infinite confusion to the unenlightened… or make the road to enlightenment eaiser.

i’m sure there is a massive historic reason why…

lets hope multihashes / multibases get thought about, with the upcoming IPFS fetchers…