Why is it rebuilding a derivation whose output path is present?

nomeata · May 28, 2021, 1:18pm

I wonder if something is broken about my nix installation, but this looks odd:

This derivation has been built, it seem:

~ $ nix-build /nix/store/fw1rn6nh4mydpsg4zvb5wq4ylbfivcwr-pcre-8.44.drv
/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44

but building something that depends on it seems to try to build it:

~ $ nix-build /nix/store/nfqilgw5sv0dsv7m0wvgh0jqi65dc520-pcre-light-0.4.1.0.drv
these derivations will be built:
  /nix/store/fw1rn6nh4mydpsg4zvb5wq4ylbfivcwr-pcre-8.44.drv
  /nix/store/nfqilgw5sv0dsv7m0wvgh0jqi65dc520-pcre-light-0.4.1.0.drv
these paths will be fetched (143.30 MiB download, 1409.56 MiB unpacked):
  /nix/store/0qpfsz8xj1g2fbw28k1049pxci0jmmac-gnugrep-3.4
  /nix/store/6xxam2hm83h5hbihm6ppxyvssi86wjw0-call-stack-0.1.0
  /nix/store/l0vn78f1q59q8j41qcnfhygwpk65fw7i-HUnit-1.6.0.0
  /nix/store/s7kbnrgsf495zg5gj2ka8r23sga19kzn-gcc-wrapper-9.2.0
  /nix/store/wx2xwdxjc1zvb2vlbng98k8g6bqa6z5j-ghc-8.6.5
  /nix/store/xylrsrjwnhmr6pgz2c7zzj5nl41qps8a-stdenv-linux
…

and at the end it fails in a weird way, I guess because install tries to override what’s already there:

copying path '/nix/store/0qpfsz8xj1g2fbw28k1049pxci0jmmac-gnugrep-3.4' from 'https://kaleidogen.cachix.org'...
building '/nix/store/fw1rn6nh4mydpsg4zvb5wq4ylbfivcwr-pcre-8.44.drv'...
unpacking sources
unpacking source archive /nix/store/zpf0j0qqw8bhm1knfr1x29zzq8pc85wj-pcre-8.44.tar.bz2
source root is pcre-8.44
setting SOURCE_DATE_EPOCH to timestamp 1581528922 of file pcre-8.44/config.h.generic
…
installing
install flags: SHELL=/nix/store/5b7a4ynm23gm375q1fzbsh4sqc806864-bootstrap-tools/bin/bash pkgconfigdir=/nix/store/nnsdf11zg16bq5b6kf2q75i63c7qdh6n-pcre-8.44-dev/lib/pkgconfig m4datadir=/nix/store/nnsdf11zg16bq5b6kf2q75i63c7qdh6n-pcre-8.44-dev/share/aclocal aclocaldir=/nix/store/nnsdf11zg16bq5b6kf2q75i63c7qdh6n-pcre-8.44-dev/share/aclocal install
make  install-am
make[1]: Entering directory '/tmp/nix-build-pcre-8.44.drv-0/pcre-8.44'
make[2]: Entering directory '/tmp/nix-build-pcre-8.44.drv-0/pcre-8.44'
 /nix/store/5b7a4ynm23gm375q1fzbsh4sqc806864-bootstrap-tools/bin/mkdir -p '/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44/lib'
 /nix/store/5b7a4ynm23gm375q1fzbsh4sqc806864-bootstrap-tools/bin/bash ./libtool   --mode=install /nix/store/5b7a4ynm23gm375q1fzbsh4sqc806864-bootstrap-tools/bin/install -c   libpcre.la libpcreposix.la '/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44/lib'
libtool: install: /nix/store/5b7a4ynm23gm375q1fzbsh4sqc806864-bootstrap-tools/bin/install -c .libs/libpcre.so.1.2.12 /nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44/lib/libpcre.so.1.2.12
install: cannot remove '/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44/lib/libpcre.so.1.2.12': Permission denied
make[2]: *** [Makefile:1534: install-libLTLIBRARIES] Error 1
make[2]: Leaving directory '/tmp/nix-build-pcre-8.44.drv-0/pcre-8.44'
make[1]: *** [Makefile:3066: install-am] Error 2
make[1]: Leaving directory '/tmp/nix-build-pcre-8.44.drv-0/pcre-8.44'
make: *** [Makefile:3060: install] Error 2
builder for '/nix/store/fw1rn6nh4mydpsg4zvb5wq4ylbfivcwr-pcre-8.44.drv' failed with exit code 2
cannot build derivation '/nix/store/nfqilgw5sv0dsv7m0wvgh0jqi65dc520-pcre-light-0.4.1.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/nfqilgw5sv0dsv7m0wvgh0jqi65dc520-pcre-light-0.4.1.0.drv' failed

How would that happen?

nomeata · May 28, 2021, 1:35pm

I completely wiped my installation, and started the build (via hercules-ci, not locally), and I spotted this in the logs. Not sure if it is related.

while retrieving dependencies: CppStdException "Exception: path '/nix/store/nnsdf11zg16bq5b6kf2q75i63c7qdh6n-pcre-8.44-dev' does not exist and cannot be created; type: nix::Error"
unable to retrieve dependency; attempting fallback to local build

Anyways, even after wiping /nix I get

copying path '/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44' from 'https://kaleidogen.cachix.org'
…

and yet it tries to build that path:

libtool: install: /nix/store/5b7a4ynm23gm375q1fzbsh4sqc806864-bootstrap-tools/bin/install -c .libs/libpcre.so.1.2.12 /nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44/lib/libpcre.so.1.2.12
install: cannot remove '/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44/lib/libpcre.so.1.2.12': Permission denied

Maybe the cachix cache is corrupt somehow?

nomeata · May 28, 2021, 1:51pm

Ah, it seems that somehow the cachix cache has one output of that derivation, but not all:

~ $ nix show-derivation /nix/store/fw1rn6nh4mydpsg4zvb5wq4ylbfivcwr-pcre-8.44.drv
{
  "/nix/store/fw1rn6nh4mydpsg4zvb5wq4ylbfivcwr-pcre-8.44.drv": {
    "outputs": {
      "bin": {
        "path": "/nix/store/mds1n66wamkh4gg650m08knf5jbyrdsd-pcre-8.44-bin"
      },
      "dev": {
        "path": "/nix/store/nnsdf11zg16bq5b6kf2q75i63c7qdh6n-pcre-8.44-dev"
      },
      "doc": {
        "path": "/nix/store/fk0lydjnqlhhl5p0zs4ji4s1nw62wlwb-pcre-8.44-doc"
      },
      "man": {
        "path": "/nix/store/psigq4y7yx3ck5py6qmw95rfn6nw3vb8-pcre-8.44-man"
      },
      "out": {
        "path": "/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44"
      }
    },
…

~ $ nix-store --delete --ignore-liveness /nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44
finding garbage collector roots...
deleting '/nix/store/0qpfsz8xj1g2fbw28k1049pxci0jmmac-gnugrep-3.4'
deleting '/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44'
deleting '/nix/store/trash'
deleting unused links...
note: currently hard linking saves -0.00 MiB
2 store paths deleted, 0.72 MiB freed
~ $ nix-store --realize /nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44
these paths will be fetched (0.16 MiB download, 0.47 MiB unpacked):
  /nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44
copying path '/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44' from 'https://kaleidogen.cachix.org'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44
~ $ nix-store --realize /nix/store/nnsdf11zg16bq5b6kf2q75i63c7qdh6n-pcre-8.44-dev
don't know how to build these paths:
  /nix/store/nnsdf11zg16bq5b6kf2q75i63c7qdh6n-pcre-8.44-dev

Is that something that should happen?

Looks like I could dig myself out of this by

~ $ nix-store --delete --ignore-liveness /nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44

then disabling the affected cachix cache, and then running

~ $ nix-build /nix/store/fw1rn6nh4mydpsg4zvb5wq4ylbfivcwr-pcre-8.44.drv

but I wonder if this is something that can be prevented completely?

cfhammill · March 16, 2022, 3:07pm

I encountered a very similar issue, with the recent file size issue with cudatoolkit (builds using specific .nar file from cache.nixos.org fail with 503 · Issue #207 · NixOS/infra · GitHub), I tried disabling substituters, it for some reason tried to rebuild a bunch of c libraries that were already in the nix store, the whole thing fails trying to build libunistring when it attempts to copy to the store.

libtool: install: /nix/store/p4s4jf7aq6v6z9iazll1aiqwb34aqxq9-bootstrap-tools/bin/install -c .libs/libunistring.so.2.1.0 /nix/store/8ckxc8biqqfdwyhr0w70jgrcb4h7a4y5-libunistring-0.9.10/lib/libunistring.so.2.1.0
       > /nix/store/p4s4jf7aq6v6z9iazll1aiqwb34aqxq9-bootstrap-tools/bin/install: cannot remove '/nix/store/8ckxc8biqqfdwyhr0w70jgrcb4h7a4y5-libunistring-0.9.10/lib/libunistring.so.2.1.0': Permission denied
       > make[3]: *** [Makefile:3620: install-libLTLIBRARIES] Error 1
       > make[3]: Leaving directory '/tmp/nix-build-libunistring-0.9.10.drv-0/libunistring-0.9.10/lib'
       > make[2]: *** [Makefile:4422: install-am] Error 2
       > make[2]: Leaving directory '/tmp/nix-build-libunistring-0.9.10.drv-0/libunistring-0.9.10/lib'
       > make[1]: *** [Makefile:4416: install] Error 2
       > make[1]: Leaving directory '/tmp/nix-build-libunistring-0.9.10.drv-0/libunistring-0.9.10/lib'
       > make: *** [Makefile:1559: install-recursive] Error 1
       For full logs, run 'nix log /nix/store/nq5zrwpzxs20qvl54ks3frj14qhfalqp-libunistring-0.9.10.drv'

The command

nix develop
claims to try to fetch/build ~300 packages

nix --substituters "" develop

is trying to install 900 packages. I suspect libunistring is just the first of many packages that would have this issue. I also can’t remove the file from the store even with ignore-liveness because apparently it has other referents.

I’m only using cache.nixos.org, so this is with the default binary cache.

vcunat · March 16, 2022, 4:46pm

The build probably needed libunistring.dev or something, so getting that from cache should work around it. Overall this is a bug in Nix; I think I’ve already seen it, maybe it’s even long fixed if you’re still using Nix 2.3.x.

cfhammill · March 16, 2022, 6:14pm

Thanks! I didn’t even realize there was a dev attr. I upgraded to 2.7 yesterday hoping it would fix things but to no avail. I tried getting libunistring.dev, now it’s failing to build curl - I was suspecting this was the first of many of this type of issue.