Why is it rebuilding a derivation whose output path is present?

I wonder if something is broken about my nix installation, but this looks odd:

This derivation has been built, it seem:

~ $ nix-build /nix/store/fw1rn6nh4mydpsg4zvb5wq4ylbfivcwr-pcre-8.44.drv
/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44

but building something that depends on it seems to try to build it:

~ $ nix-build /nix/store/nfqilgw5sv0dsv7m0wvgh0jqi65dc520-pcre-light-0.4.1.0.drv
these derivations will be built:
  /nix/store/fw1rn6nh4mydpsg4zvb5wq4ylbfivcwr-pcre-8.44.drv
  /nix/store/nfqilgw5sv0dsv7m0wvgh0jqi65dc520-pcre-light-0.4.1.0.drv
these paths will be fetched (143.30 MiB download, 1409.56 MiB unpacked):
  /nix/store/0qpfsz8xj1g2fbw28k1049pxci0jmmac-gnugrep-3.4
  /nix/store/6xxam2hm83h5hbihm6ppxyvssi86wjw0-call-stack-0.1.0
  /nix/store/l0vn78f1q59q8j41qcnfhygwpk65fw7i-HUnit-1.6.0.0
  /nix/store/s7kbnrgsf495zg5gj2ka8r23sga19kzn-gcc-wrapper-9.2.0
  /nix/store/wx2xwdxjc1zvb2vlbng98k8g6bqa6z5j-ghc-8.6.5
  /nix/store/xylrsrjwnhmr6pgz2c7zzj5nl41qps8a-stdenv-linux
…

and at the end it fails in a weird way, I guess because install tries to override what’s already there:

copying path '/nix/store/0qpfsz8xj1g2fbw28k1049pxci0jmmac-gnugrep-3.4' from 'https://kaleidogen.cachix.org'...
building '/nix/store/fw1rn6nh4mydpsg4zvb5wq4ylbfivcwr-pcre-8.44.drv'...
unpacking sources
unpacking source archive /nix/store/zpf0j0qqw8bhm1knfr1x29zzq8pc85wj-pcre-8.44.tar.bz2
source root is pcre-8.44
setting SOURCE_DATE_EPOCH to timestamp 1581528922 of file pcre-8.44/config.h.generic
…
installing
install flags: SHELL=/nix/store/5b7a4ynm23gm375q1fzbsh4sqc806864-bootstrap-tools/bin/bash pkgconfigdir=/nix/store/nnsdf11zg16bq5b6kf2q75i63c7qdh6n-pcre-8.44-dev/lib/pkgconfig m4datadir=/nix/store/nnsdf11zg16bq5b6kf2q75i63c7qdh6n-pcre-8.44-dev/share/aclocal aclocaldir=/nix/store/nnsdf11zg16bq5b6kf2q75i63c7qdh6n-pcre-8.44-dev/share/aclocal install
make  install-am
make[1]: Entering directory '/tmp/nix-build-pcre-8.44.drv-0/pcre-8.44'
make[2]: Entering directory '/tmp/nix-build-pcre-8.44.drv-0/pcre-8.44'
 /nix/store/5b7a4ynm23gm375q1fzbsh4sqc806864-bootstrap-tools/bin/mkdir -p '/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44/lib'
 /nix/store/5b7a4ynm23gm375q1fzbsh4sqc806864-bootstrap-tools/bin/bash ./libtool   --mode=install /nix/store/5b7a4ynm23gm375q1fzbsh4sqc806864-bootstrap-tools/bin/install -c   libpcre.la libpcreposix.la '/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44/lib'
libtool: install: /nix/store/5b7a4ynm23gm375q1fzbsh4sqc806864-bootstrap-tools/bin/install -c .libs/libpcre.so.1.2.12 /nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44/lib/libpcre.so.1.2.12
install: cannot remove '/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44/lib/libpcre.so.1.2.12': Permission denied
make[2]: *** [Makefile:1534: install-libLTLIBRARIES] Error 1
make[2]: Leaving directory '/tmp/nix-build-pcre-8.44.drv-0/pcre-8.44'
make[1]: *** [Makefile:3066: install-am] Error 2
make[1]: Leaving directory '/tmp/nix-build-pcre-8.44.drv-0/pcre-8.44'
make: *** [Makefile:3060: install] Error 2
builder for '/nix/store/fw1rn6nh4mydpsg4zvb5wq4ylbfivcwr-pcre-8.44.drv' failed with exit code 2
cannot build derivation '/nix/store/nfqilgw5sv0dsv7m0wvgh0jqi65dc520-pcre-light-0.4.1.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/nfqilgw5sv0dsv7m0wvgh0jqi65dc520-pcre-light-0.4.1.0.drv' failed

How would that happen?

I completely wiped my installation, and started the build (via hercules-ci, not locally), and I spotted this in the logs. Not sure if it is related.

while retrieving dependencies: CppStdException "Exception: path '/nix/store/nnsdf11zg16bq5b6kf2q75i63c7qdh6n-pcre-8.44-dev' does not exist and cannot be created; type: nix::Error"
unable to retrieve dependency; attempting fallback to local build

Anyways, even after wiping /nix I get

copying path '/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44' from 'https://kaleidogen.cachix.org'
…

and yet it tries to build that path:

libtool: install: /nix/store/5b7a4ynm23gm375q1fzbsh4sqc806864-bootstrap-tools/bin/install -c .libs/libpcre.so.1.2.12 /nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44/lib/libpcre.so.1.2.12
install: cannot remove '/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44/lib/libpcre.so.1.2.12': Permission denied

Maybe the cachix cache is corrupt somehow?

Ah, it seems that somehow the cachix cache has one output of that derivation, but not all:

~ $ nix show-derivation /nix/store/fw1rn6nh4mydpsg4zvb5wq4ylbfivcwr-pcre-8.44.drv
{
  "/nix/store/fw1rn6nh4mydpsg4zvb5wq4ylbfivcwr-pcre-8.44.drv": {
    "outputs": {
      "bin": {
        "path": "/nix/store/mds1n66wamkh4gg650m08knf5jbyrdsd-pcre-8.44-bin"
      },
      "dev": {
        "path": "/nix/store/nnsdf11zg16bq5b6kf2q75i63c7qdh6n-pcre-8.44-dev"
      },
      "doc": {
        "path": "/nix/store/fk0lydjnqlhhl5p0zs4ji4s1nw62wlwb-pcre-8.44-doc"
      },
      "man": {
        "path": "/nix/store/psigq4y7yx3ck5py6qmw95rfn6nw3vb8-pcre-8.44-man"
      },
      "out": {
        "path": "/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44"
      }
    },
…
~ $ nix-store --delete --ignore-liveness /nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44
finding garbage collector roots...
deleting '/nix/store/0qpfsz8xj1g2fbw28k1049pxci0jmmac-gnugrep-3.4'
deleting '/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44'
deleting '/nix/store/trash'
deleting unused links...
note: currently hard linking saves -0.00 MiB
2 store paths deleted, 0.72 MiB freed
~ $ nix-store --realize /nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44
these paths will be fetched (0.16 MiB download, 0.47 MiB unpacked):
  /nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44
copying path '/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44' from 'https://kaleidogen.cachix.org'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44
~ $ nix-store --realize /nix/store/nnsdf11zg16bq5b6kf2q75i63c7qdh6n-pcre-8.44-dev
don't know how to build these paths:
  /nix/store/nnsdf11zg16bq5b6kf2q75i63c7qdh6n-pcre-8.44-dev

Is that something that should happen?

Looks like I could dig myself out of this by

~ $ nix-store --delete --ignore-liveness /nix/store/j72c0vfp2lnzsnpai3rf9w207c50cqik-pcre-8.44

then disabling the affected cachix cache, and then running

~ $ nix-build /nix/store/fw1rn6nh4mydpsg4zvb5wq4ylbfivcwr-pcre-8.44.drv

but I wonder if this is something that can be prevented completely?

I encountered a very similar issue, with the recent file size issue with cudatoolkit (builds using specific .nar file from cache.nixos.org fail with 503 · Issue #207 · NixOS/nixos-org-configurations · GitHub), I tried disabling substituters, it for some reason tried to rebuild a bunch of c libraries that were already in the nix store, the whole thing fails trying to build libunistring when it attempts to copy to the store.

libtool: install: /nix/store/p4s4jf7aq6v6z9iazll1aiqwb34aqxq9-bootstrap-tools/bin/install -c .libs/libunistring.so.2.1.0 /nix/store/8ckxc8biqqfdwyhr0w70jgrcb4h7a4y5-libunistring-0.9.10/lib/libunistring.so.2.1.0
       > /nix/store/p4s4jf7aq6v6z9iazll1aiqwb34aqxq9-bootstrap-tools/bin/install: cannot remove '/nix/store/8ckxc8biqqfdwyhr0w70jgrcb4h7a4y5-libunistring-0.9.10/lib/libunistring.so.2.1.0': Permission denied
       > make[3]: *** [Makefile:3620: install-libLTLIBRARIES] Error 1
       > make[3]: Leaving directory '/tmp/nix-build-libunistring-0.9.10.drv-0/libunistring-0.9.10/lib'
       > make[2]: *** [Makefile:4422: install-am] Error 2
       > make[2]: Leaving directory '/tmp/nix-build-libunistring-0.9.10.drv-0/libunistring-0.9.10/lib'
       > make[1]: *** [Makefile:4416: install] Error 2
       > make[1]: Leaving directory '/tmp/nix-build-libunistring-0.9.10.drv-0/libunistring-0.9.10/lib'
       > make: *** [Makefile:1559: install-recursive] Error 1
       For full logs, run 'nix log /nix/store/nq5zrwpzxs20qvl54ks3frj14qhfalqp-libunistring-0.9.10.drv'

The command

nix develop
claims to try to fetch/build ~300 packages

nix --substituters "" develop

is trying to install 900 packages. I suspect libunistring is just the first of many packages that would have this issue. I also can’t remove the file from the store even with ignore-liveness because apparently it has other referents.

I’m only using cache.nixos.org, so this is with the default binary cache.

The build probably needed libunistring.dev or something, so getting that from cache should work around it. Overall this is a bug in Nix; I think I’ve already seen it, maybe it’s even long fixed if you’re still using Nix 2.3.x.

Thanks! I didn’t even realize there was a dev attr. I upgraded to 2.7 yesterday hoping it would fix things but to no avail. I tried getting libunistring.dev, now it’s failing to build curl - I was suspecting this was the first of many of this type of issue.