Nix wants to build everything from sources with the same nixpkgs. How to diagnose?

I have multi-user setup of nix on RHEL7 and pinned nixpkgs in my .nix file that creates an environment of a large number of packages (python, many python packages, texlive, etc.). At some point trying to build the environment I noticed that nix-build all of a sudden wants to rebuild everything from scratch (libc, gcc, python, jdk, etc.) which takes days on the small VM I am using. I never ran GC so it doesn’t even need to go to the binary cache to get the compiled libc and gcc. The only change I know of is adding one more python package to the script.

How do I diagnose why nix-build would want to start building the whole world from scratch? I see that it builds, say libc, with a different hash than before. I tried comparing the drv files but quickly got lost. Is there anything more basic than libc which would help me pinpoint the reason for the full rebuild?

nix-diff is the best tool I’ve found to diagnose this.

The way I use it is like:

$ nix run nixpkgs.nix-diff -c nix-diff $(nix-instantiate ./old-nixpkgs -A my-attr) $(nix-instantiate ./new-nixpkgs -A my-attr)
1 Like

Thanks! My nixpkgs is the same in both cases. I ran it on the two libc derivations and basically got these few lines repeated for many inputs:

  • The arguments do not match
      • -e
      - /nix/store/
      + /nix/store/
  • The input named `bootstrap-stage2-gcc-wrapper-` differs
    • These two derivations have already been compared
  • The input named `bootstrap-stage2-stdenv-linux` differs
    - /nix/store/9p9lv9fasriagylsja0qzw4ldg38jpci-bootstrap-stage2-stdenv-linux.drv:{out}
    + /nix/store/xi2fplqc1w7pgjligkld743paaywa3kz-bootstrap-stage2-stdenv-linux.drv:{out}
    • The builders do not match
        - /nix/store/n9acaakxahkv1q3av11l93p7rgd4xqsf-bootstrap-tools/bin/bash
        + /nix/store/qgf0p7p4in4pw0xl1grgxdnkd57p3l5l-bootstrap-tools/bin/bash
    • The arguments do not match
        • -e
        - /nix/store/
        + /nix/store/

I have compared the pairs of shell scripts and they are all identical but the permissions differ, for instance

$ ls -l /nix/store/*
-rwxrwxr-x 1 root root 2055 Dec 31  1969 /nix/store/
-r-xr-xr-x 2 root root 2055 Dec 31  1969 /nix/store/

$ ls -l /nix/store/*
-r-xr-xr-x 2 root root 34 Dec 31  1969 /nix/store/
-rwxrwxr-x 1 root root 34 Dec 31  1969 /nix/store/

Is that suspicious?

I don’t think that’s an issue - that won’t go into the drv hash. Although it is a little weird.

There’s a few things that go into stage2 that you may have accidentally changed the Nix expressions for. For instance, perl, gcc, bison, glibc all go into that.

Wait what… Nix isn’t supposed to allow the write permission. Everything under /nix/store should be 555 or 444.