Nix copying a store-path into the store

Development Nix
1

I noticed some curious behavior of nix run after rebuilding the system from a flake. Specifically it spends some significant amount of time copying a store-path into the store (which from my understanding should result in the same store-path as its essentially a fixed input derivation). The store-path in question is nixpkgs’s source. Either i am mistaken about how the input hash is actually calculated or this is simply a bug. Either way it’s quite expensive to copy 400ish MiB after every rebuild so ideally we would avoid doing so.

Here is a screenshot of what i mean specifically:

462
462694×56 8.18 KB

EDIT: probably related to nix copies inputs already in the nix store · Issue #11228 · NixOS/nix · GitHub

5 Likes
2

When using NixOS with flakes, this step goes away because nixpkgs# will be always in store:

This is what the result looks like:

 jq < /etc/nix/registry.json
{
  "flakes": [
    {
      "exact": true,
      "from": {
        "id": "nixpkgs",
        "type": "indirect"
      },
      "to": {
        "path": "/nix/store/i9dsnmsjhs9rbagb9b1z3mp66q6v67pz-source",
        "type": "path"
      }
    },
    {
      "exact": true,
      "from": {
        "id": "self",
        "type": "indirect"
      },
      "to": {
        "path": "/nix/store/8wxyl9xi0vpv5ygji5sbf1jx1r64h3gi-source",
        "type": "path"
      }
    }
  ],
  "version": 2
}
3

Apologies in advance for dumping a lot here. This bothered me a bit too much after coming across it again this morning, and after finding a lot of somewhat related-ish issues, new and old, I found this thread and saw you had recently responded. I hope this is an alright place :slight_smile:

I still experience this behavior on Nix 2.24.12 (and I believe onwards, but I haven’t had the time to confirm yet) with that setup

Specifically, here’s the relevant section of my /etc/nix/registry.json:

    {
      "exact": true,
      "from": {
        "id": "nixpkgs",
        "type": "indirect"
      },
      "to": {
        "path": "/nix/store/b19xi89pgm1arig3l9xqn1h81zjd1j91-source",
        "type": "path"
      }
    },

And when running nix eval:

image
image781×57 13.7 KB

This also ends up affecting the default $NIX_PATH set by the same module

image
image772×58 13.1 KB

I believe this, nix copies inputs already in the nix store · Issue #11228 · NixOS/nix · GitHub, and maybe some of nix will pointlessly re-fetch registry entries if they are defined as store paths · Issue #7075 · NixOS/nix · GitHub are signs of an underlying issue that would be the culprit of what I came across in nixos/nixpkgs-flake: enable flakes in systems built from flakes, make ISO behave well by lf- · Pull Request #374460 · NixOS/nixpkgs · GitHub. It appears to have also worsened since Nix 2.18, with the source path names workaround no longer functioning (I believe it still works in Lix)

My current workaround for this copying when using NIX_PATH lookup paths (like <nixpkgs>) is symlinking my Flake inputs to the top-level of my system derivation like so

{
  lib,
  pkgs,
  inputs,
  ...
}:

let
  flakeInputs = pkgs.linkFarm "flake-inputs" (
    lib.mapAttrs (lib.const (flake: flake.outPath)) inputs
  );
in

{
  nix = {
    nixPath = lib.mkForce (
      lib.mapAttrsToList (name: lib.const "${name}=/run/current-system/inputs/${name}") inputs
    );
  };

  system.extraSystemBuilderCmds = ''
    ln -s ${flakeInputs} $out/inputs
  '';
}

(Taken from my full config. There are no workarounds for Flakes being copied since they are seemingly always copied at least once)

This doesn’t completely solve the problem though, as even when using files (from $NIX_PATH lookup paths, plain store paths, anything that’s not a Flake), this behavior will still show up in nix develop (and AFAICT, only nix develop):

image
image917×79 27.8 KB

I’m not sure how recent this regression is, but I’m assuming it’s also from Nix 2.19+ as it doesn’t occur in Lix (I am able to confirm this on Lix 2.91.1 and 2.92)

I also want to CC @jade in case they’re interested and have the time, as this is fairly related to the issues we were discussing in the aforementioned module PR. Hopefully we can apply some workarounds on the NixOS side so people are less likely to come across these issues until they’re fixed upstream

4

Mhm. Not sure how to reproduce this:

All these operations are fast for me:

nix-shell-env % nix eval nixpkgs#hello
«derivation /nix/store/4g4pbpxsxqbmzksbwq2jrnvls7f03k6v-hello-2.12.1.drv»

~/git/nixos-infra/non-critical-infra main
nix-shell-env % which nix
nix: aliased to noglob nix

~/git/nixos-infra/non-critical-infra main
nix-shell-env % real-which nix
/nix/store/m3xhfza117560y4zif2ziwr1787bamx6-nix-2.27.0pre20250307_749ffbe/bin/nix

~/git/nixos-infra/non-critical-infra main
nix-shell-env % nix-shell -p nix

[nix-shell:~/git/nixos-infra/non-critical-infra]$ nix eval nixpkgs#hello
«derivation /nix/store/4g4pbpxsxqbmzksbwq2jrnvls7f03k6v-hello-2.12.1.drv»

[nix-shell:~/git/nixos-infra/non-critical-infra]$ nix --version
nix (Nix) 2.24.12

I can see a copy when doing --eval-store /tmp/foo but this is expected because it has to copy nixpkgs from /nix/store to /tmp/foo

[nix-shell:~/git/nixos-infra/non-critical-infra]$ time nix eval --eval-store /tmp/foo nixpkgs#hello
«derivation /nix/store/4g4pbpxsxqbmzksbwq2jrnvls7f03k6v-hello-2.12.1.drv»

real    0m5.947s
user    0m2.076s
sys     0m3.848s
5

I’ve found this difficult to reproduce as well, since it seems the issue just won’t occur after it’s been copied once. Usually it’ll happen the first time I run a command after switching to an updated configuration (specifically not running nix flake update to download and hash the inputs locally. Just pulling and nixos-rebuild switch) or when using any inputs pinned in the registry (in the same way as the NixOS module) that my system doesn’t use directly

The example I gave above falls under the latter case, where my atlas system is built against my nixpkgs-stable input, but it’s nixpkgs registry entry still points to the root nixpkgs input. Likewise, my personal systems that are built against nixpkgs will have the same issue when evaluating the nixpkgs-stable registry entry

image
image789×101 36.6 KB

EDIT: This time it decided to also trigger with the base nixpkgs registry entry. Probably because I have stopped using Flakes in CLI due to this issue, so this is the first time I’m using it?

image
image771×74 28.8 KB

Anyways, relevant registry.json section

    {
      "exact": true,
      "from": {
        "id": "nixpkgs",
        "type": "indirect"
      },
      "to": {
        "path": "/nix/store/jk6xpbfh10gz6q5cqw8b2f7xk0pl7hkv-source",
        "type": "path"
      }
    },
    {
      "exact": true,
      "from": {
        "id": "nixpkgs-stable",
        "type": "indirect"
      },
      "to": {
        "path": "/nix/store/070ny7v4l1lbwx44ac3bsagzwx9mjdcy-source",
        "type": "path"
      }
    },

Usually I would think to just delete the copy it creates so it would think to create a new one…but I have no idea where it’s copying an existing store path to

EDIT: Here’s exactly how I link my Flake inputs to the registry for reference

6

Well path type is probably wrong, it has different effects from e.g. git type inputs. Sharing the flake reference would be safer.

7

If it only happens when not running nix flake update than this might be expected to be slower? But I am also unsure how you can reproduce this path input if you don’t copy it into the nix store with nixos-rebuild. Copying from a nix store path to the same path also doesn’t seem to make any sense to me. This would only happen if for some reason nix no longer has this path marked as a valid path?

8

For nixpkgs it should be fine. At least I haven’t seen any downsides.

9

fyi the caches involved are either in /root/.cache/nix or in ~/.cache/nix and given it’s flakes bugs in particular it’s probably the latter since it’s likely client side.

there’s a setup you can do with NIX_DAEMON_SOCKET_PATH on both daemon and client, --store somepath on the daemon and --store daemon on the client in which you may be able to reproduce it with a fresh nix-daemon running with a fresh store. both of those settings need absolute paths. maybe you can repro with that.

though, i would also argue that hunting down a bug in cppnix is a waste of time if you can simply use lix and not worry about it (i am, however, biased), since it’s unlikely to be fixed in a timely manner unless you have a patch for it. the flake registry stuff we fixed immediately on lix’s release remains unfixed to this day aiui, for example.

semi-joking: the biggest nix ux improvement that could be made by NixOS on this issue is replacing nix with lix by default but that is politically sensitive even if you had every technical and project management justification in the world.

10

fwiw, i’ve attempted to reproduce this issue during the past few hours and found out (via these lines) that it’s pretty much consistently reproducible by doing the following:

# remove fetcher cache
rm -v ~/.cache/nix/fetcher-cache-*
# evaluate something
nix eval nixpkgs#hello
# observe "copying '/nix/store/...-source' to the store"

i’ve noticed 2.18 doesn’t have the “copying” line but 2.19 does, so i decided to do the stupid thing of bisecting it instead of just looking at the commits. with the following rudimentary bisect script, git found ea38605 which added the line.

bisect script 
#!/usr/bin/env bash
set -euxo pipefail

nom build . || exit 125
nix run . -- eval --raw n#hello -vvvv --debug |& (! rg "copying '/nix/store.*' to the store...")
# ^ this nix is 2.18 in a `nix shell`.

i don’t think this changed anything drastically, because the eval still completes under 3 seconds.

digging (bisecting) further, i found this pr that seems to be the culprit of the slow eval. (builds fail for some commits due to failing tests, so those are skipped)

bisect script 
#!/usr/bin/env bash
set -euxo pipefail

rm -vf ~/.cache/nix/*.sqlite*
nom build . || exit 125
timeout 3 nix run . -- eval --raw n#hello -vv

i believe Nix#11701 is supposed to remediate this issue, but this is still present on master. haven’t looked into it much further since it’s late.

extra stuff
5 Likes
11

I was having this issue recently and it turned out to be this

Mine is of course a home-manager setup so it’ll be different, but the bottom line was an instance of nixpkgs in nix-path without content hash (not sure if this is the correct reason, but anytime I have a source without narHash in a flake lock it copies it to store) . My flake:nixpkgs is also setup here

I’d recommend checking what is being added to your nix-path and how its being added and ensure its either a flake or something with content hash, example from my registry.json

> cat ~/.config/nix/registry.json
{
  "flakes": [
    {
      "exact": true,
      "from": {
        "id": "nixpkgs",
        "type": "indirect"
      },
      "to": {
        "lastModified": 1717179513,
        "narHash": "sha256-vboIEwIQojofItm2xGCdZCzW96U85l9nDW3ifMuAIdM=",
        "path": "/nix/store/wzx1ba5hqqfa23vfrvqmfmkpj25p37mr-source",
        "rev": "63dacb46bf939521bdc93981b4cbb7ecb58427a0",
        "type": "path"
      }
    }
  ],
  "version": 2
}
12

Using flake over to does seem to fix it. Seems it would line up with what (I’m pretty sure is) the code path would want

Still not sure exactly what causes this though, as the PR mentioned here

digging (bisecting) further, i found this pr that seems to be the culprit of the slow eval. (builds fail for some commits due to failing tests, so those are skipped)

Only landed in Nix 2.25, but this happens in Nix 2.24 as well. Lix has a nearly identical condition here, so I’m guessing this may be more to do with caching. Enabling debug logs (-vvvvvv) seems to confirm this:

In Nix 2.24.12:

copying '/nix/store/b19xi89pgm1arig3l9xqn1h81zjd1j91-source'...
performing daemon worker op: 11
performing daemon worker op: 1
performing daemon worker op: 26
did not find cache entry for 'fetchToStore:{"fingerprint":"path:93b552aafbf4af5af9de7508fc87cf1c2a6991401d75e5e7f38dd89400b90d1d:","method":"nar","name":"source","path":"/","store":"/nix/store"}'
copying '/nix/store/b19xi89pgm1arig3l9xqn1h81zjd1j91-source/' to the store...

In Lix 2.91.1:

copying '/nix/store/b19xi89pgm1arig3l9xqn1h81zjd1j91-source'...
performing daemon worker op: 11
acquiring write lock on '/nix/var/nix/temproots/394424'
performing daemon worker op: 1
performing daemon worker op: 26
got tree '/nix/store/b19xi89pgm1arig3l9xqn1h81zjd1j91-source' from 'path:/nix/store/b19xi89pgm1arig3l9xqn1h81zjd1j91-source?lastModified=0&narHash=sha256-k7VSqvv0r1r53nUI/IfPHCppkUAddeXn843YlAC5DR0%3D'

Lix appears to somehow be getting the narHash of a store path without it being explicitly passed, while Nix can’t?

In any case, I’ve opened nixos/nixpkgs-flake: pass full flake to registry if possible by getchoo · Pull Request #388090 · NixOS/nixpkgs · GitHub to adopt this behavior by default in NixOS

13

Either it’s asking the database or it didn’t need the hash in the first place and whatever cppnix is doing is busted. I’m not sure which. The hash that appears in the store path is a derivative of the narHash but this mapping is not injective: the last few bits are xor’d to produce a 160 bit store path name from the 256 bit hash, so you cannot get the hash from a store path without external info.

1 Like