This snippet should no longer be required on versions of Nix >= 2.24.14. Copying of path type inputs like these have been fixed as of create cache entry for paths already in the nix store by Mic92 · Pull Request #12911 · NixOS/nix · GitHub , and I can’t replicate any kind of issue with them even on the latest Nix versions
…but it seems similar behavior has appeared recently elsewhere:
opened 04:35PM - 31 Jul 24 UTC
bug
fetching
**Describe the bug**
I have this in the nix registry.json
```
{
"exa… ct": true,
"from": {
"id": "nixpkgs",
"type": "indirect"
},
"to": {
"path": "/nix/store/<some path>",
"type": "path"
}
},
```
Running `nix eval 'nixpkgs#hello.meta` produces output `"copying /nix/store/<some path>"` every time.
I notice this does not always happen. It happens if the "path" points to something that is built. In this case, my nixpkgs source is built by calling `applyPatches`. It does not happen if the "path" points to something that is fetched directly. e.g. if I use `inputs.nixpkgs.url = github.com:nixos/nixpkgs`, then points "path" to `inputs.nixpkgs`, then `nix eval` does not copy.
**Expected behavior**
`nix` should not copy if the input is already in the nix store.
**Priorities**
Add :+1: to [issues you find important](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc).
opened 12:42PM - 25 Mar 25 UTC
bug
fetching
## Describe the bug
The output of `fetchTree { type = "file"; narHash = "..."; … url = "..."; }` seems to depend only on the `narHash`, since fetching the same file from different URLs gives the same outPath.
However, unlike a fixed-output derivation, `fetchTree` will try to perform the download (or at least attempts to connect to the URL) even if the outPath already exists. I think this is due to checking a URL-based cache, but not checking whether the store path already exists.
## Steps To Reproduce
Use `fetchTree` with a narHash to fetch a file from a URL, and note its outPath. Then try it with the same narHash and a different URL. It will attempt to connect/download, even though we already have that outPath.
Here's a concrete example, fetching the same file from multiple IPFS gateways (I got this IPFS CID using `printf 'hello world' | ipfs block add`):
Fetch from `ipfs.io` with an empty narHash, to find what the narHash should be (I'm on NixOS, but using Nix 2.27 for some unrelated git-hashing fixes):
```
$ nix repl
Nix 2.27.0pre19700101_dirty
Type :? for help.
nix-repl> builtins.fetchTree { type = "file"; url = "https://ipfs.io/ipfs/bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e"; narHash = ""; }
error:
… while calling the 'fetchTree' builtin
at «string»:1:1:
1| builtins.fetchTree { type = "file"; url = "https://ipfs.io/ipfs/bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e"; narHash = ""; }
| ^
… while fetching the input 'https://ipfs.io/ipfs/bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e?narHash=sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA%3D'
error: NAR hash mismatch in input 'https://ipfs.io/ipfs/bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e?narHash=sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA%3D', expected 'sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=' but got 'sha256-rkUEKu9bFIg12wLQRf6JtMCf+eR22rABoUvAMi0/IJM='
```
Fetching with that narHash (the file seems to have been cached):
```
nix-repl> builtins.fetchTree { type = "file"; url = "https://ipfs.io/ipfs/bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e"; narHash = "sha256-rkUEKu9bFIg12wLQRf6JtMCf+eR22rABoUvAMi0/IJM="; }
{
narHash = "sha256-rkUEKu9bFIg12wLQRf6JtMCf+eR22rABoUvAMi0/IJM=";
outPath = "/nix/store/0csgnsbvjfr2axpryskr9v7l43bzjvnd-source";
}
```
Now we know the narHash, try fetching the same file from a different URL:
```
nix-repl> builtins.fetchTree { type = "file"; url = "https://cloudflare-ipfs.com/ipfs/bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e"; narHash = "sha256-rkUEKu9bFIg12wLQRf6JtMCf+eR22rABoUvAMi0/IJM="; }
warning: error: unable to download 'https://cloudflare-ipfs.com/ipfs/bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e': Could not resolve hostname (6) Could not resolve host: cloudflare-ipfs.com; retrying in 303 ms
warning: error: unable to download 'https://cloudflare-ipfs.com/ipfs/bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e': Could not resolve hostname (6) Could not resolve host: cloudflare-ipfs.com; retrying in 645 ms
warning: error: unable to download 'https://cloudflare-ipfs.com/ipfs/bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e': Could not resolve hostname (6) Could not resolve host: cloudflare-ipfs.com; retrying in 1049 ms
warning: error: unable to download 'https://cloudflare-ipfs.com/ipfs/bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e': Could not resolve hostname (6) Could not resolve host: cloudflare-ipfs.com; retrying in 2426 ms
error:
… while calling the 'fetchTree' builtin
at «string»:1:1:
1| builtins.fetchTree { type = "file"; url = "https://cloudflare-ipfs.com/ipfs/bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e"; narHash = "sha256-rkUEKu9bFIg12wLQRf6JtMCf+eR22rABoUvAMi0/IJM="; }
| ^
… while fetching the input 'https://cloudflare-ipfs.com/ipfs/bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e?narHash=sha256-rkUEKu9bFIg12wLQRf6JtMCf%2BeR22rABoUvAMi0/IJM%3D'
error: unable to download 'https://cloudflare-ipfs.com/ipfs/bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e': Could not resolve hostname (6) Could not resolve host: cloudflare-ipfs.com
[0.0 MiB DL]
```
Cloudflare have shut down that IPFS gateway, but we still attempted to connect to it despite already having that file in our store.
If we try another, working URL then it will re-download the file, but the output is identical to using the original `ipfs.io` URL:
```
nix-repl> builtins.fetchTree { type = "file"; url = "https://gateway.pinata.cloud/ipfs/bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e"; narHash = "sha256-rkUEKu9bFIg12wLQRf6JtMCf+eR22rABoUvAMi0/IJM="; }
{
narHash = "sha256-rkUEKu9bFIg12wLQRf6JtMCf+eR22rABoUvAMi0/IJM=";
outPath = "/nix/store/0csgnsbvjfr2axpryskr9v7l43bzjvnd-source";
}
```
## Expected behavior
If the `outPath` already exists in our store, then those `fetchTree` calls should return the `{ narHash = "..."; outPath = "..."; }` result immediately, without attempting to download the URL.
## Metadata
nix-env (Nix) 2.27.0pre19700101_dirty
## Additional context
Related issues:
- https://github.com/NixOS/nix/issues/9570 seems to also be caused by `fetchTree` running "eagerly", in a way that fixed-output derivations wouldn't.
- https://github.com/NixOS/nix/issues/9077 would make `fetchTree` act more like a fixed-output derivation. It doesn't mention the outPath being independent of the input URL, or `fetchTree` being too "eager" to perform a download when it doesn't need to.
I'm currently working around this in a rather clunky way, by using a fixed-output derivation that uses `/bin/sh` to make a copy of the fetched file. This way, I can query whether the outPath already exists without having to call `fetchTree` (I use `/dev/null` instead):
```
with rec {
inherit (builtins) currentSystem derivation fetchTree getEnv pathExists;
override = getEnv "IPFS_GATEWAY";
gateway = if override == "" then "https://ipfs.io" else override;
fixed = src: derivation {
name = "source";
builder = "/bin/sh";
system = currentSystem;
outputHashMode = "nar";
outputHash = narHash;
args = [
"-c"
''read -r -d "" content < ${src}; printf '%s\n' "$content" > "$out"''
];
};
existing = (fixed "/dev/null").outPath;
file = if pathExists existing then existing else fixed (fetchTree {
inherit narHash;
type = "file";
url = "${gateway}/ipfs/${cid}";
});
};
file
```
## Checklist
- [x] checked [latest Nix manual] \([source])
- [x] checked [open bug issues and pull requests] for possible duplicates
[latest Nix manual]: https://nixos.org/manual/nix/unstable/
[source]: https://github.com/NixOS/nix/tree/master/doc/manual/source
[open bug issues and pull requests]: https://github.com/NixOS/nix/labels/bug
---
Add :+1: to [issues you find important](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc).
The main idea behind it is the indirection . As opposed to passing a store path directly, using something like the Flake registry or a symlink allows for the same NIX_PATH to immediately being resolving to the new paths (when otherwise services, shells, etc. would need to restart to get the new NIX_PATH with new paths applied). The Flake registry is just one way to do this
It’s been recommended against to use /etc as a location for symlinks, though, as it’s a bit more stateful. So if you want to add your system’s Nixpkgs to your NIX_PATH without using Flakes (and the bugs associated with them), this snippet would probably be a bit better
getchoo:
{
lib,
pkgs,
inputs,
...
}:
let
flakeInputs = pkgs.linkFarm "flake-inputs" (
lib.mapAttrs (lib.const (flake: flake.outPath)) inputs
);
in
{
nix = {
nixPath = lib.mkForce (
lib.mapAttrsToList (name: lib.const "${name}=/run/current-system/inputs/${name}") inputs
);
};
system.extraSystemBuilderCmds = ''
ln -s ${flakeInputs} $out/inputs
'';
}