That’s indeed a bummer. Technically there’s no need to have the drv to copy stuff around, but the cli interface doesn’t allow anything else afaik.
Technically you can probably save most of the bandwidth by copying the output paths from the builders and the realisations from the coordinator. That’s a bit more convoluted than what it should be of course
The below is taking 225GB 387GB of memory according to macOS, and rising:
> nix store make-content-addressed --all
warning: dumping very large path (> 256 MiB); this may run out of memory
If my machine weren’t a maxed out Macbook Pro (it’s an M1), this would have crashed long ago. I’m on nix (Nix) 2.10.3. Assuming this will fail, does anyone have any hints for debugging what’s going on? (or is it simply expected to fail when run on the entire store at once?)
$ nix store make-content-addressed --all
error: derivation '/nix/store/hi3j7zj1ig2x1qnva8csxmq4ry88gzym-hscolour-1.24.4.drv' has incorrect output '/nix/store/fmas9wlcibxc59c7dq7z6rhxjz57gfdg-hscolour-1.24.4-data', should be '/nix/store/v87cgbhgbxsmw2dzs0a3ysgd8hbcp4qd-hscolour-1.24.4-data'
nix store verify --all is happy though?
$ nix --version
nix (Nix) 2.11.0
$ nix show-derivation /nix/store/hi3j7zj1ig2x1qnva8csxmq4ry88gzym-hscolour-1.24.4.drv
error: path '/nix/store/hi3j7zj1ig2x1qnva8csxmq4ry88gzym-hscolour-1.24.4.drv' is not a valid store path
but the file exists, so presumably the store didn’t add it to the db?
nix store make-content-addressed --all
error: derivation '/nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv' has incorrect output '/nix/store/cz934gbnggvh1hi3gcfqw8gppgh38hrm-linux-headers-static-5.19', should be '/nix/store/viwsyykm4lninzhpi43gc7dv9d656ql4-linux-headers-static-5.19'
> nix show-derivation /nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv
error: path '/nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv' is not a valid store path
> exa -l /nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv
.r--r--r-- 3,0k root 1 Jan 1970 /nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv
No nix-serve implementation I have tried was able to provide the realisations endpoint to make a binary-cache useful (I have looked at nix-serve, nix-serve-ng and harmonia). I took a look at ca-derivations support by thufschmitt · Pull Request #21 · edolstra/nix-serve · GitHub, fixed the stale nixpkgs override (see GitHub - gador/nix-serve at pr-21) and managed to get nix-serve running with the realisations endpoint. This, however, didn’t really help because I either got hit by Content-addressed realisations can't always be registered · Issue #5220 · NixOS/nix · GitHub (due to different local and remote versions of the same package) or nix build still tried to rebuild everything, although the .drv file, andout path are available on the binary-cache. Of course I also tried with ssh-ng store, but nix build still tried to rebuild everything. I suspect this is because the local nix didn’t connect the derivation with the realisation of the out path. (I also tried copying the out path to the local machine, but this resulted in the error messages with conflicting versions of the same package. No garbage-collecting helped there either)
So all in all I was really happy to see it worked, but in its current form I cannot use it in production.
We really need to get this done. If I am given commit bit, I will review it and https://github.com/NixOS/hydra/pull/1180 myself; I did not write either of those so while I am obviously for the ideas I am still an independent reviewer.