The below is taking 225GB 387GB of memory according to macOS, and rising:
> nix store make-content-addressed --all
warning: dumping very large path (> 256 MiB); this may run out of memory
If my machine weren’t a maxed out Macbook Pro (it’s an M1), this would have crashed long ago. I’m on nix (Nix) 2.10.3. Assuming this will fail, does anyone have any hints for debugging what’s going on? (or is it simply expected to fail when run on the entire store at once?)
$ nix store make-content-addressed --all
error: derivation '/nix/store/hi3j7zj1ig2x1qnva8csxmq4ry88gzym-hscolour-1.24.4.drv' has incorrect output '/nix/store/fmas9wlcibxc59c7dq7z6rhxjz57gfdg-hscolour-1.24.4-data', should be '/nix/store/v87cgbhgbxsmw2dzs0a3ysgd8hbcp4qd-hscolour-1.24.4-data'
nix store verify --all is happy though?
Also,
$ nix --version
nix (Nix) 2.11.0
$ nix show-derivation /nix/store/hi3j7zj1ig2x1qnva8csxmq4ry88gzym-hscolour-1.24.4.drv
error: path '/nix/store/hi3j7zj1ig2x1qnva8csxmq4ry88gzym-hscolour-1.24.4.drv' is not a valid store path
but the file exists, so presumably the store didn’t add it to the db?
nix store make-content-addressed --all
error: derivation '/nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv' has incorrect output '/nix/store/cz934gbnggvh1hi3gcfqw8gppgh38hrm-linux-headers-static-5.19', should be '/nix/store/viwsyykm4lninzhpi43gc7dv9d656ql4-linux-headers-static-5.19'
> nix show-derivation /nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv
error: path '/nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv' is not a valid store path
> exa -l /nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv
.r--r--r-- 3,0k root 1 Jan 1970 /nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv
I have tested config.contentAddressedByDefault = true; in my flake based setup on three x86_64 machines last week and reverted again today. My experience was mixed:
It actually worked! (yay)
It took a long time to basically (re)build every package in my config. Stuff like gfortranlibreoffice etc. do take their time.
No nix-serve implementation I have tried was able to provide the realisations endpoint to make a binary-cache useful (I have looked at nix-serve, nix-serve-ng and harmonia). I took a look at ca-derivations support by thufschmitt · Pull Request #21 · edolstra/nix-serve · GitHub, fixed the stale nixpkgs override (see GitHub - gador/nix-serve at pr-21) and managed to get nix-serve running with the realisations endpoint. This, however, didn’t really help because I either got hit by Content-addressed realisations can't always be registered · Issue #5220 · NixOS/nix · GitHub (due to different local and remote versions of the same package) or nix build still tried to rebuild everything, although the .drv file, andout path are available on the binary-cache. Of course I also tried with ssh-ng store, but nix build still tried to rebuild everything. I suspect this is because the local nix didn’t connect the derivation with the realisation of the out path. (I also tried copying the out path to the local machine, but this resulted in the error messages with conflicting versions of the same package. No garbage-collecting helped there either)
So all in all I was really happy to see it worked, but in its current form I cannot use it in production.
Concurrently, on the ops side, it would be good to update hydra.nixos.org to the latest master (prior to merging that PR), which includes a Nix 2.17 → 2.19 bump. Per the tracking issue I don’t want the prod deployment to jump too many commits at time, as that makes unexpected issues (a) more likely and (b) harder to debug if they do occur.
Hoping we can finally get Nixpkgs CA derivations enabled CI’d and cached in 2024!