Content-addressed Nix − call for testers

That’s indeed a bummer. Technically there’s no need to have the drv to copy stuff around, but the cli interface doesn’t allow anything else afaik.

Technically you can probably save most of the bandwidth by copying the output paths from the builders and the realisations from the coordinator. That’s a bit more convoluted than what it should be of course :confused:

The below is taking 225GB 387GB of memory according to macOS, and rising:

> nix store make-content-addressed --all                                                                                      
warning: dumping very large path (> 256 MiB); this may run out of memory

If my machine weren’t a maxed out Macbook Pro (it’s an M1), this would have crashed long ago. I’m on nix (Nix) 2.10.3. Assuming this will fail, does anyone have any hints for debugging what’s going on? (or is it simply expected to fail when run on the entire store at once?)

That sounds like virtual memory which does not really matter. Most haskell programs have that set to 1TB.

1 Like

The OOM killer got it eventually, but in mainline operation it fluctuated between 11GiB and 40GiB of real memory.

I’m getting an odd error, “incorrect output”:

$ nix store make-content-addressed --all
error: derivation '/nix/store/hi3j7zj1ig2x1qnva8csxmq4ry88gzym-hscolour-1.24.4.drv' has incorrect output '/nix/store/fmas9wlcibxc59c7dq7z6rhxjz57gfdg-hscolour-1.24.4-data', should be '/nix/store/v87cgbhgbxsmw2dzs0a3ysgd8hbcp4qd-hscolour-1.24.4-data'

nix store verify --all is happy though?

Also,

$ nix --version
nix (Nix) 2.11.0
$ nix show-derivation /nix/store/hi3j7zj1ig2x1qnva8csxmq4ry88gzym-hscolour-1.24.4.drv
error: path '/nix/store/hi3j7zj1ig2x1qnva8csxmq4ry88gzym-hscolour-1.24.4.drv' is not a valid store path

but the file exists, so presumably the store didn’t add it to the db?

1 Like

I am having the exact same problem:

nix store make-content-addressed --all
error: derivation '/nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv' has incorrect output '/nix/store/cz934gbnggvh1hi3gcfqw8gppgh38hrm-linux-headers-static-5.19', should be '/nix/store/viwsyykm4lninzhpi43gc7dv9d656ql4-linux-headers-static-5.19'

> nix show-derivation /nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv
error: path '/nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv' is not a valid store path
> exa -l /nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv
.r--r--r-- 3,0k root  1 Jan  1970 /nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv

I have tested config.contentAddressedByDefault = true; in my flake based setup on three x86_64 machines last week and reverted again today. My experience was mixed:

So all in all I was really happy to see it worked, but in its current form I cannot use it in production.

4 Likes

I get the following error with config.contentAddressedByDefault = true;

error: opening file '/nix/store/naf68a3znxdc8zpwba1prwiyjqz7z09v-bash-5.1-p16.drv': Too many open files                                                                                                            
corrupted double-linked list                                                                                                                                                                                       

this went away after a few nixos-rebuilds. Now I am crashing with

error: derivation '/nix/store/zpc7fx6jkycw57jnigvi7b7dgyckcdnl-flac-1.4.1.drv' doesn't have expected output 'bin' (derivation-goal.cc/resolvedFinished,realisation)                                                

I had this hundreds of times. Just retry the build and it should hang at the next output not found

now I am failing with

error: creating pipe: Too many open files                                                                                                                                                                          
corrupted double-linked list  

I finally managed to build everything. But after rebooting my system hangs when loading initrd. Screenshot attached.

I’m just getting (repeatedly):

error: creating pipe: Too many open files
corrupted double-linked list
building the system configuration...

Fails in just a couple seconds and doesn’t seem to make any progress.

EDIT:

$ ulimit -n
1024
1 Like

In my tests I had this same error, I think is related to libcurl multiplexer to cache.nixos, at least, is what last messages using -vvv shows.

Tested with nix 1.15.0

Maybe it will be fixed in the next version since --max-substituter-jobs · Issue #3379 · NixOS/nix · GitHub is closed by https://github.com/NixOS/nix/pull/8299

My lsof -u root nixos rebuild files · GitHub show a lot of FIFO

fix the hash rewriting for ca-derivations by thufschmitt · Pull Request #4282 · NixOS/nix · GitHub got merged

7 Likes

Wondering if it is related to creating pipe: Too many open files

What is the status of CA derivations? @regnat are you still working on this? Is there a roadmap for the experimental feature?

3 Likes

That’s unfortunately blocked on the Hydra side of things. https://github.com/NixOS/hydra/pull/1228 is the latest attempt, but stalled because no one is able to properly review it.

3 Likes

We really need to get this done. If I am given commit bit, I will review it and https://github.com/NixOS/hydra/pull/1180 myself; I did not write either of those so while I am obviously for the ideas I am still an independent reviewer.

11 Likes