Content-addressed Nix − call for testers

Sandro · August 2, 2022, 12:30pm

That sounds like virtual memory which does not really matter. Most haskell programs have that set to 1TB.

Patrick · August 2, 2022, 5:56pm

The OOM killer got it eventually, but in mainline operation it fluctuated between 11GiB and 40GiB of real memory.

wmertens · September 10, 2022, 6:40am

I’m getting an odd error, “incorrect output”:

$ nix store make-content-addressed --all
error: derivation '/nix/store/hi3j7zj1ig2x1qnva8csxmq4ry88gzym-hscolour-1.24.4.drv' has incorrect output '/nix/store/fmas9wlcibxc59c7dq7z6rhxjz57gfdg-hscolour-1.24.4-data', should be '/nix/store/v87cgbhgbxsmw2dzs0a3ysgd8hbcp4qd-hscolour-1.24.4-data'

nix store verify --all is happy though?

Also,

$ nix --version
nix (Nix) 2.11.0
$ nix show-derivation /nix/store/hi3j7zj1ig2x1qnva8csxmq4ry88gzym-hscolour-1.24.4.drv
error: path '/nix/store/hi3j7zj1ig2x1qnva8csxmq4ry88gzym-hscolour-1.24.4.drv' is not a valid store path

but the file exists, so presumably the store didn’t add it to the db?

gador · September 25, 2022, 7:54am

I am having the exact same problem:

nix store make-content-addressed --all
error: derivation '/nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv' has incorrect output '/nix/store/cz934gbnggvh1hi3gcfqw8gppgh38hrm-linux-headers-static-5.19', should be '/nix/store/viwsyykm4lninzhpi43gc7dv9d656ql4-linux-headers-static-5.19'

> nix show-derivation /nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv
error: path '/nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv' is not a valid store path
> exa -l /nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv
.r--r--r-- 3,0k root  1 Jan  1970 /nix/store/7z63ndpzk3rvp3hhqp5mb15g7yn75mga-linux-headers-static-5.19.drv

gador · October 8, 2022, 10:02am

I have tested config.contentAddressedByDefault = true; in my flake based setup on three x86_64 machines last week and reverted again today. My experience was mixed:

It actually worked! (yay)
It took a long time to basically (re)build every package in my config. Stuff like gfortran libreoffice etc. do take their time.
My deployment tools ( deploy-rs) did not work with CA derivations due to the path being unknown during evaluation. I hacked together a fix to make it deploy (still WIP!, see Incompatibility with content-addressed derivations · Issue #164 · serokell/deploy-rs · GitHub)
I was hit repeatedly by build failures due to (probably unrelated) Nix bugs like Remote Building Experienence massively degraded since ~2-3 months or nix-build: requires non-existent output 'out' from input derivation · Issue #6572 · NixOS/nix · GitHub which slowed the rebuild
No nix-serve implementation I have tried was able to provide the realisations endpoint to make a binary-cache useful (I have looked at nix-serve, nix-serve-ng and harmonia). I took a look at ca-derivations support by thufschmitt · Pull Request #21 · edolstra/nix-serve · GitHub, fixed the stale nixpkgs override (see GitHub - gador/nix-serve at pr-21) and managed to get nix-serve running with the realisations endpoint. This, however, didn’t really help because I either got hit by Content-addressed realisations can't always be registered · Issue #5220 · NixOS/nix · GitHub (due to different local and remote versions of the same package) or nix build still tried to rebuild everything, although the .drv file, andout path are available on the binary-cache. Of course I also tried with ssh-ng store, but nix build still tried to rebuild everything. I suspect this is because the local nix didn’t connect the derivation with the realisation of the out path. (I also tried copying the out path to the local machine, but this resulted in the error messages with conflicting versions of the same package. No garbage-collecting helped there either)

So all in all I was really happy to see it worked, but in its current form I cannot use it in production.

mjbecze · October 13, 2022, 10:08am

I get the following error with config.contentAddressedByDefault = true;

error: opening file '/nix/store/naf68a3znxdc8zpwba1prwiyjqz7z09v-bash-5.1-p16.drv': Too many open files                                                                                                            
corrupted double-linked list

mjbecze · October 13, 2022, 10:15am

this went away after a few nixos-rebuilds. Now I am crashing with

error: derivation '/nix/store/zpc7fx6jkycw57jnigvi7b7dgyckcdnl-flac-1.4.1.drv' doesn't have expected output 'bin' (derivation-goal.cc/resolvedFinished,realisation)

gador · October 13, 2022, 12:38pm

I had this hundreds of times. Just retry the build and it should hang at the next output not found

mjbecze · October 15, 2022, 6:16pm

now I am failing with

error: creating pipe: Too many open files                                                                                                                                                                          
corrupted double-linked list

mjbecze · November 17, 2022, 8:23pm

I finally managed to build everything. But after rebooting my system hangs when loading initrd. Screenshot attached.

n8henrie · May 17, 2023, 1:07am

I’m just getting (repeatedly):

error: creating pipe: Too many open files
corrupted double-linked list
building the system configuration...

Fails in just a couple seconds and doesn’t seem to make any progress.

EDIT:

$ ulimit -n
1024

hugosenari · May 24, 2023, 11:53pm

In my tests I had this same error, I think is related to libcurl multiplexer to cache.nixos, at least, is what last messages using -vvv shows.

Tested with nix 1.15.0

Maybe it will be fixed in the next version since --max-substituter-jobs · Issue #3379 · NixOS/nix · GitHub is closed by https://github.com/NixOS/nix/pull/8299

hugosenari · May 25, 2023, 12:12am

My lsof -u root nixos rebuild files · GitHub show a lot of FIFO

Artturin · June 29, 2023, 3:35pm

fix the hash rewriting for ca-derivations by thufschmitt · Pull Request #4282 · NixOS/nix · GitHub got merged

hugosenari · July 11, 2023, 6:54pm

Wondering if it is related to creating pipe: Too many open files

github.com/NixOS/nix

`Bad file descriptor` when building content addressed derivation

opened 10:27AM - 10 May 22 UTC

aciceri

bug ca-derivations

**Describe the bug** When I try to build ca derivations I get sporadic errors… about "bad file descriptor"s. Sometimes they are built and sometimes not. **Steps To Reproduce** ``` $ nix build --impure --expr '(import <nixpkgs> { config.contentAddressedByDefault = true; }).htop' error (ignored): error: closing file descriptor 39: Bad file descriptor error (ignored): error: closing file descriptor 25: Bad file descriptor error: substitution of 'sha256:7abd5b35fa87ed09127c7f67000ef79752dbc9d33da6ee334a48f2bc7158eb38!out': read failed: Bad file descriptor ``` Same command again gives another output ``` error (ignored): error: cannot unlink '/tmp/nix-build-boost-build-boost-1.69.0.drv-0/boost_1_69_0/boost': Directory not empty error (ignored): error: cannot unlink '/tmp/nix-build-audit-2.8.5.drv-0/audit-2.8.5': Directory not empty error: closing file descriptor 23: Bad file descriptor ``` I've built nix from master (the [latest](https://github.com/NixOS/nix/commit/3ec979fa902c49e975a9af7dc2792fb197030e04) commit) but I had the same problem with current `nixos-unstable`'s nix (2.8) and even with `nixUnstable` on stable (2.5) ``` $ nix-env --version nix-env (Nix) 2.9.0 ``` ``` $ nix show-config accept-flake-config = false access-tokens = allow-dirty = true allow-import-from-derivation = true allow-new-privileges = false allow-symlinked-store = false allow-unsafe-native-code-during-evaluation = false allowed-impure-host-deps = allowed-uris = allowed-users = @wheel auto-optimise-store = false bash-prompt = bash-prompt-suffix = build-hook = /nix/store/7xc0d4pkxm2mly6glp5svqyp8qvfva7b-nix-2.8.0/libexec/nix/build-remote build-poll-interval = 5 build-users-group = nixbld builders = builders-use-substitutes = false commit-lockfile-summary = compress-build-log = true connect-timeout = 0 cores = 0 diff-hook = download-attempts = 5 enforce-determinism = true eval-cache = true experimental-features = ca-derivations flakes nix-command extra-platforms = aarch64-linux i686-linux fallback = false filter-syscalls = true flake-registry = https://github.com/NixOS/flake-registry/raw/master/flake-registry.json fsync-metadata = true gc-reserved-space = 8388608 hashed-mirrors = http-connections = 25 http2 = true ignored-acls = security.selinux system.nfs4_acl impersonate-linux-26 = false keep-build-log = true keep-derivations = true keep-env-derivations = false keep-failed = false keep-going = false keep-outputs = true log-lines = 10 max-build-log-size = 0 max-free = 18446744073709551615 max-jobs = 4 max-silent-time = 0 min-free = 0 min-free-check-interval = 5 nar-buffer-size = 33554432 narinfo-cache-negative-ttl = 3600 narinfo-cache-positive-ttl = 2592000 netrc-file = /etc/nix/netrc nix-path = nixpkgs=/nix/store/ni03h8nnld654lcqrs594lq1g1md818q-source nixos-config=/nix/store/317ij093b6ih587p35fk85sc6pkp58ms-nixos home-manager=/nix/store/33ybk6hkdwfd8w9xglbb42bzad7k3fpn-source plugin-files = post-build-hook = pre-build-hook = preallocate-contents = false print-missing = true pure-eval = true repeat = 0 require-sigs = true restrict-eval = false run-diff-hook = false sandbox = true sandbox-build-dir = /build sandbox-dev-shm-size = 50% sandbox-fallback = false sandbox-paths = /bin/sh=/nix/store/z7hbia18niwzbaxydjyainfbnwc90xgv-busybox-static-x86_64-unknown-linux-musl-1.35.0/bin/busybox /nix/store/d5djp7cj3jsqy4m3lg4nvhfkka9pkhlm-qemu-6.1.1 /nix/store/l5smdvmw3nim4bndhddbxikaz9mfjpwj-bash-5.1-p8 /run/binfmt secret-key-files = show-trace = false stalled-download-timeout = 300 store = auto substitute = true substituters = https://nrdxp.cachix.org https://nixpkgs-wayland.cachix.org https://nix-community.cachix.org https://hydra.iohk.io https://cache.ngi0.nixos.org/ https://arm.cachix.org https://aciceri-fleet.cachix.org https://cache.nixos.org/ https://cache.nixos.org/ https://nrdxp.cachix.org https://nix-community.cachix.org sync-before-registering = false system = x86_64-linux system-features = benchmark big-parallel kvm nixos-test tarball-ttl = 3600 timeout = 0 trace-function-calls = false trusted-public-keys = nrdxp.cachix.org-1:Fc5PSqY2Jm1TrWfm88l6cvGWwz3s93c6IOifQWnhNW4= nixpkgs-wayland.cachix.org-1:3lwxaILxMRkVhehr5StQprHdEo4IrE8sRho9R9HOLYA= nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs= hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ= cache.ngi0.nixos.org-1:KqH5CBLNSyX184S9BKZJo1LxrxJ9ltnY2uAs5c/f1MA= arm.cachix.org-1:K3XjAeWPgWkFtSS9ge5LJSLw3xgnNqyOaG7MDecmTQ8= aciceri-fleet.cachix.org-1:WiHJIK4UFTdfvWx0lG3mCR4EddyYsRhIuMGSje3/YGI= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= nrdxp.cachix.org-1:Fc5PSqY2Jm1TrWfm88l6cvGWwz3s93c6IOifQWnhNW4= nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs= trusted-substituters = trusted-users = root @wheel hydra-queue-runner use-case-hack = false use-registries = true use-sqlite-wal = true user-agent-suffix = warn-dirty = true ``` **Additional context** Sometimes I get "core dumped" in the nix daemon logs but not necessarily.

asymmetric · September 17, 2023, 2:44pm

What is the status of CA derivations? @regnat are you still working on this? Is there a roadmap for the experimental feature?

regnat · September 19, 2023, 1:06pm

That’s unfortunately blocked on the Hydra side of things. https://github.com/NixOS/hydra/pull/1228 is the latest attempt, but stalled because no one is able to properly review it.

Ericson2314 · September 19, 2023, 2:42pm

We really need to get this done. If I am given commit bit, I will review it and https://github.com/NixOS/hydra/pull/1180 myself; I did not write either of those so while I am obviously for the ideas I am still an independent reviewer.