Content-addressed Nix − call for testers

Thanks for the quick reply! These Empty Directory errors only started showing up when I tried overriding the hashes. Maybe that’s a nix problem. I’ll gc and try again, though it’ll take a while to compile everything again. If I can reproduce, I’ll start simplifying.

I forgot to check the whole log.
The gc has kicked in now, and everytime I try to build the system again I get into this error with no output

❯ nix build .\#darwinConfigurations.raphael.system
warning: Git tree '/Users/raphael/.dotfiles' is dirty
error: unexpected end-of-file

I’ll try again next week and will make sure to get the logs before I post something.

Using hydra:

 src = pkgs.fetchFromGitHub {
    owner = "regnat";
    repo = "hydra";
    rev = "2090d3a0899db4b63d3fbf93545be0020385a165";
    sha256 = "sha256-GrYR/xX/kWxxB0VoHKj0c3Pw7A6RLPexDkVNNVa7kn4=";
};

Getting errors like:

possibly transient failure building ‘/nix/store/XYZ.drv’ on ‘root@333.33.3.3’: error: dependency '/nix/store/3fs5wqlj9ipv7wjx4av81skmpcdjjh3d-jq-1.6-dev' of '/nix/store/XYZ.tif.drv' does not exist, and substitution is disabled

Reverting to package = hydra-unstable; restores behavior as expected with regards to remote building. All builds have __contentAddressed = false. Any thoughts, leads? I attempted setting use-substitutes, useSubstitutes, and builders-use-substitutes to no effect. Only solution was to revert.

This error
error: unexpected end-of-file
Always happens before the logs
querying info about the missing paths
Could it be possible that the error is in the dependency resolution?
Once the error happens, after that, trying to build will consistently yield the same error.
on a mac there is an strace equivalent called dtruss. I got a dump

dtrace: error on enabled probe ID 2186 (ID 174: syscall::read:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2186 (ID 174: syscall::read:return): invalid kernel access in action #13 at DIF offset 68
11758/0xa986d3:   1359439       4      1 madvise(0x7FF4F8641000, 0x10000, 0x9)           = 0 0
11758/0xa986d3:   1359451      11      8 close(0x8)              = 0 0
11758/0xa986d3:   1359738      71     68 open("/nix/store/hdbirzvfrxcziwpr51hpgdh6cy8rwa06-x509-store-1.6.7.tar.gz.drv\0", 0x1000000, 0xB3D20)             = 8 0
11758/0xa986d3:   1359741       4      1 fstat64(0x8, 0x7FFEE7BDF730, 0x0)               = 0 0
dtrace: error on enabled probe ID 2186 (ID 174: syscall::read:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2186 (ID 174: syscall::read:return): invalid kernel access in action #13 at DIF offset 68
11758/0xa986d3:   1359791       3      0 madvise(0x7FF4F8641000, 0x10000, 0x9)           = 0 0
11758/0xa986d3:   1359797       7      4 close(0x8)              = 0 0
11758/0xa986d3:   1359969      64     61 open("/nix/store/kscmjj7wl6clvp69835nr3vnq7rav2bn-x509-store-1.6.7-r1.cabal.drv\0", 0x1000000, 0xB3920)           = 8 0
11758/0xa986d3:   1359972       5      1 fstat64(0x8, 0x7FFEE7BDF730, 0x0)               = 0 0
dtrace: error on enabled probe ID 2186 (ID 174: syscall::read:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2186 (ID 174: syscall::read:return): invalid kernel access in action #13 at DIF offset 68
11758/0xa986d3:   1360019       3      1 madvise(0x7FF4F8641000, 0x10000, 0x9)           = 0 0
11758/0xa986d3:   1360026       7      4 close(0x8)              = 0 0
11758/0xa986d3:   1360200       4      0 madvise(0x7FF4F8651000, 0x11000, 0x9)           = 0 0
11758/0xa986d3:   1360289       3      0 madvise(0x7FF4F8651000, 0x11000, 0x9)           = 0 0
11758/0xa986d3:   1360384      66     64 open("/nix/store/mfaip0nkzinhz4c34zbwmpil9aq7jsxj-connection-0.3.1.tar.gz.drv\0", 0x1000000, 0xB9A30)             = 8 0
11758/0xa986d3:   1360386       4      1 fstat64(0x8, 0x7FFEE7BDFCF0, 0x0)               = 0 0
dtrace: error on enabled probe ID 2186 (ID 174: syscall::read:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2186 (ID 174: syscall::read:return): invalid kernel access in action #13 at DIF offset 68
11758/0xa986d3:   1360434       3      1 madvise(0x7FF4F8641000, 0x10000, 0x9)           = 0 0
11758/0xa986d3:   1360440       6      4 close(0x8)              = 0 0
11758/0xa986d3:   1360574      32     29 open("/nix/store/qxn9kdxk1j82vxra0bpav298km86dd6h-x509-validation-1.6.11.drv\0", 0x1000000, 0xB9710)              = 8 0
11758/0xa986d3:   1360577       4      1 fstat64(0x8, 0x7FFEE7BDFCF0, 0x0)               = 0 0
dtrace: error on enabled probe ID 2186 (ID 174: syscall::read:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2186 (ID 174: syscall::read:return): invalid kernel access in action #13 at DIF offset 68
11758/0xa986d3:   1360626       4      1 madvise(0x7FF4F8641000, 0x10000, 0x9)           = 0 0
11758/0xa986d3:   1360636      10      7 close(0x8)              = 0 0
11758/0xa986d3:   1360936      75     71 open("/nix/store/yw9h9w9pf2m4j386dv3z528j7fm4xff9-x509-validation-1.6.11.tar.gz.drv\0", 0x1000000, 0xB5B90)               = 8 0
11758/0xa986d3:   1360939       4      1 fstat64(0x8, 0x7FFEE7BDFA10, 0x0)               = 0 0
dtrace: error on enabled probe ID 2186 (ID 174: syscall::read:return): invalid kernel access in action #13 at DIF offset 68
dtrace: error on enabled probe ID 2186 (ID 174: syscall::read:return): invalid kernel access in action #13 at DIF offset 68
11758/0xa986d3:   1360991       3      1 madvise(0x7FF4F8641000, 0x10000, 0x9)           = 0 0
11758/0xa986d3:   1360998       7      4 close(0x8)              = 0 0
11758/0xa986d3:   1361165       3      0 madvise(0x7FF4F8651000, 0x11000, 0x9)           = 0 0
11758/0xa9870a:        85 2478356      8 psynch_cvwait(0x1087FBEC0, 0x50000000800, 0x700)                = -1 4
11758/0xa98709:        63 2479818     14 __sigwait(0x700005947F58, 0x700005947F5C, 0x0)          = -1 4
11758/0xa9870b:        85 2478350      7 psynch_cvwait(0x1087FBEC0, 0x70100000900, 0x700)                = -1 4
11758/0xa9870c:        89 2478363      6 psynch_cvwait(0x1087FBEC0, 0x70000000A00, 0x700)                = -1 260

That dump goes on and on in a similar fashion just with different nix store paths

I’m probably missing something, but why the need for trustix then?

1 Like

Any idea what this means?

warning: rewriting hashes in '/nix/store/4lqcwpw1gifjmg4wbh3dawqac5xyp8jp-alsa-lib-1.2.4.drv.chroot/nix/store/a1b3ggra2q53dxkx15pr2aipfqmkq2wn-alsa-lib-1.2.4-dev'; cross fingers
warning: rewriting hashes in '/nix/store/xd05052zxds1mc6p63hqq1fnb24my8kg-audit-2.8.5.drv.chroot/nix/store/z6lrgfxk8n2clz0p4ixpjbr7zhlyyln6-audit-2.8.5-bin'; cross fingers
warning: rewriting hashes in '/nix/store/xd05052zxds1mc6p63hqq1fnb24my8kg-audit-2.8.5.drv.chroot/nix/store/mgw2rrkaf5bmxn6d802gx94a301dlf4r-audit-2.8.5-dev'; cross fingers
warning: rewriting hashes in '/nix/store/xd05052zxds1mc6p63hqq1fnb24my8kg-audit-2.8.5.drv.chroot/nix/store/1lf6sawybw39g0ibyd26r2f6g7k95gaw-audit-2.8.5-man'; cross fingers
warning: rewriting hashes in '/nix/store/2c15mq1am6zg3g3crxv56ld4by8wlbvb-apr-1.7.0.drv.chroot/nix/store/3v7zylcnj28nd4fcrfxycscyfpv2fpgm-apr-1.7.0-dev'; cross fingers
error: a 'i686-linux' with features {} is required to build '/nix/store/0a4bp6zr910zqd5fablgghzfb864r3dp-bootstrap-tools.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}

I’m by no mean a MacOS expert, but that trace might be totally normal. The invalid kernel access things seems to just be a dtrace issue (something to do with the system integrity protection).

The unexpected end-of-file errors are often an issue with the daemon protocol and generally due to a version mismach between the client and the daemon (which is technically supported, but not as well tested as it should be). Is it the case here? If it is, can you try with the same version for both?

There’s still some part that has to be trusted when using a package that has been built by someone else. What content-addressing allows is to separate the trust from the actual storage: With an input-addressed derivation you have to trust the store path itself, while with a content-addressed one, the store path itself is self-trusted, what you have to trust is the fact that the derivation produced this store path (and because this is a simple mapping, you can in theory have a different one for each user, in which case they effectively have a different trust in the same store)

4 Likes

It looks like you’re trying to build something for i686 on a machine that only advertises itself as an x86_64. By default Nix automatically adds i686-linux to the extra-platforms of x86_64-linux machines (because it’s backwards-compatible), but if you’ve overriden it or you’re using distributed builds then you need to specify it explicitely

Either way, you still have to trust whoever has built those derivations to have produced that output path. It’s true that there’s less to build, so the amount of work needed for not trusting the storage has been reduced.

Trust wise, the benefit I see that you can now verify an output is really what it claims to be, as it previously wasn’t possible. The downside is when the output is not binary deterministic, which is quite often the case in real world (getting better every day), so you get false negatives - which is a bummer for when you want to verify something is true. Still an important feature for audits that the industry pays big bucks these days.

The two benefits I see are:

  • Global store, which allows you to trust and share FOD (although the FOD trust could have been implemented on top of the binary cache protocol with the extensional model too).

  • Early cut-off optimization that we’ve all been waiting for and the last optimization that Nix was missing from Build system a la carte paper. So changing a commend in stdenv will rebuild it, but since the output of stdenv will stay the same, there won’t be rebuilds of the world. Extremely important for shipping security updates fast enough!

10 Likes

Note that [RFC 0017] Intensional Store by wmertens · Pull Request #17 · NixOS/rfcs · GitHub looks at separating trust from the store.

You were right. Thanks!

I’m pretty confused about the trust things. Would the improvement allow me, at best, to use other nix user’s cache as a mirror but only for things already build by the official hydra and thus already in the official nix cache?

So I’ve transferred to running it in a vm, since that gives quicker cycles.

Still the same issue, the graphical UI doesn’t come up.

Error log:

May 12 23:32:56 nixos-asus sddm[712]: Setting default cursor
[test@nixos-asus:~]$ -asus sddm[712]: Could not setup default cursor
May 12 23:32:56 nixos-asus sddm[712]: Running display setup script  "/nix/store/cjm5hbc18qr5zw06g3zs45vqamqyl6la-Xsetup"
May 12 23:32:56 nixos-asus sddm[712]: Display server started.
May 12 23:32:56 nixos-asus sddm[712]: Socket server starting...
May 12 23:32:56 nixos-asus sddm[712]: Socket server started.
May 12 23:32:56 nixos-asus sddm[712]: Loading theme configuration from "/run/current-system/sw/share/sddm/themes/breeze/theme.conf
May 12 23:32:56 nixos-asus sddm[712]: Greeter starting...
May 12 23:32:56 nixos-asus sddm[712]: Error from greeter session: "execve: No such file or directory"

This is the input file: config file ca-derivations · GitHub

Built with nixos-rebuild build-vm -I nixos-config=configuration.nix, ran with QEMU_OPTS="-m 8192 -smp 2" ./result/bin/run-$hostname-vm.

I tried with two different revisions of the unstable channel, the one I tested with today was the following:

# nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.11.16-hardened1, NixOS, 21.05pre287333.63586475587 (Okapi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.4pre20210503_6d2553a`
 - channels(root): `"nixos-21.05pre287333.63586475587, nixpkgs"`
 - channels(rick): `""`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

If there’s anything I can do to debug this further, let me know. I don’t know how to easily debug these early-bootish issues.

I tried to minimize the example, but there are probably still some unnecessary things in the config. Most important is probably the xserver stuff / plasma and sddm.

Thanks a lot for the response.
Checking on my system however I have

.dotfiles/nix on  master [!] took 3s
❯ nix --version
nix (Nix) 2.4pre20210503_6d2553a
.dotfiles/nix on  master [!]
❯ which nix-daemon
/run/current-system/sw/bin/nix-daemon
.dotfiles/nix on  master [!]
❯ nix-daemon --version
nix-daemon (Nix) 2.4pre20210503_6d2553a

It’s more work, but as long as hash rewriting works, there is nothing from stopping us from having some sort of “pluggable equivalence class” thing to get rid of any false negatives w.r.t the status quo. This is also useful for things we cannot do today such as mixing cross- and native-built packages.

2 Likes

Could we also use ‘pluggable equivalency classes’ of store paths for something like Guix grafts (sending out security-patched but otherwise functionally identical versions of dependencies without/before rebuilding all the reverse deps)?

2 Likes

Very excited for this feature! I’ve tried the Raider of the unknown level (a.k.a. contentAddressedByDefault = true) on a system built from nixos-unstable using flakes. Building the system fails with :

error: output '/nix/store/m6mliw2qv94mx8sqmbdflwdd32qhqrz5-stdenv-linux' is not allowed to refer to the following paths:
         /nix/store/df8j1d5by33hfg5wn7kjjccngb6i8fv6-glibc-2.32-40

Also during the build I’ve seen a lot of warnings like this one :

warning: skipping suspicious writable file '/nix/store/6v17hy9vphp12a0f98b9wq4j20iglsmn-libelf-0.8.13/lib/libelf.a'

I’ll be avoiding doing any gc for a while to help debugging.

Awesome, thanks!

I had a similar issue with openjfx and openjdk which turned out to be an instance of ca-derivations breaks some nix-side store paths manipulations · Issue #4764 · NixOS/nix · GitHub. But I’m not sure it’s the case here :thinking: . Can you share your nixpkgs revision so that I can have a look at the Nix expression?

Yes, that’s an instance of Building wrong-hash FOD with auto-optimise-store on leaves writable files in store · Issue #4513 · NixOS/nix · GitHub which has been fixed a couple days ago.

I was using 83d907fd760d9ee4f49b4b7e4b1c6682f137b573.

I’m currently trying to build using master 354e005d6c7a910fd009b90360545fed8ddba4a2 and report back the results :slight_smile:

Edit: By the way I forgot to mention that I didn’t use cache.ngi0.nixos.org.