Content-addressed Nix − call for testers

[thufschmitt]

Hello Nixers,

One year and a half after the submission of the RFC, I’m happy to announce that thanks to the work of @edolstra, @ericson2314, and your humble servant, content-addressed Nix has now reached beta-grade quality.

This means that it’s now time to start using it.

Content what?

Without entering too much into the details − I invite you to check out this blog post or the relevant section in Eelco’s PhD thesis for that − content-addressed Nix is an extension of the Nix model bringing several new possibilities. In particular, it enables “early cutoff” (stopping a rebuild if it can be proved that the end-result will be the same as something already known), which could reduce hydra’s (and yours) load and storage a lot. It also changes the Trust model of Nix, allowing for example several users to share the same store without trusting each other.

So what’s the status?

All the features are here to make this a reality: It’s now possible to build content-addressed derivations, both locally and remotely, to move them across machines and transparently sign and substitute them.

Now this is a big change, and as such there’s certainly a number of ugly monsters hiding in some dark corners of the code and waiting for the right edge-case to wake them up.
And that’s where we need you, to track them down to the last one, and crush them all.

What can I do?

There’s several things you can do, depending on how much of an adventurer you feel.

Before anything, make sure that you have a recent-enough Nix — both client and daemon, either by using nixUnstable from a nixos-unstable from after the 4th of march may, or by fetching it directly from source.

• Level 0 — Remote adventurer : Try using some CA closures in a sandbox:

  Run a CA build of Emacs:

  $ nix shell \
    --experimental-features 'nix-command' \
    --store /tmp/my-ca-nix \
    --trusted-public-keys '' \
    --substituters https://cache.ngi0.nixos.org/ \
    /nix/store/ih1ish76pdmzcqbdcdd09z007f6bxjrf-emacs-28.1 \
    -c emacs --version
  

  See that content-addressed paths don’t need signatures, because they are self-authenticating!

  $ nix store verify \
    --experimental-features nix-command \
    --store /tmp/my-ca-nix \
    --sigs-needed 10000 \
    /nix/store/yvk5yl9fid0zlxqk1xvvzn787d8gbh00-emacs-27.2
  

  (to get more paths to test, use this dirty one-liner to fetch all the paths built by the latest evaluation:

  curl https://hydra.ngi0.nixos.org/jobset/ca-test/nixpkgs/evals -H 'Accept: application/json' | nix run nixpkgs#jq -- '.evals[0].builds[]' | xargs -I{} sh -c 'curl -s https://hydra.ngi0.nixos.org/build/{} -H "Accept: application/json" | nix run nixpkgs#jq -- -r ".buildproducts[] | .path"'
  

  )

• Level 1 — Visitor of the garden : Add the ca-derivations and ca-references experimental Nix features to your /etc/nix/nix.conf.

  This shouldn’t change anything, except make Nix take different (ca-compatible) code-paths to do the same thing.

  If it changes (or breaks) something, then congratulations, you’ve spotted a bug!

• Level 2 — Explorer of the world : Mark some individual derivations as “content addressed”.
  On a recent-enough nixpkgs, this should be as simple as setting __contentAddressed = true in your call to mkDerivation.
  If you encounter a derivation that works as input-addressed but not as content-addressed, then it’s (probably) a Nix bug.

• Level 3 — Raider of the unknown : Switch to a fully content-addressed system.
  Entering this new world is as easy as passing config.contentAddressedByDefault = true when evaluating nixpkgs.
  Be warned that although there’s a hydra instance testing some stuff, it’s not a channel blocker, and its associated binary cache won’t contain everything you’re used to.
  So you’ll have to rebuild most of your system yourself and you might encounter some unexpected breakages.

  You can use the binary cache cache.ngi0.nixos.org to speed up your builds. To do so, merge the following into your nix.conf:

  substituters = https://cache.ngi0.nixos.org/
  trusted-public-keys = cache.ngi0.nixos.org-1:KqH5CBLNSyX184S9BKZJo1LxrxJ9ltnY2uAs5c/f1MA=
  

  And like above, anything that used to work but doesn’t anymore is probably a bug, so please do submit it!

What’s next?

All this testing will give us more insight on how to finalize and stabilize the design, which in turn will feed the RFC in case it needs to be refined.

Then the big next milestone is to use this directly on https://hydra.nixos.org and populate the official binary cache with a fully content-addressed nixpkgs. And (eventually) make this the default.

EDIT (2022-06-20): Changed the emacs store path (the old one got built before Make the Nar hash non modulo by thufschmitt · Pull Request #4849 · NixOS/nix · GitHub so it didn’t have the correct Nar hash)

[payas]

I’ll be happy to be the Visitor of the Garden, once the binaries are available. Otherwise my trusty rusty will overheat the apartment to boil in already hot subtropical summer compiling the universe.

[danieldk]

First of all, this is awesome, thanks a lot to all involved to bringing this feature!

@thufschmitt

If you encounter a derivation that works as input-addressed but not as content-addressed, then it’s (probably) a Nix bug.

A naive question: what would happen in this case when building a derivation? Plus another question: what happens if __contentAddressed = true is used when the derivation is not reproducible (results in a different output contents across builds)?

Anyway, I am really exited to try this out!

[thufschmitt]

Glad you’re excited about it

A naive question: what would happen in this case when building a derivation?

The most likely outcomes (from my own testing) are either the build failing (with a more-or-less sensible error message), or a specific Nix command will crash (like I encountered recently). In case that’s your fear, I don’t anticipate very bad things like a store corruption (after all, in addition to the testsuite and my own desktop, there’s at least one machine that builds CA derivations rather heavily, and there hasn’t been any major issue with it).

what happens if __contentAddressed = true is used when the derivation is not reproducible (results in a different output contents across builds)?

It all depends of the exact scenario, but in the simplest case where there’s only one source of truth (either you’re only building locally, or there’s only one binary cache that feeds everything else), it’ll work mostly as input-addressed derivations, in that the first build will be accepted as the “truthful” build, and Nix won’t even try to rebuild it (why should it after all?)

[NobbZ]

@thufschmitt

Entering this new world is as easy as passing config.contentAddressedByDefault = 1 when evaluating nixpkgs.

So I did the following on my system and rebuilt, though not much happened…

❯ git diff HEAD^ HEAD
diff --git a/configuration.nix b/configuration.nix
index 35c6774..8e02046 100644
--- a/configuration.nix
+++ b/configuration.nix
@@ -6,6 +6,7 @@
 
 {
   nixpkgs.config.allowUnfree = true;
+  nixpkgs.config.contentAddressedByDefault = 1;
 
   imports = [ ];

[thufschmitt]

@payas

I’ll be happy to be the Visitor of the Garden, once the binaries are available. Otherwise my trusty rusty will overheat the apartment to boil in already hot subtropical summer compiling the universe.

If you’re only enabling the experimental feature, but without marking individual derivations as __contentAddressed, then you won’t have to rebuild anything (quite fortunately I should say, because otherwise my house would be awfully overheated too)

[thufschmitt]

@NobbZ first thing that comes to my mind is that you might have a too old nixpkgs − the support for this was only added a few days ago (Make it easy to try out content-addressed derivations by thufschmitt · Pull Request #120316 · NixOS/nixpkgs · GitHub)

[NobbZ]

Oh, that might be indeed, that host builds on nixos-20.09, though as that is a playbox anyway, I’ll change the main branch to build from as another try.

[payas]

That is great to know! I’ll probably switch over before weekend. I usually update my machine once a week, but I run unstable Nix and NixOS, so should be fine to report in couple of weeks.

[bbigras]

Does __contentAddressed = true work for mkShell? I got error: unknown hash algorithm ''.

[thufschmitt]

It should, yes. I think the error you get is because your nixpkgs is too old. You can still make it work, by also setting outputHashAlgo = "sha256"; outputHashMethod = "recursive";

[jonringer]

This will help a LOT with reviews on staging :). Thank you all the effort from everyone

[NobbZ]

Now I am stuck somewhere else, nix doesn’t seem to recognize its settings:

$ sudo nixos-rebuild switch --impure -L --flake . --keep-going
warning: 'nix flake info' is a deprecated alias for 'nix flake metadata'
warning: Git tree '/home/nmelzer/Projects/nixos-config' is dirty
building the system configuration...
warning: Git tree '/home/nmelzer/Projects/nixos-config' is dirty
error: experimental Nix feature 'ca-derivations' is disabled; use '--experimental-features ca-derivations' to override
(use '--show-trace' to show detailed location information)
$ bat /etc/nix/nix.conf | grep experi
experimental-features = nix-command flakes ca-derivations ca-references
$ nix --version
nix (Nix) 2.4pre20210326_dd77f71

[bbigras]

Ah, sorry. I thought I was up-to-date but the flakes of that particular project wasn’t. but thanks.

Now I’m stuck at a similar problem than @NobbZ

[SFrijters]

Cool stuff!
I’m going to try to visit the dragons, but the first issue I ran into was

error: value is an integer while a Boolean was expected

       at /nix/store/hi3hl700rmbsk24sdayacsa2mjj7nrp3-source/lib/attrsets.nix:349:29:

          348|   */
          349|   optionalAttrs = cond: as: if cond then as else {};
             |                             ^
          350|
(use '--show-trace' to show detailed location information)

Seems like my config wanted config.contentAddressedByDefault = true; instead of config.contentAddressedByDefault = 1;.

I also had the issue that @NobbZ posted, but for me that was resolved by switching on the ca-derivations and ca-references options first, rebuilding the system (no actual builds needed of course except for re-writing nix.conf), and only then switching on contentAddressedByDefault.

$ nix --version
nix (Nix) 2.4pre20210503_6d2553a

It’s currently chugging along doing a full rebuild.

[thufschmitt]

Mh, that seems to be an issue with nixos-rebuild: It’s calling Nix with --experimental-features 'nix-command flakes', which overrides the global configuration

You can probably work around this by manually building .#nixosConfigurations.$(hostname).config.system.build.toplevel and calling sudo ./result/bin/switch-to-configuration switch. But that’s worth solving in nixos-rebuild itself

(cc @bbigras )

[NobbZ]

I have those settings in the nix.conf already. Though my nix version is 5 weeks older than yours, though still 3 weeks newer than the requirement.

[NobbZ]

Nope, that doesn’t help:

$ nix build -L .#nixos/configs/$(hostname)
warning: Git tree '/home/nmelzer/Projects/nixos-config' is dirty
error: experimental Nix feature 'ca-derivations' is disabled; use '--experimental-features ca-derivations' to override
(use '--show-trace' to show detailed location information)

[thufschmitt]

@SFrijters

Seems like my config wanted config.contentAddressedByDefault = true; instead of config.contentAddressedByDefault = 1; .

Thanks, that was a typo in the post indeed

[thufschmitt]

Mh… what’s the output of nix show-config | grep experimental ? Maybe the config file got hijacked somewhere

(Or maybe try restarting the daemon in case it didn’t pick the feature?)