Content-addressed Nix − call for testers

Today trying to build commit f9e45390deced72bc7cc0b75d3a922aa10a33e08
on darwin
I’ve had the following failure

error: builder for '/nix/store/r1aivaz16mj43iwdrmmpjh4pdk1781h2-gettext-0.21.drv' failed to produce output path for output 'man' at '/nix/store/mpfz8gx1i0355za3ashaab2m6zmax3bl-gettext-0.21-man'

Here are some additional logs

make[3]: Leaving directory '/private/tmp/nix-build-gettext-0.21.drv-0/gettext-0.21'
make[2]: Leaving directory '/private/tmp/nix-build-gettext-0.21.drv-0/gettext-0.21'
make[1]: Leaving directory '/private/tmp/nix-build-gettext-0.21.drv-0/gettext-0.21'
@nix { "action": "setPhase", "phase": "fixupPhase" }
post-installation fixup
strip is /nix/store/4wdmgb53g0d24bbnkmwjbrirpx4pz8pv-bootstrap-stage0-clang/bin/strip
find: '/nix/store/mpfz8gx1i0355za3ashaab2m6zmax3bl-gettext-0.21-man': No such file or directory
strip is /nix/store/4wdmgb53g0d24bbnkmwjbrirpx4pz8pv-bootstrap-stage0-clang/bin/strip
strip is /nix/store/4wdmgb53g0d24bbnkmwjbrirpx4pz8pv-bootstrap-stage0-clang/bin/strip
strip is /nix/store/4wdmgb53g0d24bbnkmwjbrirpx4pz8pv-bootstrap-stage0-clang/bin/strip

I tried again just to see if the error was persistent, and I had a different error with the same library

>  /nix/store/4wdmgb53g0d24bbnkmwjbrirpx4pz8pv-bootstrap-stage0-clang/bin/mkdir -p '/nix/store/fdfly8j8zn75sh7h9rwbbk0j4jw259pc-gettext-0.21/share/gettext'
       >  /nix/store/4wdmgb53g0d24bbnkmwjbrirpx4pz8pv-bootstrap-stage0-clang/bin/install -c -m 644 lib/gettext.h '/nix/store/fdfly8j8zn75sh7h9rwbbk0j4jw259pc-gettext-0.21/share/gettext'
       > install: cannot create regular file '/nix/store/fdfly8j8zn75sh7h9rwbbk0j4jw259pc-gettext-0.21/share/gettext/gettext.h': Permission denied
       > make[3]: *** [Makefile:441: install-gettextsrcDATA] Error 1
       > make[3]: Leaving directory '/private/tmp/nix-build-gettext-0.21.drv-0/gettext-0.21/gnulib-local'
       > make[2]: *** [Makefile:514: install-am] Error 2
       > make[2]: Leaving directory '/private/tmp/nix-build-gettext-0.21.drv-0/gettext-0.21/gnulib-local'
       > make[1]: *** [Makefile:403: install-recursive] Error 1
       > make[1]: Leaving directory '/private/tmp/nix-build-gettext-0.21.drv-0/gettext-0.21'
       > make: *** [Makefile:711: install] Error 2

This is amazing news! :sparkles:

It would be great to have more detailed instructions how to do it. Right now it’s like a quiz without a way to see if you have the correct answer.

This is what i did:

Before anything, make sure that you have a recent-enough Nix
using nixUnstable from a nixos-unstable

  nix.package = pkgs.nixUnstable;

Level 0 — Remote adventurer :tv: : Try using some CA closures in a sandbox:

[davidak@gaming:~]$ nix shell \
>   --experimental-features 'ca-references nix-command' \
>   --store /tmp/my-ca-nix \
>   --trusted-public-keys '' \
>   --substituters https://cache.ngi0.nixos.org/ \
>   /nix/store/yvk5yl9fid0zlxqk1xvvzn787d8gbh00-emacs-27.2 \
>   -c emacs --version
GNU Emacs 27.2
Copyright (C) 2021 Free Software Foundation, Inc.
GNU Emacs comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GNU Emacs
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.

It starts. How do i see that it’s actually content-addressed?

[davidak@gaming:~]$ nix store verify \
>   --experimental-features nix-command \
>   --store /tmp/my-ca-nix \
>   --sigs-needed 10000 \
>   /nix/store/yvk5yl9fid0zlxqk1xvvzn787d8gbh00-emacs-27.2

See that content-addressed paths don’t need signatures, because they are self-authenticating!

How do i see that here? I get no output.

Level 1 — Visitor of the garden :footprints:: Add the ca-derivations and ca-references experimental Nix features to your /etc/nix/nix.conf.

I did it like this:

  nix.extraOptions = ''
    experimental-features = nix-command ca-derivations ca-references
  '';

I did a system rebuild, the nix daemon was restarted. Nothing else happened. I didn’t test anything else here.

Level 2 — Explorer of the world :cowboy_hat_face:: Mark some individual derivations as “content addressed”.

I naively added __contentAddressed to the hello package and build it. The output hash was different. I guess this is correct.

https://github.com/davidak/nixpkgs/commit/f35e1ad15ec5b3ce0997321df401b96bb853d56c

[davidak@gaming:~/code/nixpkgs]$ nix build -f . hello

[davidak@gaming:~/code/nixpkgs]$ ll result
lrwxrwxrwx 1 davidak users 54 May 20 16:35 result -> /nix/store/fx0d44m03659dw46ch99aln9c3fsliaj-hello-2.10

How can i see if it works as content-addressed? That the build does not throw an error?

Level 3 — Raider of the unknown :dragon_face:: Switch to a fully content-addressed system.

My configuration.nix now has this added lines:

  nix.package = pkgs.nixUnstable;
  nixpkgs.config.contentAddressedByDefault = true;
  nix.extraOptions = ''
    experimental-features = nix-command ca-derivations ca-references
    substituters = https://cache.ngi0.nixos.org/
    trusted-public-keys = cache.ngi0.nixos.org-1:KqH5CBLNSyX184S9BKZJo1LxrxJ9ltnY2uAs5c/f1MA=
  '';

I’m rebuilding the system now from latest unstable.

I guess lines like these mean that the content-addressing works.

warning: rewriting hashes in '/nix/store/yq3birl9c5plpa3vgkmrwmz7vx0kbb29-libxslt-1.1.34.drv.chroot/nix/store/3ag9sb8s65x9070pdzapbjr85cl1jr4k-libxslt-1.1.34-py'; cross fingers
...
Resolved derivation: '/nix/store/qqja4mcv2n6s9mrl379ds2f3jvr665sa-extra-hosts.drv' -> '/nix/store/hjanz73r8m9sd3qmwd3w9zpgd3z1x3gn-extra-hosts.drv'...
Resolved derivation: '/nix/store/jjvlwr36rcddlhdklznc4fbiblam4qav-fix-paths.patch.drv' -> '/nix/store/yx7c7vi1f80zkppzdqvycpirxx625187-fix-paths.patch.drv'...
Resolved derivation: '/nix/store/ag5zb08r5zns32d190dh4mnnpf26s6sw-fuse-3.10.3.drv' -> '/nix/store/zyiapkl5p15dkndzb1ggngsb649qx4i2-fuse-3.10.3.drv'...

This thread is a bit of a mess - how does the idea of updating wiki page with issues/workarounds/PR’s/etc sound like? https://nixos.wiki/wiki/Nix_command/make-content-addressable

Basically wiki page as GitHub URL dump and try to more or less keep it up to date

That’s a great idea! I’ve started Ca-derivations - NixOS Wiki to that effect. It’s in no way complete, but at least it’s a better entrypoint than this discourse post

@davidak Ca-derivations - NixOS Wiki might answer some of your questions

3 Likes

That’s quite a lot of (good) questions here.

Indeed you don’t (agree that it’s definitely not self-explaining). The whole point is that you never signed this path, but nix store verify does succeed because it being content-addressed means that there’s no need to add an external trust to it.

Running the same command on an input-addressed emacs will yield an error:

$ nix store verify \
>    --experimental-features nix-command \
>    --sigs-needed 10000 \
>    /nix/store/0k85iw503c40bjnfhq2wxmznnp24nwxj-emacs-27.2
path '/nix/store/0k85iw503c40bjnfhq2wxmznnp24nwxj-emacs-27.2' is untrusted
$ echo $?
2

That’s great then :slight_smile: As said in the post :

Mostly, yes. Or that something doesn’t quite work properly when using it. Or any issue that happens when using the content-addressed version that didn’t exist with the input-addressed one.

Indirectly yes.

A nice way to check the result is to run

nix store verify --recursive \
    --experimental-features nix-command \
    --sigs-needed 10000 \
    /run/current-system

to see whether some paths are not content addressed in the system.

(interestingly I just ran that, and it turns out that there’s a few (41) non content-addressed paths in my current system).

1 Like

thanks!

My system rebuild is now complete.

It failed 3 times and i had to start it again. The last is the known mailutils issue. The other worked next time.

click to extend
test_ioloop.py::TestIOLoop::test_close_all
/nix/store/p5dydjzj5kjmnlyf00w7jwy0lb90aj6r-python3.8-pyzmq-22.0.3/lib/python3.8/site-packages/zmq/tests/test_ioloop.py:111: DeprecationWarning: zmq.eventloop.ioloop is deprecated in pyzmq 17. pyzmq now works with default tornado and asyncio eventloops.
  loop = self.IOLoop.current()

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED ../../nix/store/p5dydjzj5kjmnlyf00w7jwy0lb90aj6r-python3.8-pyzmq-22.0.3/lib/python3.8/site-packages/zmq/tests/test_future.py::TestFutureSocket::test_recv_json_cancelled
==== 1 failed, 185 passed, 24 skipped, 90 deselected, 6 warnings in 52.60s =====
error: builder for '/nix/store/rwgin3n4fp9pirac32mbdbgl6cjfpxh3-python3.8-pyzmq-22.0.3.drv' failed with exit code 1
building '/nix/store/kkvhmfjxf45aw182ar8mjsxs3d4gbkd7-python3.8-unittest2-1.1.0.drv'...
error: 1 dependencies of derivation '/nix/store/6hf056vzaxx9k3afk1ds4xv8ppi72jib-python3.8-jupyter_client-6.1.12.drv' failed to build
error: 1 dependencies of derivation '/nix/store/0nnm3pl9mafip0xcba5wmb4jlmk2r8f9-python3.8-ipykernel-5.5.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/phxarvh1p2g4zh99z5dj9f69jg7fc6xw-python3.8-notebook-6.3.0.drv' failed to build
building '/nix/store/h5nfllm0dwjychmk87rwvk95n54imdl0-python3.8-urllib3-1.26.4.drv'...
error: 1 dependencies of derivation '/nix/store/c5nmx9lm7cvqxmg28bs4hfh4qd8fwpp5-python3.8-ipywidgets-7.6.3.drv' failed to build
error: 1 dependencies of derivation '/nix/store/svgbq37gqcpmxqz68mbdbjcc0kw0nw9b-python3.8-rich-10.1.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/bqycsv6y5k0508fq6gi25xnr60jl5397-python3.8-tqdm-4.60.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/bp5y4fz572s0flp8bmpv6cxaxmv800f6-python3-3.8.9-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/qd101l7brb9fx7v4a2b6vc9yad2qh3sv-noto-fonts-emoji-2020-09-16-unicode13.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/gi9nwkanlfdjfdgarrv89h3srvcvb3mm-X11-fonts.drv' failed to build
error (ignored): error: cannot unlink '/tmp/nix-build-nix-2.4pre20210503_6d2553a.drv-0/nix-test/tests': Directory not empty
error: 1 dependencies of derivation '/nix/store/fm33x8mib6jpawbx7fg00fl5hd3bi0s1-fc-00-nixos-cache.conf.drv' failed to build
error: 1 dependencies of derivation '/nix/store/x8kdjj40g8b5686in9j4x8xfr1dj6610-xserver.conf.drv' failed to build
error: 1 dependencies of derivation '/nix/store/260m8n47pbh1v9x9wh6lhmn4rrarc74h-fontconfig-conf.drv' failed to build
building '/nix/store/5jl93av6zlp8k28w5miaw43aizln38q3-python3.8-virtualenv-20.4.3.drv'...
building '/nix/store/2npwwwczzz70hc4xkjgyj455bbj38hjh-python3.8-webencodings-0.5.1.drv'...
building '/nix/store/1cb6fdd65703swdvvwy8pccgzimibkqs-python3.8-zopfli-0.1.8.drv'...
building '/nix/store/rwni22an7ksy4ikb1fwx9fwqgsrp8n88-qemu.conf.drv'...
error: 1 dependencies of derivation '/nix/store/wbrfc5n9pfllvpjjil5rd7y84qj6zbv3-system-path.drv' failed to build
building '/nix/store/jjblisx702g8gghjjhqjllc76pv8m4pa-t1lib-5.1.2.drv'...
building '/nix/store/nssim6jwk36qfmd240pyz84dy3238c7l-t1utils-1.41.drv'...
building '/nix/store/l4g7nz5nm8q3rr90xvlnv3gwfhp052fi-taglib-1.12.drv'...
building '/nix/store/sf4i82p6r42ms1qibhsamn9jmcll5asc-tcl-8.6.11.drv'...
building '/nix/store/v334vd36zdda3h8lcl48y7znihi1dm22-tcp-wrappers-7.6.q-26.drv'...
building '/nix/store/7ip1rblxm1ca6p49g1riilb1pbnkahpn-tetragonos.r49732.tar.xz.drv'...
building '/nix/store/rbwi6qyv045qclwd9yjmqgv7ryfqygxd-texi2html-5.0.drv'...
building '/nix/store/8q8zdkbrxhnjkrm52jdavs352yii7izh-texlive-ae-1.4.drv'...
building '/nix/store/lg9r1y6m79qxyi16n1jlcnwbmk4sifw5-texlive-amscls-2.20.6.drv'...
building '/nix/store/pzclqc1phf6fmnvr6i0galj61k2jwmz9-texlive-amsfonts-3.04.drv'...
building '/nix/store/p0416zdnn5jjc1gsjkfzmzbqnwl5ipms-texlive-amsmath-56514.drv'...
building '/nix/store/f9k0hrk2rfcz1mp14b1i36vkm5p5cnhz-texlive-anysize-15878.drv'...
error: 1 dependencies of derivation '/nix/store/wbp573hyykr25nr2xlcb45f796mr3n97-xserver-wrapper.drv' failed to build
error (ignored): error: cannot unlink '/tmp/nix-build-tcl-8.6.11.drv-0': Directory not empty
error: 1 dependencies of derivation '/nix/store/n8zad77lrfy5dlq5ycffx521pmldzx11-nixos-system-gaming-21.05pre289526.7a1fbc38a4b.drv' failed to build



an-1 -I/nix/store/pfc7qf99r62avq46m14wx3i8wi2hgrzc-libpng-apng-1.6.37-dev/include/libpng16 -I/nix/store/dzh9ikjl7cybjbdn7jmdlx9kj466rqqp-zlib-1.2.11-dev/include  -I/nix/store/lzhwhwmxwx307dgbpljnqlxvi8p9ihjr-texlive-bin-2021/include -I../../../texk/web2c/mplibdir  -Wimplicit -Wreturn-type -g -O2 -MT libmplibbackends_a-svgout.o -MD -MP -MF .deps/libmplibbackends_a-svgout.Tpo -c -o libmplibbackends_a-svgout.o `test -f 'svgout.c' || echo '../../../texk/web2c/'`svgout.c
config.status: creating examples/Makefile
config.status: creating examples/config/Makefile
config.status: creating examples/cpp/Makefile
CC [M]  net/openvswitch/vport-netdev.o
/nix/store/n5k6i1bs84fyb8p5191mzjzb89qd0h6s-bootstrap-tools/bin/bash ../libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -DEXEEXT=\"\" -DEXEEXT=\"\" -DEXEEXT=\"\" -I. -I..  -I../intl -I../intl -I.. -I.. -DDEPENDS_ON_LIBICONV=1 -DDEPENDS_ON_LIBINTL=1 -DLIBXML_STATIC    -g -O2 -c -o unistr/u8-mbtouc-unsafe.lo unistr/u8-mbtouc-unsafe.c
../../../texk/web2c/mplibdir/svgout.w:63:10: fatal error: mplib.h: No such file or directory
 63 | #include "mplib.h"
    |          ^~~~~~~~~
compilation terminated.
make[2]: *** [Makefile:15942: libmplibbackends_a-svgout.o] Error 1
make[2]: Leaving directory '/build/texlive-20210325-source/WorkDir/texk/web2c'
make[1]: *** [Makefile:18786: all-recursive] Error 1
make[1]: Leaving directory '/build/texlive-20210325-source/WorkDir/texk/web2c'
make: *** [Makefile:5487: all] Error 2
error: builder for '/nix/store/mmhqcqm8s3zmnafqw0dd1lzjdcx53ycr-texlive-core-big.bin-2021.drv' failed with exit code 2
error: 1 dependencies of derivation '/nix/store/287x5np0591djpbbg31p113m7slaf0ws-texlive-combined-2021.drv' failed to build
error: 1 dependencies of derivation '/nix/store/hx0gwphs36nc4bccf9yi8s5pdph5msvl-texlive-combined-2021.drv' failed to build
error: 1 dependencies of derivation '/nix/store/r9dj8rfx478ar0925071922hvxbg68sv-texlive-combined-2021.drv' failed to build
error: 1 dependencies of derivation '/nix/store/apmsv2f6qk5imrqws8hqznfkyyn634xk-asciidoc-9.0.4.drv' failed to build
error: 1 dependencies of derivation '/nix/store/av615110cld42yd86qsqjz4bs641ls94-gnome-shell-40.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/m4pc1yqrmdzr0v8pndq83w2j8kzhch2i-gnome-session-40.1.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/5cwg1waynxbyz7i16sbj1ljj1gbfghk7-pantheon.drv' failed to build
error: 1 dependencies of derivation '/nix/store/gnrril0plixadlshfad3xarszkkrg6lp-elementary-session-settings-unstable-2020-07-06.drv' failed to build
error: 1 dependencies of derivation '/nix/store/1an1imai4m5z9j9vfm7xnna6mcirv0bm-desktops.drv' failed to build
error: 1 dependencies of derivation '/nix/store/wbrfc5n9pfllvpjjil5rd7y84qj6zbv3-system-path.drv' failed to build
error (ignored): error: cannot unlink '/tmp/nix-build-mailutils-3.12.drv-0/mailutils-3.12': Directory not empty
error: 1 dependencies of derivation '/nix/store/n8zad77lrfy5dlq5ycffx521pmldzx11-nixos-system-gaming-21.05pre289526.7a1fbc38a4b.drv' failed to build



Servers ....................... pop3d imap4d comsat mda lmtpd
Clients ....................... putmail frm mail sieve messages readmsg dotlock movemail mimeview decodemail mh

*******************************************************************

Before proceeding, verify if these satisfy your requirements.
configure: WARNING: unrecognized options: --with-path-sendmail
CC      lib/decompress_unlzma.o
checking whether isnan(float) can be used without linking with libm... yes
checking for strdup... yes
building
build flags: SHELL=/nix/store/2fp2bd2p52cbdgrc9j9fq12m14kki37k-bash-4.4-p23/bin/bash
Makefile:548: *** missing rule before recipe.  Stop.
error: builder for '/nix/store/0sjmpv2l1k729r7g90qr0r9l8mjri73h-mailutils-3.12.drv' failed with exit code 2
error: 1 dependencies of derivation '/nix/store/8lq9d03jspsbf9vnv3g49sm3phkn92q1-smartmontools-7.2.drv' failed to build
error: 1 dependencies of derivation '/nix/store/wbrfc5n9pfllvpjjil5rd7y84qj6zbv3-system-path.drv' failed to build
error (ignored): error: cannot unlink '/tmp/nix-build-gettext-0.21.drv-0/gettext-0.21/gettext-tools': Directory not empty
error: 1 dependencies of derivation '/nix/store/pblryim6s9ly1dk1rj9xlh3w6sc7wvgi-unit-smartd.service.drv' failed to build
error: 1 dependencies of derivation '/nix/store/n8zad77lrfy5dlq5ycffx521pmldzx11-nixos-system-gaming-21.05pre289526.7a1fbc38a4b.drv' failed to build

The system boots and i can login into Pantheon desktop. I was logged out in GitHub, Discourse etc in Firefox.

[davidak@gaming:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.12.4, NixOS, 21.05pre289526.7a1fbc38a4b (Okapi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.4pre20210503_6d2553a`
 - channels(root): `"home-manager, nixos-21.05pre289526.7a1fbc38a4b, nixos-hardware, nixos-unstable-21.05pre289526.7a1fbc38a4b"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`


[davidak@gaming:~]$ nix store verify --recursive \
>     --experimental-features nix-command \
>     --sigs-needed 10000 \
>     /run/current-system
path '/nix/store/08p57lqsydb0ybf9g25r4prfhwflfbv0-python3.8-Jinja2-2.11.3' is untrusted
path '/nix/store/0r4lnvnr043zkiw3rwf76pgq7lhgkd85-reuse-0.12.1' is untrusted
path '/nix/store/16n426g6jbcwwdwlq7h7qmbv1v22p2v5-zlib-1.2.11' is untrusted
path '/nix/store/3d8pvcxnj6dp9fxakn588vmd4wiz63wx-fira-mono-3.206' is untrusted
path '/nix/store/4flbwghn930vyx0s70d29xr5wgp35sz3-vpnc-scripts-c0122e8' is untrusted
path '/nix/store/534d0l78q3apma8naqgfkxf2vb1p7i0x-python3.8-requests-2.25.1' is untrusted
path '/nix/store/5bg9dy337rp8r1mcc4cdrjwhw1kwmppj-openssl-1.1.1k' is untrusted
path '/nix/store/68c5b0h2w2ab57sbas1b4m073ik8hlm5-bash-4.4-p23' is untrusted
path '/nix/store/6kgfmzx90c1a6afqnbkz6qprkzss476k-mime-types-9' is untrusted
path '/nix/store/8c2hpw473d80yfw1nqyaxdkaz12rqj0w-python3.8-urllib3-1.26.4' is untrusted
path '/nix/store/8mkph96c97h87sdibvpms0x56pkzic3a-python3.8-chardet-4.0.0' is untrusted
path '/nix/store/8j0zk6lz3kgrga56lnvwr1cchr88g3sb-python3.8-setuptools-54.2.0' is untrusted
path '/nix/store/9ilyrqidrjbqvmnn8ykjc7lygdd86g7q-gcc-10.2.0-lib' is untrusted
path '/nix/store/9ysrnpijf5yi6zvp22jfp96iw85m2nlb-libffi-3.3' is untrusted
path '/nix/store/bsb6lp486p52widk6nggrykkyjmv9i76-open-sans-1.11' is untrusted
path '/nix/store/bzjd66sdi85rwj7kd3zcdsi52xzhx69c-nix-wallpaper-simple-dark-gray.png' is untrusted
path '/nix/store/dbn507rrsmgmdxwknhb3554nmkl0kvgi-gyre-fonts-2.005' is untrusted
path '/nix/store/dbxrckq9l6lr1s53db6kn4f2rajj766s-sqlite-3.35.2' is untrusted
path '/nix/store/fkndl2n7l8x9ncrkzggpcmci7933kh3x-python3.8-MarkupSafe-1.1.1' is untrusted
path '/nix/store/gm124arknrkwdcnxdg0wb802cnzh4z2m-python3.8-toml-0.10.2' is untrusted
path '/nix/store/gyvm8a6nawv9ai1wpcn6syx8m4l9ni7b-python3.8-setuptools_scm-6.0.1' is untrusted
path '/nix/store/h2hnqj2x6j4jqvr9dry7aakbp6jpvb4d-xz-5.2.5' is untrusted
path '/nix/store/h4zipkqpacwqd3df1d2dbl2688r72rdp-python3.8-boolean.py-3.8' is untrusted
path '/nix/store/iwk66ibxs1k0hyyjlm388rckb4hd8f5x-python3.8-pysocks-1.7.1' is untrusted
path '/nix/store/him2krcndagwpw60d0qwa1w8s0nyjxyq-ncurses-6.2' is untrusted
path '/nix/store/j1ws3ipg9nwj22194xa7ql9d7637mjv8-python3.8-brotli-1.0.9' is untrusted
path '/nix/store/jjhjkrll0n60bcczgldc4z6bmn2f24lc-libidn2-2.3.0' is untrusted
path '/nix/store/kpadhaqz7vx83rrdvnwrjndkxq9s5iil-roboto-mono-2.002-20190125' is untrusted
path '/nix/store/lan2w3ab1mvpxj3ppiw2sizh8i7rpz7s-busybox' is untrusted
path '/nix/store/lslp7akb81gzss4rsxvfl4l08ax9iwlw-python3.8-python-debian-0.1.39' is untrusted
path '/nix/store/nhy4czm0bg0mh2lpfzdkmsxvf5jc59pw-bzip2-1.0.6.0.2' is untrusted
path '/nix/store/pldw9fmb9zsn218ljd54hn3wcn87kbgm-python3.8-binaryornot-0.4.4' is untrusted
path '/nix/store/q5g87xcw06a4cxsgf0bgmhpd8ipnk0nh-libunistring-0.9.10' is untrusted
path '/nix/store/q8476s0vgzlgxmzxw7av70l5p20zbhqs-gdbm-1.19' is untrusted
path '/nix/store/sfzav51kd5gsjnfll2i2br8vg37acyd3-expat-2.2.10' is untrusted
path '/nix/store/slsqgiikh9mnzd3jw90mc2x61anj84h7-python3.8-six-1.15.0' is untrusted
path '/nix/store/srjp3lv3qfand8pwsj8np32yw1q4f7ms-intel2200BGFirmware-3.1' is untrusted
path '/nix/store/szphjhh0j68yhddyw2zh8ykr00mhd1r2-publicsuffix-list-2019-05-24' is untrusted
path '/nix/store/v3kqlbg4mim18511hij6jzxmbjywagnf-python3.8-license-expression-1.2' is untrusted
path '/nix/store/q6gfck5czr67090pwm53xrdyhpg6bx67-python3-3.8.9' is untrusted
path '/nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40' is untrusted
path '/nix/store/xmp5w9p9jb4r2nq8sa1wyc33s6ymq198-zd1211-firmware-1.5' is untrusted
path '/nix/store/xvdf2dnj66vyyi0jjwxr17qjk0v3w8fp-nix-wallpaper-simple-dark-gray_bootloader.png' is untrusted
path '/nix/store/xy13dlrqldl0c5p8lms1pdlri4gpmwip-python3.8-certifi-2020.12.5' is untrusted
path '/nix/store/y1hybm8h1kln0hg06c42m4g1wsblc0ig-freefont-ttf-20120503' is untrusted
path '/nix/store/z1lwwj3sfccycvhs7gy5lhikkl19b4gq-readline-6.3p08' is untrusted
path '/nix/store/z8qgxgsdfan42zfricbikhlvnwyq9wb7-python3.8-idna-3.1' is untrusted

What’s up with these untrusted paths?

Would be useful if nix-info mentioned that the system is config.contentAddressedByDefault = true

1 Like

yes, i also thought that. also the activated experimental nix features.

also, nix-info should be integrated into nix

do you want to create issues for that?

Nice :slight_smile: Any idea whether the two transient failures possibly due to the system being overloaded or something like that?

I’m not sure. Some of these seems to be bootstrap tools, so maybe I messed-up something when implementing config.contentAddressedByDefault and some of these don’t pick it up. Some also are fixed-output derivations, so I’ve no idea why they don’t show-up as trusted.

That might be possible. I let it run over night and when i came back, there was these errors.

I tried to reproduce it, but it built successful 5 times.

nix build --rebuild /nix/store/mmhqcqm8s3zmnafqw0dd1lzjdcx53ycr-texlive-core-big.bin-2021.drv
nix build --rebuild /nix/store/rwgin3n4fp9pirac32mbdbgl6cjfpxh3-python3.8-pyzmq-22.0.3.drv

I also had out of disk space errors and had to give /tmp more space from my RAM, for example for LLVM to build.

mount -o remount,size=26G /tmp

What are the steps that Nix does to compute the package hash to use for content addressing and how can i do it manually with standard linux tools?

For example the zcbnxwkq95lbshldnrmrbifkmhz311cc in:

[davidak@gaming:~]$ nix build -f channel:nixos-unstable hello

[davidak@gaming:~]$ ll result
lrwxrwxrwx 1 davidak users 54 May 24 09:23 result -> /nix/store/zcbnxwkq95lbshldnrmrbifkmhz311cc-hello-2.10

I found out one way to get a hash of a directory is to tar it. Is Nix doing that also?

[davidak@gaming:~]$ tar -c /nix/store/zcbnxwkq95lbshldnrmrbifkmhz311cc-hello-2.10 | sha256sum 
tar: Removing leading `/' from member names
7c953e49b5d145cd26014a4e3caab4e6c617cada50bc262008ac64fa65d8b2e4  -

You can’t really easily. The non-modulo hash is the hash of the nar archive (so basically your tar version, except that it uses Nars because tar store too much metadata for it to be properly reproducible). But the “modulo” part means that we also have to zero-out every occurence of the store path hash in the nar.
So that would be something like

nix store dump-path /nix/store/zcbnxwkq95lbshldnrmrbifkmhz311cc-hello-2.10 | sed 's/zcbnxwkq95lbshldnrmrbifkmhz311cc/00000000000000000000000000000000/g' | sha256 sum

But I’m not sure that would work out of the box, and that would just give you the ca-hash of the path, which is different from its store path. To get the store-path, you need to re-hash it as per https://github.com/NixOS/nix/blob/af4ff644d5413b603f1ac512e21536a9e903bc34/src/libstore/store-api.cc#L182-L194

(So recomputing it outside of Nix it would be quite a journey :wink: )

1 Like

There’s maybe something useful in here, you could write something in Go to do it: https://github.com/colemickens/niche/tree/master/pkg

there’s utilities for making a NAR and for finding the b32 store path hash part, you’d have to do zero-out internal store path, etc, but you could avoid doing it all by hand or in C if you wanted.

i noticed that when comparing the hash of 2 identical symlinks as tar

[davidak@gaming:~]$ ln -s /usr/bin/env test

[davidak@gaming:~]$ ln -s /usr/bin/env test2

[davidak@gaming:~]$ sha1sum test
18601d953c96cb03cac354d054f321925a54a429  test

[davidak@gaming:~]$ sha1sum test2 
18601d953c96cb03cac354d054f321925a54a429  test2

[davidak@gaming:~]$ diff test test2


[davidak@gaming:~]$ tar -cf test.tar test

[davidak@gaming:~]$ tar -cf test2.tar test2

[davidak@gaming:~]$ sha1sum test.tar
a3f2c1a2fa053bac812d8ea3d86a2193f8321b20  test.tar

[davidak@gaming:~]$ sha1sum test2.tar
67f5e84c74ec9b325e286b5818466098bd9ed6e7  test2.tar

[davidak@gaming:~]$ nix run -f channel:nixos-20.09 diffoscope test.tar test2.tar 
--- test.tar
+++ test2.tar
├── file list
│ @@ -1 +1 @@
│ -lrwxrwxrwx   0 davidak   (1000) users      (100)        0 2021-05-26 14:53:39.000000 test -> /usr/bin/env
│ +lrwxrwxrwx   0 davidak   (1000) users      (100)        0 2021-05-26 14:53:36.000000 test2 -> /usr/bin/env

Is it possible to read CA derivations during evaluation? All my attempts at doing so turned out futile as the returned outPath is something like /0n3197jb6ymfim2whssvasnf978438q3dz0dg3imj1p5n6jqv2ny:

You can read the derivation itself, but you can indeed not access its output paths given that it’s not built yet. I [RFC 0092] Computed derivations by Ericson2314 · Pull Request #92 · NixOS/rfcs · GitHub will provide a CA-compatible IFD-like mechanism that could probably be used that way, but in the meantime that will indeed not work (though it looks like you could probably use recursive-nix here. I’m not sure that it works properly with CA derivations either ATM, but at least there’s no fundamental reason for it not to work)

1 Like

Yes! I have in fact talked to them about this feature being useful for that, too.

1 Like

Awesome! Btw thanks for all your work on portability and cross-compilation in Nix and Nixpkgs. Being able to freely mix natively and cross-compiled packages seems like it would really punctuate all that work reaching a new level, and I hope we get to see it happen some time soon

To check my understanding: could mixing native with cross-compiled packages help Nixpkgs porters when they find themselves in a situation where

  • some packages successfully build natively, but not cross
  • some packages successfully build cross-compiled, but not natively

in allowing the porters to short circuit some of those dependency chains, by trying to build packages on both platforms? Could that give someone who is trying to get Nixpkgs to build more and more stuff on a new platform more freedom about the order in which they fix things? Or should we expect native and cross building failures to mostly coincide?

Wondering if the feature possibly has more advantages than distributing compilation more easily, or for running build farms or whatever.

1 Like

Yes, I think it would help with that. I think managing the trust relation without lots of tools a la trustix might be a sort of high cognative load sysadim-y chore, but getting around things that just won’t cross compile could be quite useful.

Once we have the ability to mix and match separate trust relations on one machine, “semi-risky” experiments like grafting, cross-native mixing, etc. will be less scary. I do look forward to that.

2 Likes

I’ve been talking to @toonn as he is working on bootstrapping tools for macOS.

Once CA derivations are stable enough for him to get significant speedup of the feedback loop rebuilding everything, it would be good to know so he can give it a try.

2 Likes