Content-addressed Nix − call for testers

bbigras · July 15, 2021, 9:56pm

With Content-addressed nix, is dockerTools.buildLayeredImage supposed to create the same exact layers (except one) if I just change 1 line in my app’s source code?

Right now it seems to create different layers.

thufschmitt · July 16, 2021, 11:04am

I guess it generally should if things are properly reproducible. But I don’t really know how it works behind the scenes, so I can’t tell much

bbigras · July 16, 2021, 3:59pm

It seems created = "now";was causing my problem. If I remove it, only 2/15 layers are rebuilt/repushed.

thufschmitt · July 16, 2021, 6:00pm

That’s quite a source of non-determinism indeed

Ma27 · July 25, 2021, 11:05am

@bbigras @ethancedwards can you still reproduce the issue? Because I can’t

bbigras · July 27, 2021, 6:03am

Sorry for the delay, I just got back home.

you can test with GitHub - bbigras/test-ca

❯ nix build
trace: warning: crate2nix: Passing `buildRustCrate` as argument to Cargo.nix is deprecated. If you don't customize `buildRustCrate`, replace `callPackage ./Cargo.nix {}` by `import ./Cargo.nix { inherit pkgs; }`, and if you need to customize `buildRustCrate`, use `buildRustCrateForPkgs` instead.
error: hash mismatch importing path '/nix/store/szqzgsdclhyww9iq6gavs6v5pqns4hj2-libunistring-0.9.10';
         specified: sha256:1gcnvxyhh8g3jp2gl2d9zrmp5m3vhgddzc0yihq17iz8ar9hyb2f
         got:       sha256:1f4yx9iass21g6scn6q82132hhg8r6g4wl5mb4dpxsdbdw6xsm0i
error: some substitutes for the outputs of derivation '/nix/store/65ssx3nb9smafarki03cf2fs9fxha3xm-libidn2-2.3.1.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: 1 dependencies of derivation '/nix/store/dp3216f6ya3c1j9vqxng4y86dqg66b1z-systemd-minimal-247.6.drv' failed to build
error: some substitutes for the outputs of derivation '/nix/store/gfpd64m1dcsh8lfakygw7hxb04bjs4mm-bash-4.4-p23.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/hf8dah4qyx87dkrsbi094z1j8gadr90z-cmake-3.19.7.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/jxwfyb8yga5k0fb030wbc8sbky3h0j4j-coreutils-8.32.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/8587k4i6c6k1phs45fz85q40aqm97xsc-cpio-2.13.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/kj0q15gsh3izbbmvr8kr5inigcxqbzyx-expat-2.4.1.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/ipsqmhajjggy84i9b96kdr6y6kp1i3yy-file-5.39.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/3yk3zxplc08spjfs1nd3ala654rfc6wh-gawk-5.1.0.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/pm0j1dp32kq0mix9vgwx6wl1l2b49bpk-gettext-0.21.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/afmi8m5p2ddpji3c5y6iskvs1bdb19p2-gnugrep-3.6.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/8znmvlinddlfcbk835pq85p562790d6k-gnused-4.8.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/j260ngabsiv3ff649sz5ip8jvmpd92yj-gzip-1.10.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: 1 dependencies of derivation '/nix/store/78an02w7qa4995av7b4vnfxvrsinnwk2-libfido2-1.7.0.drv' failed to build
error: some substitutes for the outputs of derivation '/nix/store/wm8hmq1451lcw2n69nhllqhi7q23qvcg-libkrb5-1.18.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/7rz190v3nm13wzk9rkcy55yzw84x78cp-linux-pam-1.5.1.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/4s4dc21q4rdyby1kd9rhf2bnz858lvs3-onig-6.9.6.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/zf5k5zxaf0fhlwwmg52ax313d6pn0ylb-perl-5.34.0.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/078jccaqh3mqbkmlxz3r1azw04kjir84-perl5.34.0-FCGI-0.79.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/k8ibzsfi1lxm7r9bkzknjng2y10a58cd-perl5.34.0-HTML-Parser-3.75.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/67qld74bgkd3bchp3hjh05904j1rwq6l-perl5.34.0-TermReadKey-2.38.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/8c6h4571y641xgf9fmblxj6z3sb8sdrw-python3-3.9.5.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/9mpkg3fkppc45y907fxsj6p8hvk4ijvk-rustc-1.53.0.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/l4zcfybwg4zf9s6pjakwag8654b9ajvx-rustc-bootstrap-1.52.1.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: 1 dependencies of derivation '/nix/store/qm24w7izn6cla6839hxld15y622gy9gz-rust_test-ca-0.1.0.drv' failed to build

Ma27 · July 27, 2021, 12:04pm

Unfortunately I get a different error:

→ nix build --experimental-features 'nix-command flakes ca-derivations' -vvvvv
using revision a7287680171c8d8dd8470ef95850a8a02cf68bec of repo '/home/ma27/Projects/test-ca'
acquiring global GC lock '/nix/var/nix/gc.lock'
acquiring read lock on '/nix/var/nix/temproots/437134'
acquiring write lock on '/nix/var/nix/temproots/437134'
downgrading to read lock on '/nix/var/nix/temproots/437134'
using cache entry '{"name":"source","rev":"a7287680171c8d8dd8470ef95850a8a02cf68bec","type":"git"}' -> '{"lastModified":1627365569,"rev":"a7287680171c8d8dd8470ef95850a8a02cf68bec","revCount":1}', '/nix/store/iaf1i8i3a9ba2xllndbi3qhfgpv6b6hf-source'
got tree '/nix/store/iaf1i8i3a9ba2xllndbi3qhfgpv6b6hf-source' from 'git+file:///home/ma27/Projects/test-ca?ref=master&rev=a7287680171c8d8dd8470ef95850a8a02cf68bec'
checking access to '/nix/store/iaf1i8i3a9ba2xllndbi3qhfgpv6b6hf-source/flake.nix'
evaluating file '/nix/store/iaf1i8i3a9ba2xllndbi3qhfgpv6b6hf-source/flake.nix'
old lock file: {
  "nodes": {
    "flake-utils": {
      "locked": {
        "lastModified": 1623875721,
        "narHash": "sha256-A8BU7bjS5GirpAUv4QA+QnJ4CceLHkcXdRp4xITDB0s=",
        "owner": "numtide",
        "repo": "flake-utils",
        "rev": "f7e004a55b120c02ecb6219596820fcd32ca8772",
        "type": "github"
      },
      "original": {
        "owner": "numtide",
        "repo": "flake-utils",
        "type": "github"
      }
    },
    "nixpkgs": {
      "locked": {
        "lastModified": 1627358328,
        "narHash": "sha256-QYKM8Bxm2FlS4luhWyGy9L5fz10r5ZAnMwc/dfn+i40=",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "e22631727b4b1ee5426397a583be3ea107bed8d6",
        "type": "github"
      },
      "original": {
        "id": "nixpkgs",
        "type": "indirect"
      }
    },
    "root": {
      "inputs": {
        "flake-utils": "flake-utils",
        "nixpkgs": "nixpkgs"
      }
    }
  },
  "root": "root",
  "version": 7
}
computing lock file node ''
computing input 'flake-utils'
keeping existing input 'flake-utils'
computing lock file node 'flake-utils'
computing input 'nixpkgs'
keeping existing input 'nixpkgs'
computing lock file node 'nixpkgs'
new lock file: {
  "nodes": {
    "flake-utils": {
      "locked": {
        "lastModified": 1623875721,
        "narHash": "sha256-A8BU7bjS5GirpAUv4QA+QnJ4CceLHkcXdRp4xITDB0s=",
        "owner": "numtide",
        "repo": "flake-utils",
        "rev": "f7e004a55b120c02ecb6219596820fcd32ca8772",
        "type": "github"
      },
      "original": {
        "owner": "numtide",
        "repo": "flake-utils",
        "type": "github"
      }
    },
    "nixpkgs": {
      "locked": {
        "lastModified": 1627358328,
        "narHash": "sha256-QYKM8Bxm2FlS4luhWyGy9L5fz10r5ZAnMwc/dfn+i40=",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "e22631727b4b1ee5426397a583be3ea107bed8d6",
        "type": "github"
      },
      "original": {
        "id": "nixpkgs",
        "type": "indirect"
      }
    },
    "root": {
      "inputs": {
        "flake-utils": "flake-utils",
        "nixpkgs": "nixpkgs"
      }
    }
  },
  "root": "root",
  "version": 7
}
using cached string attribute 'defaultPackage.x86_64-linux.type'
using cached string attribute 'defaultPackage.x86_64-linux.drvPath'
using cached string attribute 'defaultPackage.x86_64-linux.outPath'
using cached string attribute 'defaultPackage.x86_64-linux.outputName'
building of '/nix/store/qm24w7izn6cla6839hxld15y622gy9gz-rust_test-ca-0.1.0.drv!out' from .drv file: created
building of '/nix/store/qm24w7izn6cla6839hxld15y622gy9gz-rust_test-ca-0.1.0.drv!out' from .drv file: woken up
querying info about missing paths...
starting pool of 7 threads
closing daemon connection because of an exception
error: unexpected end-of-file

I’m using

→ nix --version
nix (Nix) 2.4pre20210707_02dd6bb
→ nixos-version
21.05.20210724.e810832 (Okapi)
→ nix doctor -v
Running checks against store uri: daemon
[PASS] PATH contains only one nix version.
[PASS] All profiles are gcroots.
[PASS] Client protocol matches store protocol.

It seems as the nix-daemon is crashed with SIGABRT, I’m getting the following backtrace:

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49
#1  0x00007f209080d523 in __GI_abort () at abort.c:79
#2  0x00007f209080d41f in __assert_fail_base (fmt=0x7f209096ef60 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7f20911d09c0 "stmt.stmt", file=0x7f20911d09a9 "src/libstore/sqlite.cc", line=92,
    function=<optimized out>) at assert.c:92
#3  0x00007f209081bd92 in __GI___assert_fail (assertion=assertion@entry=0x7f20911d09c0 "stmt.stmt", file=file@entry=0x7f20911d09a9 "src/libstore/sqlite.cc", line=line@entry=92,
    function=function@entry=0x7f20911d0878 "nix::SQLiteStmt::Use::Use(nix::SQLiteStmt&)") at assert.c:101
#4  0x00007f2091180b18 in nix::SQLiteStmt::Use::Use (this=<optimized out>, stmt=...) at src/libstore/sqlite.cc:92
#5  nix::SQLiteStmt::Use::Use (this=<optimized out>, stmt=...) at src/libstore/sqlite.cc:89
#6  0x00007f20910f1efd in nix::SQLiteStmt::use (this=<optimized out>) at src/libstore/sqlite.hh:80
#7  nix::LocalStore::queryRealisationCore_ (this=0x9a5db0, state=..., id=...) at src/libstore/local-store.cc:1784
#8  0x00007f20910f2883 in nix::LocalStore::queryRealisation_ (this=0x9a5db0, state=..., id=...) at src/libstore/local-store.cc:1808
#9  0x00007f20910f5cef in operator() (__closure=<optimized out>) at src/libstore/local-store.cc:1838
#10 nix::retrySQLite<std::optional<const nix::Realisation>, nix::LocalStore::queryRealisation(const nix::DrvOutput&)::<lambda()> > (fun=...) at src/libstore/sqlite.hh:113
#11 nix::LocalStore::queryRealisation (this=this@entry=0x9a5db0, id=...) at src/libstore/local-store.cc:1839
#12 0x00007f20910fc232 in nix::LocalStore::queryPartialDerivationOutputMap[abi:cxx11](nix::StorePath const&) (this=0x9a5db0, path_=...) at src/libstore/local-store.cc:1012
#13 0x00007f209111a53f in operator() (__closure=0x7fff1351dee0, bfd=...) at src/libstore/misc.cc:181
#14 0x00007f209111b272 in std::__invoke_impl<void, nix::overloaded<nix::Store::queryMissing(const std::vector<nix::DerivedPath>&, nix::StorePathSet&, nix::StorePathSet&, nix::StorePathSet&, uint64_t&, uint64_t&)::<lambda(const nix::DerivedPath&)>::<lambda(nix::DerivedPath::Built)>, nix::Store::queryMissing(const std::vector<nix::DerivedPath>&, nix::StorePathSet&, nix::StorePathSet&, nix::StorePathSet&, uint64_t&, uint64_t&)::<lambda(const nix::DerivedPath&)>::<lambda(nix::DerivedPath::Opaque)> >, const nix::DerivedPathBuilt&> (__f=...) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:60
#15 std::__invoke<nix::overloaded<nix::Store::queryMissing(const std::vector<nix::DerivedPath>&, nix::StorePathSet&, nix::StorePathSet&, nix::StorePathSet&, uint64_t&, uint64_t&)::<lambda(const nix::DerivedPath&)>::<lambda(nix::DerivedPath::Built)>, nix::Store::queryMissing(const std::vector<nix::DerivedPath>&, nix::StorePathSet&, nix::StorePathSet&, nix::StorePathSet&, uint64_t&, uint64_t&)::<lambda(const nix::DerivedPath&)>::<lambda(nix::DerivedPath::Opaque)> >, const nix::DerivedPathBuilt&> (__fn=...) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:95
#16 std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<void> (*)(nix::overloaded<nix::Store::queryMissing(const std::vector<nix::DerivedPath>&, nix::StorePathSet&, nix::StorePathSet&, nix::StorePathSet&, uint64_t&, uint64_t&)::<lambda(const nix::DerivedPath&)>::<lambda(nix::DerivedPath::Built)>, nix::Store::queryMissing(const std::vector<nix::DerivedPath>&, nix::StorePathSet&, nix::StorePathSet&, nix::StorePathSet&, uint64_t&, uint64_t&)::<lambda(const nix::DerivedPath&)>::<lambda(nix::DerivedPath::Opaque)> >&&, const std::variant<nix::DerivedPathOpaque, nix::DerivedPathBuilt>&)>, std::integer_sequence<long unsigned int, 1> >::__visit_invoke(nix::overloaded<nix::Store::queryMissing(const std::vector<nix::DerivedPath>&, nix::StorePathSet&, nix::StorePathSet&, nix::StorePathSet&, uint64_t&, uint64_t&)::<lambda(const nix::DerivedPath&)>::<lambda(nix::DerivedPath::Built)>, nix::Store::queryMissing(const std::vector<nix::DerivedPath>&, nix::StorePathSet&, nix::StorePathSet&, nix::StorePathSet&, uint64_t&, uint64_t&)::<lambda(const nix::DerivedPath&)>::<lambda(nix::DerivedPath::Opaque)> > &&, const std::variant<nix::DerivedPathOpaque, nix::DerivedPathBuilt> &) (__visitor=...,
    __vars#0=std::variant<nix::DerivedPathOpaque, nix::DerivedPathBuilt> [index 1] = {...}) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/variant:1020
#17 0x00007f209111c596 in std::__do_visit<std::__detail::__variant::__deduce_visit_result<void>, nix::overloaded<nix::Store::queryMissing(const std::vector<nix::DerivedPath>&, nix::StorePathSet&, nix::StorePathSet&, nix::StorePathSet&, uint64_t&, uint64_t&)::<lambda(const nix::DerivedPath&)>::<lambda(nix::DerivedPath::Built)>, nix::Store::queryMissing(const std::vector<nix::DerivedPath>&, nix::StorePathSet&, nix::StorePathSet&, nix::StorePathSet&, uint64_t&, uint64_t&)::<lambda(const nix::DerivedPath&)>::<lambda(nix::DerivedPath::Opaque)> >, const std::variant<nix::DerivedPathOpaque, nix::DerivedPathBuilt>&> (__visitor=...)
    at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/variant:1702
#18 std::visit<nix::overloaded<nix::Store::queryMissing(const std::vector<nix::DerivedPath>&, nix::StorePathSet&, nix::StorePathSet&, nix::StorePathSet&, uint64_t&, uint64_t&)::<lambda(const nix::DerivedPath&)>::<lambda(nix::DerivedPath::Built)>, nix::Store::queryMissing(const std::vector<nix::DerivedPath>&, nix::StorePathSet&, nix::StorePathSet&, nix::StorePathSet&, uint64_t&, uint64_t&)::<lambda(const nix::DerivedPath&)>::<lambda(nix::DerivedPath::Opaque)> >, const std::variant<nix::DerivedPathOpaque, nix::DerivedPathBuilt>&> (__visitor=...)
    at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/variant:1718
#19 operator() (req=..., __closure=0x9ea6c0) at src/libstore/misc.cc:168
#20 std::__invoke_impl<void, nix::Store::queryMissing(const std::vector<nix::DerivedPath>&, nix::StorePathSet&, nix::StorePathSet&, nix::StorePathSet&, uint64_t&, uint64_t&)::<lambda(const nix::DerivedPath&)>&, nix::DerivedPath> (__f=...) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:60
#21 std::__invoke_r<void, nix::Store::queryMissing(const std::vector<nix::DerivedPath>&, nix::StorePathSet&, nix::StorePathSet&, nix::StorePathSet&, uint64_t&, uint64_t&)::<lambda(const nix::DerivedPath&)>&, nix::DerivedPath> (__fn=...) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:110
#22 std::_Function_handler<void(nix::DerivedPath), nix::Store::queryMissing(const std::vector<nix::DerivedPath>&, nix::StorePathSet&, nix::StorePathSet&, nix::StorePathSet&, uint64_t&, uint64_t&)::<lambda(const nix::DerivedPath&)> >::_M_invoke(const std::_Any_data &, nix::DerivedPath &&) (__functor=..., __args#0=...) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/std_function.h:291
#23 0x00007f209111f5e7 in std::function<void (nix::DerivedPath)>::operator()(nix::DerivedPath) const (__args#0=..., this=0x9ac330)
    at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/std_function.h:622
#24 std::__invoke_impl<void, std::function<void (nix::DerivedPath)>&, nix::DerivedPath&>(std::__invoke_other, std::function<void (nix::DerivedPath)>&, nix::DerivedPath&) (__f=...)
    at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:60
#25 std::__invoke<std::function<void (nix::DerivedPath)>&, nix::DerivedPath&>(std::function<void (nix::DerivedPath)>&, nix::DerivedPath&) (__fn=...)
    at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:95
#26 std::_Bind<std::function<void (nix::DerivedPath)> (nix::DerivedPath)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (__args=..., this=0x9ac330)
    at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/functional:416
#27 std::_Bind<std::function<void (nix::DerivedPath)> (nix::DerivedPath)>::operator()<, void>() (this=0x9ac330) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/functional:499
#28 std::__invoke_impl<void, std::_Bind<std::function<void (nix::DerivedPath)> (nix::DerivedPath)>&>(std::__invoke_other, std::_Bind<std::function<void (nix::DerivedPath)> (nix::DerivedPath)>&) (__f=...)
    at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:60
#29 std::__invoke_r<void, std::_Bind<std::function<void (nix::DerivedPath)> (nix::DerivedPath)>&>(std::_Bind<std::function<void (nix::DerivedPath)> (nix::DerivedPath)>&) (__fn=...)
    at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:110
#30 std::_Function_handler<void (), std::_Bind<std::function<void (nix::DerivedPath)> (nix::DerivedPath)> >::_M_invoke(std::_Any_data const&) (__functor=...)
    at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/std_function.h:291
#31 0x00007f2090e3389a in std::function<void ()>::operator()() const (this=0x7fff1351e090) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/std_function.h:622
#32 nix::ThreadPool::doWork (this=0x7fff1351e3b0, mainThread=<optimized out>) at src/libutil/thread-pool.cc:145
#33 0x00007f2090e35385 in nix::ThreadPool::process (this=this@entry=0x7fff1351e3b0) at src/libutil/thread-pool.cc:61
#34 0x00007f2091117f3d in nix::Store::queryMissing (this=<optimized out>, targets=..., willBuild_=std::set with 0 elements, willSubstitute_=std::set with 0 elements, unknown_=std::set with 0 elements,
    downloadSize_=<optimized out>, narSize_=@0x7fff1351e518: 0) at src/libstore/misc.cc:234
--Type <RET> for more, q to quit, c to continue without paging--c
#35 0x00007f209105d961 in nix::Worker::run (this=this@entry=0x7fff1351e790, _topGoals=std::set with 1 element = {...}) at src/libstore/build/worker.cc:244
#36 0x00007f209100ba2a in nix::Store::buildPaths (this=0x9a6730, reqs=std::vector of length 1, capacity 0 = {...}, buildMode=<optimized out>) at src/libstore/build/entry-points.cc:25
#37 0x00007f20910779eb in nix::daemon::performOp (op=<optimized out>, to=..., from=..., clientVersion=287, recursive=<optimized out>, trusted=nix::daemon::Trusted, store=..., logger=0x99e3f0) at src/libstore/daemon.cc:520
#38 nix::daemon::processConnection(nix::ref<nix::Store>, nix::FdSource&, nix::FdSink&, nix::daemon::TrustedFlag, nix::daemon::RecursiveFlag, std::function<void (nix::Store&)>) (store=..., from=..., to=..., trusted=trusted@entry=nix::daemon::Trusted, recursive=recursive@entry=nix::daemon::NotRecursive, authHook=...) at src/libstore/daemon.cc:989
#39 0x000000000052a55c in operator() (__closure=0x9ac6a0) at src/nix/daemon.cc:244
#40 std::__invoke_impl<void, daemonLoop()::<lambda()>&> (__f=...) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:60
#41 std::__invoke_r<void, daemonLoop()::<lambda()>&> (__fn=...) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:110
#42 std::_Function_handler<void(), daemonLoop()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/std_function.h:291
#43 0x00007f2090e521dc in std::function<void ()>::operator()() const (this=<optimized out>) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/std_function.h:622
#44 operator() (__closure=0x7fff1351f5b0) at src/libutil/util.cc:1011
#45 std::__invoke_impl<void, nix::startProcess(std::function<void()>, const nix::ProcessOptions&)::<lambda()>&> (__f=...) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:60
#46 std::__invoke_r<void, nix::startProcess(std::function<void()>, const nix::ProcessOptions&)::<lambda()>&> (__fn=...) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:110
#47 std::_Function_handler<void(), nix::startProcess(std::function<void()>, const nix::ProcessOptions&)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/std_function.h:291
#48 0x00007f2090e4da79 in std::function<void ()>::operator()() const (this=0x7fff1351f5b0) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/std_function.h:622
#49 nix::doFork(bool, std::function<void()>) (allowVfork=<optimized out>, fun=...) at src/libutil/util.cc:995
#50 0x00007f2090e52069 in nix::startProcess(std::function<void ()>, nix::ProcessOptions const&) (fun=..., options=...) at src/libutil/util.cc:1023
#51 0x000000000052b571 in daemonLoop () at src/nix/daemon.cc:225
#52 runDaemon (stdio=false) at src/nix/daemon.cc:308
#53 0x000000000052bd95 in main_nix_daemon (argc=<optimized out>, argv=<optimized out>) at src/nix/daemon.cc:329
#54 0x000000000058c3a5 in std::function<void (int, char**)>::operator()(int, char**) const (__args#1=0x7fff13520768, __args#0=2, this=0x7fff1351ff90) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/std_function.h:248
#55 nix::mainWrapped (argc=2, argv=0x7fff13520768) at src/nix/main.cc:222
#56 0x00007f209137b718 in std::function<void ()>::operator()() const (this=0x7fff13520630) at /nix/store/5qjycalzb9sqzvqg65kf5zimqwjabm9g-gcc-10.3.0/include/c++/10.3.0/bits/std_function.h:622
#57 nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) (programName="nix-daemon", fun=...) at src/libmain/shared.cc:329
#58 0x00000000004798b6 in main (argc=<optimized out>, argv=<optimized out>) at src/nix/main.cc:336

cc @thufschmitt

thufschmitt · July 27, 2021, 12:46pm

@Ma27 that looks like an instance of The Nix daemon crashes when started without `--experimental-features ca-derivations`, but a client tries to force it · Issue #5017 · NixOS/nix · GitHub . Do you have experimental-features = ca-derivations on your daemon’s nix.conf or did you just pass it to the client?

thufschmitt · July 27, 2021, 12:51pm

Oh, I think your issue comes from Make the Nar hash non modulo by thufschmitt · Pull Request #4849 · NixOS/nix · GitHub .

That’s quite a pain, but we had to change the hashing of CA paths with references, meaning that previously built paths are (partially) incompatible, including a subset of cache.ngi0.nixos.org . The easiest workaround is to rebuild it locally (nix build /nix/store/65ssx3nb9smafarki03cf2fs9fxha3xm-libidn2-2.3.1.drv --option substitute false) to get the right version on the store.

Ma27 · July 29, 2021, 8:18pm

@thufschmitt the issue I originally reported was of slightly different nature since I don’t have CA enabled and this was the case with newly built paths.

I think I understood what’s happening, but this isn’t really a part of Nix I know much about, but let me explain:

on the last vagrant-bump[1] I got another one of these infamous hash-mismatch errors when trying to build it remotely:

$ nix-build /nix/store/2nwv1fm5bs0a2sgb2a014gv7hgfaj3f7-ruby2.7.4-vagrant_cloud-3.0.5.drv -j0
this derivation will be built:
  /nix/store/2nwv1fm5bs0a2sgb2a014gv7hgfaj3f7-ruby2.7.4-vagrant_cloud-3.0.5.drv
building '/nix/store/2nwv1fm5bs0a2sgb2a014gv7hgfaj3f7-ruby2.7.4-vagrant_cloud-3.0.5.drv' on 'ssh-ng://builder-ng'...
copying 1 paths...
copying path '/nix/store/j3wsx0in5hgnl3sr91i8h19b74w5hcz5-vagrant_cloud-3.0.5.gem' to 'ssh-ng://builder-ng'...
error: ca hash mismatch importing path '/nix/store/j3wsx0in5hgnl3sr91i8h19b74w5hcz5-vagrant_cloud-3.0.5.gem';
         specified: sha256:0np0d8rjca130si5iaxasbqmfbbx4l3kd9mxdsa3p5mqiia7za3b
         got:       sha256:14iwaq2d9ai4x36w2vd4gv0zwpa9yj5nv8l6rxqfs5njd69jv90h
error: builder for '/nix/store/2nwv1fm5bs0a2sgb2a014gv7hgfaj3f7-ruby2.7.4-vagrant_cloud-3.0.5.drv' failed with exit code 1

The package is buildable when doing nix-build --option builders '', but I get this error (from the remote-side) when trying to build remotely using ssh-ng. I use nix (Nix) 2.4pre20210707_02dd6bb on both sides.

The 0np0d8rjca130si5iaxasbqmfbbx4l3kd9mxdsa3p5mqiia7za3b is the one that’s returned by nix-prefetch-url (which is always done by bundix) when trying to download the source URL (https://rubygems.org/gems/vagrant_cloud-3.0.5.gem).
When I run nix-hash over the store-path returned by nix-prefetch-url (/nix/store/j3wsx0in5hgnl3sr91i8h19b74w5hcz5-vagrant_cloud-3.0.5.gem[2]) I get the following results:
```
$ nix-hash /nix/store/j3wsx0in5hgnl3sr91i8h19b74w5hcz5-vagrant_cloud-3.0.5.gem  --base32 --type sha256
08xd2w1qm62mdsf61r7chw9lsy6aimn3ipwlzklkn0jqf76lkf2v

$ nix-hash /nix/store/j3wsx0in5hgnl3sr91i8h19b74w5hcz5-vagrant_cloud-3.0.5.gem  --base32 --type sha256 --flat
14iwaq2d9ai4x36w2vd4gv0zwpa9yj5nv8l6rxqfs5njd69jv90h
```
I’m not really sure where 08xd2w1qm62mdsf61r7chw9lsy6aimn3ipwlzklkn0jqf76lkf2v comes from, however the hash-mismatch returned by my remote-builder is because it seems to get the flat hash while trying to addToStore() the path.
When I look at the hash associated with the path, I get this:
```
$ nix show-derivation /nix/store/j3wsx0in5hgnl3sr91i8h19b74w5hcz5-vagrant_cloud-3.0.5.gem | jq '.[].env|[.outputHashAlgo,.outputHashMode,.outputHash]'
[
  "sha256",
  "flat",
  "0np0d8rjca130si5iaxasbqmfbbx4l3kd9mxdsa3p5mqiia7za3b"
]
```
I don’t think I entirely understood how the results from nix-prefetch-url and nix-hash (talking about 08xd2w1qm62mdsf61r7chw9lsy6aimn3ipwlzklkn0jqf76lkf2v vs. 0np0d8rjca130si5iaxasbqmfbbx4l3kd9mxdsa3p5mqiia7za3b) differ, but since the mismatch occurred because the remote-side evaluated to the flat store-path hash (14iwaq2d9ai4x36w2vd4gv0zwpa9yj5nv8l6rxqfs5njd69jv90h from nix-hash --flat vs. 0np0d8rjca130si5iaxasbqmfbbx4l3kd9mxdsa3p5mqiia7za3b from nix-prefetch-url) it seems to be as there’s a mixup between CA hash and flat NAR hash?

Does this help you to track the issue down a bit more? I’m very sorry (and quite frustrated due to that), but I still can’t provide a reliable way to reproduce this, so that’s it for now

[1] vagrant: 2.2.17 -> 2.2.18 by Ma27 · Pull Request #131879 · NixOS/nixpkgs · GitHub
[2] I downloaded it on machines in several countries today and the content was stable, so I won’t export the store-path here now.

thufschmitt · July 30, 2021, 5:02am

That’s a bit puzzling: On the first try I had a similar error (but with different hashes), and now everything works fine:

$ export NIX_REMOTE=$PWD/store
$ nix-prefetch-url https://rubygems.org/gems/vagrant_cloud-3.0.5.gem
path is '/nix/store/j3wsx0in5hgnl3sr91i8h19b74w5hcz5-vagrant_cloud-3.0.5.gem'
0np0d8rjca130si5iaxasbqmfbbx4l3kd9mxdsa3p5mqiia7za3b
$ nix-hash $PWD/store/nix/store/j3wsx0in5hgnl3sr91i8h19b74w5hcz5-vagrant_cloud-3.0.5.gem --flat --type sha256 --base32
0np0d8rjca130si5iaxasbqmfbbx4l3kd9mxdsa3p5mqiia7za3b
$ nix copy --to daemon /nix/store/j3wsx0in5hgnl3sr91i8h19b74w5hcz5-vagrant_cloud-3.0.5.gem -v
copying 1 paths...
copying path '/nix/store/j3wsx0in5hgnl3sr91i8h19b74w5hcz5-vagrant_cloud-3.0.5.gem' to 'daemon'...

And nix-hash --flat <(curl -o - https://rubygems.org/gems/vagrant_cloud-3.0.5.gem) --type sha256 --base32 does return the right hash (0np0d8…)

So I’m not sure what to think of it…

Ma27 · July 30, 2021, 9:35am

nix-hash --flat <(curl -o - https://rubygems.org/gems/vagrant_cloud-3.0.5.gem) --type sha256 --base32 returns 0np0d8rjca130si5iaxasbqmfbbx4l3kd9mxdsa3p5mqiia7za3b for me while nix-hash /nix/store/j3wsx0in5hgnl3sr91i8h19b74w5hcz5-vagrant_cloud-3.0.5.gem --base32 --type sha256 --flat gives me 14iwaq2d9ai4x36w2vd4gv0zwpa9yj5nv8l6rxqfs5njd69jv90h.

I exported /nix/store/j3wsx0in5hgnl3sr91i8h19b74w5hcz5-vagrant_cloud-3.0.5.gem via nix-store --export and shared it via https://wolke.mbosch.me/s/yxKPAgW32na7L8G.

When I do wget https://rubygems.org/gems/vagrant_cloud-3.0.5.gem -O foo and then diffoscope /nix/store/j3wsx0in5hgnl3sr91i8h19b74w5hcz5-vagrant_cloud-3.0.5.gem foo I get the following result: gist:674ba3a9cf70a007e7395d53919773c9 · GitHub

Perhaps @edolstra also wants to take a look?

bbigras · September 1, 2021, 3:07pm

Any easy way to clean the nix store of those “skipping suspicious writable file”?

thufschmitt · September 2, 2021, 8:07am

I don’t think there’s an easy way, short of manually remounting it rw and manually changing the permissions

r-burns · September 16, 2021, 3:50am

I’m at “Raider of the unknown” level using Nix via nix-user-chroot (i.e. single-user, no daemon).

Attempting to enter a nix-shell fails with an invalid bash path:

$ nix --version                                                                                                                                           
nix (Nix) 2.4pre20210908_3c56f62
$ nix-shell -p hello
error: executing shell '/0pw21223rxbqsmm0qmp3fa1v74vvdp36r5qj9ngp7gdymb7gi68k/bin/bash': No such file or directory

(Same for nix-shell -A hello.)

It looks like this is a placeholder path which still needs to be resolved into the actual content-addressed bash store path.

thufschmitt · September 16, 2021, 8:24am

Indeed, there’s a missing rewriting for bash in nix-build.cc (which doesn’t show-up in the tests because since nix-shell evaluates (import <nixpkgs> {}).bashInteractive, the bash derivation will only be CA if the “ambiant” nixpkgs config specifies contentAddressedByDefault = true;, which isn’t the case in the tests. But I can reproduce it with echo '{ contentAddressedByDefault = true; }' > config.nix && NIXPKGS_CONFIG=$PWD/config.nix nix-shell -p hello).

Do you mind opening a nix issue for that and tag me on it?

In the meantime, you can work around it by setting NIX_BUILD_SHELL=$(which bash)

Anderssorby · November 2, 2021, 2:02pm

I turned on ca-derivations today and it broke all my builds with the error error: unexpected end-of-file.

$ nix --version
nix (Nix) 2.4pre20211006_53e4794

bbjubjub · November 2, 2021, 10:27pm

Can you try updating to the 2.4 stable release? It just came out. Even if it doesn’t fix it, it’s useful to know if it’s present in the most recent version.

iago · November 10, 2021, 4:39am

Hello!
I’m having some problems setting this up. On level 0, the first command does not work for me, and I could not find a way to make it work. Following to the next levels also did not work.

I tried on multiple NixOS machines (all share the same base config, nix version 2.5pre20211007_844dd90) and on one Ubuntu machine with a standalone nix install (nix version 2.4). In all cases I got the exact same error below.

I added --keep-going to the command, otherwise it will only show one random mismatching hash out of those four.

% nix shell --keep-going \
  --trusted-public-keys '' \
  --substituters https://cache.ngi0.nixos.org/ \
  /nix/store/yvk5yl9fid0zlxqk1xvvzn787d8gbh00-emacs-27.2 \
  -c emacs --version
error: hash mismatch importing path '/nix/store/q5dfzj5jm03asw3lzq6l0m2ivr0miixs-libunistring-0.9.10';
         specified: sha256:1fl38plbsahjcnmpnrd6yqk8wzpw0wr6wxghyzm3wzg3ww571s0w
         got:       sha256:1k4iikm5r2zsl5g3sf9wd02f4qf61qn022cnp82j4dc67b1zxbpa
error: hash mismatch importing path '/nix/store/1q7055w9c91v6x1rfg7wa53g4xxnmp0m-xkeyboard-config-2.31';
         specified: sha256:1bn05dj6zr4qfn46i3cshaq6a4xqmwv6yi8wl3yvrrwxm27cqvym
         got:       sha256:1idhbkn86aspz0yzb9dhzw7qy0qlz3hicf3sjpmxdx7jag4djw5i
error: hash mismatch importing path '/nix/store/26fv0mkix1gpagavm3h5p4dwbsw13bc7-gsettings-desktop-schemas-3.38.0';
         specified: sha256:04jacbmnxhnyhxxy2gnlxm1k9n36sfidhy3q8khln3mxr74dl3wm
         got:       sha256:0ggim4fkhgvrnrrk4yvmsbgaa1v2r5bnrwj1cgbzw2rxmxc9p497
error: hash mismatch importing path '/nix/store/s8pnmymm9bgc2i3mlccf1k8h4qclpdab-iso-codes-4.6.0';
         specified: sha256:0v5asff60kzj7iis8pzd3p6qs96nhgsydv85f402wc3lc9ccd9lk
         got:       sha256:1ar03pcn9qnwjc79k215s35kj5g13pszl0hzqllh8pib3i9bxn1m
error: build of '/nix/store/yvk5yl9fid0zlxqk1xvvzn787d8gbh00-emacs-27.2' failed

I tried with a new emacs path too (ran this to get one

curl https://hydra.ngi0.nixos.org/jobset/ca-test/nixpkgs/evals -H 'Accept: application/json' | nix run nixpkgs#jq -- '.evals[0].builds[]' | xargs -I{} sh -c 'curl -s https://hydra.ngi0.nixos.org/build/{} -H "Accept: application/json" | nix run nixpkgs#jq -- -r ".buildproducts[] | .path"' | grep emacs

)

Still can’t build, but less mismatching hashes this time

% nix shell --keep-going \
  --trusted-public-keys '' \
  --substituters https://cache.ngi0.nixos.org/ \
  /nix/store/a7bz6vqwsqw6hp54g8ymp5b46w3h361f-emacs-27.2 \
  -c emacs --version
error: hash mismatch importing path '/nix/store/s8pnmymm9bgc2i3mlccf1k8h4qclpdab-iso-codes-4.6.0';
         specified: sha256:0v5asff60kzj7iis8pzd3p6qs96nhgsydv85f402wc3lc9ccd9lk
         got:       sha256:1ar03pcn9qnwjc79k215s35kj5g13pszl0hzqllh8pib3i9bxn1m
error: build of '/nix/store/a7bz6vqwsqw6hp54g8ymp5b46w3h361f-emacs-27.2' failed

thufschmitt · November 10, 2021, 10:55am

Ah yes, that’s because Use the hash modulo for the derivation outputs · Issue #4347 · NixOS/nix · GitHub changed the way hashes are computed for CA paths with references, but the binary cache still has some narinfo files with the old-style hash.

I’ve patched all the files in the closure of /nix/store/yvk5yl9fid0zlxqk1xvvzn787d8gbh00-emacs-27.2, so that one should now be good (modulo http caching, but I’ve no control over it and I’ve no idea how long it’s supposed to cache stuff )