Compile NixOS derivations into a single file for a single host

Help
41

just the tail of nix log /nix/store/52nin4jf3g607hkbalywqz0vjcdg4z2s-nix-2.18.0pre20230731_57113a6.drv:

ran test tests/derivation-json.sh... [FAIL]
    +++(/build/source/tests/common/vars-and-functions.sh:274) trap onError ERR
    ++(common.sh:8) [[ -n '' ]]
    ++(derivation-json.sh:3) nix-instantiate simple.nix
    warning: you did not specify '--add-root'; the result might be removed by the garbage collector
    +(derivation-json.sh:3) drvPath=/build/nix-test/tests/derivation-json/store/7n22c0g55amv41d1n1xc4wcd286xzn16-simple.drv
    +(derivation-json.sh:6) nix derivation show /build/nix-test/tests/derivation-json/store/7n22c0g55amv41d1n1xc4wcd286xzn16-simple.drv
    warning: you don't have Internet access; disabling some network-dependent features
    warning: The interpretation of store paths arguments ending in `.drv` recently changed. If this command is now failing try again with '/build/nix-test/tests/derivation-json/store/7n22c0g55amv41d1n1xc4wcd286xzn16-simple.drv^*'
    ++(derivation-json.sh:7) nix derivation add
    warning: you don't have Internet access; disabling some network-dependent features
    +(derivation-json.sh:7) drvPath2=/build/nix-test/tests/derivation-json/store/7n22c0g55amv41d1n1xc4wcd286xzn16-simple.drv
    +(derivation-json.sh:8) [[ /build/nix-test/tests/derivation-json/store/7n22c0g55amv41d1n1xc4wcd286xzn16-simple.drv = \/\b\u\i\l\d\/\n\i\x\-\t\e\s\t\/\t\e\s\t\s\/\d\e\r\i\v\a\t\i\o\n\-\j\s\o\n\/\s\t\o\r\e\/\7\n\2\2\c\0\g\5\5\a\m\v\4\1\d\1\n\1\x\c\4\w>
    +(derivation-json.sh:11) cat /build/nix-test/tests/derivation-json/test-home/simple.json
    +(derivation-json.sh:11) jq '.[]'
    ++(derivation-json.sh:12) nix derivation add
    warning: you don't have Internet access; disabling some network-dependent features
    +(derivation-json.sh:12) drvPath3=/build/nix-test/tests/derivation-json/store/7n22c0g55amv41d1n1xc4wcd286xzn16-simple.drv
    +(derivation-json.sh:13) [[ /build/nix-test/tests/derivation-json/store/7n22c0g55amv41d1n1xc4wcd286xzn16-simple.drv = \/\b\u\i\l\d\/\n\i\x\-\t\e\s\t\/\t\e\s\t\s\/\d\e\r\i\v\a\t\i\o\n\-\j\s\o\n\/\s\t\o\r\e\/\7\n\2\2\c\0\g\5\5\a\m\v\4\1\d\1\n\1\x\c\4\>
    +(derivation-json.sh:16) cat /build/nix-test/tests/derivation-json/test-home/simple-unwrapped.json
    +(derivation-json.sh:16) grepQuiet 'has incorrect output'
    +(/build/source/tests/common/vars-and-functions.sh:266) grep 'has incorrect output'
    +(derivation-json.sh:16) jq '.name = "foo"'
    +(derivation-json.sh:16) expectStderr 1 nix derivation add
    +(/build/source/tests/common/vars-and-functions.sh:207) local expected res
    +(/build/source/tests/common/vars-and-functions.sh:208) expected=1
    +(/build/source/tests/common/vars-and-functions.sh:209) shift
    +(/build/source/tests/common/vars-and-functions.sh:210) nix derivation add
    +(/build/source/tests/common/vars-and-functions.sh:210) res=1
    +(/build/source/tests/common/vars-and-functions.sh:211) [[ 1 -ne 1 ]]
    +(/build/source/tests/common/vars-and-functions.sh:215) return 0
    ++(derivation-json.sh:19) nix-instantiate dependencies.nix
    warning: you did not specify '--add-root'; the result might be removed by the garbage collector
    +(derivation-json.sh:19) topDrvPath=/build/nix-test/tests/derivation-json/store/b08nmxkskc7vdj1phpngsn1rdnldqhyh-dependencies-top.drv
    +(derivation-json.sh:21) nix derivation show -r /build/nix-test/tests/derivation-json/store/b08nmxkskc7vdj1phpngsn1rdnldqhyh-dependencies-top.drv
    warning: you don't have Internet access; disabling some network-dependent features
    warning: The interpretation of store paths arguments ending in `.drv` recently changed. If this command is now failing try again with '/build/nix-test/tests/derivation-json/store/b08nmxkskc7vdj1phpngsn1rdnldqhyh-dependencies-top.drv^*'
    ++(derivation-json.sh:22) jq length /build/nix-test/tests/derivation-json/test-home/depend.json
    +(derivation-json.sh:22) [[ 4 = 4 ]]
    ++(derivation-json.sh:24) nix derivation add
    warning: you don't have Internet access; disabling some network-dependent features
    +(derivation-json.sh:24) dependPaths='/build/nix-test/tests/derivation-json/store/b08nmxkskc7vdj1phpngsn1rdnldqhyh-dependencies-top.drv
    /build/nix-test/tests/derivation-json/store/ngih9qlp3x67s455msagh3drx59pk0wa-dependencies-input-1.drv
    /build/nix-test/tests/derivation-json/store/xk3b03x2yrqyamvzz53xf315h0kgd3qf-dependencies-input-2.drv
    /build/nix-test/tests/derivation-json/store/z1a9k6a6646fqb97wdc9vkym0bf4573b-dependencies-input-0.drv'
    ++(derivation-json.sh:25) jq -r 'keys | .[]' /build/nix-test/tests/derivation-json/test-home/depend.json
    +(derivation-json.sh:25) [[ /build/nix-test/tests/derivation-json/store/b08nmxkskc7vdj1phpngsn1rdnldqhyh-dependencies-top.drv
    /build/nix-test/tests/derivation-json/store/ngih9qlp3x67s455msagh3drx59pk0wa-dependencies-input-1.drv
    /build/nix-test/tests/derivation-json/store/xk3b03x2yrqyamvzz53xf315h0kgd3qf-dependencies-input-2.drv
    /build/nix-test/tests/derivation-json/store/z1a9k6a6646fqb97wdc9vkym0bf4573b-dependencies-input-0.drv = \/\b\u\i\l\d\/\n\i\x\-\t\e\s\t\/\t\e\s\t\s\/\d\e\r\i\v\a\t\i\o\n\-\j\s\o\n\/\s\t\o\r\e\/\b\0\8\n\m\x\k\s\k\c\7\v\d\j\1\p\h\p\n\g\s\n\1\r\d\n\l\d\>
    \/\b\u\i\l\d\/\n\i\x\-\t\e\s\t\/\t\e\s\t\s\/\d\e\r\i\v\a\t\i\o\n\-\j\s\o\n\/\s\t\o\r\e\/\n\g\i\h\9\q\l\p\3\x\6\7\s\4\5\5\m\s\a\g\h\3\d\r\x\5\9\p\k\0\w\a\-\d\e\p\e\n\d\e\n\c\i\e\s\-\i\n\p\u\t\-\1\.\d\r\v\
    \/\b\u\i\l\d\/\n\i\x\-\t\e\s\t\/\t\e\s\t\s\/\d\e\r\i\v\a\t\i\o\n\-\j\s\o\n\/\s\t\o\r\e\/\x\k\3\b\0\3\x\2\y\r\q\y\a\m\v\z\z\5\3\x\f\3\1\5\h\0\k\g\d\3\q\f\-\d\e\p\e\n\d\e\n\c\i\e\s\-\i\n\p\u\t\-\2\.\d\r\v\
    \/\b\u\i\l\d\/\n\i\x\-\t\e\s\t\/\t\e\s\t\s\/\d\e\r\i\v\a\t\i\o\n\-\j\s\o\n\/\s\t\o\r\e\/\z\1\a\9\k\6\a\6\6\4\6\f\q\b\9\7\w\d\c\9\v\k\y\m\0\b\f\4\5\7\3\b\-\d\e\p\e\n\d\e\n\c\i\e\s\-\i\n\p\u\t\-\0\.\d\r\v ]]
    +(derivation-json.sh:28) diff -u /dev/fd/63 /dev/fd/62
    ++(derivation-json.sh:29) nix derivation add
    ++(derivation-json.sh:29) jq 'map_values( .outputs = { out: .env.out } )'
    ++(derivation-json.sh:29) cat /build/nix-test/tests/derivation-json/test-home/simple.json
    ++(derivation-json.sh:28) cat
    ++(derivation-json.sh:29) true
    --- /dev/fd/63      2023-08-04 06:50:52.378167776 +0000
    +++ /dev/fd/62      2023-08-04 06:50:52.378167776 +0000
    @@ -1,4 +1,3 @@
    -warning: you don't have Internet access; disabling some network-dependent features
     error:
            … while adding JSON derivation with key '/build/nix-test/tests/derivation-json/store/7n22c0g55amv41d1n1xc4wcd286xzn16-simple.drv'
     
    ++(derivation-json.sh:28) onError
    ++(/build/source/tests/common/vars-and-functions.sh:237) set +x
    derivation-json.sh: test failed at:
      main in derivation-json.sh:28
make: *** [mk/lib.mk:125: tests/derivation-json.sh.test] Error 1
make: *** Waiting for unfinished jobs....
ran test tests/db-migration.sh... [SKIP]
ran test tests/output-normalization.sh... [PASS]
ran test tests/bash-profile.sh... [PASS]
ran test tests/selfref-gc.sh... [PASS]
ran test tests/pass-as-file.sh... [PASS]
ran test tests/build.sh... [PASS]
ran test tests/build-delete.sh... [PASS]
42

Oh, I guess that’s caused by sandboxing which isn’t enabled on macOS. I pushed a fix and it works in the ubuntu CI pipeline now, could you try again?

43

I tried to use it:

  • on a new NixOS EC2
nix --extra-experimental-features flakes --extra-experimental-features nix-command run nixpkgs#tmux
nix --extra-experimental-features flakes --extra-experimental-features nix-command build --no-link --print-out-paths .#nixosConfigurations.foobar.config.system.build.toplevel

/nix/store/i3z6frs2x781rgigi06437smchrl7g13-nixos-system-foobar-23.05.20230731.b7cde1c

nix --extra-experimental-features flakes --extra-experimental-features nix-command shell github:iFreilicht/nix/enable-derivation-show-add-roundtrip
nix --extra-experimental-features flakes --extra-experimental-features nix-command derivation show -r \
/nix/store/i3z6frs2x781rgigi06437smchrl7g13-nixos-system-foobar-23.05.20230731.b7cde1c \
| gzip > something.gz
ls -lh something.gz

-rw-r--r-- 1 root root 6.0M Aug  4 10:49 something.gz
  • and then on completely different VM
nix shell github:iFreilicht/nix/enable-derivation-show-add-roundtrip
nix derivation add < /tmp/something.gz

error:
       … while adding JSON derivation with key '/nix/store/006qs3vdsv79s13671xl5axmk1iziakl-kded-5.106.0.drv'

       error: opening file '/nix/store/0vjkczq30x59l8l8is3cn1p9ryckmmkk-gsettings-desktop-schemas-44.0.drv': No such file or directory
44

I also tested on Ubuntu. Works now.

1 Like
45

I assume you mean something like gunzip < /tmp/something.gz | nix derivation add, right? Reading gzipped files directly is not supported.

Alright, so we came much farther this time! Would it be possible for you to send me the something.gz file? I suspect this error may be caused by the derivations not being in the expected order inside of it, though that error should definitely be recoverable.

46

As nix derivation show is just a thin wrapper around computeFSClosure in libstore, the latter should probably return a (reverse) topological sorting of derivations.

47

Yes, sorry, I didn’t actually copy the command.

Yes, thank you.

48

I looked at the file, and indeed, the derivations are in the wrong order. However, it’s not as easy as just doing a jq '. | to_entries | reverse | from_entries' on the file to reverse the order :see_no_evil:

Right now, this file is generated by starting with the top derivation, serializing that to json, and then going to through the input derivations of it in order and doing the same thing recursively, so it’s not guaranteed that doing this in reverse will work.

I could even replicate this in the tests now, so that’s something. So, this statement seems to be very much correct:

@mark.c I also looked more at nix-copy-closure and nix copy, and both don’t quite fit your usecase. nix-copy-closure doesn’t support writing to a binary cache, and nix copy does support that, but has no option for only copying derivations unless the target is reachable via SSH :expressionless:

49

Yeah, that was my conclusion too. Thank you for looking into that.

50

@mark.c I finished a new version of the branch and pushed it. nix derivation add should be able to re-order the derivations so they can all be added.

It’s not a great solution right now, but it’s good enough to see if there’s any problems. You can just run the same nix run command I posted above, it should automatically detect that there’s a new version on my branch and build it.

51

There was an issue I found while improving the code quality. If you tried it yesterday, give it another shot now. Everything is finished, and derivation add will now also try to substitute inputs to prevent avoidable errors if your installation is very bare-bones or you have a different version of nixpkgs on the machine you ran derivation show on.

52

Thank you very much for your excellent work.

Building now via: nix shell github:iFreilicht/nix/enable-derivation-show-add-roundtrip on fresh VMs

53
  • a blank NixOS EC2
nix build --no-link --print-out-paths .#nixosConfigurations.foobar.config.system.build.toplevel

/nix/store/hbwb7jyvd91iqz6f8g7bfddq3xhjiss6-nixos-system-foobar-23.05.20230805.9607b91

nix shell github:iFreilicht/nix/enable-derivation-show-add-roundtrip
nix --version

nix (Nix) 2.18.0pre20230807_4bd73c5

nix derivation show -r \
    /nix/store/hbwb7jyvd91iqz6f8g7bfddq3xhjiss6-nixos-system-foobar-23.05.20230805.9607b91 \
    | gzip > /tmp/something.gz
  • different VPS with the most simple configuration
nix shell github:iFreilicht/nix/enable-derivation-show-add-roundtrip
nix --version

nix (Nix) 2.18.0pre20230807_4bd73c5

gunzip --stdout /tmp/something.gz | nix derivation add

...
       'zzxmpjwf4794ic79av3wcpra27ih3ksy-libbacktrace-unstable-2022-12-16.drv' requires the input 'fqpxlv5wdhlbq9jdb5sa5xzs26gi5q3h-stdenv-linux.drv' but it could not be instantiated
       'zzxmpjwf4794ic79av3wcpra27ih3ksy-libbacktrace-unstable-2022-12-16.drv' requires the input 'gm52x6f6759sqvqczwrqr18524xglvyj-source.drv' but it could not be instantiated
       'zzxmpjwf4794ic79av3wcpra27ih3ksy-libbacktrace-unstable-2022-12-16.drv' requires the input 'm17fm11328r5d1h6mnsa8cl6x8bw7sab-bash-5.2-p15.drv' but it could not be instantiated
       'zzxmpjwf4794ic79av3wcpra27ih3ksy-libbacktrace-unstable-2022-12-16.drv' requires the input 'y4wpj23xx7w6xqpsjs3fkzpb7b0f2cjq-autoreconf-hook.drv' but it could not be instantiated
       'zzxmpjwf4794ic79av3wcpra27ih3ksy-libbacktrace-unstable-2022-12-16.drv' requires the input '6rap50ch2lynn8ar4w7bwyah9s7c6iwm-0001-libbacktrace-avoid-libtool-wrapping-tests.patch' but it could not be instantiated
       'zzxmpjwf4794ic79av3wcpra27ih3ksy-libbacktrace-unstable-2022-12-16.drv' requires the input 'gk481mqibmnlrqx414vfbxnfp4gm1zs8-0004-libbacktrace-Support-NIX_DEBUG_INFO_DIRS-environment.patch' but it could not be instantiated
       'zzxmpjwf4794ic79av3wcpra27ih3ksy-libbacktrace-unstable-2022-12-16.drv' requires the input 'k10a77kfqvgi8zi1w9q07lcbc9k10vhn-0002-libbacktrace-Allow-configuring-debug-dir.patch' but it could not be instantiated
       'zzxmpjwf4794ic79av3wcpra27ih3ksy-libbacktrace-unstable-2022-12-16.drv' requires the input 'lhym1d5wk48mxxyxh98mhlz4m7cc1qmr-0003-libbacktrace-Support-multiple-build-id-directories.patch' but it could not be instantiated

echo $?

1
54

Yeah that makes sense, stdenv-linux is also missing input sources, just like in this line:

Hmmm. That’s problematic. You see, something like this .patch file is an input source, not a derivation, and so it is not part of the output of derivation show -r. This means that if a single one of the derivations in the input depends on a source file that isn’t already in the nix store, derivation add will fail. I didn’t think this would be an issue, but should’ve known better.

This is not fixable. In a case like this, nix derivation is simply the wrong tool for the job. I did modify the code however to ensure all input sources are present first. I pushed it now, so you can try it out, but all it will do is give you a shorter and more useful error message.

So, this PR won’t be useful for your usecase, though I think it could still be useful generally.

The real solution here is to improve nix copy. Specifically, there needs to be something like an --inputs-only flag that allows skipping the outputs to give you a small file. I’ll write up a feature ticket tomorrow.

55

Thank you very much.

Just to understand… Does it mean that all the following use cases break the nix derivation?

  1. If I write a shell script and install it in environment.systemPackages
  2. If I use out-of-tree Kernel module or ZFS module?
  3. Using repo metadata or other “impurities” in the configuration (e.g. to record a Git commit id of the Flake revision the system was built from)

Googling that term, is that a description of functionality to add?

56

Yes, in the sense that these make nix derivation insufficient for your usecase. Though to be honest, I don’t think it was ever intended for this. Both commands are more meant for external tools to interact with the nix store, just like store add-path. derivation add in particular was only added in release 2.15.1.

Also, as we just saw, even if you build a very barebones configuration, derivation add is not sufficient, as the stdenv itself requires some source files.

Exactly. Tickets are units of work that programmers can work on independently. If you find a bug, you create a bug ticket, if you want to propose a feature, you create a feature ticket. People use different terms for this, on GitHub they’re called “Issues”, but “ticket” is understood by most. If you take a look at the Nix issue tracker, you can see there’s over 2000 bugs and feature requests open right now. Having them listed there helps a lot with prioritizing and organizing who works on what.
Independent programmers like me, who are not part of the core team, can look at lists like this one to see what to best work on next.

57

So it’s feature request, thank you :wink:

58

Alright, finished the feature request. Please have a look and let me know if that sounds good to you. I don’t know when I can start working on this yet, but it’s on my radar.

59

Sounds great. I commented to get some insights / help with documentation: Add `--inputs-only` to `nix copy` · Issue #8806 · NixOS/nix · GitHub

Thank you very much!

60

Alright, so after a while, one of the Nix maintainers commented on the issue:

You can nix copy a derivation and its closure.

I was under the impression that NixOS configurations were built like profiles, without a deriver, but that is not the case.

I think this confusion came from this post of yours:

This is incorrect! --query --deriver always outputs a derivation, not the output. This is why it’s important to never copy-paste the output of commands without the command you actually ran.

Full instructions

I’ve set up my own NixOS server in the meantime, so now I went through all the steps myself, and this is what worked for me:

Preparation

Make sure that nix --version returns at least 2.15.1. Older version might work, but we know for sure that 2.13.3 and 2.13.5 do not!

Building

First, build the system configuration:

$ nixos-rebuild build --flake path/to/flake
building the system configuration...
$ readlink result
/nix/store/r8y8a8gckycx95xdw8wnw1vq5dc13bll-nixos-system-foobar-23.11.20230812.f045184
$ nix-store -q --deriver result
/nix/store/gccqy8z0f78ip7kc4h8s87bb5khg08w0-nixos-system-foobar-23.11.20230812.f045184.drv

Remember the result of readlink result, that’s what we’re trying to replicate!

Copying to a local binary cache

Note the resulting path of nix-store -q --deriver is a derivation, and we can nix copy that quite easily:

$ configDrv=/nix/store/gccqy8z0f78ip7kc4h8s87bb5khg08w0-nixos-system-foobar-23.11.20230812.f045184.drv
$ nix copy $configDrv --to file:///tmp/configCache

Now, don’t be scared, it will look like this command copies like over 10GB of files, but actually, the binary cache we wrote to is pretty small, especially after compressing:

$ cd /tmp
$ du -sh configCache
20M	configCache
$ zip -r config.zip configCache
  adding: configCache/ (stored 0%)
  [ ... ]
$ du -sh config.zip
5.6M	config.zip
$ tar -czvf config.tar.gz configCache
configCache/
[...]
$ du -sh config.tar.gz
4.8M	config.tar.gz

To confirm the derivation has been copied properly, you can run this command:

$ nix store ls --store file:///tmp/configCache $configDrv
gccqy8z0f78ip7kc4h8s87bb5khg08w0-nixos-system-foobar-23.11.20230812.f045184.drv

Moving and recreating the config on another machine

Copy the file to my other machine (to avoid confusion, I copied it to the same path /tmp/config.tar.gz), uncompress it, and nix copy it to the local store:

$ cd /tmp
$ tar -xvzf config.tar.gz
x configCache/
[...]
$ configDrv=/nix/store/gccqy8z0f78ip7kc4h8s87bb5khg08w0-nixos-system-foobar-23.11.20230812.f045184.drv
$ nix copy $configDrv --from file:///tmp/configCache

Now, you can build the configuration like so, and check the result.

$ nix build "${configDrv}^*"
$ readlink result
/nix/store/r8y8a8gckycx95xdw8wnw1vq5dc13bll-nixos-system-foobar-23.11.20230812.f045184

This should be the exact same as when we ran readlink result after nixos-rebuild on the original machine.

Activating the config

Finally, activating the configuration is as easy as:

$ ./result/activate

Let me know if this works for you!

Potential caveats

BTW, I found that running the nix copy command like this:

$ nix copy $(nix-store -q --deriver result) --to file:///tmp/configCache

Did not work. I’m not entirely sure why, maybe I did something wrong, but there doesn’t seem to be an obvious problem with this.

5 Likes