Compile NixOS derivations into a single file for a single host

Help
21 
nix-store -q -R $( nix-store -q --deriver  /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba ) |
xargs nix-store --export > d.nar

then

1 Like
22

Yes, thank you! That worked!

I piped it through gzip:

| xargs nix-store --export | gzip >

and

gunzip --stdout /tmp/importme.nar.gz | nix-store --import and it imported a lot of /nix/store/...drv paths.

So now just to figure out how to nixos-rebuilt into that imported derivation.

23

First step is nix-store -r /nix/store/[…].drv

Then I think it is /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba/bin/switch-to-configuration switch. I haven’t actually run mainline NixOS in a long time, so I might be mixing up things; I hope you have some path to recover the VM if this messes up the boot…

24

I’ll test it on a new VM in cloud, so if it blows NixOS, not a big deal. Thank you :wink:

25

I’ve just run it. It took really long time to realize all those derivations but it did finish OK. Unfortunately when I run

find /nix/ -name switch-to-configuration

I get only the Nix store path that the current generation (/nix/var/nix/profiles/system) is pointing to.

26

What is the path you have built, where it comes from originally, and what is inside this path?

(I think you have obtained updated software builds, so further experiment should go faster)

27

Apologize for the late answer and thank you very much for putting up with me.

I tried the whole process again:

  • on fully configured VM
out=$(nix build --no-link --print-out-paths .#nixosConfigurations.foobar.config.system.build.toplevel)
  • out has /nix/store/2z6q5mvh95w4r0swq8b856awkyknn2f6-nixos-system-foobar-23.05.20230731.b7cde1c 
nix-store -q -R $(nix-store -q --deriver "${out}") | xargs nix-store --export | gzip > /tmp/export.nar.gz
ls -lh /tmp/export.nar.gz

-rw-r--r-- 1 root root 8.2M Aug  2 14:23 /tmp/export.nar.gz
  • on a VM that was created from almost unmodified configuration from nixos-generate-config (in other words a very basic system)
  • it will import a lot of /nix/store/*.drv 
gunzip --stdout /tmp/export.nar.gz | nix-store --import
  • now realize
nix-store -r /nix/store/2z6q5mvh95w4r0swq8b856awkyknn2f6-nixos-system-foobar-23.05.20230731.b7cde1c.drv

don't know how to build these paths:
  /nix/store/2z6q5mvh95w4r0swq8b856awkyknn2f6-nixos-system-foobar-23.05.20230731.b7cde1c.drv
error: cannot build missing derivation '/nix/store/2z6q5mvh95w4r0swq8b856awkyknn2f6-nixos-system-foobar-23.05.20230731.b7cde1c.drv'
  • is it missing? yes
ls -l /nix/store/*foobar*

ls: cannot access '/nix/store/*foobar*': No such file or directory
  • at this moment before any derivations were realized, the only switch-to-configuration is the one that created the initial simple configuration
find /nix/ -name switch-to-configuration

/nix/store/g6jzf8hqqpk6b3c6dz7yvzxfncarqs7s-nixos-system-nixos-23.05.997.ddf4688dc7a/bin/switch-to-configuration
28

And when you ran nix-store --import, did it print anything? It should print the paths imported (including nix/store/2z6q5mvh95w4r0swq8b856awkyknn2f6-nixos-system-foobar-23.05.20230731.b7cde1c.drv)

Also, nix-store -q --deriver "${out}" — was the shell environment variable out set? Your commands don’t show this. What this command prints on its own?

29

/nix/store/2z6q5mvh95w4r0swq8b856awkyknn2f6-nixos-system-foobar-23.05.20230731.b7cde1c

(the output of nix build --no-link --print-out-paths .#nixosConfigurations.foobar.config.system.build.toplevel)

Yes, many nix store paths, but not /nix/store/2z6q5mvh95w4r0swq8b856awkyknn2f6-nixos-system-foobar-23.05.20230731.b7cde1c

30

Can you check whether /nix/store/2z6q5mvh95w4r0swq8b856awkyknn2f6-nixos-system-foobar-23.05.20230731.b7cde1c was missing from the export (e.g. not in the tarball) or just not imported?
Does the number of imported derivations match the number of derivations in the tarball?

31

I’ll need to try again to build it / export it / import it - which I’m happy to do but…

This process of exporting tens of derivations is miles away from my initial goal which was to somehow serialize/compile/collapse a large repository with many files with configuration for many hosts into a single file/a few files for a single host. The GZIPed export has 8M which is more than my entire uncompressed source…

32

Yeah, I was already wondering about that. Have you tried the approach I mentioned above?

With this you would only pack up the flake.nix, flake.lock and the relevant subtree. That solves your trust problem.

Regarding complexity of evaluation: You have to decide which side to take in the trade-off. Push a small config to the target and evaluate on the target or pre-evaluate on a bigger machine and push a bigger intermediate result. If you can communicate with the the target machine you might be able to limit the export to missing store paths, potentially reducing the size of your tarball.

33

The suggestion to copy files, not the whole repo - I do that since beginning. Very early (probably my mistake) I realized that I can just copy files and I don’t need to copy .git/. Thank you for that suggestion.

I didn’t follow up about the suggestion using Haumea. When I started with Nix / NixOS I looked at Haumea, mkFlake, mkHome, Snowfall Lib and couple of others. I like the approach but I firmly decided to create my own naming / file structure. I like the simplicity of Nix language and I like that I can build my own framework-ish structure rather than using someone’s else idea.

I appreciate the suggestion but I really don’t like these framework-ish projects.

I decided that I’m OK if the target host does all the work and it takes a lot of time. I was slightly worried about memory consumption but I don’t have problem with that even on a cheap $5 VPS. The only problem is that it takes a minute or two to finish the evaluation but that’s OK with me even if the time increases in the future.

What I’m not OK is that I have to ship 100 files (and in future probably more) to the host.

Let’s take an example of a configuration that reads JSON (JSON is in the repo or it is in other repo with flake) that has an array with name, IP and a few options. The flake.nix would then map over the array and generate nixosConfigurations.foo, nixosConfigurations.bar… for each host with different name, IP and some configuration based on those few options in the JSON. That configuration would be included / excluded simply by mkIf.

Such solution can be easily implemented in Nix language and by using flakes and when I call nixos-rebuild --flake .#foo the Nix compiler has to have some intermediate steps where in some point it will work only on that part of code that works with nixosConfigurations.foo and that has mkIf true (not mkIf <some function of variable>).

The only thing I need is to get that code from that specific step of evaluation and save it as a blob of Nix language that doesn’t have JSON (already read by builtins.fromJSON), doesn’t have comments, doesn’t have packages."aarch64-linux".hello since I’m not on ARM and I’m not using hello package anywhere in the configuration…

But if this is not possible - that’s OK. I can’t have everything. I love Nix anyway :wink: .

Thank you.

34

Just so you know, because of all this trouble, I started a PR that will allow you to simply run nix derivation show -r and nix derivation add to transfer all the derivations required by the system configuration in one file and actually read it again.

As long as you can embed all info in derivations (meaning they don’t rely on input sources that you have to copy over manually), this will be the single file solution you’re looking for.

I also didn’t see you mention trying nix-copy-closure. I linked it above, and you can specify a file:// argument in --to and zip the resulting folder. (wrong, only nix copy can do that, and it’s not useful here, see Compile NixOS derivations into a single file for a single host - #48 by iFreilicht.

1 Like
35

I’m sorry for troubling you but I’m so happy that you actually want to change the core code to enable this feature. Thank you so much!

So I’m assuming I’ll be able to do something like

nix build --no-link --print-out-paths .#nixosConfigurations.foobar.config.system.build.toplevel

derivation show -r \
/nix/store/2z6q5mvh95w4r0swq8b856awkyknn2f6-nixos-system-foobar-23.05.20230731.b7cde1c.drv \
> something

and then

nix derivation add  < /tmp/something

/nix/store/2z6q5mvh95w4r0swq8b856awkyknn2f6-nixos-system-foobar-23.05.20230731.b7cde1c/bin/switch-to-configuration boot

Let me check it.

THANK YOU!

36

My pleasure! Nix has a lot of corners like this that you would expect to work but that actually don’t, and I really want to fix those. User-facing features like this are a great opportunity for myself to become more familiar with the code-base and should enable me to make more contributions in the future.

Yes, that’s the idea exactly. As you can see in the PR, I also added documentation for this. You can also try it out yourself already:

$ nix shell github:iFreilicht/nix/enable-derivation-show-add-roundtrip
# This will build nix from source off of my branch, it will take a few minutes
$ nix --version # Just to make sure it's using the right version
nix (Nix) 2.18.0pre20230731_57113a6
$ nix derivation show 'nixpkgs#hello' | nix derivation add # Sanity check
/nix/store/8rgfc52k0529kypam0jy5p1a4jsj4dbq-hello-2.12.1.drv

$ nix derivation show -r $system_foobar > something
$ nix derivation add < something

If this works, doesn’t work, or you have feedback on the docs, feel free to comment on the PR in GitHub. Getting feedback and real-life tests from users on changes like this is very valuable!

1 Like
37

Signing up for Github account now!

Hmm, I need to figure out how to do it. I’ll try

nix run github:nixos/nixpkgs/nixos-unstable#nix

38

No, you have to use the exact command that I posted:

nix shell github:iFreilicht/nix/enable-derivation-show-add-roundtrip

You can also use nix run, but that seems less convenient.

39

Just tried it, but the tests fail:

ran test tests/derivation-json.sh... [FAIL]
40

Huh, that’s not good, that means that test is flaky and potentially not reproducible. Could you give me the rest of the output? Ideally like so:

$ nix log github:iFreilicht/nix/enable-derivation-show-add-roundtrip | sed -n '/derivation-json.sh/,$p'
ran test tests/derivation-json.sh... [FAIL]
...