Compile NixOS derivations into a single file for a single host

Help
1

Hello,

First thank you for this great community. I’m so happy about the vast resource the Nix project and this forum provides.

Finally, I got into the state that I wanted to get to since I started with NixOS. I have a single repo for different hosts, some of them are servers, some of them are desktops and I have a large modularized configuration that is assembled for a specific host by a lot of mkIfs. And it works :wink: .

I have currently two problems (good problems to have btw :wink: ):

  1. When I run nixos-rebuild, the first step “evaluating derivation” takes a lot of time on small cheap VM with a single shared CPU core. And that makes complete sense since the configuration is large and complicated.
  2. I have to ship the whole repo to each potentially untrusted (or at least less trusted) VM in the public cloud that I want to update and nixos-rebuild. Obviously, the repo contains details about my home network, my laptop etc. - which is not ideal. I can delete it after nixos-rebuild or copy it to tmpfs but that’s not the point.

So I wonder, is there a possibility to somehow compile the whole config into a single file with the resulting derivation for a given host? So I can copy only that single file and flake.nix and flake.lock? Or maybe just the single file?

I know that some DevOps tools for NixOS already exist. Tools that help you to manage your fleet in Push/Pull fashion (Chef, Puppet, Ansible style) and that would be probably the solution for my problem. But I don’t think I’m ready for such a tool. At my stage of Nix journey I’d prefer some simpler solution so that I can understand how it works and frankly, I don’t have a large number of computers anyway so I might never adopt a proper DevOps tool.

Please let me know.

Thank you.

3 Likes
2

You will get an unreadable and unmaintainable config if you put it in one file, there a reason why always no one does that, besides the fact that it’s then not modular.

3

nixos-rebuild can act a simple deployment tool itself using --target-host (and --build-host). This will offload evaluation to your local machine, building to the given build or local host and finally deploy the final result to the target host. You need to be a trusted user on the remote machine for this (or use root/sudo, see the nixos-rebuild(1) man page).

I’d also recommend to pass the --use-substitutes flag to prevent copying stuff from your local machine to the target host that could be downloaded from cache.nixos.org directly (since the VM likely has a better internet connection and ssh transfers are slow).

Note that this is not precisely what you wanted, I think you were asking for a way to somehow serialize the evaluation result into a single file and transfer it to the remote machine were you’d save on the evaluation work, but would be building the configuration. Serializing an evaluation result into a single file is not possible, but you can in principle copy the necessary derivation files using nix-copy-closure. Such a feature is not really provided by stock tools in a convenient way, but can be wired up with a custom script. Since building the configuration is also a pain on a low memory VM, I imagine you are better off with nixos-rebuild --target-host anyways.

4 Likes
4

Apologize, I probably didn’t explain it well enough. I don’t want to create the config manually. I want somehow to transform (compile) my config into a single file for a single host and then use that file only to change / update config for that host.

1 Like
5

Awesome. I’ll check this out. Thank you.

Yes. I used the term compilation but serialization makes more sense. I remember testing tool name nix-diff and if I remember correctly the tool shows you among other things the final derivation which is a huge blob of Nix language. Technically readable by a human but not really (like trying to read compiled javascript).

I’ll try that. I’d prefer the serialized solution since it is more practical for my simple use case since I might not always have a simple SSH access. But I didn’t know about the memory requirements which might force me to the first solution anyway.

Thank you.

6

You are root at the target machine, right? You can evaluate the main derivation for the system, do nix-store -qR to find out all of its dependencies (which are also derivations), then feed them all at once as arguments to nix-store --export, redirecting the output to my-system.closure.nar or something. nix-store --import with input from this file will add all the necessary derivations to the store, and then you can nix-store -r the main system derivation (and then use its activation script).

2 Likes
7

Yes.

That’s definitely interesting idea. I didn’t know that can be done. Thank you.

The only problem is that I’d have to apply the configuration first on the target machine and then I’d be able to compile it. My idea is more like the mentioned nixos-rebuild --target-host where I’d get the nar file (or whatever else) on the host that has the full configuration and then ship it to the target machine (via SSH or not) and then apply it.

8

Ah sure, you can also build the full configuration on the local machine, then look at its closure, nix-store --export the closure of the output instead of the closure of the derivations, and then activate it on the target after import.

Yet another approach: create a static-file binary cache locally, push the system closure there, tarball the entire cache, realise the known system output path on the target using the copied-over binary cache (by its new filesystem location, of course). This is messier but might be better integrated with the new CLI which still has glaring omissions in the functionality

9

Yes, as I mentioned yesterday in a different post:

nix build --no-link --print-out-paths .#nixosConfigurations."${myHost}".config.system.build.toplevel
10

I recently reorganized my configurations from a similar motivation:
I have one subdir per machine and a machine-agnostic top-level flake, which uses haumea to import whatever systems are present.

That way I can sync individual system configurations to the target systems and share the same top-level flake.{nix,lock}.

A useful aspect here is that nixos-rebuild --flake relaxes a bit on the flake definition. It only expects a flake.nix, not a Git-repo, so one can also sync with a variety of tools (rsync, meld, syncthing,…).

11

I don’t see anyone has addressed this, but I only took a quick glance, so apologies if it has already been mentioned, but your system’s derivation file is exactly that, a single file describing your system which has been “compiled” or “evaulated” from a Nix expression.

I take advantage of this fact to control the cost of Nix evaluation in a CI tool of mine by centralizing all evaluation to one host and simply distributing the resulting derivation files to an array of builder machines to begin their work immediately without any additional evaluation cost.

So yeah, you can simply evaluate the derivations locally, push them over to a remote system to actually build. One caveat though, nix copy currently does every copy synchronously which takes a millenia for any non-trivial derivation closure such as a NixOS system, so I used a hack to simply use nix-store --export to stream the derivations and their dependencies over a compressed ssh connection instead, which is much faster (maybe nix copy should just do that?).

But yeah, the short answer is, you can evaluate all your system derivations on your local host, and distribute the resulting drv files yourself to the host you wish to deploy them on. If you would like a concrete example of how I do that with nix-store --export, see this code snippet.

In general, I hope Nix gains some ability to make working with derivations more direct and user friendly eventually, since the derivation file format is essentially the equivalent of a Nix evaluation cache. If you have the final derivation file, you can outright skip eval all together. This is already somewhat the case with the new --store flag, but it could still use some improvement imo.

2 Likes
12

Instead of calling import and export manually and doing the transfer yourself, nix-copy-closure might be a good option as well.

13

Ouh, yeah, I use this feature since beginning. I always rsync the whole repo except .git/.

@iFreilicht @nrdxp @rudolf : Thank you very much.

I’m still a bit confused how to combine all the pieces. Let’s say I want to copy as little as possible and I want to use actual files (not tools that can communicate over SSH).

I start like this:

nix build --no-link --print-out-paths .#nixosConfigurations.foobar.config.system.build.toplevel

which gives me:

/nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba

then I use (instead of nix-store --export)

nix store dump-path /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba > foobar.nar

Which will create a small (less then MiB) file foobar.nar.

And now I can somehow import it and apply it on the “foobar” machine? Could you please tell me how?

Thank you.

14

That’s one of the omissions @7c6f434c was talking about; there’s no inverse for nix store dump-path.

I tried a few things:

I think it should be possible to just copy the derivation (not the output path) like so:

$ nix derivation show /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba > foobar.drv.json
# move that file to the other machine
$ cat foobar.drv.json | nix derivation add
$ nix-store --realise /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba

Unfortunately, when I tried this out on my machine with a simple package, the nix derivation add step would fail with an assertion error. After writing a full-on bug-report on this, I found out that you have to transform that output of derivation show to be able to pass it to derivation add:

nix derivation show /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba \
| jq '.[]' > foobar.drv.json

I also tried using nix-store --export and nix-store --import but that fails with a signature error.

The lowest-tech solution is also the one that works (here I’m using grep as an example):

$ nix-store --query --deriver /nix/store/rn59ig1q28l4i3cd1lzszkjmxpl6i1r0-gnugrep-3.7
/nix/store/y565239srnxvr85kw7kc61py1s3x982l-gnugrep-3.7.drv
$ cp /nix/store/y565239srnxvr85kw7kc61py1s3x982l-gnugrep-3.7.drv gnugrep-3.7.drv
# Now copy that derivation file to your other machine, in whatever way you want.
$ nix store add-file gnugrep-3.7.drv
/nix/store/2r4f3yx819gz1nwl27mx10ba2jszc23j-gnugrep-3.7.drv
$ nix-store --realise /nix/store/2r4f3yx819gz1nwl27mx10ba2jszc23j-gnugrep-3.7.drv
/nix/store/rn59ig1q28l4i3cd1lzszkjmxpl6i1r0-gnugrep-3.7

Note that the final output path is identical to the one we started with.

I think this should work perfectly fine in terms of getting all the packages. About how to activate it, I’m not sure you should just be able to call

$outPath/bin/switch-to-configuration

where $outPath is whatever nix-store --realise returned.

EDIT: Ok now after revising this answer multiple times I actually read your first post again and you specifically don’t want to evaluate the derivation again. I’m not sure if my solution will save you any time, but please try it out and report back.

If the evaluation still takes long, you can also copy the built configuration, all its dependencies are downloaded from the binary cache anyway.

So instead of using dump-path, you could just zip or tar everything inside the output path, move the zip file over to the other machine, and then use nix store add-path to add the whole directory to the nix store.

15

maybe not in the new cli, but there is still nix-store --import for importing a serialized nar or stream of nars.

16

Yes, but as I wrote in my post, I tried it and it failed with an error about a missing signature.

17

For me it complains when I try as a user, but works as root (preferably with NIX_REMOTE= nix-store --import just in case)

18

I tried

nix build --no-link --print-out-paths .#nixosConfigurations.foobar.config.system.build.toplevel
nix store dump-path /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba > a.nar
nix-store --export  /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba > b.nar

And I get error both with

nix-store --import < a.nar
nix-store --import < b.nar

The first is complaining about not being exported with --export and the second is complaining about a bad option.

19

Could you please show how the second is complaining? It should complain about missing deps;

nix-store -q -R  /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba |
xargs nix-store --export > c.nar

Should give an exported closure (although probably also with things that target already has), which should be importable (as root)

20

Now I’m getting different error:

error: path '/nix/store/1vb2wkzpzppj0177xpnhr8nrd3f9ijby-system-path' is not valid

I tried the solution you proposed but I quickly realized that I’m exporting everything → so I’d get many GiB to import. That is interesting command to keep in my tool belt but this doesn’t make sense for my current problem. I want to get a small file/a few files that represents compiled/serialized configuration. It should be smaller than my source, not larger.

Thank you @7c6f434c