Compile NixOS derivations into a single file for a single host

[mark.c]

@sternenseemann

nixos-rebuild can act a simple deployment tool itself

Awesome. I’ll check this out. Thank you.

@sternenseemann

serialize the evaluation result into a single file and transfer it to the remote machine

Yes. I used the term compilation but serialization makes more sense. I remember testing tool name nix-diff and if I remember correctly the tool shows you among other things the final derivation which is a huge blob of Nix language. Technically readable by a human but not really (like trying to read compiled javascript).

@sternenseemann

but you can in principle copy the necessary derivation files using nix-copy-closure. Such a feature is not really provided by stock tools in a convenient way, but can be wired up with a custom script.

I’ll try that. I’d prefer the serialized solution since it is more practical for my simple use case since I might not always have a simple SSH access. But I didn’t know about the memory requirements which might force me to the first solution anyway.

Thank you.

[7c6f434c]

You are root at the target machine, right? You can evaluate the main derivation for the system, do nix-store -qR to find out all of its dependencies (which are also derivations), then feed them all at once as arguments to nix-store --export, redirecting the output to my-system.closure.nar or something. nix-store --import with input from this file will add all the necessary derivations to the store, and then you can nix-store -r the main system derivation (and then use its activation script).

[mark.c]

@7c6f434c

You are root at the target machine, right?

Yes.

That’s definitely interesting idea. I didn’t know that can be done. Thank you.

The only problem is that I’d have to apply the configuration first on the target machine and then I’d be able to compile it. My idea is more like the mentioned nixos-rebuild --target-host where I’d get the nar file (or whatever else) on the host that has the full configuration and then ship it to the target machine (via SSH or not) and then apply it.

[7c6f434c]

Ah sure, you can also build the full configuration on the local machine, then look at its closure, nix-store --export the closure of the output instead of the closure of the derivations, and then activate it on the target after import.

Yet another approach: create a static-file binary cache locally, push the system closure there, tarball the entire cache, realise the known system output path on the target using the copied-over binary cache (by its new filesystem location, of course). This is messier but might be better integrated with the new CLI which still has glaring omissions in the functionality

[rudolf]

@7c6f434c

you can also build the full configuration on the local machine

Yes, as I mentioned yesterday in a different post:

nix build --no-link --print-out-paths .#nixosConfigurations."${myHost}".config.system.build.toplevel

[wamserma]

@mark.c

2. I have to ship the whole repo to each potentially untrusted (or at least less trusted) VM in the public cloud that I want to update and nixos-rebuild. Obviously, the repo contains details about my home network, my laptop etc. - which is not ideal. I can delete it after nixos-rebuild or copy it to tmpfs but that’s not the point.

I recently reorganized my configurations from a similar motivation:
I have one subdir per machine and a machine-agnostic top-level flake, which uses haumea to import whatever systems are present.

That way I can sync individual system configurations to the target systems and share the same top-level flake.{nix,lock}.

A useful aspect here is that nixos-rebuild --flake relaxes a bit on the flake definition. It only expects a flake.nix, not a Git-repo, so one can also sync with a variety of tools (rsync, meld, syncthing,…).

[nrdxp]

I don’t see anyone has addressed this, but I only took a quick glance, so apologies if it has already been mentioned, but your system’s derivation file is exactly that, a single file describing your system which has been “compiled” or “evaulated” from a Nix expression.

I take advantage of this fact to control the cost of Nix evaluation in a CI tool of mine by centralizing all evaluation to one host and simply distributing the resulting derivation files to an array of builder machines to begin their work immediately without any additional evaluation cost.

So yeah, you can simply evaluate the derivations locally, push them over to a remote system to actually build. One caveat though, nix copy currently does every copy synchronously which takes a millenia for any non-trivial derivation closure such as a NixOS system, so I used a hack to simply use nix-store --export to stream the derivations and their dependencies over a compressed ssh connection instead, which is much faster (maybe nix copy should just do that?).

But yeah, the short answer is, you can evaluate all your system derivations on your local host, and distribute the resulting drv files yourself to the host you wish to deploy them on. If you would like a concrete example of how I do that with nix-store --export, see this code snippet.

In general, I hope Nix gains some ability to make working with derivations more direct and user friendly eventually, since the derivation file format is essentially the equivalent of a Nix evaluation cache. If you have the final derivation file, you can outright skip eval all together. This is already somewhat the case with the new --store flag, but it could still use some improvement imo.

[iFreilicht]

Instead of calling import and export manually and doing the transfer yourself, nix-copy-closure might be a good option as well.

[mark.c]

@wamserma

nixos-rebuild --flake relaxes a bit on the flake definition. It only expects a flake.nix, not a Git-repo,

Ouh, yeah, I use this feature since beginning. I always rsync the whole repo except .git/.

@rudolf

nix build --no-link --print-out-paths .#nixosConfigurations."${myHost}".config.system.build.toplevel

@nrdxp

nix-store --export, see this code snippet.

@iFreilicht

transfer yourself, nix-copy-closure might be a good option as well

@iFreilicht @nrdxp @rudolf : Thank you very much.

I’m still a bit confused how to combine all the pieces. Let’s say I want to copy as little as possible and I want to use actual files (not tools that can communicate over SSH).

I start like this:

nix build --no-link --print-out-paths .#nixosConfigurations.foobar.config.system.build.toplevel

which gives me:

/nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba

then I use (instead of nix-store --export)

nix store dump-path /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba > foobar.nar

Which will create a small (less then MiB) file foobar.nar.

And now I can somehow import it and apply it on the “foobar” machine? Could you please tell me how?

Thank you.

[iFreilicht]

That’s one of the omissions @7c6f434c was talking about; there’s no inverse for nix store dump-path.

I tried a few things:

I think it should be possible to just copy the derivation (not the output path) like so:

$ nix derivation show /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba > foobar.drv.json
# move that file to the other machine
$ cat foobar.drv.json | nix derivation add
$ nix-store --realise /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba

Unfortunately, when I tried this out on my machine with a simple package, the nix derivation add step would fail with an assertion error. After writing a full-on bug-report on this, I found out that you have to transform that output of derivation show to be able to pass it to derivation add:

nix derivation show /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba \
| jq '.[]' > foobar.drv.json

I also tried using nix-store --export and nix-store --import but that fails with a signature error.

The lowest-tech solution is also the one that works (here I’m using grep as an example):

$ nix-store --query --deriver /nix/store/rn59ig1q28l4i3cd1lzszkjmxpl6i1r0-gnugrep-3.7
/nix/store/y565239srnxvr85kw7kc61py1s3x982l-gnugrep-3.7.drv
$ cp /nix/store/y565239srnxvr85kw7kc61py1s3x982l-gnugrep-3.7.drv gnugrep-3.7.drv
# Now copy that derivation file to your other machine, in whatever way you want.
$ nix store add-file gnugrep-3.7.drv
/nix/store/2r4f3yx819gz1nwl27mx10ba2jszc23j-gnugrep-3.7.drv
$ nix-store --realise /nix/store/2r4f3yx819gz1nwl27mx10ba2jszc23j-gnugrep-3.7.drv
/nix/store/rn59ig1q28l4i3cd1lzszkjmxpl6i1r0-gnugrep-3.7

Note that the final output path is identical to the one we started with.

I think this should work perfectly fine in terms of getting all the packages. About how to activate it, I’m not sure you should just be able to call

$outPath/bin/switch-to-configuration

where $outPath is whatever nix-store --realise returned.

EDIT: Ok now after revising this answer multiple times I actually read your first post again and you specifically don’t want to evaluate the derivation again. I’m not sure if my solution will save you any time, but please try it out and report back.

If the evaluation still takes long, you can also copy the built configuration, all its dependencies are downloaded from the binary cache anyway.

So instead of using dump-path, you could just zip or tar everything inside the output path, move the zip file over to the other machine, and then use nix store add-path to add the whole directory to the nix store.

[nrdxp]

maybe not in the new cli, but there is still nix-store --import for importing a serialized nar or stream of nars.

[iFreilicht]

Yes, but as I wrote in my post, I tried it and it failed with an error about a missing signature.

[7c6f434c]

For me it complains when I try as a user, but works as root (preferably with NIX_REMOTE= nix-store --import just in case)

[mark.c]

I tried

nix build --no-link --print-out-paths .#nixosConfigurations.foobar.config.system.build.toplevel
nix store dump-path /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba > a.nar
nix-store --export  /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba > b.nar

And I get error both with

nix-store --import < a.nar
nix-store --import < b.nar

The first is complaining about not being exported with --export and the second is complaining about a bad option.

[7c6f434c]

Could you please show how the second is complaining? It should complain about missing deps;

nix-store -q -R  /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba |
xargs nix-store --export > c.nar

Should give an exported closure (although probably also with things that target already has), which should be importable (as root)

[mark.c]

@7c6f434c

how the second is complaining?

Now I’m getting different error:

error: path '/nix/store/1vb2wkzpzppj0177xpnhr8nrd3f9ijby-system-path' is not valid

I tried the solution you proposed but I quickly realized that I’m exporting everything → so I’d get many GiB to import. That is interesting command to keep in my tool belt but this doesn’t make sense for my current problem. I want to get a small file/a few files that represents compiled/serialized configuration. It should be smaller than my source, not larger.

Thank you @7c6f434c

[7c6f434c]

nix-store -q -R $( nix-store -q --deriver  /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba ) |
xargs nix-store --export > d.nar

then

[mark.c]

Yes, thank you! That worked!

I piped it through gzip:

| xargs nix-store --export | gzip >

and

gunzip --stdout /tmp/importme.nar.gz | nix-store --import and it imported a lot of /nix/store/...drv paths.

So now just to figure out how to nixos-rebuilt into that imported derivation.

[7c6f434c]

First step is nix-store -r /nix/store/[…].drv

Then I think it is /nix/store/any5gh2i9yqbsafs0v6mbapa7412a2jw-nixos-system-foobar-23.05.20230723.ac1acba/bin/switch-to-configuration switch. I haven’t actually run mainline NixOS in a long time, so I might be mixing up things; I hope you have some path to recover the VM if this messes up the boot…

[mark.c]

I’ll test it on a new VM in cloud, so if it blows NixOS, not a big deal. Thank you