Thank you very much.
Just to understand… Does it mean that all the following use cases break the nix derivation?
environment.systemPackages
Googling that term, is that a description of functionality to add?
Yes, in the sense that these make nix derivation insufficient for your usecase. Though to be honest, I don’t think it was ever intended for this. Both commands are more meant for external tools to interact with the nix store, just like store add-path. derivation add in particular was only added in release 2.15.1.
Also, as we just saw, even if you build a very barebones configuration, derivation add is not sufficient, as the stdenv itself requires some source files.
Exactly. Tickets are units of work that programmers can work on independently. If you find a bug, you create a bug ticket, if you want to propose a feature, you create a feature ticket. People use different terms for this, on GitHub they’re called “Issues”, but “ticket” is understood by most. If you take a look at the Nix issue tracker, you can see there’s over 2000 bugs and feature requests open right now. Having them listed there helps a lot with prioritizing and organizing who works on what.
Independent programmers like me, who are not part of the core team, can look at lists like this one to see what to best work on next.
So it’s feature request, thank you 
Alright, finished the feature request. Please have a look and let me know if that sounds good to you. I don’t know when I can start working on this yet, but it’s on my radar.
Sounds great. I commented to get some insights / help with documentation: Add `--inputs-only` to `nix copy` · Issue #8806 · NixOS/nix · GitHub
Thank you very much!
Alright, so after a while, one of the Nix maintainers commented on the issue:
You can
nix copya derivation and its closure.
I was under the impression that NixOS configurations were built like profiles, without a deriver, but that is not the case.
I think this confusion came from this post of yours:
This is incorrect! --query --deriver always outputs a derivation, not the output. This is why it’s important to never copy-paste the output of commands without the command you actually ran.
I’ve set up my own NixOS server in the meantime, so now I went through all the steps myself, and this is what worked for me:
Make sure that nix --version returns at least 2.15.1. Older version might work, but we know for sure that 2.13.3 and 2.13.5 do not!
First, build the system configuration:
$ nixos-rebuild build --flake path/to/flake
building the system configuration...
$ readlink result
/nix/store/r8y8a8gckycx95xdw8wnw1vq5dc13bll-nixos-system-foobar-23.11.20230812.f045184
$ nix-store -q --deriver result
/nix/store/gccqy8z0f78ip7kc4h8s87bb5khg08w0-nixos-system-foobar-23.11.20230812.f045184.drv
Remember the result of readlink result, that’s what we’re trying to replicate!
Note the resulting path of nix-store -q --deriver is a derivation, and we can nix copy that quite easily:
$ configDrv=/nix/store/gccqy8z0f78ip7kc4h8s87bb5khg08w0-nixos-system-foobar-23.11.20230812.f045184.drv
$ nix copy $configDrv --to file:///tmp/configCache
Now, don’t be scared, it will look like this command copies like over 10GB of files, but actually, the binary cache we wrote to is pretty small, especially after compressing:
$ cd /tmp
$ du -sh configCache
20M configCache
$ zip -r config.zip configCache
adding: configCache/ (stored 0%)
[ ... ]
$ du -sh config.zip
5.6M config.zip
$ tar -czvf config.tar.gz configCache
configCache/
[...]
$ du -sh config.tar.gz
4.8M config.tar.gz
To confirm the derivation has been copied properly, you can run this command:
$ nix store ls --store file:///tmp/configCache $configDrv
gccqy8z0f78ip7kc4h8s87bb5khg08w0-nixos-system-foobar-23.11.20230812.f045184.drv
Copy the file to my other machine (to avoid confusion, I copied it to the same path /tmp/config.tar.gz), uncompress it, and nix copy it to the local store:
$ cd /tmp
$ tar -xvzf config.tar.gz
x configCache/
[...]
$ configDrv=/nix/store/gccqy8z0f78ip7kc4h8s87bb5khg08w0-nixos-system-foobar-23.11.20230812.f045184.drv
$ nix copy $configDrv --from file:///tmp/configCache
Now, you can build the configuration like so, and check the result.
$ nix build "${configDrv}^*"
$ readlink result
/nix/store/r8y8a8gckycx95xdw8wnw1vq5dc13bll-nixos-system-foobar-23.11.20230812.f045184
This should be the exact same as when we ran readlink result after nixos-rebuild on the original machine.
Finally, activating the configuration is as easy as:
$ ./result/activate
Let me know if this works for you!
BTW, I found that running the nix copy command like this:
$ nix copy $(nix-store -q --deriver result) --to file:///tmp/configCache
Did not work. I’m not entirely sure why, maybe I did something wrong, but there doesn’t seem to be an obvious problem with this.
Thank you so much the the great write up. I’ll test it today!
Strangely I failed on this step. Also nix copy $configDrv --to file:///tmp/configCache copied thousands of xxx.narinfo files and the full size of the directory was over 3 GiB. I’ll try again this evening.
Thank you.
What do you mean by that? What was the error message?
Hmm, could it be that you’re using some software that has its whole binary distribution as the input? Most unfree software does this, as well as packages ending in -bin.
Apologize for the delay. I tested it again today:
cat configuration.nix
{ modulesPath, pkgs, ... }: {
imports = [ "${modulesPath}/virtualisation/amazon-image.nix" ];
environment.systemPackages = with pkgs; [ htop vim tmux tree bottom ];
nix =
{
settings =
{
experimental-features = [ "nix-command" "flakes" ];
warn-dirty = false;
auto-optimise-store = true;
};
};
}
nix --extra-experimental-features flakes --extra-experimental-features nix-command flake show
nixos-rebuild dry-build --flake /etc/nixos#mynix |& grep /nix/store | wc -l
601
nixos-rebuild build --flake /etc/nixos#mynix
readlink result
/nix/store/070ybkrrnzhh7mzv8m6r013s6yr9g247-nixos-system-unnamed-23.05.20230829.2ab91c8
nix-store -q --deriver result
/nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv
nix --extra-experimental-features flakes --extra-experimental-features nix-command copy /nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv --to file:///tmp/mynix
tar -czf /tmp/mynix.tar.gz /tmp/mynix
du -sh /tmp/mynix*
291M /tmp/mynix
287M /tmp/mynix.tar.gz
du -sh /tmp/mynix*
291M /tmp/mynix
287M /tmp/mynix.tar.gz
nix --extra-experimental-features flakes --extra-experimental-features nix-command store ls --store file:///tmp/nixos /nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv
error: path '/nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv' is not a valid store path
nix --extra-experimental-features flakes --extra-experimental-features nix-command copy /nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv --from file:///tmp/nixos
error: cannot build missing derivation '/nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv'
Based on what you’ve shown, it like you should be using file:///tmp/mynix instead of file:///tmp/nixos on the EC2 instance.
Thank you. That was copy/paste issue. I’ve just tried again:
[root@ip-10-0-0-51:/tmp]# du -sh /tmp/mynix*
291M /tmp/mynix
287M /tmp/mynix.tar.gz
[root@ip-10-0-0-51:/tmp]# nix --extra-experimental-features flakes --extra-experimental-features nix-command store ls --store file:///tmp/mynix /nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv
error: path '/nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv' is not a valid store path
[root@ip-10-0-0-51:/tmp]# nix --extra-experimental-features flakes --extra-experimental-features nix-command copy /nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv --from file:///tmp/mynix
error: cannot build missing derivation '/nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv'
Thank you @iFreilicht for the instructions. Unfortunately I get the same error as @mark.c . Is it possible that the instructions work only when there is a very small difference between the source (builder) and the target?
I’ve just tried it again and I’m sorry, it doesn’t work.
[root@ip-10-0-0-94:/etc/nixos]# cat configuration.nix flake.nix
{ modulesPath, pkgs, ... }: {
imports = [ "${modulesPath}/virtualisation/amazon-image.nix" ];
environment.systemPackages = with pkgs; [ htop vim tmux tree bottom ];
nix =
{
settings =
{
experimental-features = [ "nix-command" "flakes" ];
warn-dirty = false;
auto-optimise-store = true;
};
};
}
{
inputs =
{
nixpkgs.url = "github:nixos/nixpkgs/nixos-23.05";
};
outputs = { self, nixpkgs }:
let
system = "x86_64-linux";
pkgs =
import nixpkgs
{
inherit system;
};
lib = nixpkgs.lib;
in
{
nixosConfigurations =
{
abc = lib.nixosSystem
{
inherit system;
modules = [ ./configuration.nix ];
};
};
};
}
[root@ip-10-0-0-94:/etc/nixos]# cd
[root@ip-10-0-0-94:~]# nixos-rebuild build --flake /etc/nixos#abc
building the system configuration...
warning: creating lock file '/etc/nixos/flake.lock'
trace: warning: system.stateVersion is not set, defaulting to 23.05. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.
[root@ip-10-0-0-94:~]#
[root@ip-10-0-0-94:~]# readlink result
/nix/store/379k42cmcaaklb8mgd3xv4f23dfi4wg2-nixos-system-unnamed-23.05.20230907.4f77ea6
[root@ip-10-0-0-94:~]# nix-store -q --deriver result
/nix/store/5k836ssrihmrcmmx8m8z74fbkvf1f3h6-nixos-system-unnamed-23.05.20230907.4f77ea6.drv
[root@ip-10-0-0-94:~]# nix --extra-experimental-features flakes --extra-experimental-features nix-command copy /nix/store/5k836ssrihmrcmmx8m8z74fbkvf1f3h6-nixos-system-unnamed-23.05.20230907.4f77ea6.drv --to file:///tmp/abc
[root@ip-10-0-0-94:~]# tar -czf /tmp/abc.tar.gz /tmp/abc/
tar: Removing leading `/' from member names
[root@ip-10-0-0-94:~]# du -sh /tmp/abc*
283M /tmp/abc
279M /tmp/abc.tar.gz
[root@ip-10-0-0-94:~]#
[root@ip-10-0-0-219:~]# du -sh /tmp/abc*
283M /tmp/abc
279M /tmp/abc.tar.gz
[root@ip-10-0-0-219:~]# nix --extra-experimental-features flakes --extra-experimental-features nix-command copy /nix/store/5k836ssrihmrcmmx8m8z74fbkvf1f3h6-nixos-system-unnamed-23.05.20230907.4f77ea6.drv --from file:///tmp/abc
error: cannot build missing derivation '/nix/store/5k836ssrihmrcmmx8m8z74fbkvf1f3h6-nixos-system-unnamed-23.05.20230907.4f77ea6.drv'
[root@ip-10-0-0-219:~]#
Could you please review your instructions @iFreilicht ?
Thank you
Please refer to the private message I sent you. I will have a look in the week after 16th of September.
You may already know this, but you can build on one machine and push to another by just adding “–target-host (ssh-username)@(host.domain)” to the nixos-rebuild command.
Of course, this requires that the user has the ability to execute sudo for the necessary commands. In my case I have set my root user to use “*” as the password hash so that I can use a .ssh key to log into it.
This gives the push/pull functionality that nix-ops provides without setting up nix-ops. My small experience with nix-ops is that it also gives more real-time status of every system it is connected to.
Hi @genson ,
We discussed this option earlier but I totally understand that you didn’t read every single post - the thread is very long…
The idea and my need is to take a flake repo with a single or multiple NixOS configurations and somehow boil it down to a single (small) file that will target a single host and won’t contain any comments or functions or mkIf etc.
When I opened this thread a few months ago I foolishly assumed that this is a simple thing to do since nix probably has something like a compilation phase / stage when all mkIf and other things are converted to the actual values, etc. But it turned out this is much more complicated
I am a little suspicious of this line in your flake. This might be what causes the size of the cache to baloon to over 200MB. I don’t know what’s in that module, so hard to say, but all the other tools are definitely very simple.
Ok, that’s pretty bad 
Does the second command (nix store ls ...) successfully return the derivation on the machine you built the configuration on?
I just started renting a small VPC on Hetzner and will try these steps, including the transfer, from my home machine to that VPC later.
Either way, it is pretty annoying that --from doesn’t fail with a proper error if the passed directory doesn’t even exist.
Ok so I tried it once more now. I followed everything exactly as I described, and it all worked.
I have three machines:
horse, my current laptopjunction, my home servergateway, my Hetzner VPSThe logs below are literally verbatim copied from me following the instructions. I only changed the
way I set $configDrv to make mistakes less likely and omitted some details with [...].
You can see the full NixOS configurations and the flake used to build them in
my dotfiles repo on github
felix@horse:~$ ssh junction
Last login: Fri Aug 25 22:21:35 2023 from [...]
felix@junction:~$ nixos-rebuild build --flake path:/home/felix/.dotfiles
building the system configuration...
felix@junction:~$ readlink result
/nix/store/40jck40wcbjh4xhdpm301fk45f1vky8b-nixos-system-junction-23.11.20230812.f045184
felix@junction:~$ configDrv=$(nix-store -q --deriver result)
felix@junction:~$ echo "$configDrv"
/nix/store/cm8r7al3q69ys5dy7hg4f962973j8dz6-nixos-system-junction-23.11.20230812.f045184.drv
felix@junction:~$ nix copy $configDrv --to file:///tmp/configCache
felix@junction:~$ cd /tmp
felix@junction:/tmp$ du -sh configCache
21M configCache
felix@junction:/tmp$ tar -czvf config.tar.gz configCache
configCache/
configCache/v7ihfx32zv7bdha6i9dd6a3r0knzs8j3.narinfo
[...]
configCache/5qynqmq6rd2ahaf4760bj2b2ksqg80hy.narinfo
configCache/vq1bx1j10363hnl2by0fcchm7a8nl3ha.narinfo
felix@junction:/tmp$ du -sh config.tar.gz
5.6M config.tar.gz
felix@junction:/tmp$ nix store ls --store file:///tmp/configCache $configDrv
cm8r7al3q69ys5dy7hg4f962973j8dz6-nixos-system-junction-23.11.20230812.f045184.drv
felix@junction:/tmp$ exit
Connection to junction closed.
felix@horse:~$ scp junction:/tmp/config.tar.gz gateway:/tmp
felix@horse:~$ ssh gateway
Last login: Fri Aug 25 22:12:32 2023 from [...]
felix@gateway:~$ cd /tmp
felix@gateway:/tmp$ tar -xvzf config.tar.gz
configCache/
configCache/v7ihfx32zv7bdha6i9dd6a3r0knzs8j3.narinfo
[...]
configCache/5qynqmq6rd2ahaf4760bj2b2ksqg80hy.narinfo
configCache/vq1bx1j10363hnl2by0fcchm7a8nl3ha.narinfo
felix@gateway:/tmp$ configDrv=/nix/store/cm8r7al3q69ys5dy7hg4f962973j8dz6-nixos-system-junction-23.11.20230812.f045184.drv
felix@gateway:/tmp$ nix copy $configDrv --from file:///tmp/configCache
warning: The interpretation of store paths arguments ending in `.drv` recently changed. If this command is now failing try again with '/nix/store/cm8r7al3q69ys5dy7hg4f962973j8dz6-nixos-system-junction-23.11.20230812.f045184.drv^*'
felix@gateway:/tmp$ nix build "${configDrv}^*"
felix@gateway:/tmp$ nix store ls --store file:///tmp/configCache $configDrv
cm8r7al3q69ys5dy7hg4f962973j8dz6-nixos-system-junction-23.11.20230812.f045184.drv
I’m really puzzled now, the process itself absolutely works. It seems to be properly implemented in Nix itself, its unlikely the culprit here.
Again, the amazon-image.nix import seems to me like a likely contributor to the huge size of the cache nix copy created, but I can’t say for sure. Maybe I’ll have time soon to rent an EC2 instance myself and try it there as well.
Can you try this without importing amazon-image.nix? Or can you maybe share that file somehow?
Ah, one other thing that I’m noticing just now is your invocation of tar:
tar -czf /tmp/abc.tar.gz /tmp/abc/
This causes the archive to be nested, so you might have to use --from file:///tmp/abc/abc when running nix copy.