Compile NixOS derivations into a single file for a single host

mark.c · August 25, 2023, 10:09am

Thank you so much the the great write up. I’ll test it today!

mark.c · August 25, 2023, 11:33am

Strangely I failed on this step. Also nix copy $configDrv --to file:///tmp/configCache copied thousands of xxx.narinfo files and the full size of the directory was over 3 GiB. I’ll try again this evening.

Thank you.

iFreilicht · August 29, 2023, 5:37am

What do you mean by that? What was the error message?

Hmm, could it be that you’re using some software that has its whole binary distribution as the input? Most unfree software does this, as well as packages ending in -bin.

mark.c · August 30, 2023, 12:17pm

Apologize for the delay. I tested it again today:

NixOS EC2
very simple configuration

cat configuration.nix

{ modulesPath, pkgs, ... }: {
  imports = [ "${modulesPath}/virtualisation/amazon-image.nix" ];

  environment.systemPackages = with pkgs; [ htop vim tmux tree bottom ];

  nix =
    {
      settings =
        {
          experimental-features = [ "nix-command" "flakes" ];
          warn-dirty = false;
          auto-optimise-store = true;
        };
    };
}

nix --extra-experimental-features flakes --extra-experimental-features nix-command flake show
nixos-rebuild dry-build --flake /etc/nixos#mynix |& grep /nix/store | wc -l

nixos-rebuild build --flake /etc/nixos#mynix
readlink result

/nix/store/070ybkrrnzhh7mzv8m6r013s6yr9g247-nixos-system-unnamed-23.05.20230829.2ab91c8

nix-store -q --deriver result

/nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv

nix --extra-experimental-features flakes --extra-experimental-features nix-command copy /nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv --to file:///tmp/mynix
tar -czf /tmp/mynix.tar.gz /tmp/mynix

du -sh /tmp/mynix*

291M    /tmp/mynix
287M    /tmp/mynix.tar.gz

new blank NixOS EC2

du -sh /tmp/mynix*

291M    /tmp/mynix
287M    /tmp/mynix.tar.gz

nix --extra-experimental-features flakes --extra-experimental-features nix-command store ls --store file:///tmp/nixos /nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv

error: path '/nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv' is not a valid store path

nix --extra-experimental-features flakes --extra-experimental-features nix-command copy /nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv --from file:///tmp/nixos

error: cannot build missing derivation '/nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv'

drewkett · August 30, 2023, 1:56pm

Based on what you’ve shown, it like you should be using file:///tmp/mynix instead of file:///tmp/nixos on the EC2 instance.

mark.c · August 30, 2023, 4:49pm

Thank you. That was copy/paste issue. I’ve just tried again:

[root@ip-10-0-0-51:/tmp]# du -sh /tmp/mynix*
291M    /tmp/mynix
287M    /tmp/mynix.tar.gz

[root@ip-10-0-0-51:/tmp]# nix --extra-experimental-features flakes --extra-experimental-features nix-command store ls --store file:///tmp/mynix /nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv
error: path '/nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv' is not a valid store path

[root@ip-10-0-0-51:/tmp]# nix --extra-experimental-features flakes --extra-experimental-features nix-command copy /nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv --from file:///tmp/mynix
error: cannot build missing derivation '/nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv'

honda · September 6, 2023, 2:40pm

Thank you @iFreilicht for the instructions. Unfortunately I get the same error as @mark.c . Is it possible that the instructions work only when there is a very small difference between the source (builder) and the target?

mark.c · September 9, 2023, 2:35pm

I’ve just tried it again and I’m sorry, it doesn’t work.

[root@ip-10-0-0-94:/etc/nixos]# cat configuration.nix flake.nix
{ modulesPath, pkgs, ... }: {
  imports = [ "${modulesPath}/virtualisation/amazon-image.nix" ];

  environment.systemPackages = with pkgs; [ htop vim tmux tree bottom ];

  nix =
    {
      settings =
        {
          experimental-features = [ "nix-command" "flakes" ];
          warn-dirty = false;
          auto-optimise-store = true;
        };
    };
}


{
  inputs =
    {
      nixpkgs.url = "github:nixos/nixpkgs/nixos-23.05";
    };

  outputs = { self, nixpkgs }:
    let
      system = "x86_64-linux";
      pkgs =
        import nixpkgs
          {
            inherit system;
          };
      lib = nixpkgs.lib;
    in
      {
        nixosConfigurations =
          {
            abc = lib.nixosSystem
              {
                inherit system;
                modules = [ ./configuration.nix ];
              };
          };
      };
}

[root@ip-10-0-0-94:/etc/nixos]# cd

[root@ip-10-0-0-94:~]# nixos-rebuild build --flake /etc/nixos#abc
building the system configuration...
warning: creating lock file '/etc/nixos/flake.lock'
trace: warning: system.stateVersion is not set, defaulting to 23.05. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.

[root@ip-10-0-0-94:~]#

[root@ip-10-0-0-94:~]# readlink result
/nix/store/379k42cmcaaklb8mgd3xv4f23dfi4wg2-nixos-system-unnamed-23.05.20230907.4f77ea6

[root@ip-10-0-0-94:~]# nix-store -q --deriver result
/nix/store/5k836ssrihmrcmmx8m8z74fbkvf1f3h6-nixos-system-unnamed-23.05.20230907.4f77ea6.drv

[root@ip-10-0-0-94:~]# nix --extra-experimental-features flakes --extra-experimental-features nix-command copy /nix/store/5k836ssrihmrcmmx8m8z74fbkvf1f3h6-nixos-system-unnamed-23.05.20230907.4f77ea6.drv --to file:///tmp/abc

[root@ip-10-0-0-94:~]# tar -czf /tmp/abc.tar.gz /tmp/abc/
tar: Removing leading `/' from member names

[root@ip-10-0-0-94:~]# du -sh /tmp/abc*
283M    /tmp/abc
279M    /tmp/abc.tar.gz

[root@ip-10-0-0-94:~]#

another EC2

[root@ip-10-0-0-219:~]# du -sh /tmp/abc*
283M    /tmp/abc
279M    /tmp/abc.tar.gz

[root@ip-10-0-0-219:~]# nix --extra-experimental-features flakes --extra-experimental-features nix-command copy /nix/store/5k836ssrihmrcmmx8m8z74fbkvf1f3h6-nixos-system-unnamed-23.05.20230907.4f77ea6.drv --from file:///tmp/abc
error: cannot build missing derivation '/nix/store/5k836ssrihmrcmmx8m8z74fbkvf1f3h6-nixos-system-unnamed-23.05.20230907.4f77ea6.drv'

[root@ip-10-0-0-219:~]#

Could you please review your instructions @iFreilicht ?

Thank you

iFreilicht · September 10, 2023, 8:02am

Please refer to the private message I sent you. I will have a look in the week after 16th of September.

genson · September 11, 2023, 4:18am

You may already know this, but you can build on one machine and push to another by just adding “–target-host (ssh-username)@(host.domain)” to the nixos-rebuild command.

Of course, this requires that the user has the ability to execute sudo for the necessary commands. In my case I have set my root user to use “*” as the password hash so that I can use a .ssh key to log into it.

This gives the push/pull functionality that nix-ops provides without setting up nix-ops. My small experience with nix-ops is that it also gives more real-time status of every system it is connected to.

mark.c · September 23, 2023, 5:15am

Hi @genson ,

We discussed this option earlier but I totally understand that you didn’t read every single post - the thread is very long…

The idea and my need is to take a flake repo with a single or multiple NixOS configurations and somehow boil it down to a single (small) file that will target a single host and won’t contain any comments or functions or mkIf etc.

When I opened this thread a few months ago I foolishly assumed that this is a simple thing to do since nix probably has something like a compilation phase / stage when all mkIf and other things are converted to the actual values, etc. But it turned out this is much more complicated

iFreilicht · October 12, 2023, 12:41pm

I am a little suspicious of this line in your flake. This might be what causes the size of the cache to baloon to over 200MB. I don’t know what’s in that module, so hard to say, but all the other tools are definitely very simple.

mark.c:

[root@ip-10-0-0-51:/tmp]# du -sh /tmp/mynix*
291M    /tmp/mynix
287M    /tmp/mynix.tar.gz

[root@ip-10-0-0-51:/tmp]# nix --extra-experimental-features flakes --extra-experimental-features nix-command store ls --store file:///tmp/mynix /nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv
error: path '/nix/store/rn6brlsf6z80ffhgqdkf63b8020hl0hd-nixos-system-unnamed-23.05.20230829.2ab91c8.drv' is not a valid store path

Ok, that’s pretty bad

Does the second command (nix store ls ...) successfully return the derivation on the machine you built the configuration on?

I just started renting a small VPC on Hetzner and will try these steps, including the transfer, from my home machine to that VPC later.

Either way, it is pretty annoying that --from doesn’t fail with a proper error if the passed directory doesn’t even exist.

iFreilicht · October 13, 2023, 8:38am

Ok so I tried it once more now. I followed everything exactly as I described, and it all worked.
I have three machines:

horse, my current laptop
junction, my home server
gateway, my Hetzner VPS

The logs below are literally verbatim copied from me following the instructions. I only changed the
way I set $configDrv to make mistakes less likely and omitted some details with [...].
You can see the full NixOS configurations and the flake used to build them in
my dotfiles repo on github

felix@horse:~$ ssh junction
Last login: Fri Aug 25 22:21:35 2023 from [...]
felix@junction:~$ nixos-rebuild build --flake path:/home/felix/.dotfiles
building the system configuration...
felix@junction:~$ readlink result
/nix/store/40jck40wcbjh4xhdpm301fk45f1vky8b-nixos-system-junction-23.11.20230812.f045184
felix@junction:~$ configDrv=$(nix-store -q --deriver result)
felix@junction:~$ echo "$configDrv"
/nix/store/cm8r7al3q69ys5dy7hg4f962973j8dz6-nixos-system-junction-23.11.20230812.f045184.drv
felix@junction:~$ nix copy $configDrv --to file:///tmp/configCache
felix@junction:~$ cd /tmp
felix@junction:/tmp$ du -sh configCache
21M     configCache
felix@junction:/tmp$ tar -czvf config.tar.gz configCache
configCache/
configCache/v7ihfx32zv7bdha6i9dd6a3r0knzs8j3.narinfo
[...]
configCache/5qynqmq6rd2ahaf4760bj2b2ksqg80hy.narinfo
configCache/vq1bx1j10363hnl2by0fcchm7a8nl3ha.narinfo
felix@junction:/tmp$ du -sh config.tar.gz
5.6M    config.tar.gz
felix@junction:/tmp$ nix store ls --store file:///tmp/configCache $configDrv
cm8r7al3q69ys5dy7hg4f962973j8dz6-nixos-system-junction-23.11.20230812.f045184.drv
felix@junction:/tmp$ exit
Connection to junction closed.
felix@horse:~$ scp junction:/tmp/config.tar.gz gateway:/tmp
felix@horse:~$ ssh gateway
Last login: Fri Aug 25 22:12:32 2023 from [...]
felix@gateway:~$ cd /tmp
felix@gateway:/tmp$ tar -xvzf config.tar.gz
configCache/
configCache/v7ihfx32zv7bdha6i9dd6a3r0knzs8j3.narinfo
[...]
configCache/5qynqmq6rd2ahaf4760bj2b2ksqg80hy.narinfo
configCache/vq1bx1j10363hnl2by0fcchm7a8nl3ha.narinfo
felix@gateway:/tmp$ configDrv=/nix/store/cm8r7al3q69ys5dy7hg4f962973j8dz6-nixos-system-junction-23.11.20230812.f045184.drv
felix@gateway:/tmp$ nix copy $configDrv --from file:///tmp/configCache
warning: The interpretation of store paths arguments ending in `.drv` recently changed. If this command is now failing try again with '/nix/store/cm8r7al3q69ys5dy7hg4f962973j8dz6-nixos-system-junction-23.11.20230812.f045184.drv^*'
felix@gateway:/tmp$ nix build "${configDrv}^*"
felix@gateway:/tmp$ nix store ls --store file:///tmp/configCache $configDrv
cm8r7al3q69ys5dy7hg4f962973j8dz6-nixos-system-junction-23.11.20230812.f045184.drv

I’m really puzzled now, the process itself absolutely works. It seems to be properly implemented in Nix itself, its unlikely the culprit here.

Again, the amazon-image.nix import seems to me like a likely contributor to the huge size of the cache nix copy created, but I can’t say for sure. Maybe I’ll have time soon to rent an EC2 instance myself and try it there as well.

Can you try this without importing amazon-image.nix? Or can you maybe share that file somehow?

iFreilicht · October 13, 2023, 8:47am

Ah, one other thing that I’m noticing just now is your invocation of tar:

tar -czf /tmp/abc.tar.gz /tmp/abc/

This causes the archive to be nested, so you might have to use --from file:///tmp/abc/abc when running nix copy.

mark.c · October 13, 2023, 9:22am

Ouh! I need to tar -czf /tmp/abc.tar.gz -C /tmp abc

I’ll try again! Thank you!

rudolf · October 13, 2023, 9:31am

The amazon-image file is just:

github.com

NixOS/nixpkgs/blob/4d75ea462e1f420c68ff43173e2524ce5ab25c86/nixos/modules/virtualisation/amazon-image.nix

# Configuration for Amazon EC2 instances. (Note that this file is a
# misnomer - it should be "amazon-config.nix" or so, not
# "amazon-image.nix", since it's used not only to build images but
# also to reconfigure instances. However, we can't rename it because
# existing "configuration.nix" files on EC2 instances refer to it.)

{ config, lib, pkgs, ... }:

with lib;

let
  cfg = config.ec2;
in

{
  imports = [
    ../profiles/headless.nix
    # Note: While we do use the headless profile, we also explicitly
    # turn on the serial console on ttyS0 below. This is because
    # AWS does support accessing the serial console:

This file has been truncated. show original

mark.c · October 13, 2023, 6:52pm

first EC2

ssh firstnix
cd /etc/nixos
cat configuration.nix flake.nix

{ modulesPath, pkgs, ... }: {
  imports = [ "${modulesPath}/virtualisation/amazon-image.nix" ];

  environment.systemPackages = with pkgs; [ htop vim tmux tree bottom ];

  nix =
    {
      settings =
        {
          experimental-features = [ "nix-command" "flakes" ];
          warn-dirty = false;
          auto-optimise-store = true;
        };
    };
}
{
  inputs =
    {
      nixpkgs.url = "github:nixos/nixpkgs/nixos-23.05";
    };

  outputs = { self, nixpkgs }:
    let
      system = "x86_64-linux";
      pkgs =
        import nixpkgs
          {
            inherit system;
          };
      lib = nixpkgs.lib;
    in
      {
        nixosConfigurations =
          {
            abc = lib.nixosSystem
              {
                inherit system;
                modules = [ ./configuration.nix ];
              };
          };
      };
}

nix build --no-link --print-out-paths /etc/nixos#nixosConfigurations.abc.config.system.build.toplevel

/nix/store/s38ck7liswgiskxh8smccywzkkdrixzl-nixos-system-unnamed-23.05.20231011.bd1cde4

nix-store -q --deriver /nix/store/s38ck7liswgiskxh8smccywzkkdrixzl-nixos-system-unnamed-23.05.20231011.bd1cde4

/nix/store/dmyyml7l7kbbcs6jvnckqrk5wldaq1f1-nixos-system-unnamed-23.05.20231011.bd1cde4.drv

nix copy /nix/store/dmyyml7l7kbbcs6jvnckqrk5wldaq1f1-nixos-system-unnamed-23.05.20231011.bd1cde4.drv --to file:///tmp/abc
tar -czf /tmp/abc.tar.gz -C /tmp abc
du -sh /tmp/abc.tar.gz /tmp/abc

288M    /tmp/abc.tar.gz
291M    /tmp/abc

second EC2

ssh secondnix
tar -xf /tmp/abc.tar.gz -C /tmp
ls -l /tmp/abc | grep narinfo | wc -l

ls -l /tmp/abc/nar | grep nar.xz | wc -l

nix copy /nix/store/dmyyml7l7kbbcs6jvnckqrk5wldaq1f1-nixos-system-unnamed-23.05.20231011.bd1cde4.drv --from file:///tmp/abc

error: cannot build missing derivation '/nix/store/dmyyml7l7kbbcs6jvnckqrk5wldaq1f1-nixos-system-unnamed-23.05.20231011.bd1cde4.drv'

Weird, I’ll try again tomorrow.

iFreilicht · October 13, 2023, 7:48pm

Maybe to make absolutely sure, run only and exactly the commands that I did, including the cd into /tmp? Either way, something about your flake makes it huge, I’ll try to build it myself this weekend and see if I can replicate this.

mark.c · October 14, 2023, 9:08am

Absolutely.

ssh first
mkdir -p ~/.config/nix
tee ~/.config/nix/nix.conf <<< 'experimental-features = nix-command flakes'
nix run nixpkgs#vim -- /etc/nixos/configuration.nix
nix run nixpkgs#vim -- /etc/nixos/flake.nix
nixos-rebuild build --flake /etc/nixos#abc
readlink result

/nix/store/s38ck7liswgiskxh8smccywzkkdrixzl-nixos-system-unnamed-23.05.20231011.bd1cde4

configDrv=$(nix-store -q --deriver result)
echo "$configDrv"

/nix/store/dmyyml7l7kbbcs6jvnckqrk5wldaq1f1-nixos-system-unnamed-23.05.20231011.bd1cde4.drv

nix copy $configDrv --to file:///tmp/configCache
cd /tmp
du -sh configCache

291M    configCache

tar -czvf config.tar.gz configCache
du -sh config.tar.gz

288M    config.tar.gz

nix store ls --store file:///tmp/configCache $configDrv

error: path '/nix/store/dmyyml7l7kbbcs6jvnckqrk5wldaq1f1-nixos-system-unnamed-23.05.20231011.bd1cde4.drv' is not a valid store path

iFreilicht · October 14, 2023, 8:35pm

@mark.c You’re not going to believe this.

It still worked, and I built the exact same configuration as you:

nixos-rebuild build --flake .#abc
readlink result

/nix/store/s38ck7liswgiskxh8smccywzkkdrixzl-nixos-system-unnamed-23.05.20231011.bd1cde4

configDrv=$(nix-store -q --deriver result)
echo "$configDrv"

/nix/store/dmyyml7l7kbbcs6jvnckqrk5wldaq1f1-nixos-system-unnamed-23.05.20231011.bd1cde4.drv

nix copy $configDrv --to file:///tmp/configCache
cd /tmp
du -sh configCache
25M     configCache

tar -czvf config.tar.gz configCache
du -sh config.tar.gz

6.4M     config.tar.gz

nix store ls --store file:///tmp/configCache $configDrv

dmyyml7l7kbbcs6jvnckqrk5wldaq1f1-nixos-system-unnamed-23.05.20231011.bd1cde4.drv

What the hell is going on here? The derivation was perfectly reproduced on my machine, the hash proves it. Why is your version of nix behaving so differently? Not only does it work, but nix copy produced a much smaller cache on my machine than on yours.

If you run nix --version, what does it return?
Also, what does realpath $(which nix) return?

Maybe this behavior is a bug in a newer or older version of nix?