Impermanence + sops-nix + nixos-anywhere lead to missing `hashedPasswordFile`s

Help
1

Im currently struggling to combine nix-impermanence, sops-nix, and nixos-anywhere. The specific problem is that after boot, I cannot log into any of the users, but during deployment with nixos-anywhere it says that if picked up the decryption key for the secrets…

To make things more concrete:

# bootstrap.sh, run from macOS

#!/bin/bash

temp=$(mktemp -d)

cleanup() {
   rm -rf "$temp"
}
trap cleanup EXIT

mkdir -p "$temp/etc/ssh"
chmod 755 "$temp/etc/ssh"

cp ~/.ssh/id_ed25519 "$temp/etc/ssh/ssh_host_ed25519_key"
chmod 600 "$temp/etc/ssh/ssh_host_ed25519_key"

nix run github:nix-community/nixos-anywhere --              \
   --flake '.#hostname'                                     \
   --build-on remote                                       \
   --extra-files "$temp"                                   \
   --disk-encryption-keys /tmp/secret.key /tmp/secret.key  \
   --target-host nixos@$1

# relevant snippets of flake.nix

{ inputs
, pkgs
, config
, ...
}:
let
  secretspath = builtins.toString inputs.nix-secrets;
in
{
  imports =
    [
      inputs.disko.nixosModules.disko
      inputs.impermanence.nixosModules.impermanence
      ./disk-config.nix
    ];

  boot = {
    loader = {
      systemd-boot.enable = true;
      efi.canTouchEfiVariables = true;
    };
    kernelModules = [ "dm_mod" "dm_crypt" ];
  };

  sops = {
    defaultSopsFile = "${secretspath}/secrets/eiger.yaml";
    secrets = {
      user-worker-password = {
        key = "user/worker/password";
        neededForUsers = true;
      };
    };
    age = {
      sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
    };
  };


  environment.persistence."/persist" = {
    directories = [
      "/etc/nixos"
      "/passwords"
      "/var/lib"
      "/var/log"
      "/var/lib/sops-nix"
      secretspath
    ];
    files = [
      "/etc/machine-id"

      "/etc/ssh/ssh_host_ed25519_key"
      "/etc/ssh/ssh_host_ed25519_key.pub"
      "/etc/ssh/ssh_host_rsa_key"
      "/etc/ssh/ssh_host_rsa_key.pub"
    ];
  };

  fileSystems."/persist".neededForBoot = true;

  fileSystems."/" = {
    device = "tmpfs";
    fsType = "tmpfs";
    options = [ "defaults" "size=2G" "mode=755" ];
  };

  users = {
    groups.media = { };
    mutableUsers = false;
    users = {
      root.hashedPassword = "mkpasswd -c test so that I can debug";
      worker = {
        hashedPasswordFile = config.sops.secrets."user-worker-password".path;
        isNormalUser = true;
        extraGroups = [ "media" ];
      };
    };
  };

  nix.settings.allowed-users = [ "root" ];
};

The output from the bootstrap.sh file is something like this

Lots of disko commands and messages related to it

### Building the system closure ###
warning: Git tree '/Users/iilak/prg/internal/nix-config' is dirty
evaluation warning: The option `hardware.opengl.extraPackages' defined in `/nix/store/01avwgmdmbmmj3f9qy5g2l6xa94h1mci-source/hosts/eiger/jellyfin.nix' has been renamed to `hardware.graphics.extraPackages'.
evaluation warning: The option `hardware.opengl.enable' defined in `/nix/store/01avwgmdmbmmj3f9qy5g2l6xa94h1mci-source/hosts/eiger/jellyfin.nix' has been renamed to `hardware.graphics.enable'.
Warning: Permanently added '192.168.198.132' (ED25519) to the list of known hosts.
### Copying extra files ###
Pseudo-terminal will not be allocated because stdin is not a terminal.
Warning: Permanently added '192.168.198.132' (ED25519) to the list of known hosts.
Warning: Permanently added '192.168.198.132' (ED25519) to the list of known hosts.
Connection to 192.168.198.132 closed.
### Installing NixOS ###
Pseudo-terminal will not be allocated because stdin is not a terminal.
Warning: Permanently added '192.168.198.132' (ED25519) to the list of known hosts.
installing the boot loader...
setting up secrets for users...
Cannot read ssh key '/etc/ssh/ssh_host_rsa_key': open /etc/ssh/ssh_host_rsa_key: no such file or directory
sops-install-secrets: Imported /etc/ssh/ssh_host_ed25519_key as age key with fingerprint age1e2pselcrjk5zl89f00zen4n94zemu5wqhu900zf9w8jg6rl68d3saptlhp
Warning: Source directory '/persist/etc' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/var' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/var/lib' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/nix' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/nix/store' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/etc/nixos' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/passwords' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/var/log' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/var/lib/sops-nix' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/nix/store/hpgfvihqlmccrsl5bs73iz4ql0bv6xm2-source' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/etc/ssh' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
setting up /etc...
Creating initial /etc/machine-id
A file already exists at /etc/ssh/ssh_host_ed25519_key!
Activation script snippet 'persist-files' failed (1)
Created "/boot/EFI".
Created "/boot/EFI/systemd".
Created "/boot/EFI/BOOT".
Created "/boot/loader".
Created "/boot/loader/keys".
Created "/boot/loader/entries".
Created "/boot/EFI/Linux".
Copied "/nix/store/3rp3z9c3clhcq432cgas4apm4abansfh-systemd-257.5/lib/systemd/boot/efi/systemd-bootaa64.efi" to "/boot/EFI/systemd/systemd-bootaa64.efi".
Copied "/nix/store/3rp3z9c3clhcq432cgas4apm4abansfh-systemd-257.5/lib/systemd/boot/efi/systemd-bootaa64.efi" to "/boot/EFI/BOOT/BOOTAA64.EFI".
Random seed file /boot/loader/random-seed successfully written (32 bytes).
Successfully initialized system token in EFI variable with 32 bytes.
Created EFI boot entry "Linux Boot Manager".
installation finished!
### Rebooting ###
Pseudo-terminal will not be allocated because stdin is not a terminal.
Warning: Permanently added '192.168.198.132' (ED25519) to the list of known hosts.
### Waiting for the machine to become unreachable due to reboot ###
kex_exchange_identification: read: Connection reset by peer
Connection reset by 192.168.198.132 port 22
### Done! ###

The confusing pieces for me:

...
Warning: Source directory '/persist/nix/store/hpgfvihqlmccrsl5bs73iz4ql0bv6xm2-source' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
...
A file already exists at /etc/ssh/ssh_host_ed25519_key!
Activation script snippet 'persist-files' failed (1)

The first line looks to me like its trying to copy ${secretsPath} to /persist, which probably no necessary since the /nix/persist should already be persisted anyway.

The more confusing thing to me is A file already exists at /etc/ssh/ssh_host_ed25519_key!. Is this the file created by the ISO or is it the one that nixos-anywhere should have copied according to the --extra-files command? The correct file must have been copied to /mnt/etc/ssh, otherwise the build of the system would have not worked since it could not have decrypted the secrets of nix-secrets. But after reboot /run/secrets does not exist, so the hashedPasswordFile for worker does not exist either… Looking at /etc/ssh/ssh_host_ed25519_key it is probably the one created by the ISO (which is btw. correctly symlinked to /persist/etc/ssh), since the public key contains root@hostname instead of username@macOS, which is what was in the original file used to decrypt the secrets during build.

Anybody an idea what Im doing wrong here?

P.S. Not sure if its relevant, but here is the disko part:

_: {
  disko.devices = {
    disk = {
      main = {
        type = "disk";
        device = "/dev/nvme0n1";
        content = {
          type = "gpt";
          partitions = {
            boot = {
              size = "1M";
              type = "EF02";
            };
            ESP = {
              size = "512M";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot";
                mountOptions = [ "defaults" "umask=0077" ];
              };
            };
            luks = {
              size = "100%";
              content = {
                type = "luks";
                name = "crypted";
                passwordFile = "/tmp/secret.key";
                settings = {
                  allowDiscards = true;
                  bypassWorkqueues = true;
                };
                content = {
                  type = "btrfs";
                  extraArgs = [ "-f" ];
                  subvolumes = {
                    "/nix" = {
                      mountpoint = "/nix";
                      mountOptions = [ "compress=zstd" "noatime" ];
                    };
                    "/persist" = {
                      mountpoint = "/persist";
                      mountOptions = [ "compress=zstd" "noatime" ];
                    };
                    "/home" = {
                      mountpoint = "/home";
                      mountOptions = [ "compress=zstd" "noatime" ];
                    };
                    "/swap" = {
                      mountpoint = "/swap";
                      swap.swapfile.size = "4G";
                    };
                  };
                };
              };
            };
          };
        };
      };
    };
  };
}
3 Likes
2

I’ve never personally used nixos-anywhere so this might be wrong. Without more information, it looks like an incomplete sops setup. Try adding “user..openssh.authorizedKeys.keys = [config.sops.secrets.“pathShouldCorrespondToPersistentSetInYaml/sops_encrypted/ed25519_key/id_ed25519”.path];” if it does not already exist in your nix configuration. Not in the home configuration *unless home-manager standalone without nixos operating system is being used, but I’m going to assume your home manager is an integrated nixos submodule. If the worker’s permissions are still not set up correctly, it won’t be capable of creating and interacting with the “/run/secrets” directory. Also, another guess, but if the worker is supposed to be a builder, look into “build-users-group” - you should have stuff like “nix-users” groups set up for the worker to work as a builder. If adding openssh.authorizedKeys.keys to the worker does not fix the issue, I would recommend debugging user permissions issues by also adding the users.root.hashedPasswordFile = config.sops.secrets.root-password.path; as well as “user.root.openssh.authorizedKeys.keys”. Oh, I bet this is the main issue, since it’s an immutable setup, add age.generateKey = lib.mkDefault false; on the nix home-manager configuration or nix configuration if you don’t use home-manager. I would also double check key = “user/worker/password”; maybe instead do age = {
sshKeyPaths = [ “/etc/ssh/ssh_host_ed25519_key” ];
keyFile = lib.mkDefault “${config.xdg.configHome}/sops/age/keys.txt”;
};
Again, without more information, I don’t know if you use xdg or have home-manager, sops implementations changed based on your nix / nixos setup. Hope my response is not too illegible, if it is hard to read maybe someone else can word it better it and feel free to correct any mistakes.

3

Hi, I’m facing a very similar issue and wondering if you ever solved it?

4

I’m running a similar issue and would like to know if you’ve solved it!

5

It appears there’s a problem with sops-nix, during the last 2 updates the ~/.config/sops-nix/secrets/ dir where such secrets were saved, does not contain them anymore.

All it contains now is broken symlinks to where the secrets used to be stored on the disk (in a temporary dir).

Probably this might be the cause of what you’re witnessing now. I still haven’t found an answer as to why this is occurring but still looking, if I find anything will update you

LE: it appears after adding this key = ““ field, sops-nix started working fine and my secrets were properly mounted fix(sops): secrets not mounted · dminca/nix-config@f7f9f49 · GitHub

LLE: okay that wasn’t it, it appears (at least on MacOS) after a successful nix run, you’ll have to execute the sops-nix-user script from the nix store (you get a pop-up that a new process will run in background), and it’s sitting on a path like this /nix/store/gzxz6r2hxf5awp1krg4pn88rirmfcrsi-sops-nix-user. Once you run it, the secrets are mounted (there’s no output)

6

I ran into a similar issue myself and spent way too long debugging it. Unlike your situation, though, I had nothing in /persist/etc/sshand the files in /etc/ssh were not symlinked.

I fixed that by telling nixos-anywhere to copy the SSH keys directly to /persist/etc/ssh instead:

# setting up the extra-files directory
install -d -m755 "$temp/persist/etc/ssh"
cp ssh_host_ed25519_key* "$temp/persist/etc/ssh/"
chmod 600 "$temp/persist/etc/ssh/ssh_host_"*

I also switched /etc/ssh to be persisted as a directory rather than individual files. Since impermanence uses bind mounts for directories (rather than symlinking individual files), this fixes the “file already exists” conflicts from any existing SSH keys during activation:

environment.persistence."/persist".directories = [
  "/etc/ssh"
];

And finally, I also configured sops to pull the key from /persist/etc instead of /etc to avoid any timing issues between impermanence and sops during boot.

sops.age = {
  sshKeyPaths = ["/persist/etc/ssh/ssh_host_ed25519_key"];
};

I’m not entirely sure if the last is necessary, but it started working after all three changes.

1 Like
7

Why use impermanence if you are not selective about what you persist?

It shouldn’t be necessary. What I do to prevent the error is:


services.openssh = {
    hostKeys = [
      {
        path = "/persist/etc/ssh/ssh_host_ed25519_key";
        type = "ed25519";
      }
      {
        path = "/persist/etc/ssh/ssh_host_rsa_key";
        type = "rsa";
        bits = 4096;
      }
    ];
  };