Impermanence + sops-nix + nixos-anywhere lead to missing `hashedPasswordFile`s

iilak · July 9, 2025, 7:32pm

Im currently struggling to combine nix-impermanence, sops-nix, and nixos-anywhere. The specific problem is that after boot, I cannot log into any of the users, but during deployment with nixos-anywhere it says that if picked up the decryption key for the secrets…

To make things more concrete:

# bootstrap.sh, run from macOS

#!/bin/bash

temp=$(mktemp -d)

cleanup() {
   rm -rf "$temp"
}
trap cleanup EXIT

mkdir -p "$temp/etc/ssh"
chmod 755 "$temp/etc/ssh"

cp ~/.ssh/id_ed25519 "$temp/etc/ssh/ssh_host_ed25519_key"
chmod 600 "$temp/etc/ssh/ssh_host_ed25519_key"

nix run github:nix-community/nixos-anywhere --              \
   --flake '.#hostname'                                     \
   --build-on remote                                       \
   --extra-files "$temp"                                   \
   --disk-encryption-keys /tmp/secret.key /tmp/secret.key  \
   --target-host nixos@$1

# relevant snippets of flake.nix

{ inputs
, pkgs
, config
, ...
}:
let
  secretspath = builtins.toString inputs.nix-secrets;
in
{
  imports =
    [
      inputs.disko.nixosModules.disko
      inputs.impermanence.nixosModules.impermanence
      ./disk-config.nix
    ];

  boot = {
    loader = {
      systemd-boot.enable = true;
      efi.canTouchEfiVariables = true;
    };
    kernelModules = [ "dm_mod" "dm_crypt" ];
  };

  sops = {
    defaultSopsFile = "${secretspath}/secrets/eiger.yaml";
    secrets = {
      user-worker-password = {
        key = "user/worker/password";
        neededForUsers = true;
      };
    };
    age = {
      sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
    };
  };


  environment.persistence."/persist" = {
    directories = [
      "/etc/nixos"
      "/passwords"
      "/var/lib"
      "/var/log"
      "/var/lib/sops-nix"
      secretspath
    ];
    files = [
      "/etc/machine-id"

      "/etc/ssh/ssh_host_ed25519_key"
      "/etc/ssh/ssh_host_ed25519_key.pub"
      "/etc/ssh/ssh_host_rsa_key"
      "/etc/ssh/ssh_host_rsa_key.pub"
    ];
  };

  fileSystems."/persist".neededForBoot = true;

  fileSystems."/" = {
    device = "tmpfs";
    fsType = "tmpfs";
    options = [ "defaults" "size=2G" "mode=755" ];
  };

  users = {
    groups.media = { };
    mutableUsers = false;
    users = {
      root.hashedPassword = "mkpasswd -c test so that I can debug";
      worker = {
        hashedPasswordFile = config.sops.secrets."user-worker-password".path;
        isNormalUser = true;
        extraGroups = [ "media" ];
      };
    };
  };

  nix.settings.allowed-users = [ "root" ];
};

The output from the bootstrap.sh file is something like this

Lots of disko commands and messages related to it

### Building the system closure ###
warning: Git tree '/Users/iilak/prg/internal/nix-config' is dirty
evaluation warning: The option `hardware.opengl.extraPackages' defined in `/nix/store/01avwgmdmbmmj3f9qy5g2l6xa94h1mci-source/hosts/eiger/jellyfin.nix' has been renamed to `hardware.graphics.extraPackages'.
evaluation warning: The option `hardware.opengl.enable' defined in `/nix/store/01avwgmdmbmmj3f9qy5g2l6xa94h1mci-source/hosts/eiger/jellyfin.nix' has been renamed to `hardware.graphics.enable'.
Warning: Permanently added '192.168.198.132' (ED25519) to the list of known hosts.
### Copying extra files ###
Pseudo-terminal will not be allocated because stdin is not a terminal.
Warning: Permanently added '192.168.198.132' (ED25519) to the list of known hosts.
Warning: Permanently added '192.168.198.132' (ED25519) to the list of known hosts.
Connection to 192.168.198.132 closed.
### Installing NixOS ###
Pseudo-terminal will not be allocated because stdin is not a terminal.
Warning: Permanently added '192.168.198.132' (ED25519) to the list of known hosts.
installing the boot loader...
setting up secrets for users...
Cannot read ssh key '/etc/ssh/ssh_host_rsa_key': open /etc/ssh/ssh_host_rsa_key: no such file or directory
sops-install-secrets: Imported /etc/ssh/ssh_host_ed25519_key as age key with fingerprint age1e2pselcrjk5zl89f00zen4n94zemu5wqhu900zf9w8jg6rl68d3saptlhp
Warning: Source directory '/persist/etc' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/var' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/var/lib' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/nix' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/nix/store' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/etc/nixos' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/passwords' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/var/log' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/var/lib/sops-nix' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/nix/store/hpgfvihqlmccrsl5bs73iz4ql0bv6xm2-source' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
Warning: Source directory '/persist/etc/ssh' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
setting up /etc...
Creating initial /etc/machine-id
A file already exists at /etc/ssh/ssh_host_ed25519_key!
Activation script snippet 'persist-files' failed (1)
Created "/boot/EFI".
Created "/boot/EFI/systemd".
Created "/boot/EFI/BOOT".
Created "/boot/loader".
Created "/boot/loader/keys".
Created "/boot/loader/entries".
Created "/boot/EFI/Linux".
Copied "/nix/store/3rp3z9c3clhcq432cgas4apm4abansfh-systemd-257.5/lib/systemd/boot/efi/systemd-bootaa64.efi" to "/boot/EFI/systemd/systemd-bootaa64.efi".
Copied "/nix/store/3rp3z9c3clhcq432cgas4apm4abansfh-systemd-257.5/lib/systemd/boot/efi/systemd-bootaa64.efi" to "/boot/EFI/BOOT/BOOTAA64.EFI".
Random seed file /boot/loader/random-seed successfully written (32 bytes).
Successfully initialized system token in EFI variable with 32 bytes.
Created EFI boot entry "Linux Boot Manager".
installation finished!
### Rebooting ###
Pseudo-terminal will not be allocated because stdin is not a terminal.
Warning: Permanently added '192.168.198.132' (ED25519) to the list of known hosts.
### Waiting for the machine to become unreachable due to reboot ###
kex_exchange_identification: read: Connection reset by peer
Connection reset by 192.168.198.132 port 22
### Done! ###

The confusing pieces for me:

...
Warning: Source directory '/persist/nix/store/hpgfvihqlmccrsl5bs73iz4ql0bv6xm2-source' does not exist; it will be created for you with the following permissions: owner: 'root:root', mode: '0755'.
...
A file already exists at /etc/ssh/ssh_host_ed25519_key!
Activation script snippet 'persist-files' failed (1)

The first line looks to me like its trying to copy ${secretsPath} to /persist, which probably no necessary since the /nix/persist should already be persisted anyway.

The more confusing thing to me is A file already exists at /etc/ssh/ssh_host_ed25519_key!. Is this the file created by the ISO or is it the one that nixos-anywhere should have copied according to the --extra-files command? The correct file must have been copied to /mnt/etc/ssh, otherwise the build of the system would have not worked since it could not have decrypted the secrets of nix-secrets. But after reboot /run/secrets does not exist, so the hashedPasswordFile for worker does not exist either… Looking at /etc/ssh/ssh_host_ed25519_key it is probably the one created by the ISO (which is btw. correctly symlinked to /persist/etc/ssh), since the public key contains root@hostname instead of username@macOS, which is what was in the original file used to decrypt the secrets during build.

Anybody an idea what Im doing wrong here?

P.S. Not sure if its relevant, but here is the disko part:

_: {
  disko.devices = {
    disk = {
      main = {
        type = "disk";
        device = "/dev/nvme0n1";
        content = {
          type = "gpt";
          partitions = {
            boot = {
              size = "1M";
              type = "EF02";
            };
            ESP = {
              size = "512M";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot";
                mountOptions = [ "defaults" "umask=0077" ];
              };
            };
            luks = {
              size = "100%";
              content = {
                type = "luks";
                name = "crypted";
                passwordFile = "/tmp/secret.key";
                settings = {
                  allowDiscards = true;
                  bypassWorkqueues = true;
                };
                content = {
                  type = "btrfs";
                  extraArgs = [ "-f" ];
                  subvolumes = {
                    "/nix" = {
                      mountpoint = "/nix";
                      mountOptions = [ "compress=zstd" "noatime" ];
                    };
                    "/persist" = {
                      mountpoint = "/persist";
                      mountOptions = [ "compress=zstd" "noatime" ];
                    };
                    "/home" = {
                      mountpoint = "/home";
                      mountOptions = [ "compress=zstd" "noatime" ];
                    };
                    "/swap" = {
                      mountpoint = "/swap";
                      swap.swapfile.size = "4G";
                    };
                  };
                };
              };
            };
          };
        };
      };
    };
  };
}

Eastwdd · July 11, 2025, 12:21pm

I’ve never personally used nixos-anywhere so this might be wrong. Without more information, it looks like an incomplete sops setup. Try adding “user..openssh.authorizedKeys.keys = [config.sops.secrets.“pathShouldCorrespondToPersistentSetInYaml/sops_encrypted/ed25519_key/id_ed25519”.path];” if it does not already exist in your nix configuration. Not in the home configuration *unless home-manager standalone without nixos operating system is being used, but I’m going to assume your home manager is an integrated nixos submodule. If the worker’s permissions are still not set up correctly, it won’t be capable of creating and interacting with the “/run/secrets” directory. Also, another guess, but if the worker is supposed to be a builder, look into “build-users-group” - you should have stuff like “nix-users” groups set up for the worker to work as a builder. If adding openssh.authorizedKeys.keys to the worker does not fix the issue, I would recommend debugging user permissions issues by also adding the users.root.hashedPasswordFile = config.sops.secrets.root-password.path; as well as “user.root.openssh.authorizedKeys.keys”. Oh, I bet this is the main issue, since it’s an immutable setup, add age.generateKey = lib.mkDefault false; on the nix home-manager configuration or nix configuration if you don’t use home-manager. I would also double check key = “user/worker/password”; maybe instead do age = {
sshKeyPaths = [ “/etc/ssh/ssh_host_ed25519_key” ];
keyFile = lib.mkDefault “${config.xdg.configHome}/sops/age/keys.txt”;
};
Again, without more information, I don’t know if you use xdg or have home-manager, sops implementations changed based on your nix / nixos setup. Hope my response is not too illegible, if it is hard to read maybe someone else can word it better it and feel free to correct any mistakes.