Early boot remote decryption

Help
1

Hi,

I’m a bit lost trying to set up remote decyprtion of my hard drive via SSH on my Raspberry Pi 4b. This is the relevant part of my configuration:

boot.loader.systemd-boot.enable = true;
boot.loader.systemd-boot.configurationLimit = 10;
boot.loader.systemd-boot.editor = false;
boot.loader.efi.canTouchEfiVariables = false;

# Network card drivers
boot.initrd.kernelModules = [ "genet" ];

# It may be necessary to wait a bit for devices to be initialized.
# See: https://github.com/NixOS/nixpkgs/issues/98741
boot.initrd.preLVMCommands = lib.mkOrder 400 "sleep 1";

boot.initrd.network = {
  enable = true;
  ssh = {
    enable = true;
    port = 32983;
    hostKeys = [ "/root/initrd-ssh-key" ];
    # All users being a member of the "wheel" group are allowed to connect and enter the password.
    authorizedKeys = with lib; concatLists (mapAttrsToList (name: user: if elem "wheel" user.extraGroups then user.openssh.authorizedKeys.keys else []) config.users.users);
  };
};

I generated an SSH host key before rebuilding the system (ssh-keygen -t ed25519 -N "" -f /root/initrd-ssh-key). I see the following error messages at the first boot stage:

chmod: /root/initrd-ssh-key: No such file or directory
Unable to load host key: /root/initrd-ssh-key
sshd: no hostkeys available -- exiting.

I researched my problem a bit and found a similar issue: initrd ssh: cp: cannot stat '/etc/secrets/initrd/ssh_host_ed25519_key': No such file or directory · Issue #84976 · NixOS/nixpkgs · GitHub

So, I tried setting hostKeys = [ /root/initrd-ssh-key ];, since this was suggested to resolve the problem. However, I end up with the same error:

chmod: /etc/ssh/nitrd-ssh-key: No such file or directory
Unable to load host key: /etc/ssh/nitrd-ssh-key
sshd: no hostkeys available -- exiting.

Only this time, the host key is being looked up at a location I didn’t specify anywhere in my configuration. Does anybody know what is going on here? Is this a problem with systemd-boot? I know a similar configuration was working for me on my old RPi 3b with the extlinux boot loader.

Just in case it’s relevant, here’s the metadata on my RPi:

  • system: "aarch64-linux"
  • host os: Linux 5.10.52, NixOS, 21.11 (Porcupine)
  • multi-user?: yes
  • sandbox: yes
  • version: nix-env (Nix) 2.5pre20211007_844dd90
  • nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos
2

You’ll could try to add that key as a secret the initial ram disk. I think something like this might do:

boot.initrd.secrets = {
  "/root/initrd-ssh-key" = "/root/initrd-ssh-key";
};

https://nixos.org/manual/nixos/stable/options.html#opt-boot.initrd.secrets

Its worth noting that that key will not be encrypted in the initrd.

3

Thank you for the reply. I wasn’t sure which hostKeys setting to go for, path vs string, but it doesn’t matter: both lead to the same error, even with the additional configuration you suggested.

I’m thinking of changing to the extlinux boot loader and trying the configuration that worked for me on my old RPi.

4

Is there a way to debug my problem, e.g. spawning a root shell after sshd fails? I’m straight away being asked for my decryption password and can’t interrupt that process.

5

I imagine you could use the “boot.initrd.network.postCommand” to launch a shell for debugging.

For additional reference, here’s the ssh/initrd config I’m using. My systems do display the zfs unlock prompt on their connected display, but after logging in and unlocking with SSH that process gets killed.

What kind of disk encryption are you using?

6

I’m not using it on a Pi, but I was storing my key under /boot, and it works, but the secrets option above seems like a better route that I wasn’t familiar with.

hostKeys = [ /boot/my_host_key ];

7

Thanks, I spawned a root shell using this in my config: boot.initrd.networkd.postCommands = lib.mkAfter ''/bin/ash'';. This is how the root directory looks (du -sh /):

0    	bin
0    	dev
0    	etc
0    	init
0    	lib
25.1M	nix
0       proc
4.0K    root
1.5M    run
0    	sys
0    	tmp
0    	var

where bin and init are actually symlinks to places in the Nix store. Does that look normal to you?

I used LUKS to encrypt the root partition, not sure anymore if type 1/2, and then just formatted it using ext4.

In the meantime I got it working by switching to the extlinux bootloader again. In this case the key is copied to the Nix store and linked at the specified location. Though now I have to pass the --impure flag to the nixos-rebuild command, otherwise error: access to path '/var/lib/initrd-ssh-key' is forbidden in restricted mode is thrown. This didn’t happen with the systemd-bootloader though, I have no idea what is going on :S

8

I’m not sure what’s going on here. My hunch is the initrd gets built differently for a Pi than x86_64 systems (my remote decryption stuff was all uefi & x86_64). If I get some time I’ll dig out one of the Pis I have somewhere and see if I can get something working.

9

If you look closely, it’s not the same error when using path ([ /root/initrd-ssh-key ]) compared to using a string ([ "/root/initrd-ssh-key" ]). For some reason the i in initrd in the path gets dropped when using path, it says nitrd-ssh-key instead of initrd-ssh-key.

I’m having the same issue on my x86_64 system. I can’t get the host keys in to initrd properly.

Any ideas?

10

I am also running into this same issue, on a x86 QEMU VM running nixos 22.05 with GRUB Legacy boot.

In particular, nixos-rebuild switch isn’t copying boot.initrd.secrets (and related variables like boot.initrd.network.ssh.hostKeys) into the initrd image.

This causes the error in stage-1-init, where chmod can’t find the ssh key, and sshd exits out immediately due to no hostkeys.

And furthermore: When I did finally manage to get a key into the initrd – changing boot.initrd.network.ssh.hostKeys would not delete it from the initrd – which I checked using @maxsc 's trick above spawning a root shell in boot.initrd.network.postCommands. This might also be related to issues like:
https://github.com/NixOS/nixpkgs/issues/101462#issuecomment-1172926129
https://github.com/NixOS/nixpkgs/issues/114594

So, it seems like there’s some issue where nixos-rebuild is not properly recreating the initrd after changes to configuration.nix, and instead wrongly uses parts of a pre-existing initrd from previous generations.

I finally managed to get the keys into initrd by removing the keys from my config, rebuilding, deleting all previous generations, adding the keys back, and then rebuilding again. In particular:

  1. Remove/comment any configurations related to boot.initrd.secrets or boot.initrd.network.ssh.hostKeys.
  2. Run nixos-rebuild switch (first time).
  3. Reboot.
  4. Run nix-collect-garbage -d.
  5. Add back the boot.initrd configurations.
  6. Run nixos-rebuild switch (second time).

Then, the keys successfully wound up in the initrd.

3 Likes