I’ve followed this tutorial to use systemd.initrd. So far it works nicely and I am able to login via ssh to my remote systems initrd whenever I need to unlock my LUKS device after attaching the required FIDO2 passkey.
Now I wanted to attach the FIDO2 passkey via USB/IP, more pecisely usbip-ssh. And I somehow managed to get it working, but not very reliable. It took me a long time to fiddle around with the parameters RestartSec, startLImitBurst and startLimitIntervalSec to get it somewhat working in 9 out of 10 cases.
My config part for the systemd unnit is the following:
boot.initrd.systemd.services.usbipkey =
{
description = "USBIP-SSH served FIDO2 key";
wantedBy = ["initrd.target"];
after = [ "systemd-network.service" ];
before = ["systemd-cryptsetup@cryptroot.service"];
requiredBy = ["systemd-cryptsetup@cryptroot.service"];
startLimitBurst = 12;
startLimitIntervalSec = 60;
unitConfig.DefaultDependencies = "no";
path = with pkgs;
[
usbip-ssh
openssh
perl
];
script=
''
usbip-ssh -p 2222 -i /secrets/boot/ssh/ssh_host_ed25519_key -o StrictHostKeyChecking=accept-new root@192.168.2.116 Nitrokey
'';
serviceConfig =
{
Type = "simple";
RestartSec = "5s";
Restart = "always";
};
};
boot.initrd.systemd.extraBin =
{
usbip-ssh = "${pkgs.usbip-ssh}/bin/usbip-ssh";
vim = "${pkgs.vim}/bin/vim";
};
boot.initrd.systemd.storePaths =
[
pkgs.perl
pkgs.usbip-ssh
pkgs.openssh
];
boot.initrd.kernelModules = [ "usbip-host" "vhci-hcd" ];
boot.initrd.systemd.settings.Manager = { LogLevel="debug"; };
I guess I’m mising a requirement for my unit to start without error on the first try or to have some other unit that is not a oneshot service as systemd-cryptsetup@.service for the after target.
But so far I couldn’t really figure out why the unit fails the first few times to begin with and using a unit from the dependency tree of systemd-cryptsetup@.service resulted in similiar problems.
Is there a better way to find out why the unit fails the first few times the setting boot.initrd.systemd.settings.Manager = { LogLevel="debug"; };?
Or is there anybody who has done something similiar and would be able to point me in the right direction? Especially with the luks unlocking unit that should wait for my unit to be running properly. Would making my unit something other than Type=simple help?
I would be really grateful for any help.
Maybe to the ssh command maybe add “-o LogLevel=DEBUG” and then look at the journal?
Ok I tried a few things but nothing really works. Setting the SSH debug level did not provide any more data than before. It seems the service gets killed by some other cause than ssh problems (restart of depending units like network or secret management maybe..). Here is the full log:
Blockquote
Dec 20 11:32:03 zima usbipkey-start[120]: perl: warning: Setting locale failed.
Dec 20 11:32:03 zima usbipkey-start[120]: perl: warning: Please check that your locale settings:
Dec 20 11:32:03 zima usbipkey-start[120]: LANGUAGE = (unset),
Dec 20 11:32:03 zima usbipkey-start[120]: LC_ALL = (unset),
Dec 20 11:32:03 zima usbipkey-start[120]: LC_CTYPE = (unset),
Dec 20 11:32:03 zima usbipkey-start[120]: LC_NUMERIC = (unset),
Dec 20 11:32:03 zima usbipkey-start[120]: LC_COLLATE = (unset),
Dec 20 11:32:03 zima usbipkey-start[120]: LC_TIME = (unset),
Dec 20 11:32:03 zima usbipkey-start[120]: LC_MESSAGES = (unset),
Dec 20 11:32:03 zima usbipkey-start[120]: LC_MONETARY = (unset),
Dec 20 11:32:03 zima usbipkey-start[120]: LC_ADDRESS = (unset),
Dec 20 11:32:03 zima usbipkey-start[120]: LC_IDENTIFICATION = (unset),
Dec 20 11:32:03 zima usbipkey-start[120]: LC_MEASUREMENT = (unset),
Dec 20 11:32:03 zima usbipkey-start[120]: LC_PAPER = (unset),
Dec 20 11:32:03 zima usbipkey-start[120]: LC_TELEPHONE = (unset),
Dec 20 11:32:03 zima usbipkey-start[120]: LC_NAME = (unset),
Dec 20 11:32:03 zima usbipkey-start[120]: LANG = “C.UTF-8”
Dec 20 11:32:03 zima usbipkey-start[120]: are supported and installed on your system.
Dec 20 11:32:03 zima usbipkey-start[120]: perl: warning: Falling back to the standard locale (“C”).
Dec 20 11:32:03 zima systemd[1]: usbipkey.service: Child 114 belongs to usbipkey.service.
Dec 20 11:32:03 zima systemd[1]: usbipkey.service: Main process exited, code=exited, status=255/EXCEPTION
Dec 20 11:32:03 zima systemd[1]: usbipkey.service: Failed with result ‘exit-code’.
Dec 20 11:32:03 zima systemd[1]: usbipkey.service: Service will restart (restart setting)
Dec 20 11:32:03 zima systemd[1]: usbipkey.service: Changed running → failed-before-auto-restart
Dec 20 11:32:03 zima systemd[1]: usbipkey.service: Consumed 192ms CPU time, 7.7M memory peak.
Dec 20 11:32:03 zima systemd[1]: usbipkey.service: Next restart interval calculated as: 5s
Dec 20 11:32:03 zima systemd[1]: usbipkey.service: Changed failed-before-auto-restart → auto-restart
Dec 20 11:32:03 zima systemd[1]: usbipkey.service: Control group is empty.
Dec 20 11:32:08 zima systemd[1]: usbipkey.service: Service restart interval 5s expired, scheduling restart.
Dec 20 11:32:08 zima systemd[1]: usbipkey.service: Trying to enqueue job usbipkey.service/start/restart-dependencies
Dec 20 11:32:08 zima systemd[1]: usbipkey.service: Installed new job usbipkey.service/start as 71
Dec 20 11:32:08 zima systemd[1]: usbipkey.service: Enqueued job usbipkey.service/start as 71
Dec 20 11:32:08 zima systemd[1]: usbipkey.service: Scheduled restart job, restart counter is at 1.
Dec 20 11:32:08 zima systemd[1]: usbipkey.service: Changed auto-restart → auto-restart-queued
Dec 20 11:32:08 zima systemd[1]: usbipkey.service: stopping held back, waiting for: systemd-cryptsetup@cryptroot.service
Dec 20 11:32:08 zima systemd[1]: usbipkey.service: Will spawn child (service_enter_start): /nix/store/qak5p1jayh3717g8pcs5gx7x15fpa8hs-unit-script-usbipkey-start/bin/usbipkey-start
Dec 20 11:32:08 zima systemd[1]: usbipkey.service: Passing 0 fds to service
Dec 20 11:32:08 zima systemd[1]: usbipkey.service: About to execute: /nix/store/qak5p1jayh3717g8pcs5gx7x15fpa8hs-unit-script-usbipkey-start/bin/usbipkey-start
Dec 20 11:32:08 zima systemd[1]: Serializing sd-executor-state to memfd.
Dec 20 11:32:08 zima systemd[1]: usbipkey.service: Forked /nix/store/qak5p1jayh3717g8pcs5gx7x15fpa8hs-unit-script-usbipkey-start/bin/usbipkey-start as 212 (via CLONE_INTO_CGROUP)
Dec 20 11:32:08 zima systemd[1]: usbipkey.service: Changed auto-restart-queued → start
It seems to be failing exactly once, to me this is a indicator that some network unit or serveice that my service depends on restarts… That is the only reason I can think of.
But regardless of to why it restarts, I guess my real problem is that I need to find a way to trigger a restart of systemd-cryptsetup@cryptroot.service when my service is in state active (running). Since a service providing a network shared hardware token may restart due to network problems at any time.
Using something like ExecStartPost="systemctl restart systemd-cryptsetup@cryptroot.service" was not working either. Not sure it would make a real difference for service type simple.
Ok, I looked at it from a completely wrong angle. I now have it working by what I think is the more systemd like way of doing things.
In the end what my service does is providing a USB FIDO2 token. So an entry in /dev and a corresponding systemd device file. I created a udev rule to make the device name more predictable and switched the RequiredBy dependency to the cryptsetup service for a WantedBy. I also added a condition that the service is only started when the created device unit doesn’t exist. This way I ensure that I can also plugin the USB token directly at the server.
Next step was creating a oneshot unit that has BindsTo and after to the hardware device unit and upholds the cryptsetup service, while also being upheld by the usbipkey service.
The only thing that is still open is to prevent systemd from going into the emergency target when the key is not provided for a certain time. But that should be no problem with a bit of further reading.