I have a laptop and a couple of other systems running NixOS that I’ve opted into using BTRFS snapshots for impermanence, Secure Boot using Lanzaboote, SystemD initrd for TPM unlocking of my LUKS partition, and Flakes to manage everything… Despite most of these features being marked as experimental, everything has been working near flawlessly. My system boots up in less than ten seconds without needing to enter a passphrase, with Secure Boot enabled, and system and home configurations defined in flakes that make it super easy to update configurations for multiple systems and users. The “last frontier” so to speak is hibernation and swap. (And FDE.) Here is my repo that drives all of this.
… Then a couple of days ago I realized that I couldn’t use systemd initrd and specify boot.initrd.postDeviceCommands to reset my root subvolume to a blank partition. I got it working without systemd as initrd and didn’t check back on it, so all my system state has been accumulating cruft for a while now. Unfortunately using systemd as initrd is necessary for the TPM unlocking to work, so I have a one-or-the-other situation here.
I haven’t found any indication that anyone has worked around this issue using impermanence via BTRFS or ZFS. Hypothetically it wouldn’t be an issue if using TMPFS, but that’s suboptimal for best performance.
Based on some nixpkgs PRs (167153, 168269) it seems like it should be possible to declare a unit service spec to perform the necessary rollback in stage 1 - has anyone succeeded in doing so, or would know how to modify a NixOS configuration in support of this? I have very little experience when it comes to systemd, and utilizing it via Nix seems very confusing.
I’m using impermanence with systemd initrd and this service config. I iterated on it in the matrix impermanence chat room with invaluable help from @lilyinstarlight.
Edit: Here’s where I ended up for the BTRFS version, borrowing from this blog post:
boot.initrd.systemd.services.rollback = {
description = "Rollback BTRFS root subvolume to a pristine state";
wantedBy = [
"initrd.target"
];
after = [
# LUKS/TPM process
"systemd-cryptsetup@enc.service"
];
before = [
"sysroot.mount"
];
unitConfig.DefaultDependencies = "no";
serviceConfig.Type = "oneshot";
script = ''
mkdir -p /mnt
# We first mount the btrfs root to /mnt
# so we can manipulate btrfs subvolumes.
mount -o subvol=/ /dev/mapper/enc /mnt
# While we're tempted to just delete /root and create
# a new snapshot from /root-blank, /root is already
# populated at this point with a number of subvolumes,
# which makes `btrfs subvolume delete` fail.
# So, we remove them first.
#
# /root contains subvolumes:
# - /root/var/lib/portables
# - /root/var/lib/machines
#
# I suspect these are related to systemd-nspawn, but
# since I don't use it I'm not 100% sure.
# Anyhow, deleting these subvolumes hasn't resulted
# in any issues so far, except for fairly
# benign-looking errors from systemd-tmpfiles.
btrfs subvolume list -o /mnt/root |
cut -f9 -d' ' |
while read subvolume; do
echo "deleting /$subvolume subvolume..."
btrfs subvolume delete "/mnt/$subvolume"
done &&
echo "deleting /root subvolume..." &&
btrfs subvolume delete /mnt/root
echo "restoring blank /root subvolume..."
btrfs subvolume snapshot /mnt/root-blank /mnt/root
# Once we're done rolling back to a blank snapshot,
# we can unmount /mnt and continue on the boot process.
umount /mnt
'';
};
Hi, I have tried with your way, but my service is failed, because it can’t found the disk device at the initrd.target stage. I tried adding systemd-udev-settle.service to the after list, but it didn’t work. Do you have any suggestions? Thanks!
update: adding dev-disk-by\x2did-ata\x2dPhison_SATA_SSD_2143332432454365.device to after list also not work…
Thanks for your help! But it does’t work for me, after adding initrd-root-device.target I try to add ls /dev to the script, I don’t see my disk in the results… Maybe it has something to do with the fact that I’m using a USB portable SSD.
Make sure you double escape those \x symbols. In nix, the \x string becomes x, but you need it to become \x, so you need to actually write \\x.
Wouldn’t have anything to do with that, other than if you don’t have the kernel modules for usb in the initrd. But if it boots at all then this wouldn’t be the problem.
This can work, but only if you’re doing a normal device-based root FS. For the non-root FS, you still need to order after the correct device, and something like ZFS will need to be ordered after the import service instead.
This device has already been escaped, it was originally: /dev/disk/by-id/ata-Phison_SATA_SSD_2143332432454365. I’ve heard that systemd unit doesn’t allow using “-”
Yes, systemd needs you to replace - with \x2d, but if you just put that as a nix string you’ll get x2d instead of \x2d. You need the actual string in the resulting file to have that \, but nix is removing it through nix’s own string escaping syntax. So to get a file containing \x2d you need to write \\x2d in your nix string.
That is to say:
builtins.toFile "not-escaped" "\x2d"
Produces a file containing x2d in it, while
builtins.toFile "escaped" "\\x2d"
produces a file with \x2d in it, which is what you need so that systemd sees the escaping
{ disk, swap, ... }:
let
restore = "false";
in
{
imports = [ (import ../disko/main-efi-btrfs.nix { inherit disk swap; }) ];
boot.initrd.systemd.enable = lib.mkForce true;
boot.initrd.systemd.services.rollback = {
description = "Rollback BTRFS @ subvolume to a pristine state";
wantedBy = [ "initrd.target" ];
after = [ "dev-disk-by\\x2did-ata\\x2dPhison_SATA_SSD_2143332432454365.device" ];
before = [ "sysroot.mount" ];
unitConfig.DefaultDependencies = "no";
serviceConfig.Type = "oneshot";
script = ''
echo "Starting rollback service..."
mount_point="/.btrfs/main/main"
mkdir -p "$mount_point"
mount -t btrfs -o subvol=/ "${disk}-part2" "$mount_point"
# Something balalababal.....
'';
};
boot.initrd.systemd.services.restore = {
description = "Restore a specific BTRFS @ subvolume from backup";
wantedBy = [ "initrd.target" ];
after = [ "rollback.service" ];
before = [ "sysroot.mount" ];
unitConfig.DefaultDependencies = "no";
serviceConfig.Type = "oneshot";
serviceConfig.Environment = "RESTORE_ACTIVE=${restore}";
script = ''
# Something blabalbalalal....
'';
};
}
It will fail with:
Sep 02 22:18:59 localhost rollback-start[85]: Starting rollback service...
Sep 02 22:18:59 localhost rollback-start[85]: Created mount point directory: /.btrfs/main/main
Sep 02 22:18:59 localhost rollback-start[91]: mount: /.btrfs/main/main: special device /dev/disk/by-id/ata-Phison_SATA_SSD_2143332432454365-part2 does not exist.
Sep 02 22:18:59 localhost rollback-start[91]: dmesg(1) may have more information after failed mount system call.
Sep 02 22:18:59 localhost systemd[1]: rollback.service: Main process exited, code=exited, status=32/n/a
wait… are you doing LUKS? Because then you need to depend on the LUKS volume, not the physical disk. Also, you probably need to add a wants or bindsTo dependency on said disk, not just an after ordering.
No, I’m not doing LUKS, my disk is unencrypted… I will try wants or bindsTo
//update
It works! By adding requires = [ "dev-disk-by\\x2did-ata\\x2dPhison_SATA_SSD_0C3307050CA900023301.device" ]; wants or bindsTo maybe also can do that, I will test that witch is better.
Thank you!
requires is probably correct. Unlike wants, it’ll make sure the service doesn’t run if the disk fails to appear (i.e. it’s unplugged or something). And bindsTo won’t add anything of value, since all it does is make it so your unit would stop if the dependency would enter any inactive state, which isn’t really relevant for a oneshot like this.