Trouble cleaning root with ZFS rollback

Help
1

I set up NixOS using disko with ZFS root and a root, nix and persisted data set. My problem is that rolling back to zroot@blank changes nothing in the filesystem. It actually worked at some point, but now it fails to erase newly written files.

The only thing I can imagine, is that somehow the creation of that snapshot at install time simply happens every reboot and as such includes all newly written files. But is this the thing I am missing?

I have impermanence set up too in my NixOS flake, but it does only persist the files I want it to and the files I want it to erase are not declaratively persisted.

{ lib, ... }:

let
        mountOptions = {
                mountpoint = "legacy";
                atime = "off";
                "com.sun:auto-snapshot" = builtins.toString false;
        };

in
{
        disko.devices = {
                disk = {
                        main = {
                                type = "disk";
                                device = lib.mkDefault "/dev/nvme0n1";
                                content = {
                                        type = "gpt";
                                        partitions = {
                                                boot = {
                                                        size = "512M";
                                                        type = "EF00";
                                                        content = {
                                                                type = "filesystem";
                                                                format = "vfat";
                                                                mountpoint = "/boot";
                                                                mountOptions = [ "umask=0077" ];
                                                        };
                                                };

                                                swap = {
                                                        size = "16G";
                                                        content = {
                                                                type = "swap";
                                                                randomEncryption = true;
                                                                priority = 100;
                                                        };
                                                };

                                                root = {
                                                        size = "100%";
                                                        content = {
                                                                name = "luks";
                                                                type = "luks";
                                                                passwordFile = "/tmp/passwordFile";
                                                                settings.allowDiscards = true;
                                                                content = {
                                                                        type = "zfs";
                                                                        pool = "zroot";
                                                                };
                                                        };
                                                };
                                        };
                                };
                        };
                };

                zpool = {
                        zroot = {
                                type = "zpool";
                                rootFsOptions = {
                                        compression = "zstd";
                                        "com.sun:auto-snapshot" = builtins.toString false;
                                        mountpoint = "none";
                                };
                                mountpoint = "/";
                                postCreateHook = ''
                                        zfs list -t snapshot -H -o name | grep -E '^zroot@blank$' || zfs snapshot zroot@blank
                                '';

                                datasets = {
                                        root = {
                                                type = "zfs_fs";
                                                options = mountOptions;
                                                mountpoint = "/";
                                        };
                                        nix = {
                                                type = "zfs_fs";
                                                options = mountOptions;
                                                mountpoint = "/nix";
                                        };
                                        persist = {
                                                type = "zfs_fs";
                                                options = mountOptions;
                                                mountpoint = "/persist";
                                        };
                                };
                        };
                };
        };

        fileSystems."/persist".neededForBoot = true;
}

If this is not enough of the configuration to point out my mistakes, let me know and I will provide more.

2

I assume that the fs creation and the snapshot are fine?

Disko should not run on every boot AFAIK.

You can check the content of the snapshot by mounting it.

If everything is fine, you might want to share the part that rolls back.

Also it’s important if you use scripted or systems initialization.

For me I was wondering the same when I played around with systemd init/perlless; there the scripted hooks don’t exist/work.

3

Filesystems are fine and usable.

boot.initrd = {
    postResumeCommands = lib.mkAfter ''
        zfs rollback -r zroot@blank
    '';
};

This does the rollback part. I also checked and it seems like it is running at boot.

Zfs also shows that the required snapshot exists.

NAME           USED    AVAIL    REFER   MOUNTPOINT
zroot@blank    0B      -        24k     -

I mounted the snapshot and it was empty. Then rolling back should delete all files, right?

I also checked and the files I want it to delete are not persisted and do not exist under /persist.

4

Snapshot looks fine to me, are you by chance using systems based init? As you are using the script based hooks for rollback, that would explain why it’s not working.

Please check if boot.initrd.systemd.enable is set to true, and if yes you need to create a systemd unit for the rollback that runs in initrd

5

Systemd initrd is disabled and by my experience I get an error when trying to use postResumeCommands with systemd initrd. At some point I switched to systemd init and back, but it is now disabled.

6

Okay, the error message is then “new" (added after I switched).

But then I am out of low hanging fruits as reason.

Sorry. Maybe someone else has an idea

1 Like
7

Double check question: shouldnt your snapshot be something like zroot/root@blank according the Disko config you attached, as I understood you want to reset / don’t you?

IMHO you currently try to rollback a snapshot on “pool level” and then mount your populated dataset root

2 Likes
8

You’re right! Just checked and either changing rootFsOptions.mountpoint to “/“ and deleting the root zfs data set in disko or rolling back to zroot/root@blank (its existence must first be declared in disko during install time) should work. Now all irrelevant files are gone!

Thank you very much! Would never have thought of this alone!

2 Likes