Devices not visable using `initrd.systemd` with btrfs

1

I use btrfs with impermanence.
It uses a volume pool located in /dev/pool/root.
I am unable to access this device in a oneshot service I use to revert the root to clean state for impermanence. What can I do?

    boot.initrd.systemd.services.rollback = {
      description = "Rollback BTRFS root subvolume to a pristine state";
      wantedBy = [
        "initrd.target"
      ];
      before = [
        "sysroot.mount"
      ];
      unitConfig.DefaultDependencies = "no";
      serviceConfig.Type = "oneshot";
      script = ''
        mkdir /btrfs_tmp
        mount /dev/pool/root /btrfs_tmp
        if [[ -e /btrfs_tmp/root ]]; then
            mkdir -p /btrfs_tmp/old_roots
            timestamp=$(date --date="@$(stat -c %Y /btrfs_tmp/root)" "+%Y-%m-%-d_%H:%M:%S")
            mv /btrfs_tmp/root "/btrfs_tmp/old_roots/$timestamp"
        fi

        delete_subvolume_recursively() {
            IFS=$'\n'
            for i in $(btrfs subvolume list -o "$1" | cut -f 9- -d ' '); do
                delete_subvolume_recursively "/btrfs_tmp/$i"
            done
            btrfs subvolume delete "$1"
        }

        for i in $(find /btrfs_tmp/old_roots/ -maxdepth 1 -mtime +${builtins.toString cfg.removeTmpFilesOlderThan}); do
            delete_subvolume_recursively "$i"
        done

        btrfs subvolume create /btrfs_tmp/root
        umount /btrfs_tmp
      '';
    };
2

First of all, you probably need boot.initrd.services.lvm.enable = true; if you don’t have it already.

Secondly, you probably need to order the service after the device with wants = ["dev-pool-root.device"]; and after = ["dev-pool-root.device"];.

3

@maralorn has an impermanent setup which wipes the root manually:

Perhaps you can use his code as a reference here.

4

Seems unfortunate to do the equivalent of rm -rf when btrfs subvolumes can handle this so much nicer and faster.

5

The approach has some merit as it is portable to practically any filesystem.

6

This made it see /dev/pool/root.

However, now I am experiencing this issue.

I am using disko though, so I don’t know where I can use fsType. Can you help?

I do have this line:
boot.initrd.supportedFilesystems = ["btrfs"];

7

@ElvishJerricco I did want to quickly request you review my reply. Once again thanks for your help!

8

That comment you quoted is probably not relevant to you. I’d be surprised if disko isn’t automatically setting fsType correctly, and if it’s not then

this is doing the trick anyway.

What makes you think you’re experiencing that same issue? There can be a lot of things that cause /sysroot to fail to mount.

9

Just because /sysroot wasn’t mounting. That was the only issue I could find mentioning it

10

Maybe it has something to do with the wantedBy.
Maybe basic instead of initrd. Because it’s already an initrd service, like you mentioned before.

wantedBy = [ "basic.target" ];

before = [

"initrd-root-fs.target"

"sysroot-var-lib-nixos.mount"

];

after = [ "sysroot.mount" ];

Perhaps

11

No, I wouldn’t expect that to be an improvement. In fact ordering after = ["sysroot.mount"]; should certainly break it. We’ll have to see some logs to know what happened.

1 Like
12

It fails to mount /sysroot/var/lib, according to the logs.

Mounting /sysroot/nix...
Mounting /sysroot/persist...
Mounting /sysroot/var/lib...
Starting Mountpoints configured in the  Real Root...
Mounted /sysroot/nix...
Failed to mount /sysroot/var/lib.

@ElvishJerricco

Then it cant mount the /sysroot/nix/.ro-store, but i think that is because var/lib failed.

It does mount /sysroot just not the /sysroot/var/lib part.

13

When it fails in initrd, it should give you access to the emergency shell if you have boot.initrd.systemd.emergencyAccess set. If not, you can add rd.systemd.debug_shell to the kernel params if you’re using a very recent nixos unstable, and then press ctrl-alt-f9 to go to the debug shell. From there you can use commands like systemctl status and journalctl to get a more detailed log about the mount units that failed

2 Likes
14

Okay, it says bad fs type.
I might be able to circumvent using systemd in the initrd. So I will try that. It’s for microvm.nix

15

wait a sec, yea, if /var/lib is a btrfs subvolume, then you can’t just be deleting it like this. You have to also recreate it.

16

It’s not
Here is my disko config.

17

I do know that using microvm.nix requires /var/lib to be used. Maybe that is causing some sort of issue.

18

Okay, here is the code that is looking for /var/lib and .ro-store

        {
          source = "/var/lib/microvms/${config.networking.hostName}/storage/journal";
          mountPoint = "/var/log/journal";
          tag = "journal";
          proto = "virtiofs";
          socket = "journal.sock";
        }
        {
          proto = "virtiofs";
          source = "/var/lib/microvms/${config.networking.hostName}/storage/etc/ssh";
          mountPoint = "/etc/ssh";
          tag = "ssh";
        }
        {
          proto = "virtiofs";
          source = "/var/lib/microvms/${config.networking.hostName}/storage/var/lib";
          mountPoint = "/var/lib";
          tag = "var-lib";
        }
        {
          source = "/nix/store";
          mountPoint = "/nix/.ro-store";
          tag = "ro-store";
          proto = "virtiofs";
        }
        {
          proto = "virtiofs";
          source = "/run/secrets/${config.networking.hostName}";
          mountPoint = "/run/secrets";
          tag = "secrets";
        }

From microvms, this is causing the current issue

19

Alright, after some troubleshooting it seems that i was using the guest module for microvms, which messes with the systemd initrd. Now that i am no longer using it, your original solution works, thank you!