Can someone review/criticise my disko + ZFS setup?

Help
1

I recently bought a NAS and I would like to get some review on my disko-ZFS-setup, if someone can spare a few minutes :smile:

The setup has not been tested yet, as the device arrived days before a work trip, but I will be able to play starting next weekend.

Hardware

  • ugreen
    • With 8GB RAM
    • Intel N100
    • Space for 4 HDDs
  • 2x 1TB 990 Samsung Pro SSD
  • 2x 22TB Harddrive (one WD, one Toshiba)

The NAS Hardware has 4 slots, my upgrade plan is to add more 22TB (or bigger) drives to the vdev once the 22TB are running out of free space.

Usecase / Software

I will the use the device for:

  • Backups (borg)
  • Storing Movies, Music, Pictures, Documents (these are all inside different git-annex repositories right now, which I will keep)
  • Services:
    • Navidrome for music streaming
    • Jellyfin for Movie Streaming to kodi
    • Kodi for watching movies and connecting also Spotify, Amazon Prime, Youtube, Arte Mediathek, SomaFM, etc
    • Immich (not 100% decided on this one yet, but looks good)
    • Syncthing “backups” (encrypted remote for syncthing)

The device will be connected to a screen / audio system in my living-room for watching movies and listening to music. Movies will maybe be streamed to devices in the LAN, music will be streamed to devices outside of the LAN as well (tailscale ftw).

ZFS Setup

1 SSD will hold /boot and the nixos root filesystem, both SSDs will hold a 16GB SWAP (because why not…)
The other SSD is used for databases (immich needs a postgres) and caches (the immich machinelearning service needs some cache space, maybe there’s other stuff that could use cache). The syncthing I want to set up will also be on there.

The “media” datasets as well as the database dataset are encrypted and will be decrypted by logging in via SSH and doing the decrypt, the relevant services start after the dataset(s) have been mounted (still need to figure out how to do this).

nixos root, backups, syncthing cache do not need encryption (the later two are incrypted client-side already).

The harddrives will be Z1, not mirror, as I want to be able to grow the vdev later on to up to 4 drives.

Disko

Here’s the disko code I came up with so far (which isn’t tested yet). I just want to know whether the general idea is alright.
I’ve read a bunch of tutorials and guides on the ZFS options I want to use, but I am still not 100% certain whether I did everything alright.

I am especially concerned about the LARC2 cache, where I am not sure whether this needs to be a small partition only (something like 16GB) or whether I rather want something quite big.
The usecase of the device is 2 users, but not even 2 users watching movies in parallel, only maybe running backups while watching a movie, which we do not care whether it takes 10 or 30 minutes to run the backup.

_:

let
  dataDiskDefaults = {
    type = "disk";
    content = {
      type = "gpt";
      partitions.zfs = {
        size = "100%";
        content.type = "zfs";
      };
    };
  };
in
{
  disko.devices = {
    disk = {
      ssd1 = {
        type = "disk";
        device = null; # TODO
        content = {
          type = "gpt";

          partitions = {
            ESP = {
              size = "512M";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot";
                mountOptions = [ "umask=0077" ];
              };
            };

            zfs = {
              size = "100%";
              content = {
                type = "zfs";
                pool = "zroot";
              };
            };

            swap = {
              size = "16G";
              content = {
                type = "swap";
                randomEncryption = true;
                priority = 100;
              };
            };
          };
        };
      };

      ssd2 = {
        type = "disk";
        device = null; # TODO
        content = {
          type = "gpt";

          partitions = {
            ssd2data1 = {
              # TODO: See ssd2zfs_larc2
              size = "100%";
              content = {
                type = "zfs";
                pool = "zssddata";
              };
            };

            swap = {
              size = "16G";
              content = {
                type = "swap";
                randomEncryption = true;
                priority = 100;
              };
            };

            ssd2zfs_larc2 = {
              # TODO: I don't yet understand how big the LARC2 cache should be
              # and whether it actually is possible / a good idea to have a dedicated partition for it
              size = "100%";
              content = {
                type = "zfs";
                pool = "zdata";
              };
            };
          };
        };
      };

      data1 = dataDiskDefaults // {
        device = ""; # TODO
        content.partitions.zfs.content.pool = "zdata";
      };

      data2 = dataDiskDefaults // {
        device = ""; # TODO
        content.partitions.zfs.content.pool = "zdata";
      };
    };

    zpool = {
      zroot = {
        type = "zpool";
        mode = {
          topology = {
            type = "topology";
            vdev = [
              {
                members = [
                  "ssd1"
                ];
              }
            ];
          };
        };

        rootFsOptions = {
          compression = "off";
          "com.sun:auto-snapshot" = "false";
        };

        datasets = {
          "nixroot" = {
            type = "zfs_fs";
            options = {
              mountpoint = "/";

              acltype = "posixacl";
              xattr = "sa";
              atime = "off";
              refreservation = "1G";
              compression = "zstd";
            };
          };
        };
      };

      zssddata = {
        type = "zpool";
        mode = {
          topology = {
            type = "topology";
            vdev = [
              {
                members = [
                  "ssd2data1"
                ];
              }
            ];
          };
        };

        rootFsOptions = {
          compression = "off";
          "com.sun:auto-snapshot" = "false";
        };

        datasets = {
          "cache/immich" = {
            type = "zfs_fs";
            options = {
              mountpoint = "/zfs_immich_cache";

              acltype = "posixacl";
              xattr = "sa";
              atime = "off";
              refreservation = "1G";
              compression = "zstd";
              quota = "50G";

              # TODO
            };
          };

          "cache/syncthing" = {
            type = "zfs_fs";
            options = {
              mountpoint = "/zfs_syncthing_cache";

              acltype = "posixacl";
              xattr = "sa";
              atime = "off";
              refreservation = "1G";
              compression = "zstd";
              quota = "200G";
            };
          };

          "db/postgres" = {
            type = "zfs_fs";
            options = {
              mountpoint = "none";
              encryption = "aes-256-gcm";
              keyformat = "passphrase";

              acltype = "posixacl";
              compression = "off";
              xattr = "sa";
              atime = "off";
              refreservation = "1G";

              # TODO
            };
          };
        };
      };

      zdata = {
        type = "zpool";
        options.cachefile = "none";

        rootFsOptions = {
          # No compression on the root device
          # compression = "zstd";
          compression = "off";

          # Do not snapshot the root device
          "com.sun:auto-snapshot" = "false";
        };

        mode = {
          topology = {
            type = "topology";
            vdev = [
              {
                mode = "raidz1";
                members = [
                  "data1"
                  "data2"
                ];
              }
            ];

            cache = ["ssd2zfs_larc2"];
          };
        };

        datasets = {
          "media/images" = {
            type = "zfs_fs";
            options = {
              mountpoint = "none";
              encryption = "aes-256-gcm";
              keyformat = "passphrase";

              acltype = "posixacl";
              compression = "off";
              xattr = "sa";
              atime = "off";
              refreservation = "1G";
            };
          };

          "media/music" = {
            type = "zfs_fs";
            options = {
              mountpoint = "none";
              encryption = "aes-256-gcm";
              keyformat = "passphrase";

              acltype = "posixacl";
              compression = "off";
              xattr = "sa";
              atime = "off";
              refreservation = "1G";
            };
          };

          "media/movies" = {
            type = "zfs_fs";
            options = {
              mountpoint = "none";
              encryption = "aes-256-gcm";
              keyformat = "passphrase";

              acltype = "posixacl";
              compression = "off";
              xattr = "sa";
              atime = "off";
              refreservation = "1G";
            };
          };

          "media/library" = {
            type = "zfs_fs";
            options = {
              mountpoint = "none";
              encryption = "aes-256-gcm";
              keyformat = "passphrase";

              acltype = "posixacl";
              compression = "off";
              xattr = "sa";
              atime = "off";
              refreservation = "1G";
            };
          };

          backups = {
            type = "zfs_fs";
            mountpoint = "/zfs_backups";
            options = {
              acltype = "posixacl";
              compression = "off";
              xattr = "sa";
              atime = "off";
              refreservation = "1G";
            };
          };

          mirrors = {
            type = "zfs_fs";
            mountpoint = "/zfs_mirrors";
            options = {
              acltype = "posixacl";
              compression = "off";
              xattr = "sa";
              atime = "off";
              refreservation = "1G";
            };
          };
        };
      };
    };
  };
}

I hope I did not forget anything here.

2
  • I think at least this can be moved into the rootFsOptions block for each pool. Doesn’t need to be specified per-dataset. You can take a look at my configs for some working examples.
  • You don’t need to include the “com.sun:auto-snapshot” line at all. In any case, I’d recommend setting up snapshots/ZFS backups using Sanoid/Syncoid at a minimum and then using Restic as a non-ZFS backup option (Borg is fine, but check out Restic if you haven’t).
  • I wouldn’t worry about the LARC2 cache at all. For a minimal use case like this, ZFS will pick what it needs. You can check the Practical ZFS forum for some additional guidance.
  • I’d encourage you test your disko code in a VM before you get the actual hardware. It’s a bit of setup to get the disk layout mostly the same, but worth it. Couple with something like nixos-anywhere.
  • Your setup would be significantly simplified by combining the two SSDs into a mirror and not trying to use a separate SSD for the caches/databases. Again, for home use, it’s really not worth the added complexity! It will also up your read speeds as a bonus.
  • 8 Gb RAM will be fine, but up it to 16 or 32 Gb when you can…you’ll likely notice the difference.

Good luck!

3 Likes
3

You mean that I do not add one or do you mean that I just assign something like 100GB and ZFS adjusts properly?

Because I’ve read somewhere that too much of an LARC2 can cause the normal ARC to grow, and given the fact that I only have 8 gigs of RAM (for now), I don’t want to put too much pressure on that 8 gigs, especially because apps like immich, jellyfin and kodi also need their fair share of RAM (especially immich as I read).

I’d encourage you test your disko code in a VM before you get the actual hardware. It’s a bit of setup to get the disk layout mostly the same, but worth it.

I already have the hardware :laughing: But I am not in a hurry, so playing around, even reinstalling multiple times, is totally fine for me!

Your setup would be significantly simplified by combining the two SSDs into a mirror and not trying to use a separate SSD for the caches/databases. Again, for home use, it’s really not worth the added complexity! It will also up your read speeds as a bonus.

Just to make sure: I assume you mean putting them into RAID-0, right? And then adding datasets for databases/cache etc on that?

8 Gb RAM will be fine, but up it to 16 or 32 Gb when you can…you’ll likely notice the difference.

The device supports up to 16Gb (at least what the vendor says) and has only one RAM slot, but sure, I will look into that if I notice that 8Gb is too small. Of course we will see once the device is used - my usecase is probably small enough that I don’t mind.

4

You shouldn’t need to do anything at all. Per Klara (well respected company and BSD administors):

When should I use L2ARC?

For most users, the answer to this question is simple—you shouldn’t. The L2ARC needs system RAM to index it—which means that L2ARC comes at the expense of ARC. Since ARC is an order of magnitude or so faster than L2ARC and uses a much better caching algorithm, you need a rather large and hot working set for L2ARC to become worth having.

| If you’re curious about how ARC works and why it’s so effective, check out our article on applying the ARC algorithm toZFS.

In general, if you have budget which could be spent either on more RAM or on CACHE vdev devices—buy the RAM! You shouldn’t typically consider L2ARC until you’ve already maxed out the RAM for your system.

No, RAID-1 mirror. Install your system and data (except media) onto the SSDs. You should have plenty of space with 1Tb SSDs. Most Nix services store their data in /var/lib/xxxx. I haven’t moved it to disko yet, but here are the datasets for my home server. Obviously, you can move the /mnt/media datasets to your larger drives instead.

2 Likes
5

Nice, thanks for sharing that!

Is there a specific reason you’re booting with grub?

You’re probably right. I do not plan to run too many service, 1TB should be plenty, yes. Doubling the read-speed should be a good idea I guess :thinking: