Advice for 2 drive setup for NixOS

Help
1

I’m planning to switch over to NixOS after setting things up and using NixOS on a VM for a couple of months. The one thing I haven’t really been able to find information about is setting up NixOS in a way where there is a boot drive and a “main drive” and the best way to tackle that.

My current set up is windows but I have a 250gb NVME drive for boot and a 2tb drive for basically everything else; storage, games, documents, photos, etc. I was hoping to replicate this on NixOS. I know the nix store can grow fast if you don’t garbage collect every so often so I was hoping to move that over to the second drive as well.

What’s the best way to achieve this and is there a filesysyem that might work better? I’ve only ever used ext4 when using linux before but I’ve heard of btrfs and zfs so I’m willing to learn more in that department as well if anything is reconmmended

1 Like
2

I’d do something simple like this (unless you really want to play with btrfs/zfs):

fileSystems = {
  "/" = {
    label = "hostname-root";
    fsType = "ext4";
  };

  "/boot" = {
    label = "hostname-boot";
    fsType = "vfat";
    options = [ "umask=0077" ];
  };

  "/home" = {
    label = "hostname-home";
    fsType = "ext4";
  };

  "/nix" = {
    label = "hostname-nix";
    fsType = "ext4";
  };
};

swapDevices = [ { label = "hostname-swap"; } ];

Where the hostname-home partition (and optionally hostname-nix) is on the 2TB drive and all the others are on the 250GB NVME.

For that setup you’d have to specify labels when creating the filesystems, like so:

$ sudo mkfs.vfat -F32 -n hostname-boot /dev/some-disk1
$ sudo mkfs.ext4 -L hostname-root /dev/some-disk2
$ sudo mkswap -L hostname-swap /dev/some-disk3

Personally I’d keep the nix store on the 250GB drive and maybe setup automatic garbage collection (nix.gc.automatic = true;) but that’s personal choice in how you want to partition things.

EDIT: fixed the permissions I had in fileSystems."/boot".options

3 Likes
3

If you’re a techy guy you can do a zfs with raid0. Then do separate datasets for /home, /boot or whatever

1 Like
4

I recommend making both drives btrfs, albeit seperate, then having / being on the 250GB drive and making bind mounts or some subvolume things for /home and /nix on the 2TB one.

2 Likes
5

With only 1 NVME and 1 HDD I would recommend keeping almost everything on the NVME drive. I would then mount the HDD to something like /mnt/HDD and create folders that I would link to elsewhere, such as creating /mnt/HDD/Library and linking to that at ~/Library. The logic for this is that the NVME will be significantly faster for random reads, so you want the OS files (/nix/store) on there for quick loading as well as your users home directory as keeping dotfiles on the HDD can slightly slow loading compared to the NVME. I don’t recommend keeping /nix/store on the HDD but instead configuring automatic garbage collection as you want the quick access of the NVME disk.

2 Likes
6

You could combine the 2 drives into one “virtual drive” with LVM ( so no space gets unused due to the fragmentation ), and then inside, use either normal ext4 or better yet, BTRFS for easy snapshots and its good sub-volume features. Make sure to allow LVM to passthru trim/discard ops so your sdd gets more lifetime.

1 Like
7

Btrfs is the FS you use when you like your data to get corrupted (not to mention the extra hassle of setting up rebalancing and co).
Zfs is developed out-of-tree, so depending on which kernel you use, you may run into issues with zfs not being compatible yet. There’s also just absurd zfs-specific interaction bugs like ENOENT: no such file or directory, copyfile '/project/.pnpm-store/... -> '/project/node_modules/... · Issue #5803 · pnpm/pnpm · GitHub

Respectfully, RAID0 is insane - one drive failure/corruption = data loss.

If you want a solid basis, use ext4. Or use XFS (it’s slightly faster and often more reliable) but it does have the limitation of not being able to shrink the FS once it’s created (you can only grow it).

8

Ehhh. This is largely outdated caution. btrfs is even the default on various distros, including Fedora, now.

Rebalancing isn’t essential anymore (at least metadata balancing actually causes more harm than good). It used to be required to prevent disks becoming unwriteable over time, but that has been solved.

Setting up a periodic scrub is just as required and as much work as it is for zfs, and arguably pretty pointless if you’re not using RAID since you can’t do anything about corrupt data anyway.

Just avoid RAID5/6, but if you’re going to be using it on two disks with disparate size and performance, you probably shouldn’t use RAID at all, so there’s really no risk here. RAID setups are super uncommon on desktop use cases anyway, especially in this modern age of expensive flash storage.

Btrfs has excellent documentation, too, I’d suggest having a read through: Welcome to BTRFS documentation! — BTRFS documentation

All that said, average desktop users probably don’t make much use of btrfs’ (or zfs’) features. COW is neat, but hardly matters in day-to-day use. Compression will largely not work for desktop user’s files, anything large will already be in a compressed file format. Block-level duplication isn’t fast enough for general-purpose use.

Snapshots can be handy, but they’re not replacements for backups, and desktop users will not put the effort in to ensure backup integrity, let alone care about the downtime snapshots can help reduce - if they back up data at all. Plus, we lack the UI needed to make them ergonomic for people who don’t think too hard about their file systems.

Overall I think I agree that just ext4/xfs are fine if you don’t know why you’d choose something else.

5 Likes
9

Sure, the same way as using lvm or even single disk :slight_smile: I just said about possibilities. Using anything but raid0 in a set up where one drive is 8 times larger is quite wastefull.

10

No, because the same data has two points of failure instead of just one. RAID0 is the waste here.

11

I’m sorry, maybe I don’t understand English enough, can you explain what you wrote?

12

raid0 stripes the data, so if either disk fails then you are hosed, so you sum the probabilities rather than multiple them. That’s a bad outcome. Given most people are rocking nvme or at least ssd these days raid0 as a performance thing is basically dead at this point.

2 Likes
13

Thank you. (something something to make discourse happy about amount of characters…)