I am working in configuring a small home server running NixOS. The plan is to have a bunch of VMs (libvirtd) that store their data in a ZFS pool. The VMs themselves will also run NixOS. I chose NixOS so I could check in all the nix files and rebuild the system if anything breaks. I don’t actually like the responsibility of running my own infra, so I want a solution that is as maintainable as possible.
I was surprised to see that NixOS neither has options to declare ZFS datasets (except legacy mountpoints, and I’m not sure if the datasets get created automatically and whether I can set options like compression), or to declare libvirtd configuration including VMs, networks, storage pools, etc.
As it stands, it feels like a lot of the things I expected to get from NixOS are not really there, and I’m wondering if I’m missing something. I’ve started writing derivations for libvirtd XML files, and will probably continue this effort, but I am starting to wonder whether all of this is actually worth it or whether I could have used a more traditional distro together with Ansible or so.
Is it that nobody has gotten around to do this yet, or because it’s too hard to solve in general, or some other reason? Or am I just doing something too obscure, so too few people have the same problem as me? I did find some half finished prior art, but wasn’t really able to understand all of their nix code and wasn’t sure they were actually solving the problem I have.
I don’t know about libvirtd or ZFS but I found that a lot options have a settings or extraOptions or similar attribute which you can use to customize special use cases.
Usually they take the normal configuration format of the program you’re using.
And yes I reckon for the most part it’s just a lot of work to program it all.
When I look at how much time it just takes to write my systems config, I don’t really want to know how many hours people are putting into creating NixOS.
About using Ansible on another distro.
I’ve used Ansible before switching to NixOS and I feel way more confident in my NixOS config still working when I create a new system as I ever did with Ansible and I would say it allowed me to have better configs and reduce my workload a lot.
I like the idea of being able to add ZFS datasets. I wouldn’t want to give NixOS the power to delete ZFS datasets unless I could make my NixOS the single point of truth for my ZFS setup, and I can’t do that because (and I’m guessing this is common, although I don’t really know) I run ZFS on my root partition, and NixOS, very reasonably, can’t install its own root partition. So I see my ZFS setup as something lower-level than the OS - almost as if it’s hardware.
There’s a bit of a philosophical conundrum surrounding the question of ZFS datasets. I don’t know if this is the reason the facility doesn’t exist, or not.
It would be really great to be able to write nixos config that says “I want these datasets, with these properties, to exist”, and have them created or settings changed at an appropriate point in the boot process if they don’t already. I want this.
The conundrum is about what to do when that config is removed (and to a lesser degree, some of the property settings that can have cascading side-effects). Should the datasets be removed again? General user expectation would be no, but that’s in (at least some) tension with the way the rest¹ of nixos works.
For better or worse (closer to worse, in my opinion) as a result we end up in a situation where zfs manipulations of the host state are imperative, and generally part of manual setup. That’s not entirely wrong — it’s the same for partitioning and making all the other kinds of filesystems — but it feels like more of a missed opportunity given the additional flexibility of zfs.
1: There are some examples of settings that don’t work this way, and do keep persistent state. Users and passwords have them, and there are options that are phrased like “ensure this database exists” as well. It’s hard to avoid the feeling that there could and should be room for something similar with zfs.
On vpsAdminOS, which is an OS built with Nix that is not NixOS, made by @snajpa, ZFS datasets and pools are configured declaratively Pools - vpsAdminOS