Hey fellow nixers!
I am seeking some wisdom for using NixOS as a homelab hypervisor and would appreciate any thoughts and ideas to help me figure out the right path forward.
I have two server systems that I use for my homelab, one is more powerful than the other. Both of them run Proxmox at the moment and they are clustered, with the bigger emphasis on the chunkier one when scheduling the VMs. I am running two VM instances of the OPNsense (one on each server) in an HA setup with automatic failover and all this sort of fun. I have quite a few VMs (some of which are NixOS based) and LXC containers (some of which are also NixOS based).
As much as I appreciate Debian, there is no denying that NixOS, when used well, is far superior with its declarative management and ability to quickly and safely roll-back. So I have been considering switching over from Proxmox to NixOS as a hypervisor (instead of upgrading to the latest major release of Proxmox), but I am struggling to find quite the right set of tools to get me what I actually need. So hopefully someone can point me in the right direction.
Essentially, what I would like to have is an ability to run an arbitrary number of virtual machines of different types across two nodes. Ideally it would be possible to migrate VMs across the servers if I need to take one of them down for maintenance.
I would appreciate a degree of overview, monitoring and reporting for the running VMs - CPU and memory usage, I/O delays, etc (I can build some of this myself on top of Prometheus or something like that, but if there is something that I can use out of the box that would be preferred). I would like to be able to run VMs with different guest OSes (Windows, FreeBSD, Linux). I require PCIe passthrough to attach ethernet cards to OPNSense VMs.
There will be two different categories of things that I plan to run in my homelab:
- “production” services, that are things like fileshare, proxy service, DNS service, etc;
- “development” service that will be all over the place, e.g. today I want to run and explore Nomad orchestrator, tomorrow it will be Kubernetes, in a week I want microvms and Firecracker. The point here is to experiment and be able to set-up and tear down the infrastructure declaratively without affecting my “production” workloads;
I have been looking at OpenNebula, OpenStack or Mist.io, but none of them actually integrate and work well with NixOS as far as I understand. On the Nix side, I have been looking at microvm.nix, but so far it only supports NixOS/Linux and not Unix or Windows and also doesn’t solve the observability and monitoring part.
I am sure many of you have some fancy cool homelab set-ups and can suggest some ideas on how I can achieve the above. Many thanks in advance!