I have been playing around with Nixos and am hoping to move over to my daily driver soon. Occasionally I still need to run Windows and also want to be able to run some non-NixOS VMs and containers. While some things could of course be run in NixOS it is convenient to just run premade dockers for some things or I want full isolation. In my initial testing I was able to get VirtualBox to more or less work, an could even boot a physical windows partition in the VM (as root).
Since my new PC is pretty beefy, I also wanted to use it to run some network services for the house and possibly on a secure VPN. Initially, I had considered running Proxmox, but everyone warns strongly against using it locally and there were some notes about issues when you need to upgrade the underlying Debian. After looking at Virt Manager (Virt-manager - NixOS Wiki), it seems like this would generally be better than running VMs as Type 2 in virtualbox and could essentially work to accomplish the same goals as Proxmox, but using NixOS for all the benefits.
I am looking for some thoughts on my plan and how to tweak this to make the most sense from a security + performance background. Generally, my idea was to create a minimal NixOS host that runs libvirtd and virt-manager. Then my main daily NixOS install would be a guest VM, as would my Window VM. I would pass through my GPU and most of my CPU cores to the main NixOS VM. The Windows VM will be generally for some hardware specific software or stuff that really just works better in Windows (no video games, don’t need GPU pass through there, but when its possible to split them, would do it). I may have some occasional full VMs or dockers, and would reallocate resources as needed.
As a general approach does this make sense? I understand I could just run NixOS directly and then run docker, containers, etc. there, but the reason I wanted my daily driver to be a VM as well was so that way if I need to “reboot” I can keep all my other network services up - e.g., the media server in a docker stays up while I apply a new configuration to NixOS.
Ideally, the “host NixOS” runs nothing other than what is required to get into the VMs making it stable and more secure. I think I would just need the minimum to run virt-manager - not sure if I can forgo a desktop environment and just launch from there and then still have GUI guests locally? Can I just boot a minimal NixOS host into memory each time I need to reboot, which would be infrequent if ever?
Since it is all local, once I am full screen in my main “Guest NixOS” it should essentially feel the same as native (the new pc has very modern hardware and I am coming from something way less powered).
Am I missing anything? If I passthrough my GPU to my “guest nixos” - it will process the same as native with a minimal amount of delay, and as long as I plug my monitor in the motherboards video card it shouldn’t really matter just for the display aspect (?)
I am also looking for some tips on dealing with the dockers. I see some stuff about converting dockers to LXC and then virt-manager can manage them directly. I also think another strategy would be to run another VM which then runs all of the docker containers natively?
tl;dr - Want to run multiple end-user OS and services with isolation at the same level. Use absolute minimal NixOS with libvirtd as the Type 1 hypervisor. Run my main NixOS in a VM, then run any other VMs and containers as guests to the “host NixOS” with the goals of isolation and being able to easily do multiple things at once without one thing possibly taking down the other.
worst case scenario you’ll learn something, i’m sure