I’ve recently decided to move from the cloud to on-prem for some services I run. I can use a pre-built solution like ProxMox, but what’s the fun in that? Moreover, my experience with these debian based distributions is that eventually you back yourself into such a terrible corner of mutable configuration that the system becomes untenable eventually and you have to start from scratch, and back up and restart all your VMs.
I’ve been running NixOS on my home machines for years now and have never run into this kind of problem for my desktop case. Now, I want to bring this zen-like experience to the data center.
I see that NixOS has packages for LXD, Ceph, and openvswitch. I want to use these to create a private cloud-like environment, managed mainly via nixops (should work with the libvirt backend). I have three servers that I’ll set up for this as a test bed. My dream is to have a git repository containing a declarative nix configuration of all my infrastructure. Submodules in this git repository would represent nixops deployments. Pushes would change the configuration of all the servers, and trigger any nixops re-deployments.
Does anyone have experience with this? I can already tell I’m going to have to do some work on nixpkgs, since it doesn’t seem like the LXD included has Ceph or LVM-thinpool support, but that’s alright… I’m more than happy to put in the work now to get the beautiful Nix experience.
My main area of concern is networking. I’m a bit out of my league here with all the terms, and I’m not sure if nixos already has support for these kinds of things. I would like to be able to create private networks with custom iptables rules for NAT (only have 3 public IPs, so some services like HTTP will have to be reverse proxied by virtual host).
I believe all of this is possible, I just wouldn’t expect the experience to be refined or enjoyable.
You will likely have to do a lot of manual work to integrate the different services, and still have some configuration mismatch.
In the long run, it will probably be easier than having a bunch of mutable configurations everywhere. But the road will be paved with the bodies of countless hours trying to solve certain edge cases
I’m in what might be a quite similar situation to yours, planning to build a hyper-converged home server that’s both a NAS and a hypervisor. And I can’t find anything that’s quite what I like.
TrueNAS SCALE is quite inflexible w.r.t. ZFS, Proxmox seems quite flexible, but it doesn’t offer much in terms of NAS features, so I’d have to roll many services to make it work, etc.
And I’m already having nightmares just thinking about doing a dist-upgrade on a hypervisor host…
If Proxmox was fully throw-away, then that’s not big deal, but it wouldn’t be, there’s quite a lot of stateful configuration that’s needed.
So I was thinking to use NixOS as, essentially a bare-metal hypervisor. ZFS pool management by hand, virt-manager and LXD for clients.
But use NixOS for the alerting, S.M.A.R.T. monitoring, etc.
The absolute gold with doing it using NixOS would be much, much less fear w.r.t. system updates.
I think it’s a great idea. Proxmox, Ubuntu server, et al are nice… they work out of the box. But I know what’s going to happen here. The statefulness means you ultimately need an expert team to diagnose the inevitably surprising and esoteric failures. NixOS is much more maintainable for a single user like myself. I have Nix boxes (deployed via NixOps) that have run services without interruption for years even with upgrades (reproducibility is amazing for this). My experience in the past with Ubuntu boxes has left me absolutely convinced that that model is not sustainable.
Yes, indeed. I’ve spent some time looking at nixpkgs and it sure looks very promising. Smartd, then there’s an alert / notification deamon (prometheus or something). I’ve found declarative dockers, sounds great. I am currently thinking, maybe I can use ONOS as a GUI for the vSwitch. Would be neat.
Then there are other nice goodies like auto-snapshots, keeping multiple EFI partitions in sync, zfs snapshot replication, etc.
I’ll probably spin up a docker container or something as a NAS filer to have a GUI and it’s getting pretty nice already.
For KVM VMs, I think virt-manager and for LXD, I’m thinking LXDMosaic.
Once I have bit of a config, we should totally try and see if it’s something that might be interesting to others
