I’ve recently decided to move from the cloud to on-prem for some services I run. I can use a pre-built solution like ProxMox, but what’s the fun in that? Moreover, my experience with these debian based distributions is that eventually you back yourself into such a terrible corner of mutable configuration that the system becomes untenable eventually and you have to start from scratch, and back up and restart all your VMs.
I’ve been running NixOS on my home machines for years now and have never run into this kind of problem for my desktop case. Now, I want to bring this zen-like experience to the data center.
I see that NixOS has packages for LXD, Ceph, and openvswitch. I want to use these to create a private cloud-like environment, managed mainly via nixops (should work with the libvirt backend). I have three servers that I’ll set up for this as a test bed. My dream is to have a git repository containing a declarative nix configuration of all my infrastructure. Submodules in this git repository would represent nixops deployments. Pushes would change the configuration of all the servers, and trigger any nixops re-deployments.
Does anyone have experience with this? I can already tell I’m going to have to do some work on nixpkgs, since it doesn’t seem like the LXD included has Ceph or LVM-thinpool support, but that’s alright… I’m more than happy to put in the work now to get the beautiful Nix experience.
My main area of concern is networking. I’m a bit out of my league here with all the terms, and I’m not sure if nixos already has support for these kinds of things. I would like to be able to create private networks with custom iptables rules for NAT (only have 3 public IPs, so some services like HTTP will have to be reverse proxied by virtual host).