In hopes of improving my experience with setting up NixOS virtual machines, I found myself designing and implementing a system to declaratively build guest configurations that I could then plug into my libvirt hypervisor. I settled on using direct kernel boot to load the kernel and tweaked initrd, and an additional /boot filesystem to supply the second stage of NixOS, as well as a nix store passthrough. This turned out to work quite well for me even in a very rough draft, and builds upon the features of Nix in a very straightforward way. Despite that, I could not find any similar projects. As is tradition, I decided to share the software in the hope that it would be useful.
Think of it as a mix between
Some quick highlights:
Low footprint The guest’s program is stored in the host’s nix store. This allows structural sharing and hard-linking to kick in, so each guest will take up little additional space.
Automation No manual steps need to be performed in a VM to install it.
Quick deployment We don’t need to copy closures into each VM, the store is shared so it’s already there.
Rollbacks of course we have those.
Orthogonality The concept doesn’t imply a choice of hypervisor, filesystem layout, or host OS, it can work with a variety of configurations.
Harvard architecture The presence of mutable data is explicitly declared and encapsulated separately from the program. Stateless VMs can also be created by omitting writable storage.
EDIT: I realized that
fileSystems.<name>.autoFormat is already in 21.05.
I am happy to announce the first release of Miniguest. It includes support for LXC and Libvirt+QEMU, with plans for more, a workflow for configuring guest domains in supported hypervisors, and declarative guests in addition to imperative ones. As the project is a flake, the
main branch will now always point to the latest release.
How do I run the template with lxc?
miniguest template my-guests/
sudo miniguest install ".#container"
miniguest create -t lxc container.
extra-config is a file that you supply and that should contain a valid
# WARNING: make sure root is uid-mapped, otherwise you might experience store corruption in the host!
Does that mean that I have to set
config.users.users.root.uid on the host and the guest to the same value?
No both of those should stay at 0. However, you would need to allocate some subuids to root, similar to what the LXD module does.
I had to learn about id-mapping in the process of making Miniguest, I will eventually add a quick explainer to the doc.
For libvirt I get:
sudo miniguest install ".#container"
miniguest create -t libvirt container
--connect qemu:///system \
-n container --os-variant nixos-unstable \
--memory 1536 \
--disk none \
--boot kernel=/etc/miniguests/container/kernel,initrd=/etc/miniguests/container/initrd \
--filesystem /nix/store,nix-store,readonly=yes,accessmode=squash \
ERROR unable to stat: /etc/miniguests/container/kernel: No such file or directory
❯ ls -T /etc/miniguests/container/
├── boot -> /nix/store/fsp4q3qw4vz779aj6kz7rlf2j430iz48-miniguest-nixos-21.11.20210825.88226ea/boot
This is expected: the container profile is designed for a container rather than a VM. To adapt it, you would need to change
boot.miniguest.guestType and add a root fileSystem. Also check the
stateless guest for an example.
I should note
miniguest create does not look at /etc/miniguests at all.
If you want to use libvirt-lxc that should be possible, but it’s not in the release. importing from LXC into libvirt thru
domxml-from-native could also be a pathway.
Would you share some use cases where Miniguest would be the best option? Or who do you see as “target audience”?
Just getting into virtualization in NixOS, have a very minimal container knowledge, and started using VMs via Vagrant because QEMU/libvirt still feels intimidating. I think the Related work section is clear how Miniguest is different from VMs and containers, but because of my lack of experience I still find it hard to understand the difference.
Thank you for putting this together!