Miniguest: lightweight and declarative guest operating systems profiles

In hopes of improving my experience with setting up NixOS virtual machines, I found myself designing and implementing a system to declaratively build guest configurations that I could then plug into my libvirt hypervisor. I settled on using direct kernel boot to load the kernel and tweaked initrd, and an additional /boot filesystem to supply the second stage of NixOS, as well as a nix store passthrough. This turned out to work quite well for me even in a very rough draft, and builds upon the features of Nix in a very straightforward way. Despite that, I could not find any similar projects. As is tradition, I decided to share the software in the hope that it would be useful.

Think of it as a mix between nixos-container and nixos-rebuild build-vm

Some quick highlights:

  • Low footprint The guest’s program is stored in the host’s nix store. This allows structural sharing and hard-linking to kick in, so each guest will take up little additional space.

  • Automation No manual steps need to be performed in a VM to install it.

  • Quick deployment We don’t need to copy closures into each VM, the store is shared so it’s already there.

  • Rollbacks of course we have those.

  • Orthogonality The concept doesn’t imply a choice of hypervisor, filesystem layout, or host OS, it can work with a variety of configurations.

  • Harvard architecture The presence of mutable data is explicitly declared and encapsulated separately from the program. Stateless VMs can also be created by omitting writable storage.

EDIT: I realized that fileSystems.<name>.autoFormat is already in 21.05.

13 Likes

I am happy to announce the first release of Miniguest. It includes support for LXC and Libvirt+QEMU, with plans for more, a workflow for configuring guest domains in supported hypervisors, and declarative guests in addition to imperative ones. As the project is a flake, the main branch will now always point to the latest release.

3 Likes

How do I run the template with lxc?

miniguest template my-guests/
cd my-guests
sudo miniguest install ".#container"
# ???

Run miniguest create -t lxc container. extra-config is a file that you supply and that should contain a valid lxc.idmap.

1 Like

# WARNING: make sure root is uid-mapped, otherwise you might experience store corruption in the host!

Does that mean that I have to set config.users.users.root.uid on the host and the guest to the same value?

No both of those should stay at 0. However, you would need to allocate some subuids to root, similar to what the LXD module does.

I had to learn about id-mapping in the process of making Miniguest, I will eventually add a quick explainer to the doc.

1 Like

For libvirt I get:

sudo miniguest install ".#container"

miniguest create -t libvirt container

virt-install \
  --connect qemu:///system \
  -n container --os-variant nixos-unstable \
  --memory 1536 \
  --disk none \
  --import \
  --boot kernel=/etc/miniguests/container/kernel,initrd=/etc/miniguests/container/initrd \
  --filesystem /nix/store,nix-store,readonly=yes,accessmode=squash \
  --filesystem /etc/miniguests/container/boot,boot,readonly=yes,accessmode=squash
[...]
ERROR    unable to stat: /etc/miniguests/container/kernel: No such file or directory

❯ ls -T /etc/miniguests/container/
/etc/miniguests/container
├── boot -> /nix/store/fsp4q3qw4vz779aj6kz7rlf2j430iz48-miniguest-nixos-21.11.20210825.88226ea/boot
└── manifest.json

This is expected: the container profile is designed for a container rather than a VM. To adapt it, you would need to change boot.miniguest.guestType and add a root fileSystem. Also check the stateless guest for an example.

I should note miniguest create does not look at /etc/miniguests at all.

If you want to use libvirt-lxc that should be possible, but it’s not in the release. importing from LXC into libvirt thru domxml-from-native could also be a pathway.

1 Like

Would you share some use cases where Miniguest would be the best option? Or who do you see as “target audience”?

Just getting into virtualization in NixOS, have a very minimal container knowledge, and started using VMs via Vagrant because QEMU/libvirt still feels intimidating. I think the Related work section is clear how Miniguest is different from VMs and containers, but because of my lack of experience I still find it hard to understand the difference.

Thank you for putting this together!