Set up Vagrant with libvirt (QEMU/KVM) on NixOS

[toraritte]

This post is a distilled version of Quickly spin up a VM containing non-NixOS but also added parts that were only mentioned implicitly or missing. It is also worth skimming through the nixos.wiki articles Virtualization in NixOS and libvirt but they look somewhat messy and it was hard to decipher which parts are relevant to which topics - but to be fair, without all these contributions I wouldn’t even know where to start so thank you so much!

These generic steps worked for me:

1. Enable libvirt by adding the following lines to /etc/nixos/configuration.nix:

   virtualisation.libvirtd.enable = true;
   boot.kernelModules = [ "kvm-amd" "kvm-intel" ];
   

2. Add your user to the libvirtd group (and maybe to the qemu-libvirtd as well) by modifying the users.users.<your-user>.extraGroups in /etc/nixos/configuration.nix.

   For example, this is how it looks for me:

      users.users.toraritte = {
        createHome = true;
        isNormalUser = true;
        extraGroups =
          [ "qemu-libvirtd" "libvirtd" 
            "wheel" "video" "audio" "disk" "networkmanager" 
          ]; 
        group = "users";
        home = "/home/toraritte";
        uid = 1027;
      };
   

   Note
   If you see errors similar to the ones below, then it means that your actual user is not a member of the libvirtd group.

   $ vagrant up --provider=libvirt
   Bringing machine 'default' up with 'libvirt' provider...
   Error while connecting to Libvirt: Error making a connection to libvirt URI qemu:///system?no_verify=1&keyfile=/home/toraritte/.ssh/id_rsa:
   Call to virConnectOpen failed: authentication unavailable: no polkit agent available to authenticate action 'org.libvirt.unix.manage'
   

3. Rebuild your NixOS configuration with

   sudo nixos-rebuild switch
   

4. (optional) In order to not to mess up your current working dir:

   mkdir -p vagrant/vm
   cd vagrant/vm
   

5. Add the Vagrant executable to your environment. Either with nix-env, nix-shell, in /etc/nixos/configuration.nix, etc. My preferred method is

   nix-shell -p vagrant
   

6. Choose a libvirt-compatible Vagrant box

   Info What Vagrant boxes are to be used with vagrant init below

7. Start the VM (chose Arch Linux for testing, but there are plenty other libvirtd-compatible Vagrant boxes)

   vagrant init generic/arch
   vagrant up --provider=libvirt
   

   Note
   Set the

   export VAGRANT_DEFAULT_PROVIDER=libvirt
   

   environment variable if you don’t want to specify the --provider on the command line every single time.

8. Connect to the VM

   vagrant ssh
   

   Info Use vagrant status to find the ID of the VM you need, if there are multiple running. (sudo virsh list will also show some useful info.)

9. Stop VM when finished with it

   exit
   vagrant down
   

Info
If you want to reset the VM to a blank slate, all it takes is (from inside the directory where Vagrantfile is located):

vagrant destroy
vagrant up

The grahamc/nix-install-matrix repo brings Vagrant-usage to the next level by automating provisioning a list of Vagrant boxes (i.e., Linux distros mostly), running predefined shell scripts for testing purposes, and generating a report in the end. (Some of my messy notes, still a work in progress; another caveat is that this project uses VirtualBox as the Vagrant provider, but pretty sure that one can switch to libvirt easily.)

Acknowledgements

@jacg Without your original post, this would’ve been very painful to figure out from scratch. Thanks!

@tomberek Thank you for the countless ideas and tips in this topic, and for answering my countless questions!

[anticipatedfart]

This is really great, thanks! Getting started with the topic of virtualisation is quite confusing. This is just what I needed.

[lilyball]

I’m going through this right now to set up my machine, thank you for posting this! Here’s some things I’ve noticed though.

@toraritte

boot.kernelModules = [ "kvm-amd" "kvm-intel" ];

It looks like nixos-generate-config is supposed to enable these in your hardware-configuration.nix if your hardware supports it, so you shouldn’t be writing this manually. If nixos-generate-config isn’t adding them, it’s because /proc/cpuinfo doesn’t list the relevant CPU flags. In my case, it turned out that Intel Virtualization Technology was disabled in my UEFI, and after booting into UEFI and turning it on, nixos-generate-config was able to detect it and add the kernel module for me.

@toraritte

Add your user to the libvirtd group (and maybe to the qemu-libvirtd as well)

AS near as I can tell, the qemu-libvirtd group is not something end users are supposed to care about. I’m still going through the setup myself, but searching for info on this group online tells me that I should ignore it.

Also it looks like the default value of virtualisation.libvirtd.qemu.runAsRoot (as of version 24.05.20240309.3030f18) is true which means it won’t be using qemu-libvirtd anyway.

---

Looking at Libvirt - NixOS Wiki, I also see that suggesting some config options that might be useful:

virtualisation.libvirtd = {
  qemu = {
    package = pkgs.qemu_kvm; # only emulates host arch, smaller download
    swtpm.enable = true; # allows for creating emulated TPM
    ovmf.packages = [(pkgs.OVMF.override {
      secureBoot = true;
      tpmSupport = true;
    }).fd]; # or use pkgs.OVMFFull.fd, which enables more stuff
  };
};

[toraritte]

Thank you for your notes! Unfortunately, it was so long ago that I can’t remember why I chose to use those settings anymore, but only that it was a “trial and error” affair, and perhaps my then-system was misconfigured too. (Have been using Nix since 2017, but I’m still at a beginner level.)

I should probably move my original post to the NixOS wiki, so that anyone can correct it. I know that Discourse has a wiki feature too, but I don’t have the trust level to turn it into a wiki post.

[mike1]

With the situation at the Nixos Wiki, why not add this to nix.dev.
Maybe in the recipes?