Libvirt, Session Mode, and VirtioFS

Help
1

Hi,

I am trying to move my virtual machine host over from Gentoo to NixOS to reduce the over all maintenance time investment and get better zfs support plus all the cool NixOS features. I have a running test system with somewhat running libvirt+qemu.

The main issue is that libvirt seems to be setup in Session Mode instead of System Mode, causing VirtioFS to not work since it is currently not supported in Session Mode. Is there a way set up libvirt in System Mode?

Documentation regarding libvirt is pretty sparse and the available options for the configuration file don’t hint at a way to configure libvirt any other way.

I attempted to run the libvirtd service daemon with a custom config but that seems to break things by causing the libvirtd systemd service to exit unexpectedly.

Any help would be appreciated, thank you.

1 Like
2

If you configure it using virtualisation.libvirtd, you should get a regular system instance.

Maybe you can share your config?

3

Hi peterhoeg, thanks for taking the time to respond. Here is my configuration:

Configuration 
{ config, pkgs, ... }:
{
  imports = [
    ./hardware-configuration.nix
  ];

  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;
  boot.extraModprobeConfig = "options amd_iommu=on iommu=pt kvm_amd.nested=1";
  boot.kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;
  boot.supportedFilesystems = [ "zfs" ];

  networking.hostName = "<hostname>";
  networking.hostId = "<hostid>";
  networking.networkmanager.enable = true;

  # User management
  users.groups.<groupname>.gid = 2000;

  users.users.<username> = {
    isNormalUser = true;
    uid = 1000;
    extraGroups = [ "wheel" "networkmanager" ];
    packages = with pkgs; [];
  };

  # Packages in system profile. (Search for packages: nix search wget)
  environment.systemPackages = with pkgs; [
    wget
    neovim
    git
    fish

    linuxKernel.packages.linux_6_1.vendor-reset
    linuxKernel.packages.linux_6_1.zfs
  ];

  # Virtualisation
  virtualisation.libvirtd = {
    enable = true;
    #extraOptions = [ "--config /etc/libvirt/libvirtd.conf" ];
    qemu.ovmf.enable = true;
    qemu.runAsRoot = true;
  };

  # List services that you want to enable:
  services.openssh.enable = true;

  # Before changing this value read the documentation for this option
  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
  system.stateVersion = "22.11"; # Did you read the comment?
}
4

I worked through the problems I was facing. It was mix of uniqueness of NixOS, Systemd, and assumptions from my end.

Libvirt Mode

Libvirt is running in system mode. Here is my current configuration:

Configuration 
   networking.firewall = {
    allowedTCPPortRanges = [
      # spice
      { from = 5900; to = 5999; }
    ];
    allowedTCPPorts = [
      # libvirt
      16509
    ];
  };

  environment.systemPackages = with pkgs; [
    ...
   virtiofsd
  ];

  virtualisation.libvirtd = {
    enable = true;
    qemu.ovmf.enable = true;
  };

Libvirt/QEmu files are in /var/lib/libvirt, and I am running virsh using the root user.

VirtioFS

For virtiofs, currently its usage in NixOS is not well supported so we need to add an explicit option for it within the libvirt guest declaration:

Github Issue

Guest filesystem xml entry 
    <filesystem type='mount' accessmode='passthrough'>
      <driver type='virtiofs'/>
      <binary path='/run/current-system/sw/bin/virtiofsd'/>
     ...
    </filesystem>

Unexpected

Using libvirt over tcp is a bit unruly, not sure why. To start out with, ssh + virsh works without issue. Using virt-manager does not work well, with proper ports exposed it will not connect unless libvirtd.service is stopped and libvirt-tcp.socket is restarted.

I will mark this as solved as the unexpected issue with the tcp socket activation is not completely related to this, and there is a work around.