NixOS 💚 Xen Project — Reviving the Xen Project Hypervisor on Nixpkgs

[sigmasquadron]

NixOS Xen Project Hypervisor

---

Hello everyone! We’re proud to announce that after a long hiatus, the Xen Project Hypervisor is once again available for general consumption on NixOS 24.11!

You’ve heard of KVM…

Xen is a virtualisation technology defined as a type-1 hypervisor, which allows multiple virtual machines, known as domains, to run concurrently with the host on the physical machine. On a typical type-2 hypervisor, like KVM, the virtual machines run as applications on top of the host. NixOS runs as the privileged Domain 0, and can paravirtualise or fully virtualise Unprivileged Domains.

Xen is well-known for its impeccable security record, and is the go-to solution for hyper-scale cloud infrastructures. We recommend Xen for anyone who needs lightweight, secure virtual machines for untrusted workflows.

How do I get it?

You can turn your existing NixOS Unstable installation into a Xen Domain 0 by setting the virtualisation.xen.enable option to true. Check the other Xen options for a more granular configuration! After rebooting into the Xen Kernel, you’ll be able to use the xl command to manage your domains.

Xen is the ideal solution for declarative environments, as virtual machines are defined with xl.cfg files and are created/destroyed atomically. If you need help writing xl.cfg files, check the documentation.

Here's an example file to get you started!

name='example-domain'
memory='2048' # This VM will use 2048 MiB of RAM. If you haven't set virtualisation.xen.dom0Resources.memory, the total memory available to the Domain 0 will balloon down.
vcpus=2 # This VM will use two of your logical cores.
type='hvm' # This makes Xen fully virtualise the VM, like KVM and other hypervisors do.
disk= [
'/path/to/where/you/want/to/store/the/virtual/disk.qcow2,qcow2,hda,w',
'file:/path/to/a/nixos-installation.iso,hdc:cdrom,r'
]
boot='cd' # Fun fact: This doesn't mean it'll boot from the CD, it means it'll try the disk 'c' first , then it'll try the CD-ROM 'd'.
vnc=1 # You can access a Xen VM through the serial console, or through VNC.

Note: As this package and module were only recently refactored, you should expect some bugs. Let us know if you encounter any issues!

The Team:

We’ve also started a whole team for maintaining the Xen packages! Together, we’re maintaining both the guest utilities for Unprivileged Domains, and the hypervisor tooling for the Domain 0. Here’s the crew:

• @CertainLach - Maintainer
• @hehongbo - Maintainer
• @sigmasquadron - Maintainer

You can find us on the Xen Project Hypervisor Team page in the nixos.org website!

We’d also like to extend an open invitation for anyone who wishes to help us maintain Xen!

Let us know if you’d like to help us test, update and keep Xen working for the foreseeable future! Simply open a Pull Request on Nixpkgs adding yourself to the Xen team on maintainers/team-list.nix.

We aren’t done yet!

As you may have noticed, it is a bit cumbersome to write xl.cfg files. To remedy that, we’ll be making a set of NixOS options that can declaratively build Xen Domains. This module will function in a similar way to the systemd services module, where you can create and configure arbitrary attribute sets that define each system service. This is still in its early stages, so let us know if there are any features you’d like to see on this translation from nix to xl.cfg.

We’re also planning to improve the guest experience and write detailed documentation for using the Xen Hypervisor on NixOS.

Thank you.

Our sincere thanks for reading our announcement. We hope to expand the Xen userbase on NixOS and achieve true first-class support for the Xen Project Hypervisor. Once more, we remind that anyone interested in the hypervisor can help us out by joining the team!

---

Happy virtualising.
— The Xen Project Hypervisor Maintenance Team

[proofconstruction]

Thank you all for this incredible work! With Xen support, especially once the domain-builder options are merged, NixOS inches closer to becoming a first-class hosting platform. I’m excited to see what comes next!

[Nebucatnetzer]

@sigmasquadron

On a typical type-2 hypervisor, like KVM

I always thought KVM is a type-1 hypervisor? Alongside Hyper-V, ESXi, etc.
Virtualbox, Desktop VMWare and similar products are type-2 hypervisors.

Cool effort from your side nonetheless💪.

[sigmasquadron]

The line between type-1 and type-2 with KVM is very blurry. Some call it type-1.5 because it’s not completely type-2 like VirtualBox, but not completely type-1 like Xen. With KVM, your VMs aren’t running alongside the host completely - only certain KVM-accelerated parts of them are. I personally consider it type-2 because it depends on QEMU, which is incontestably type-2, to virtualise anything.

edit: discourse hard.

[sirphobos]

so this will make possible to build something similar to qubeos, but with nixos as domain0?

definitely looking forward to seeing some beginner-level examples,
especially on how to forward hardware like gpu to one VM or another,
and how to route one VM’s network through another VM

Thank you for all the work done!

[sigmasquadron]

@sirphobos

so this will make possible to build something similar to qubeos, but with nixos as domain0?

Yes! Take a look at #341215 by Lach.

@sirphobos

definitely looking forward to seeing some beginner-level examples,
especially on how to forward hardware like gpu to one VM or another,
and how to route one VM’s network through another VM

GPU acceleration isn’t exactly the most beginner-level thing to do. There is a lot of upstream development work regarding GPU acceleration in Xen, which will hopefully make it easier to securely passthrough GPUs.
Regarding networking, we’re tracking general Xen documentation in #343391.

[sirphobos]

this may be a bit of an distraction, but talking about non-enterprise gpu passthrough - i really wish that intel’s i915 virtualization branch gets stable soon (how long has it been there? years!). then one could split single Xe graphics into multiple virtual ones, and passthrough one to firefox container, another into some other gpu-boostable app container…

[Yanall-Boutros]

When I add

virtualisation.xen.enable = true;

I get

 Xen does not support the legacy script-based Stage 1 initrd.

I tried adding

  nixpkgs.config.boot.initrd.systemd.enable = false;
  boot.initrd.systemd.enable = false;

rebuilding, then adding the virtualisation.xen.enable line but I still get the same error. Is there anything I’m missing?

[delliott]

The source of the error is this assertion:

NixOS/nixpkgs/blob/e24b4c09e963677b1beea49d411cd315a024ad3a/nixos/modules/virtualisation/xen-dom0.nix#L618-L621

Which requires the systemd initrd to be enabled not disabled.

IE, you need

boot.initrd.systemd.enable = true;

[tlaurion]

Qubesos dom0 pull request Qubes packages (Its alive!) by CertainLach · Pull Request #341215 · NixOS/nixpkgs · GitHub

[gspia]

Thanks all for these interesting things. I am taking my first steps with xen (and with virtualisations in practise).

When trying out to configure the systems, the following virtualisation.xen params caused boot process to fail:

boot.params = [
  “vga=ask” # Useful for non-headless systems with screens bigger than 640x480.
  “dom0=pvh” # Uses the PVH virtualisation mode for the Domain 0, instead of PV.
];

The boot process logs show

march 08 17:55:02 ltname systemd-modules-load[332]: Inserted module ‘nvidia’
march 08 17:55:02 ltname kernel: NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64 580.119.02 Release Build (nixbld@)
march 08 17:55:02 ltname kernel: thermal thermal_zone0: acpitz: critical temperature reached

That might be a bug somewhere, because without those the boot succeeds and xen seems to be usable.

Do you have recommendations on how to set the dom0Resources? (memory and maxVCPUs e.g. out of 64 and 32)

& Do you have any easy instructions, how I can manually setup e.g. ubuntu in xen VM including the networking?

[sshine]

@sigmasquadron: I’m curious how the Nix + Xen project is doing. NixOS.org team page only mentions the 6.topic: xen-project tag on GitHub, the Matrix room, and this thread. Judging by the tags, there’s quite a lot of recent activity. I will dig into nixpkgs to assess the combined maturity.

As for the documentation, which is a stated goal, what would be a good location?

[sigmasquadron]

I am very surprised it even works at all on your system. NVIDIA is notoriously unreliable on Linux, and Xen is already mostly unsupported on graphical (non-server) hardware.

As for the options, they’re there in case you need them. If you don’t have a reason to limit resources to the dom0, then you can leave them alone.

As for easy instructions, I’m afraid there are none. Xen has terrible UX, as it is mostly intended as a C API that makes VMs. The difficulty of spinning up a simple Ubuntu VM depends on what toolstack you’re using. If you’re using xl, which comes bundled with Xen, you’ll have a terrible time. Most of us have written our own programs and scripts to interface with Xen in a more reasonable way.

[sigmasquadron]

Hi. Very sorry to the both of you for not replying, I don’t check Discourse all that often.

@sshine

@sigmasquadron: I’m curious how the Nix + Xen project is doing. NixOS.org team page only mentions the 6.topic: xen-project tag on GitHub, the Matrix room, and this thread. Judging by the tags, there’s quite a lot of recent activity. I will dig into nixpkgs to assess the combined maturity.

As for the documentation, which is a stated goal, what would be a good location?

We’re doing well, I think. I wouldn’t say we’re quite production-ready yet, but I would say we’re doing considerably better than some distros out there.

Here’s a basic idea of my goals for Xen, in no particular order:

• NixOS tests, and once they’re mature and extensive enough, channel blocking. The issue with that is that the test VM environment can’t run a nested hypervisor. This blocks our goal of having Xen on NixOS being “enterprise ready” or “production ready”
• Improving the module. Xen was my first PR for Nixpkgs, and I was incredibly inexperienced back then. I’ve learned many things about module design since then, and there are many improvements to be made on the dom0 and domU modules.
• Declarative domains. As I mentioned above, Xen toolstacks are pretty unique and varied. I am currently developing a toolstack that makes sense in a declarative environment like NixOS. This has been my goal for the past two years, and will likely take a few more years of my life until it’s done.
• Packaging other toolstacks. I would like to get Orchestra in Nixpkgs at some point. Libvirt works pretty decently as far as I’m aware, but I’ve noticed it was recently broken due to a QEMU update, so back to the first item: we need testing.

Things we have been doing well, though:

• Security. We might not be production ready yet, but I’d bet my life on our security maintenance of Xen. We’re constrained by the way Hydra works, of course, but security updates get merged to master very quickly now.

As for documentation, we have the Wiki page. I don’t particularly like it (even though I wrote it), as it kind of states the obvious and duplicates the information present in the option definitions. The thing about documentation for our module is that we don’t really have much to document. It’s Xen. If you enable the option, you get Xen. The hard part with using Xen is navigating the two-decade-old mess of toolstacks and niche features and type-1 hypervisor quirks. I do want a NixOS manual entry at some point, but right now there’s nothing NixOS-specfic to add there. This will change if we package other toolstacks and when we have a declarative domains implementation.