Building QEMU/KVM VMs

Help
1

Hi everybody,

I want to build a full QEMU/KVM VM that doesn’t share anything with the host, and therefore I can’t use nixos-rebuild build-vm.

I searched this question and I found a post that describes building virtualBoxOVA by importing <nixos/nixos/modules/virtualisation/virtualbox-image.nix>.

So I looked at https://github.com/NixOS/nixpkgs/blob/52c495b2d9fb8918f71be2646bc626804cda8a73/nixos/modules/virtualisation/

and imported:

modules =
    [
    <nixos/nixos/modules/virtualisation/qemu-vm.nix>
    ...
    ];

and in REPL I looked at

:lf .
builtins.attrNames    nixosConfigurations."mysystem".config.system.build

but I didn’t notice anything new.

Could you please tell what do I need to import?

Also is there a better way how to specify imports?

I use a flake for my NixOS config and importing <nixos/.../...nix seems like pre-flake configuration. For example HM is imported as home-manager.nixosModules.home-manager and not <home-manager/.../...nix.

Thank you.

2

You can use fallocate to create a disk image. Then, create partitions n the disk image using fdisk or parted. Next, format the partitions and mount them. Then you can use nixos-install to install NixOS into the disk image. You should be able to convert the raw image to qcow2, if you want.

4

The problem with nixos-rebuild build-vm is that it shares /nix store. That’s great for testing but not so much for a general use as a VM that can run on a different host.

5

You’re looking for either virtualisation.useNixStoreImage or virtualisation.useBootLoader then.

But are you sure this is actually what you want? If you build the VM, you’ll need to copy the entire derivation to whatever device you plan to use it on anyway, and the closure will always contain everything it needs.

6

Not speaking for the OP’s specific needs, but a common reason to need this is when the intended VM host is not running nix/nixos; indeed it may not be a host you control at all.

7

Yes, of course. I want to get qcow2 file with many GiB in size.

Exactly.

Is my question surprising / unusual? Isn’t this a common thing people do to use Nix to get Raspberry Pi images, containers, VirtualBox VMs, AWS AMIs and they all have all bits they need to run on any hardware / host?

8

First time I heard about Nix is when I researched for an alternative to Hashicorp Packer to build AWS, Azure, QEMU… images

It is surprising that people don’t “speak” more often about building topic. Most of questions here on the forum are about building executable and configuration (NixOS)…

Me too!

I’m thinking about hacking https://github.com/NixOS/nixpkgs/blob/52c495b2d9fb8918f71be2646bc626804cda8a73/nixos/modules/virtualisation/digital-ocean-image.nix but at this moment I have no experience in building other things then compiling software.

Would you recommend some blog / documentation how people build custom images like Raspberry Pi, qcow2, AMI?

9

Yes please. Anyone knowledgeable in building rpi4/vm images?

10

Could this help you?

1 Like
11

Yes, that helps. Thank you!

12

If you’re building VM why not build and install the VM within say, virt-manager? That would keep it pretty separate aside from passing through your network config and such? Or am I misunderstanding the specific ask here? Sorry if I am!

1 Like
13

Hi @ziasquinn , don’t apologize :slight_smile:

Virt-manager gives you control over libvirt which gives you an easy way to store VM configuration (network adapters, connected disks…). That means you don’t need to run qemu with 5 lines of arguments which you would have to for a complex VM config without libvirt.

What is discussed here is to use power of Nix to build a VM disk that you can than connect via libvirt (virt-manager) and run a VM. Of course you can just create a VM in virt-manager, connect NixOS ISO and install it like you would install it on a laptop. But that’s manual approach. Nix gives you an option to create that a disk with all configuration that you have and then you can choose how to run it (for example locally or with a cloud provider).

Same thing applies to rpi4. You can dd pre-build image, copy config and rebuild. Or you can use power of Nix to build an image that will already have your configuration and dd that.

4 Likes
14

Hey @rudolf thanks for the explanation. That makes a lotta sense. Wow! I’m so used to getting clowned on all the time in normal linux spaces that was really refreshing for once.

1 Like
15

Building a Raspberry Pi 4 image for flashing on an SD card is easy enough. I use this module to set up the SD bits while suppressing the installer profile:

{ nixpkgs, ... }:
{
  imports = [
    "${nixpkgs}/nixos/modules/installer/sd-card/sd-image.nix"
    "${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"
  ];
  disabledModules = [
    "profiles/all-hardware.nix"
    "profiles/base.nix"
  ];
}

Then, nix build .#nixosConfigurations.<system>.config.system.build.sdImage.

16

Nix community is the most peaceful opensource community I’ve ever joined (both discourse and PRs on github).

4 Likes
17

I’ve just tried nixos-generators and it is working. I got nice large self-contained qcow2 file.

The only question / problem. Is is possible to store the qcow2 file somewhere else then /nix/store?

I’m happy to store everything that is needed to build the VM file in /nix/store but the resulting file is too large for the partition where I have the /nix/store.

18

You may try to set different:

export NIX_STORE_DIR=/mytest/store
# or
export TMPDIR=/mytest/temp
# or
export NIX_BUILD_TOP=/mytest/build
19

NIX_BUILD_TOP doesn’t do anything and NIX_STORE_DIR fails probably because I just created an empty directory. I know TMPDIR and I use it during the build but that’s just for temp files during the build.

20

You will have to wait for [RFC 0152]: local-overlay store…

1 Like
21

Or run a container with a volume on the correct partition and build qcow inside. The downside is that it won’t just build qcow there but it’ll have to build / download everything needed for the qcow and system inside.

1 Like