How to build a standalone NixOS QEMU VM?

Help
1

I want to build simple NixOS VM that doesn’t use the host’s store.

nixpkgs doesn’t seem to contain a build helper (like virtualisation/virtualbox-image.nix for VirtualBox), so I’m using the script below.
The resulting system fails at booting because GRUB can’t read the filesystem. More details at the end.

#! /usr/bin/env nix-shell
#! nix-shell -i bash -p qemu

set -eo pipefail

[[ $UID == 0 ]] || { echo "Must be run as root"; exit 1; }

mkdir -p /tmp/build-vm/mnt
cd /tmp/build-vm

# Create and mount image
qemu-img create -f qcow2 img.qcow2 10G
modprobe nbd
qemu-nbd --connect /dev/nbd1 img.qcow2

# Format, partition
echo "type=83" | sfdisk /dev/nbd1
mkfs.ext4 -U 00000000-0000-0000-0000-12345678beef -L root /dev/nbd1p1

nix-build --out-link sys - <<'EOF'
(import <nixpkgs/nixos> {
  configuration = ({ lib, ... }: with lib; {
    fileSystems."/" = {
      device = "/dev/disk/by-uuid/00000000-0000-0000-0000-12345678beef";
      fsType = "ext4";
    };
    boot.loader.grub.enable = true;
    boot.loader.grub.device = "/dev/nbd1";
    services.mingetty.autologinUser = "root";
    services.qemuGuest.enable = true;
    services.timesyncd.enable = false;
    networking.dhcpcd.extraConfig = "noarp";
  });
}).system
EOF

# Mount root partition as a loopback device. Otherwise nixos-install fails with:
# grub-install: error: cannot find a GRUB drive for /dev/nbd1p1. Check your device.map.
losetup /dev/loop1 /dev/nbd1p1
mount /dev/loop1 mnt

nixos-install --system $(realpath sys) --root $(realpath mnt) --no-root-passwd
cat mnt/boot/grub/grub.cfg

umount mnt
losetup --detach /dev/loop1
qemu-nbd --disconnect /dev/nbd1

qemu-kvm \
  -cpu kvm64 \
  -m 1024 \
  -smp 2 \
  -device virtio-rng-pci \
  -net nic,netdev=user.0,model=virtio -netdev user,id=user.0 \
  -drive index=0,id=drive0,file=img.qcow2,cache=writeback,werror=report,if=virtio \
  -vga std -usb -device usb-tablet,bus=usb-bus.0 \
  -nographic

This is the GRUB output while booting:

Welcome to GRUB!

error: no such device: 00000000-0000-0000-0000-12345678beef.
error: unknown filesystem.
Entering rescue mode...

Here are some outputs from GRUB’s rescue shell:

$ ls
(hd0) (fd0)

$ ls (hd0)
(hd0): Filesystem is unknown.

$ ls (fd0)
(fd0): Filesystem is unknown.
error: failure reading sector 0x2 from `fd0'.

Can someone give me a hint how to make this work?

4 Likes
2

I’d be grateful for any slight hint.

3

Hi,

I think you are looking for nixos/lib/make-disk-image.nix

If you look inside nixpkgs you can see how to use it, for example:

  system.build.qcow = import <nixpkgs/nixos/lib/make-disk-image.nix> {
    inherit lib config pkgs;
    diskSize = 10240;
    format = "qcow2";
  };

The result of building this attribute will be a qcow2 image of the current
configuration ready to be run with qemu.

Excerpts from Gestur Bjarkason via NixOS Discourse’s message of February 2, 2020 1:40 pm:

4

Hi @eonpatapon, how would I do this interactively (to put it in @gesturbjarkason’s script) ?
Right now I have

nix-build '<nixpkgs/nixos>' - <<'EOF'
(import <nixpkgs/nixos/lib/make-disk-image.nix> {
    inherit lib config;
    pkgs = import <nixpkgs> { inherit (pkgs) system; };
    diskSize = 10240;
    format = "qcow2";
}).system.build.qcow2
EOF

but get

error: undefined variable ‘lib’ at (string):1:50

5

You haven’t told where lib should come from…

Perhaps the following might work:

nix-build '<nixpkgs/nixos>' - <<'EOF'
let pkgs = import <nixpkgs> {}; in
(import <nixpkgs/nixos/lib/make-disk-image.nix> {
    inherit (pkgs) lib config;
    pkgs = import <nixpkgs> { inherit (pkgs) system; };
    diskSize = 10240;
    format = "qcow2";
}).system.build.qcow2
EOF
1 Like
6

I’m almost there I guess, getting

error: value is a string while a set was expected, at (string):2:1
with your change @NobbZ.

7

How would I debug an expression like the above ?
I. e. to find out what exactly is the string expected to be a set there, is it the .system.build.qcow2 part in (pseudocode) let [[binding]]; in [[anonymous config in parentheses]].system.build.qcow2 which results in a string here ? How to find out using i. e. nix-instantiate ?
As I understand .system.build.qcow2 is saying give (build) me the attribute [[config]].system.build.qcow2 which should under covers result in qcow file in the result folder. How to prove that ?

8

EDIT: You’re invited to read the followup to this question here.

Based on the excellent Mayflower article I did some mods (based on @Elyhaka’s idea) reusing the custom-iso.nix the article provides. I plan to modify the expression to output a qcow file instead of an iso file and change the op’s original shell script accordingly with that. Result would be a standalone shell script to create the qcow file usable in qemu.

nix-build --out-link vm - <<'EOF'
{ nixpkgs ? <nixpkgs>, system ? "x86_64-linux", pkgs ? import <nixpkgs/nixos> {}, ... }:
let
  myisoconfig = { pkgs, lib, ... }: with lib; with pkgs; {
    imports = [
      "${nixpkgs}/nixos/modules/profiles/qemu-guest.nix"
      "${nixpkgs}/nixos/modules/installer/cd-dvd/channel.nix"
    ];
  };
  
  evalNixos = config: import "${nixpkgs}/nixos/lib/make-disk-image.nix" {
    inherit (pkgs) lib pkgs config;
    diskSize = 8192;
    format = "qcow2";
    configFile = pkgs.writeText "configuration.nix"
      ''
        {
          # TODO Write text of myisoconfig above here as file
        }
      '';
  };
in { vm = (evalNixos myisoconfig).config.system.build.qcow; }
EOF

Error:

error: file ‘nixos-config’ was not found in the Nix search path (add it using $NIX_PATH or -I), at /home/me/.nix-defexpr/channels/nixpkgs/nixos/default.nix:1:60

What still bothers me is the error message. It is rooted in line 2 (…) import <nixpkgs/nixos> (…) and I just want to use myisoconfig from my expression instead of providing -I nixos-config interactively.

Any pointers welcome!

2 Likes