How to build a NixOS VM with nix, in which nixos-rebuild is a no-op?

Help
1

I need to test some changes in a QEMU VM that require running nixos-rebuild, for example my GRUB PR.

For this purpose I’d like to build a VM as described on Mayflower’s post on this topic.

The problem is that even though I use the same config and nixpkgs version inside and outside the VM, nixos-rebuild downloads GBs of store paths, even though they already exist via the read-only + overlay-file-system mount of /nix/store of the host and I can ls them from the guest.

I believe that this is because of what’s described in virtualisation.pathsInNixDB:

The list of paths whose closure is registered in the Nix database in the VM. All other paths in the host Nix store appear in the guest Nix store as well, but are considered garbage (because they are not registered in the Nix database in the guest).

How can I configure my VM that either

  • all of the host store paths are registered in the nix DB of the guest, or
  • all store paths relevant to the VM are registered in the guest,

so that nixos-rebuild is a no-op inside the VM and causes no downloads?

5 Likes
2

The issue here is that when you’re using Nix inside the VM, the store paths from the host aren’t registered in Nix’s SQLite database. On the other hand however all the store paths from the host machine should be available within the VM (except of course if the VM is running inside a Nix build process).

So essentially, you could either use nix-store --load-db to initialise the store database (this is btw. what pathsInNixDB does) or just run the build on the host.

The latter could be done by allowing access to the Nix daemon socket (usually in /nix/var/nix/daemon-socket/socket) of the host or even use one of the remote store backends (Introduction - Nix Reference Manual - “The store abstraction…”).

Another possibility would be to just evaluate and build the new machine configuration directly on the host and just run the activation script in the guest (eg. similar to this).

2 Likes
3

Overlay stores (not yet implemented!) would also help with this.

What I like to do when debugging boot issues is boot into a live system, create qcow2 images with the real disks as backing files, then use QEMU with those to make changes affecting e.g. the boot sector without affecting my “production” system.

1 Like
4

Can you give some example codes?

I’ve tried to simulate a rebuild after a channel update in a NixOS test but I can’t get the guest to use the host store with pathsInNixDB. You can see the test here.

5

Yes, the example from Nixos-vm-examples: A collection of examples on how to build NixOS VMs

6

Sorry, It was meant as a reply to @aszlig. I still couldn’t manage to reuse the host store paths.

7

Unfortunately it’s a bit hard to say without details, but if it’s about the linked VM test, please note that unlike directly executing the script resulting from config.system.build.vm you won’t get full access to the hosts store since Nix builds are sandboxed by default and thus only have access to the store paths of the inputs (that is all the derivations it’s referencing, eg. via string context) of the current derivation.