Building and customising vm images

I want to build some VM images with a few customisations. For the sake of a concrete example, let’s consider an Amazon AMI with zfs root, and I want to change the pool and filesystem layout the builder will create.

So the starting point before any customisation would be:

$ nix-build /data/foss/nixpkgs/nixos/release.nix -A amazonImageZfs.x86_64-linux

However, even this fails, with nixos-install inside the vm getting an error trying to allocate memory partway through building the new store.

It isn’t surprising that a zfs-using system needs more memory when doing intensive filesystem writes, but I guess I’m a little surprised that the config hasn’t been adjusted to give that installer vm more memory, since it fails quite consistently at least for me.

It’s also a little disappointing, because if it had been, it would probably have given me a clue about how to approach passing in other customisations and my main question. I can see the virtualization.memorySize option, but it seems to only be valid in nixos tests.

So, core questions:

  • in the first instance, how do I pass an option to give the vm used by nix-build in the above more RAM?
  • more generally, how do I pass other customisations to control the resulting build?
  • for extra credit, if there’s a way to make this nicer with flakes, I’m happy to go in that direction

I’ve looked over the code and there are clear constructions in there that seem to be intended to facilitate this, but I have no idea how to use or access them. Is there an example of how to use all this?

1 Like

I raised an issue for the build failure since it’s a genuine problem with default settings

I generate custom VM images too. Some are larger and have run out of memory inside the VM run. My expressions use nixos/lib/make-disk-image.nix, which passes a hard-coded value (1024 megs) to Qemu/KVM for guest memory.

I don’t know how or where attributes from release.nix invoke a VM run, so I can’t say whether it is the same problem. Speculating a bit, maybe the typical VM run use cases don’t need more memory, so it hasn’t been a problem for many users or regular NixOS release activities.

One way to fix the make-disk-image function would be to add a memSize ? 1024 argument so that callers may override it as needed. I have worked around it in my private repo by making a custom-disk-image function that does that. If that is useful it could be merged upstream.

If there are other workarounds, I’d love to learn about them too.

I started a PR for this at #218043

Not sure how to expose memSize for a specific image, or in general, without adding a parameter to each call site, e.g., the many ‘image’ expressions in nixos/modules/virtualisation. I’m hoping someone with more knowledge will advise on further improvements.

1 Like

What would be neat is some singular knob, like the virtualisation.memorySize module option, to populate the value of memSize parameter for the dozen or so make-disk-image usage sites. I’m not sure if it is possible, but even if it were, it seems slightly weird to mix the guest configuration with the image-generating step.

Is there such a thing as a library module?