Building OpenStack images and development VMs from a macOS host

1

I’m starting a new project that will have a fair bit of system configuration wrangling involved, and I’d like to define my configuration declaratively, but it seems like I need to be running Linux to build a Linux image.

Basically, I’d like to offer two things: a lowered barrier to entry for contributors by having a straightforward way to start up a development environment locally (without necessarily running NixOS themselves), and a way to generate VM images for deployment in an OpenStack cloud. I can see that there are tools for this, but I can’t seem to make them work on macOS.

First, I tried to use https://github.com/nix-community/vagrant-nixos-plugin but it appears nixos - Vagrant Cloud is more or less abandoned.

Second, I saw https://github.com/NixOS/docker and thought I could use it with GitHub - nix-community/nixos-generators: Collection of image builders [maintainer=@Lassulus] and canned Linux VM I’m already running for Docker on macOS. However, my attempt to generate a VM image from within a VM … didn’t work.

ex.nix:

{pkgs, ...}:
{
  services.nginx.enable = true;
}

nixos-generate -f openstack -c ex.nix

downloaded a bunch of stuff, but failed with

error: a 'x86_64-linux' with features {kvm} is required to build '/nix/store/3v45gv7y68zlba2lagj4awpk7hdrvm5p-nixos-disk-image.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, nixos-test}

after some digging, I found https://github.com/nix-community/nixbox/issues/38#issuecomment-583354822 which tells me kvm is required to build an image.

So… what do I do? The prospect of resorting to Ansible “scripts” in YAML fills me with dread. On the other hand, I can’t very well ask every contributor to switch to NixOS. I could try Vagrant with something like https://github.com/elitak/nixos-infect but that seems… risky.

Is there an option I haven’t tried?

1 Like
2

Very interesting question, I am also constantly looking for convenient ways to cross-build for Linux from macOS. Anyone with experience in this area?

3

I never tried it myself but you could try https://github.com/nix-community/linuxkit-nix.

4

I build ISO’s from a VM. Then I boot and run from the .ISO, and never actually install nixos in production environments. This forces the prod nixos instances to be read only. Any state that’s required can be done from a network share, etc.

I use this: GitHub - nix-community/nixos-generators: Collection of image builders [maintainer=@Lassulus] to build the ISO’s.

I’ve had zero luck trying to build a VM image from within a VM, though I never tried very hard. the linuxkit-nix idea from lewo seems interesting, but I’m quite happy just running from ISO’s that I don’t think I’ll bother unless I hit some need the ISO’s can’t solve for me.

5

Thanks, that’s an interesting approach. I could see that being tedious for iterating on the configuration, though, especially testing things like “whether the VM booted from ISO correctly mounts the network share exposed by the other VM with the other ISO that exposes the network mount”.

Linuxkit-nix appears to involve installing nix for macOS, which iirc creates a bunch of users and installs a daemon, something I don’t want to ask of contributors. (My recollection may be out of date, since it’s been a long time since I’ve tried.) I haven’t tested it myself either, but if it’s using LinuxKit I suspect it’ll have the same kvm-within-vm problem. It’s not totally clear to me why NixOS needs a VM to build a VM, and whether that requirement could be worked around.

It’s actually looking like building the missing Vagrant images for myself/the project might be easiest, though I’m less jazzed about the prospect of maintaining them. This also looks promising, though alas no recent activity: https://github.com/NixOS/nixpkgs/pull/76071

Obviously I should just be running NixOS on my laptop, but previous attempts were less enjoyable than running NixOS on servers :slight_smile:

6

Maybe footloose can help in some way? I just found the tool and I’ve just beginning evaluating it for my purposes, but it seemed worth mentioning it here

7

The good news in regards to nixos, if you get it to build, chances are it will run fine.

But I don’t disagree it can be a touch of a pain to debug, but that’s true of debugging any declarative VM, regardless of how it boots/the OS used.

We run SSHD on the VM’s so one can still login remotely to troubleshoot, if needed.

We are taking a slow deliberate approach to conversion to nixOS, so as new services come up, I’m building them in nixOS instead of our traditional model. As I get bored, I’m converting existing infra over. So I don’t have huge across-VM dependency issues to handle.