Getting started with NixOS – is this a good workflow? (Installing NixOS in a virtual machine, and deploying the config to the hardware when I'm happy)

Help
1

Hi,

As a control freak I really love the idea behind Nix and NixOS, however switching to a different Linux distro is a huge step (my current Arch installation is many years old).

Is my assumption correct, that the following workflow would work?

  1. Install NixOS via QEMU in a virtual machine
  2. Port stuff over, piece-by-piece
  3. Use my current OS and NixOS productively at the same time
  4. When I’m happy, deploy that NixOS config to the real hardware
  5. Now I have a working NixOS installation

That would make things much easier for me.

One potential issue I see, is that I develop software in Rust, and that the Rust compiler is extremely resource hungry (when it compiles stuff), so I really need all my CPU cores for that.

I’m not sure if there is a way with QEMU to run in “cooperative” mode, meaning that the host OS (Arch) and the client OS (NixOS) both have access to all CPU cores, somehow.

Another potential issue is, that there are likely differences between the real hardware and what QEMU emulates. Is this something to worry about?

I tried to get started with Nix on Arch, but I’d rather directly switch to NixOS.

I estimate that switching to NixOS and setting up a usable system would take me at least two full weeks. So the above workflow would be ideal for me, because I can spread that two weeks over a span of several months, and still always have a working system.

2

Hi, I started somewhat similar, and created a VM (using VirtualBox though) that used a real physical disk, so that I could either run my NixOS as a VM or boot it natively.

One thing I noticed was that I had to run the generation of the hardware-configuration.nix once in a VM-context, and once after booting natively, because they added different modules (makes sense), and I effectively merged them for both to work.

So yeah, your general approach should work, just be prepared for some of the closer-to-the-metal bits of your config to (maybe) require some tweaking when you switch from VM to native :slight_smile: Graphics driver etc. can be most prominent, because they differ greatly between VM and native.

That said, the switch from a non-NixOS distro (i.e., an FHS distro) to NixOS can hit really really hard. I was suddenly not able to compile some (larger) C++ project anymore. Like not at all. It should be much better with Rust, which has great support in nix & nixos now :slight_smile: But yeah, trying NixOS out in a VM first is really the way to go, especially if you don’t want to lose (a lot of) productivity. However, it will initially be harder than, and different from, any distro you have used so far, so be prepared for that to hit anyway :slight_smile:

This forum and the nix community in general are great though, and I’m sure you’ll be able to find help/solutions for everything you face along the way! Good luck! :slight_smile:

3 Likes
3

I agree, this is quite sensible. nixos-rebuild in fact even has a build-vm command, which you can use to build VMs and test your configuration before you deploy it (only works on NixOS, sadly, but still useful and shows the use case is expected).

Little recommendations to your current workflow though:

I wouldn’t recommend that. While it’s not always harmful, the hardware-configuration.nix file is largely supposed to be host-specific. Mix-and-match will cause problems most of the time.

I’d recommend using nixos-rebuild’s -I to specify a different entrypoint, and have that entrypoint do something like this:

# /etc/nixos/vm-configuration/default.nix
#
# Install with `nixos-install -I 'nixos-config=/mnt/etc/nixos/vm-configuration/default.nix'`
# (after copying your configuration directory to that place).
{
  imports = [
    ../configuration.nix  # This is where you put configuration shared between both targets
    ./hardware-configuration.nix
  ];

  # Also any other vm-specific options you want to set here
  # If you want to, you can use this to make it the default for the host once installed:
  # nix.nixPath = [ "nixos-config=/etc/nixos/vm-configuration/default.nix" ];
}

You can make a similar directory for every machine you deploy to, and just ask nixos-rebuild to build a different entrypoint for each machine.

This way you can leave the hardware-configuration.nix file for each target untouched, so you can easily regenerate it if you ever want to update it. It also gives you a clean upgrade path to flakes, as you can just set a different hostname for each of these entrypoints.

Also, if hardware-configuration.nix contains anything you think it shouldn’t, it’s better just to override that configuration in your other .nix files than to modify hardware-configuration.nix.

Just for reference, this is likely because the development files of your dependencies are not installed when you add them to environment.systemPackages. This is sometimes due to split packaging (dev stuff isn’t always installed unless you ask for it), but I also believe the env variable the linker uses to find the libraries isn’t set up either.

Typically the solution for this is to add those packages in a nix-shell instead, as it will make the development libraries available.

I wonder if this gotcha could be fixed by adding an option that does install dev things, and enable it by default. Then we’d not differ from other distros in at least one way…

2 Likes
4

Thank you, @futile and @TLATER, that sounds fantastic. Also thanks for the advice!

5

If you really need each cpu cycle, you can always suspend your QEMU-VMs.

You can also give Nix on Arch another look, by starting with home-manager
Remembering my own journey into NixOS I crammed everything in my system config and decided to start with home-manager only after getting familiar with NixOS, which in retrospective was not the best idea (I also have a machine that has to be kept on Ubuntu, but all customization is done in home-manager, so I can reuse it across machines).
My advice: keep you system config lean, put software you use on a daily basis (browser, mail client, etc.) in home-manager and everything that is project-related (including compilers) in project-specific nix shells/flakes

2 Likes
6

Thank you, @wamserma!

Remembering my own journey into NixOS I crammed everything in my system config and decided to start with home-manager only after getting familiar with NixOS, which in retrospective was not the best idea

I was planning to do the same (starting to use home-manager on NixOS), but it seems I should reconsider that.

If you really need each cpu cycle, you can always suspend your QEMU-VMs.

The thing is, that programming is the main thing I do, so if I keep the coding related stuff on the host, then there isn’t much to port to NixOS.

Maybe the best approach would be to port stuff to NixOS whenever I have some free time, but still continue to do all of my work on Arch. Then, when my NixOS is complete (and I have experimented with it a bit), I’ll switch to it in one go.

In the meantime, I can use home-manager on Arch to get even more exposure to the Nix ecosystem.

7

My intention was the other way round! When you are switching to NixOS, start early with home-manager!

There seem to be quite some people who use Nix on Arch. For programming, I suggest you do not install all your tools via home-manager, but rather set up direnv and your favourite editor via home-manager and drop a bit of nix into each of your project directories where you configure your project’s toolchain. This way you get a nice separation and each project can use it’s own version of the toolchain without messing with the tooling for other projects.
→ Effortless dev environments with Nix and direnv

https://devenv.sh/ is also useful if your projects are bit more complex than “hello world”.

2 Likes
8

While I generally agree this is the optimal setup…

Can be very frustrating. So can managing it all with home-manager.

At least initially, don’t get too upset if you don’t manage to get your python/npm/rust/go/whatever language with a native package manager to work. Especially when you’re just starting out with nix, you’ll probably want to fall back to using said native package manager quite often.

Knock yourself out with C/C++ projects though. They’re good to learn with, and you can’t easily access libraries in them without an appropriate nix shell.

1 Like
9

FWIW, that’s exactly what I did (from Arch too), and I have not regretted it since (6-7 years later). Even if Nix has some warts left to iron out in the enterprise space, as a personal development OS for your workstation, I’d say it is totally unmatched.

2 Likes