Running NixOS clusters

Help
1

NixOS adoption has a few stumbling blocks. Besides the obvious I would like to gather some communal wizdom on the following related topics:

Cloud Images

It seems like NixOS is still considered an odd duckling by cloud providers.
At least there seems limited support to provide a base image.

Now my current approach is:

  1. create a Debian system
  2. use cloudinit to nixos-infect the system (unfortunately no flake support)
  3. then manually clone my flake and switch

Now I guess this could then be used as a basis to clone machines?
I am not sure about the implications though (mac addresses, host keys etc).

Is there a better way? There needs to be a better way.

I want to be be able to create two cloud machines on Hetzner and bring up a system with my flake applied.

Has anyone gotten such a setup working?

Managing many machines/servers

Is there anyone here running more than 10 servers (50+?) on NixOS?
How does nix scale from a management point of view?
A single file that holds all nixosConfigurations?
How do you apply changes? deploy-rs and friends?

And I am also a little worried about resources to evaluate my flake.
My little flake for less than a handful of machines can already take quite long to evaluate (on a powerful machine). What will happen with 100+ nixosConfigurations? (I didn’t try yet)

And every server will download and build the derivation?
I guess some local caching might help? Like

Binary Cache - NixOS Wiki
cachix
atticd

And then it also feels like there is a certain overlap with terraform/opentofu.

I found:

GitHub - nix-community/terraform-nixos: A set of Terraform modules that are designed to deploy NixOS [maintainer=@adrian-gierakowski]

but I am curious how this works in reality.
The integration and the flow is still a bit unclear.
Are hcl files derived from the nix config?

I also saw NixOps. But it seems that’s on its way out - with a pending re-write.

If you have some feedback or repos to share, that would be great.
Thanks!

3 Likes
2

what is the part that you consider so terrible here? the actual steps involved, or that you have to manually run the steps yourself?

NixOps has a hetznerCloud plug-in that makes this pretty much a one step operation… maybe that alleviates your pain here?

of course NixOps isn’t maintained very well these days, but there is work being done on a rewrite! and I’m the meantime you could always try my project teraflops which i find an effective way to manage cloud machines

i think the most i have currently run in a single deployment was about 150 - this was done using the above mentioned teraflops (which simply uses colmena) under the hood. in scenarios like this i find it important to have sufficient resources on the machine you deploy from, to utilize nix-eval-jobs (which colmena supports), and have a bit of patience :wink:

i will mention that the more common number of NixOS machines per deploy that i work with is closer to ~20 or 30… but i am happy with the results of my largest deploy still

i split my deploys into multiple files based on the role of the machine so my project contains the usual flake.nix/lock, a database. nix for nodes running the database, a frontend.nix which holds the web stuff from developers, worker.nix for backend workers, etc…

deployments handled via colmena (via teraflops as mentioned above) but other solutions like deploy-rs might work well too? specifically colmena works so well for me on larger deploys because it uses nix-eval-jobs giving it much improved preference… so personally i highly recommend colmena

right! this is the basis of teraflops - it’s mostly a small python script to integrate terraform with colmena so it handles everything end to end :muscle:

as you mentioned there are other options in the space as well - you should also check out another excellent project called terranix to get an idea of what’s available

this specifically was the goal when i wrote teraflops - tight integration between terraform and a nix deployment tool (colmena) - you write everything in a single language (bye bye hcl, hello nix) and can reference your terraform objects from your nix nodes as well as your nix nodes from your terraform objects

i have mentioned my project quite a bit here because you specifically wanted to know what other people were doing… but i don’t want this to sound like i am saying this is the only way i can see doing things - it is just the solution i landed on

there are a ton of other great solutions out there and the one you should pick just depends on what resonates best for you

hope that helped

3

The manual steps.
I want to be able tell e.g. opentofu to create a machine - and then have it setup. Done.
Certainly a real base image would be better - but I don’t mind the infect so much.

Yes, it would, but neither option sounds great.
Using an EOL project and waiting for the rewrite.

I’m the meantime you could always try my project teraflops which i find an effective way to manage cloud machines

I’ll check it out! Looks pretty nice.

Thanks for that number. That does sound promising. Although it also sounds like a bottleneck for bigger deployments - not really what I need to worry about. But it might impact adoption in other areas.

specifically colmena works so well for me on larger deploys because it uses nix-eval-jobs giving it much improved preference… so personally i highly recommend colmena

Could you expand on that?

Does colmena work across archs?
I am deploying from nix on darwin to linux on amd64 here.

hope that helped

That totally did. Much appreciated!

4

sure, see Parallelism - Colmena (Unstable)

i believe it does - it has multiple platforms listed and built in nixpkgs

i haven’t done it… but it makes sense that it would work fine

1 Like
5

By default, Colmena evaluates nodes in batches according to available RAM using Nix’s built-in single-threaded evaluator. Experimental support is available for using nix-eval-jobs as the evaluator.
When nix-eval-jobs is enabled via --evaluator streaming, evaluation is parallelized with deployment processes kicked off as individual nodes finish evaluating.

TBH that leaves me with a lot more questions :slight_smile:

When you start a deploy from your laptop - which machine evaluates what?
So it’s not like your laptop would evaluate for all 150 servers and then start the deploy?
Is there partial evaluation? (It sounds like there is)

I’ll give it another try.
With deploy-rs it worked when I used remoteBuild = true;

When you deployed those 150 servers - did you have some sort of local cache the machines shared? Or were those 150 servers saturating your downlink, downloading all the nixOS packages? :slight_smile:

6

hopefully i can answer your questions with enough clarity

  • i run the deploy from a machine that has a fair bit of resources available to it, but with colmena you can specify remote builders too
  • check out the nix-eval-jobs documentation for some details on that but with colmena any one node can reference the configuration of any other given node… so there is some (somewhat) lazy evaluation going on, though not that lazy IIRC
  • there hasn’t been a big need to have a local cache aside from our hydra instance - the first deploy hits the network hard sure… but periodic minor updates aren’t a problem at all - if this ever became a problem it would be pretty easy to take care of
1 Like