I’m looking for some advice on managing many, potentially hundreds of NixOS machines, remotely. I’m already running with flakes on my personal machine, and I can easily see how this can be adopted to several machine, using multiple hosts with some shared nix configuration files split into modules. One issues I’m running into is the ‘hardware-configuration.nix’ file…
to build and activate the configuration on a single remote host. My issue is that even though the remote host already has a hardware-configuration.nix, it cannot seem to find/use it. I think that the “obvious” solution is to copy the remote host hardware_configuration.nix to my repo and include it there, but this will scale poorly, as I then expect I would need the hardware-config for every single host to also be in my repository. Many of these machines are identical hardware-wise, but the hardware config also includes IDs for devices and such…
I hope I’m explaining this well… does anyone now of a good solution for this? Or maybe someone has experience in managing many identical NixOS machines in a smart way, using flakes?
So assuming we can groups machines in hardware groups, i.e. those with equal hardware, one could just write their own custom hardware-configuration (based upon the one that is generated by nixos-generate-config) and then specify that the disk should be label by labels instead of UUIDs ?
As an example:
flake.nix
flake.lock
hosts/
base-thinkpad <- this includes the common.nix and the thinkpad-hardware config
base-dell
common.nix
modules/
hardware/
dell-laptop-hardware.nix
thinkpad-laptop-hardware.nix
I’ve included the contents of hardware-configuration.nix in the host-specific files ekster.nix and gekko.nix
Each host does a git pull of /etc/nixos/ and a nixos-rebuild switch every 30 minutes (inspired by puppet). One host is in charge of updating flake.lock.
My flake.nix currently has 14 entries but could potentially hold hundreds.
Thanks for sharing! This is a very neat way of doing it, but there’s one thing I’m not able to easily solve. Let’s say we have 100 Laptops, but each of them might want slightly different software. One may want audacity to edit audio and another might want GIMP. It seems to me that the only way to do this (aside from using containers as an example) is to have 1 entry pr. specific setup ie.
nixosConfigurations = {
"host1" = {
system = ...
modules = [
[...]
./common.nix
./host1.nix (<- In here we specify 'audacity' as a systemPackage)
];
};
"host2" = {
system = ...
modules = [
[...]
./common.nix
./host2.nix (<- In here we specify 'GIMP' as a systemPackage)
];
};
};
"hostX" .....
It gets even more complicated when one considers combinations of different software. this could potentially result in hundreds if not thousands of nixosConfigurations in the flake.
I’ve been experimenting with a building a service that can dynamically create ‘configuration.nix’ files, based upon client requests and some base configuration. As an example a machine with id ‘123’ would every 30 minutes send a request to a central server
GET /configuration?id=‘123’
The server would then create a configuration based on some state and knowledge about what kind of the client the pc with that id is, as an example if that specifc id has requested GIMP or Audacity, this would be included in the systemPackages for the resulting configuration.nix.
If the client then registers a difference between the received upstream configuration and the one that is currently in /etc/nixos it would fetch the new one and trigger a nixos-rebuild switch.
I’m unsure if this is bad use of nix and if it goes against how it should be used though.
I carry all my specific software and configurations with me in <user>.nix and same for my daughter in <daughter>.nix.
Software like chromium, firefox, thunderbird, libreoffice and gimp is enabled in desktop.nix, with user-specific configuration for these and extra software in <user/daughter>.nix. She uses gnome, I use sway.
With this /etc/nixos/flake.nix has as many entries as you have hosts to manage, and /.config/home-manager/flake.nix has as many entries as you have users to manage.
Do you think this setup would scale well to hundreds, maybe even thousands of users? There would probably need to be some form of GUI such that users could request new packages, as they would not be technical, but this could probably be achieved with a simple flask app that parses and allows the user to modify their <user>.nix in some simple and controlled way.
I’d say the method for users to request new software or other changes is independent of the underlying infrastructure. Depends on your users as well.
You’re right there. I just can’t tell if it’s easier to maintain differences from a common configuration as entries in a flake, with corresponding host/user-configurations, or entries in a database that are used to dynamically build configurations.
I think this setup can scale, but have not tried. Would you be in such a position?
Hopefully yes. Right now We’re looking at a proof of concept and initially it will be a very small user base, but I’m trying to make the best decisions early.
Thanks a lot! Just a small update, after playing around a little bit, I think I’m closer to doing it the “right” way. I have created the following flake:
{
description = "Dynamic NixOS config based on hostnames";
inputs.nixpkgs.url = "nixpkgs/nixos-25.05";
outputs = { self, nixpkgs, ... }:
let
# For now we assume x86 64bit Architecture.
pkgs = nixpkgs.legacyPackages.x86_64-linux;
# We load our hostname -> package mapper from a git versioned JSON file.
packageMap = builtins.fromJSON (builtins.readFile ./extra-packages.json);
#
# We define function 'getExtraPkgs'
# This function accepts a 'hostname' as a paramater and check if the packageMap
# variable has a value for the supplied hostname. If it does, return the package list
# else return an empty list.
#
getExtraPkgs = hostname:
let pkgNames = if builtins.hasAttr hostname packageMap then packageMap.${hostname} else [];
in map (name: pkgs.${name}) pkgNames;
#
# We define function 'makeSystemConfig'
# This function accepts a hostname and generates a nixosSystem configuration
# based on the common.nix configuration and potential extra packages as found
# by the getExtraPkgs function.
#
makeSystemConfig = hostname:
nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
./configuration.nix
{
networking.hostName = hostname;
environment.systemPackages = getExtraPkgs hostname;
}
];
};
in
{
# Dynamically generates the nixosConfigurations for each machine.
nixosConfigurations = builtins.listToAttrs (
map (hostname: {
name = hostname;
value = makeSystemConfig hostname;
}) (builtins.attrNames packageMap) # Iterate over all hostnames in the JSON
);
};
}
Along side a git-versioned JSON file, as an example:
This way I can specify a general configuration.nix that imports modules needed across all machines and also provide af JSON file that maps hostnames → [ nixpkgs ] that will be included during the nixos-rebuild. It’s possible to test configuration with
Which will produce a ‘result/bin/run/laptop-001-vm’ which will start QEMU. I’ve specifically chosen JSON as the mapping file, as this can easily be parsed by the builtins.fromJSON function and just as well I can built a simple application that will allow less technical user to update the file for their relevant host, that will parse and allow editing of the JSON contents in a safe and controlled way. There are still many things that can be improved, but I think this is an ok starting point. And obvious improvment would be to allow more architectures than 64bit x86, but this is not important for the POC.