Best practices / code structure for large deployments

One thing the Nix ecosystem seems to be lacking is a common standard for how infrastructure code should be organized. This is understandably outside the scope of what Nix aims to achieve, since Nix has so many different uses and install bases, but I think it would be interesting to have a discussion about it.

There seems to be a lot of discussion already about Nix’s other use-cases, such as being a build tool or using NixOps/Morph for small/medium size deployments. Where I don’t see much discussion is for larger scale deployments (500+ nodes) where you’d traditionally use something like Puppet with the “roles and profiles” pattern and an ENC like Hiera.

Are there any best-practices for this sort of thing? How to organize your code? What should or shouldn’t go into a module? How to program in little differences per region/environment while keeping everything dry?

It’d be kind of nice to come up with a common structure. Combined with the new flakes feature it would make getting started with Nix/NixOS in a large environment much easier.

4 Likes

Hopefully you’ll get a much better answer but here is the structure of my badly and over engineered configuration.nix (take hints from filenames)

configuration.nix

	nixpkgsTarball = rec {

		unstable = fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/nixos-unstable.tar.gz;
		
		#Github master. Untested and self-build.
		master   = fetchTarball https://github.com/NixOS/nixpkgs/archive/master.tar.gz;

		"20.03" = fetchTarball {
			url		 = archiveUrlBuilder "b2935fbeceaea0b64df4401545d7c8ea29102120";
			sha256 = "1z1631v7q2c8mavy7xnvfx0wz34zd49jqmjg66nk0qgsi605m3qp";
		};

		"19.03" = fetchTarball 
		{
			url = (archiveUrlBuilder "e758436f98683a77b8cd119886c9ded8b8d80759");
			sha256 = "0wwl7hh47nq1nb69nmnrpiy4an18614vkikvljj2x86pd164781m";
		};

		"2019.10.01" = fetchTarball 
		{
			url = archiveUrlBuilder "b3b123a104f448e1f377b162cbd52af461fdecb2";
			sha256 = "02d5q0hbpbq1k0ry7ws9ai78qy7piz91aygxx7wbfdgqwl37ij9x";
		};
		
		localCheckout = /home/zx/workspace/git/nixpkgs;
		qjournalCtlPR = fetchTarball ( 
			archiveUrlBuilder "8cd1f275c11d447f42161adf49242e7629695897" 
		);


   ...
...
...
 

	imports =
	[
		./custom/onedrive.nix
		# ./custom/open-with/open-with.nix
		# Include the results of the hardware scan.
		(
			if ( productVersion == 	"ThinkCentre M93p" ) 
			then
			./subconfigs/hw.desktop/config.nix
			else
			./subconfigs/hw.vaio/config.nix
		)

		./subconfigs/font.nix
		./subconfigs/zfs.nix
		./subconfigs/grub.nix
                ./subconfigs/packages.nix
	];

users.nix

{ pkgs, mainUser, mainGroup, homeMaker }: 






builtins.listToAttrs
(
	map
	(
		incompleteUser:
		{
			name  = incompleteUser.name;
			value = incompleteUser.value // 
			{
				home  = homeMaker incompleteUser.name;
				group = mainGroup;
				isNormalUser = true;
				extraGroups = [
					# Sudo
					"wheel"

					# Network manager
					"networkmanager"

					# Run locate using locate db.
					# Only for autogenerated db
					"locate"

					# [ CommentDb: 1 ]
					"video"

					"lxd"
				];
				shell = pkgs.versioned."19.09".zsh;
				# mkpasswd -m sha-512 # Careful, mkpasswd is its own package
				hashedPassword=
					"$6$IV2x36/8$LYL29lIpAMeRg4xasdasasdiF1veU5uAKnasdfFmBIAZ1xHpZQF4tbpWl4D/";
			};
		}
	)
	(
		[ 
			{
				name = mainUser;
				value = {
						uid = 1000;
				};
			}
			
			{
				name = "head";
				value = {
						uid = 20000;
				};
			}
		] ++ 
		( 
			map 
			( 
				id: 
					let uname = "tmpu${builtins.toString id}"; 
					in
				{
					name  = uname;
					value = 
					{
						uid = 20000 + id;
					};
				}
			)
			( pkgs.lib.lists.range 1 9  )
		)
	)
)

Then packages.nix further imports editors.nix, systemmanagement.nix, media.nix etc.

I had multiple disks and two computers and I wanted to make an arrangement that would help me plug and play without thinking much. Thing is there is no set answer I think, you just have to use your own categorization skills and categorize options into nix files.

This is really easy to do with the existing NixOS module system. We use role modules like this at work:

{ config, lib, ... }:

with lib;

let

  cfg = config.our.roles.our-role;

in {

  options.our.roles.our-role = {

    enable = mkEnableOption "our role";

  };

  config = mkIf cfg.enable {

    our.basic.enable = true;
    our.networking.enable = true;

  };

}

(We use the “our” namespace, so none of our module names will conflict with existing NixOS modules and to tip us off that we are dealing with our configuration options.)

Some of those question and a good part of what’s motivating your post is being addressed by:

We would be very pleased to have your (real or hypothetical) use cases on board and I’d specifically ask you to open up github discussion on that repository.

This is especially a good moment, since we are at the brink of expanding into the enterprise use cases, by first incorporating kubenix as a kubernetes resource builder.

The per-region requirement as well as efficient nix evaluation at scale are probably points we can investigate further together.

We also have the suites concept to consolidate deployment scenarios and have an “inoculating iso” and are further researching the identity bootstrapping workflows (network & cryptographic), which are thought for ease and scaleability of bare metal deployments.

Unfortunate that I saw your post only until now.


EDIT: there is also a project which looks at those problems from a hashicorp ecosystem angle (and seems to have a little bit of mind share with devos):

In my company we are using the *-nixpkgs patttern, where we inherit upstream nixpkgs, and we basically overlay with our own, and layout the same way that nixpkgs lays out. We also then use logical and hardware profiles.

This then can be used with any combination of snowflake machines, auto-updating fleet, etc, all from one tree. When we first created this, we were also building it with hydra, and managing hydra with the same source tree as well. The hydra-built version of it now lives over at https://github.com/Holo-Host/holo-nixpkgs and has evolved/changed since we worked on it. But still built by hydra, still following the same core pattern. Deployed machines from holo-nixpkgs also auto-update themselves, so that one can be a good one to look at for large scale auto updating fleet of machines.

An example template lives at GitHub - holocene-systems/template-nixpkgs haven not got around to converting it all to use flakes yet, but it could be done.

1 Like