Best practices / code structure for large deployments

One thing the Nix ecosystem seems to be lacking is a common standard for how infrastructure code should be organized. This is understandably outside the scope of what Nix aims to achieve, since Nix has so many different uses and install bases, but I think it would be interesting to have a discussion about it.

There seems to be a lot of discussion already about Nix’s other use-cases, such as being a build tool or using NixOps/Morph for small/medium size deployments. Where I don’t see much discussion is for larger scale deployments (500+ nodes) where you’d traditionally use something like Puppet with the “roles and profiles” pattern and an ENC like Hiera.

Are there any best-practices for this sort of thing? How to organize your code? What should or shouldn’t go into a module? How to program in little differences per region/environment while keeping everything dry?

It’d be kind of nice to come up with a common structure. Combined with the new flakes feature it would make getting started with Nix/NixOS in a large environment much easier.

2 Likes

Hopefully you’ll get a much better answer but here is the structure of my badly and over engineered configuration.nix (take hints from filenames)

configuration.nix

	nixpkgsTarball = rec {

		unstable = fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/nixos-unstable.tar.gz;
		
		#Github master. Untested and self-build.
		master   = fetchTarball https://github.com/NixOS/nixpkgs/archive/master.tar.gz;

		"20.03" = fetchTarball {
			url		 = archiveUrlBuilder "b2935fbeceaea0b64df4401545d7c8ea29102120";
			sha256 = "1z1631v7q2c8mavy7xnvfx0wz34zd49jqmjg66nk0qgsi605m3qp";
		};

		"19.03" = fetchTarball 
		{
			url = (archiveUrlBuilder "e758436f98683a77b8cd119886c9ded8b8d80759");
			sha256 = "0wwl7hh47nq1nb69nmnrpiy4an18614vkikvljj2x86pd164781m";
		};

		"2019.10.01" = fetchTarball 
		{
			url = archiveUrlBuilder "b3b123a104f448e1f377b162cbd52af461fdecb2";
			sha256 = "02d5q0hbpbq1k0ry7ws9ai78qy7piz91aygxx7wbfdgqwl37ij9x";
		};
		
		localCheckout = /home/zx/workspace/git/nixpkgs;
		qjournalCtlPR = fetchTarball ( 
			archiveUrlBuilder "8cd1f275c11d447f42161adf49242e7629695897" 
		);


   ...
...
...
 

	imports =
	[
		./custom/onedrive.nix
		# ./custom/open-with/open-with.nix
		# Include the results of the hardware scan.
		(
			if ( productVersion == 	"ThinkCentre M93p" ) 
			then
			./subconfigs/hw.desktop/config.nix
			else
			./subconfigs/hw.vaio/config.nix
		)

		./subconfigs/font.nix
		./subconfigs/zfs.nix
		./subconfigs/grub.nix
                ./subconfigs/packages.nix
	];

users.nix

{ pkgs, mainUser, mainGroup, homeMaker }: 






builtins.listToAttrs
(
	map
	(
		incompleteUser:
		{
			name  = incompleteUser.name;
			value = incompleteUser.value // 
			{
				home  = homeMaker incompleteUser.name;
				group = mainGroup;
				isNormalUser = true;
				extraGroups = [
					# Sudo
					"wheel"

					# Network manager
					"networkmanager"

					# Run locate using locate db.
					# Only for autogenerated db
					"locate"

					# [ CommentDb: 1 ]
					"video"

					"lxd"
				];
				shell = pkgs.versioned."19.09".zsh;
				# mkpasswd -m sha-512 # Careful, mkpasswd is its own package
				hashedPassword=
					"$6$IV2x36/8$LYL29lIpAMeRg4xasdasasdiF1veU5uAKnasdfFmBIAZ1xHpZQF4tbpWl4D/";
			};
		}
	)
	(
		[ 
			{
				name = mainUser;
				value = {
						uid = 1000;
				};
			}
			
			{
				name = "head";
				value = {
						uid = 20000;
				};
			}
		] ++ 
		( 
			map 
			( 
				id: 
					let uname = "tmpu${builtins.toString id}"; 
					in
				{
					name  = uname;
					value = 
					{
						uid = 20000 + id;
					};
				}
			)
			( pkgs.lib.lists.range 1 9  )
		)
	)
)

Then packages.nix further imports editors.nix, systemmanagement.nix, media.nix etc.

I had multiple disks and two computers and I wanted to make an arrangement that would help me plug and play without thinking much. Thing is there is no set answer I think, you just have to use your own categorization skills and categorize options into nix files.

This is really easy to do with the existing NixOS module system. We use role modules like this at work:

{ config, lib, ... }:

with lib;

let

  cfg = config.our.roles.our-role;

in {

  options.our.roles.our-role = {

    enable = mkEnableOption "our role";

  };

  config = mkIf cfg.enable {

    our.basic.enable = true;
    our.networking.enable = true;

  };

}

(We use the “our” namespace, so none of our module names will conflict with existing NixOS modules and to tip us off that we are dealing with our configuration options.)