Thoughts on the friction of classic system administration vs. devops and automation engineering vs. Nix

the indentation would be incorrect, but still produce valid bash.

ngl it was a really tough thing to internalise that generated code doesn’t need to be properly indented. Though to be fair, I had the luxury of a gradual introduction thanks to Puppet’s EPP templating which, like Jinja2, has some weird whitespace magic going on.
I spent minutes looking at the code touched in [the PR I wrote this for][1] and didn’t figure out a simple and readable way to have it look good without a trailing space or something (or moving the entire thing to a list of arguments and lib.optionals).


The above was something I started typing in a reply on a merged PR, but it felt too informal to actually submit.
As someone who started out writing C and whose first Linux distro sticking to the harddrive for more than two days was Gentoo, who got comfy with the lower level parts of Linux right as LXC and namespaces got their finishing touches[2], who later started writing automation code in Puppet, eventually wrote entire pieces of software to take over tasks which would otherwise have to be a regularly running automation snippet, and who moved from Gentoo to NixOS just a few years back.… I have a different relationship with computers than most people, though not necessarily too different from the people over here in this Discourse.
What I want to point at for a moment is the friction Nix and NixOS can cause. I had a conversation with my roommate lately about the configs generated by NixOS modules not having fully commented and documented defaults embedded into the resulting config file[3]. Of course we can’t expect nixpkgs to duplicate every single setting, their defaults, keep everything always up to date, and so on[4]. That’d be a ridiculous maintenance burden, not to mention it would likely render most generated files completely unreadable because of the lack of context on when to embed comments (like not having the same comment duplicated onto half the lines of the config). It did however get me thinking about the duality of friction when introducing people.

On one hand introducing someone to NixOS gives immediate returns on investment when they can literally point at a config and ask “what’s wrong”, rather than having to do a debugging session with screenshare or whatnot, and also because of all the prepackaged but more so the preconfigured software – the pieces we have modules for.
On the other hand, while I’ve always used strace as a very immediate means of debugging[5], the move to NixOS has reinforced that, given that a lot more things rely on environment variables for instance.
I’m not saying that NixOS is in any capacity more difficult to debug than other distros – just different IMHO – but moving something like service configs from /etc to the ExecStart of unit files does change some aspects.
Again, not saying that it’s in any way bad to change these aspects.
I really prefer units to be self-contained, it makes for more streamlined debugging in the end, since I don’t have to check 3 different places for which config may be used, I can just get the one config that systemd tells me is the correct one (using systemctl show -p ExecStart $service if needed).
Of course this extends to plain Nix itself, with a difference between ${var} and $var in a string used by a build making the difference between string interpolation on Nix and shell level and all that.
My point is, if you don’t already know a lot about the systems and underlying parts, it’s most likely a wildly different experience, and not one I can recommend wholeheartedly[6].

Just to reiterate: I like the status quo.
All this is about is that the layers of abstractions in regards to Nix and NixOS have largely hit a complexity peak somewhere in the middle, where it is easier to find an issue in the most top-level code (such as a typo), or the most low-level parts (like a process’ strace). As soon as you go two or three levels of abstraction from one direction, it’s often better to switch to the other end of the stack and debug from there, hoping to be able to avoid the middle part where there’ll be five layers of generated nested shell-scripts so to say[7].
The good thing is that by shoving a lot of complexity into the unit files directly we’re bundling all of that up so that debugging one specific part of a system only requires us to look at that part, which hopefully keeps Nix and NixOS away from the onion that we see in something like Jsonnet generating Helm charts which use container images built from another form of templating etc., as those parts are usually built to be opaque and hide complexity[8] or are just another layer on top like Puppet ensuring a fixed list of installed packages next to the code that deploys the config files for those tools, but without a visible connection between the two on the target machine, unlike Nix which would then enforce the connection by having the tool and the config both explicitly in the unit file. I hope I got across what I mean with that.


Alright, time to cut this ramble off before people tell me to write a blog post instead. I’ll be honest, I just wanted to get this whole thing off my mind so while I’m interested in your take on it, I’m not asking for one if you don’t have one readily available or feel compelled to share your thoughts on some of the things I touched on.
Because I know I’m often hesitant to reply to minor points even though I want to simply because it doesn’t feel substantial enough to do so, have this question to tack your point onto; How and more importantly when in someone’s beginnings with Linux distros or package management (or whatever you use Nix/NixOS for) would you be most comfortable to give an elevator pitch? Like.… how familiar with the workings of something else (say, Debian) would you want someone to be before suggesting they dip their flippers into this wonderful mess of a beautiful nightmare with us?
Oh crap, that last part was speaking my mixed feelings about touching computers out loud, wasn’t it.…


  1. Just for the sake of it, it was about string interpolation in Nix which is, luckily, very whitespace agnostic in its formatting. I feel like I need to qualify the ‘agnostic’ part and Discourse doesn’t seem to like nested footnotes, at least not in the preview, so here we go; when interpolation strings in Nix it just interpolates at the exact place and doesn’t touch the surrounding string, while templating languages usually have something like <% foo %> vs. <%- foo %> for whitespace trimming outside the interpolation. While that may make things a bit more awkward to write sometimes, it definitely makes it more explicit. ↩︎

  2. In terms of basic functionality, of course a lot has happened since.… 3.2? ↩︎

  3. read: nginx.conf listing all of the defaults, and potentially even having descriptions for the different location match syntax types, instead of just putting there what’s either needed or specified by the user ↩︎

  4. Talking about actual Nix-side of comment generation, not just the common $pkg/share/$pkg/doc/example.conf stuff, or docs in general. More of a semantic “here’s what the generated code does” thing emitted from nixpkgs which is borderline impossible and just not feasible. ↩︎

  5. (A few) Ops Lessons We All Learn The Hard Way № 50 ↩︎

  6. I will keep low-key recommending it anyway. I don’t usually advice people on distros to use or such, but the whole “send me your config” approach is just so much better than having to check on their machine what’s wrong. Not to mention the blissful lack of “but I didn’t change anything” and “it works on one machine but not the other” situations thanks to reproducibility. ↩︎

  7. Probably needless to say that I’m a big proponent of using execve(2) over system(3) (read: sh -c) wherever feasible. ↩︎

  8. I’m talking about OCI images being only configured using environment variables, whereas in Nix/NixOS you’d end up with build-time differences which at runtime can be looked at more directly, whereas the OCI parts focus heavily on runtime configuration. Just as an example. ↩︎

Heh, yeah, nix and NixOS are in an awkward place. It proposes a radical shift in how systems should be managed, which clearly has major advantages across the stack, but to achieve any practical system with it you need to work within the confines of software and ecosystems that very much do not subscribe to this worldview (yet?).

The result is this awkward clash where you need to understand both worlds at least to a certain extent to be able to bridge the gaps where necessary.

It sucks, but this chicken-and-egg problem is all over our industry. Think IPv6 vs IPv4. I don’t think there’s a silver bullet to solving these issues, realistically if you work in the software world you just kind of have to learn everything, and just cope with the fact that reality will always be kind of in-between old and new.

Of course, there will always be people who don’t adapt, and that’s why every once in a while I still encounter an IBM system running Fortran…