I’ve been trying to figure out secrets in NixOS (using flakes) and I’m totally lost. It seems like nixos is insanely fragmented when it comes to secrets management.
I’m configuring promtail, which takes in a configuration. In the url field, there is a secret value that I don’t want to expose in my git repo.
How can I simply template in a secret value like this? Everything I’ve read exclusively talks about secret files, but surely there is some way to do something like this?
Not only do you want to keep it out of git, you probably want to keep it out of the nix store (the nix store is publicly readable, many people throw their build contents through caches, etc). Both properties will drive you towards having to keep the secrets in their own distinct context - and particularly the latter requirement is not really compatible with templating.
This is why many in the community use sops-nix or agenix, which both encrypt your secrets to host-bound keys and then provide nixos integrations that enable run-time decryption and mounting of the secrets, and make their path available via nix to interpolate into your configuration.
So, instead of build time templating, I’d suggest aiming for activation-time mounting and runtime filesystem discovery. EDIT: Oh neat, I guess sops-nix can do activation-time templating too, very neat.
If you don’t care about secrets leaking into the nix store, then your options become quite numerous as its “roll your own secret and templating management however you otherwise would” but I really don’t recommend that.
Interesting, but it seems like that template is only for configuration files. While I might be able to make that work in this case, I’m still hoping that I could do it inline in the nix config itself
The basic problem is that (mostly, for a general solution) we want to keep the decrypted secrets out of nix evaluation entirely, and only decrypt them on the target host (using the host ssh key, for example).
This keeps the decrypted secrets out of git as nix input / history, and also out of the world-readable nix store, and any binary cache, build hosts, logs, and other places they shouldn’t be and are a risk of disclosure. Fundamentally, that means that useful nix features like assembly of application config text files that expect secrets directly in them can’t work with the secret values.
Solutions like agenix and others instead use other tools and methods during activation (long after nix has finished running) to put secrets in the right places, either as files or using a different substitution and text-assembly / templating engine. Even if it’s a nice one, at runtime, it’s still different and you need to be aware of the difference.
That’s why it feels like a “bad fit” and a bit scattered - because it is, and there are several alternative solutions, none of which are a part of nix proper.
I made Scalpel for this reason. It’s more of a workflow than an actual tool. But the basic idea is:
Instead of putting your actual secret in the config file, you put a placeholder
You find out how to extract the store-path of the generated config file
Scalpel will, at runtime, create a variant of the config file in a temporary location where the placeholders are replaced with the actual secret
You replace the store-generated config file with the generated config file
Unfortunately, there is no cookie-cutter solution for steps 2) and 4) and they will depend on the service and how it receives its config file. So this will require some tinkering. It works well enough for me usually, but it’s probably too much of a specialty thing to advertise widely.
It’s meant to be used in conjunction with agenix or SOPS. They will provide the secure secret storage and Scalpel puts the secrets in the configuration.
To add to that, there aren’t really any declarative-provisioning systems out there that actually have a good solution to this. Fundamentally, secrets should not be part of your declarative configuration, because by definition they should only be where they are used.
Kubernetes usually solves it via an awkward secret service (often vendor-specific or one of many fragmented FOSS tools - like the very sops sops-nix uses), ansible has its vault which has the same exact limitations NixOS has, ditto with docker secrets, systemd credentials, etc. None of these tools can really be considered “perfect” under all use cases, and are quite complex considering how simple and fundamental the problem is.
Secrets just aren’t an easy thing to solve, and as usual lack of standardization and 40 years’ worth of legacy software is making a convenient solution impractical. See also this post a while ago from @ElvishJerricco that likens secrets more to data, which I found really insightful: Alternative way to handle secrets - #2 by ElvishJerricco
Then use sops-nix or other to configure env.
Something like change the command line of our service to one that loads env form secret file and then start it.
Oh thanks, I didn’t know that! Yeah I think this + @polygon’s Scalpel should be able to cover most of my needs. I’ll try those things out and report back with results
I’ve been handling inline secrets with git-crypt, but after reading this thread I might switch to something different. Just for posterity, though, I have a file secrets/git-crypt.nix:
Noob question, but what does the secrets being publicly available in the nix store actually mean? Does that just mean that on the machine those secrets are readable somewhere in plain text, or is it worse than that?
They end up plaintext in /nix/store, and importantly with a+rwx permissions, so all users can read them (but not actually write because the store mount is special).
On really simple setups that are just personal machines, maybe it’s not that big of a deal, but even there it breaks a lot of the features that desktop security is based around. Containers and flatpaks, for example, can’t easily prevent their processes from reading files in /nix/store, and a lot of the systemd hardening NixOS does becomes effectively useless.
If you take your root password from git-crypt, for example, that basically means every user on the system (including all the system users) has full root permissions.
When you do remote builds the secrets will also end up on those builders, though admittedly that’s less common for personal configurations.
Okay I see, mostly I care about securing docker containers as some of them will be exposed to the internet. Wouldn’t docker containers be isolated such that they can’t access /nix/store unless I specifically give them access to that directory, or am I wrong?
I think I can live with users on the machine having access to this particular secret, but it’s not great if any docker container can see it.
NixOS containers do give full access to /nix/store, which people might stumble over when switching technologies, for example, and this is completely ignoring the potential for ACE bugs being turned into full privilege escalation. I wouldn’t do this on a host facing the public internet.
Even when using docker to run services, being able to launch a hardened systemd unit (which uses the kernel’s containerization features as well, but cannot easily prevent access to /nix/store because the binaries are ultimately in there) is very convenient. NixOS’ nginx module is really good, for example, and nginx use is ubiquitous as a reverse proxy for docker setups.
Just grab sops-nix (or agenix for simpler use cases) and have reasonable secret management to begin with - it’s not really much additional effort in exchange for not breaking services’ expectations around secret permissions.