When managing secrets inside of NixOS, one really unfortunate side-effect is that you very often kind of have to ad-hoc plumb everything through yourself, including possibly templating your secrets into configuration files, manually determining how to give them the correct privileges, etc. This is unfortunate, though, since I think it would be totally possible to make NixOS modules handle a lot more of the aspects of templating and plumbing secret values into runtime configuration.
I’m guessing people have put some thought into this in the past, but I couldn’t find much looking for discussions or RFCs about secrets integration, so I wanted to make a thread dedicated to it, in hopes that maybe the status quo can be moved forward.
I think there are three distinct problems:
-
Secret retrieval/decryption/etc. NixOS can solve this: it won’t stop anyone from manually managing secrets files or using third-party secrets management tools. That said, it is not strictly necessary, and I don’t really want to push a proposal for this right now.
-
Templating secrets into configuration files. I think it would be good if NixOS itself solved this problem, as it’s hard to solve generally otherwise.
-
Plumbing secrets and templated configuration files into daemons. I think that NixOS modules should be solving this problem in a uniform way.
Templating
Often times today, to plumb secrets into configuration files, secrets are injected directly into files using plaintext templating or substitution. This is very powerful and is a nice fallback to have, but not necessarily the right tool for most jobs. With many configuration files using Nixpkgs’s JSON- or INI-based types, having to fall back to text templating is unideal. Escaping of special characters in secrets becomes difficult.
Because of this, I think that it would be ideal if there was some kind of standard library for doing formats with runtime replacements that can be performed e.g. on system activation, or in systemd preStart, such that you could have, for example, some kind of deferred JSON with runtime substitutions. Such that you could do something like:
{ config, lib, pkgs, ... }:
let
json = pkgs.formats.jsonWithSubstitutions { };
mkConfigFile = json.generate config.myConfig;
in
{
options = {
myConfig = lib.mkOption {
type = json.type;
};
};
config = {
myConfig = {
username = "hello";
password = lib.secrets.mkSubstitution "$CREDENTIALS_DIRECTORY/password";
};
systemd.services.myService = {
serviceConfig = {
ExecStartPre = "${mkConfigFile} $RUNTIME_DIRECTORY/config.json";
ExecStart = "${pkgs.myService}/bin/myService --config=$RUNTIME_DIRECTORY/config.json"
};
};
};
}
(If needed, something like this could also be implemented using an activation script, but I think this is better.)
As for how this would work, I’m envisioning something pretty simple. The mkSubstitution calls would need to return something that could be plucked out later, perhaps a set with special attributes (maybe there’s something more elegant that could be done?), such that you could separate it into a “static” configuration and a list of substitutions, a la:
{
value = {
"username" = "hello";
};
substitutions = [
{
valuePath = ".password";
replacement = "$CREDENTIALS_DIRECTORY/password";
}
];
}
And then this set could be used to generate some sort of script that can perform these substitutions and output to a file. I’m thinking something that is roughly equivalent to this:
#!/nix/store/.../bin/bash
>$1 jq ".password" = $(< "$CREDENTIALS_DIRECTORY/password") /nix/store/...-value.json
…but ideally, implemented in a cleaner fashion that can properly escape, in this case, JSON values. But this is the general idea.
Most of the mechanics for this idea could be shared across multiple different formats; only the actual performance of the substitutions needs to be respun for each format.
Plumbing
Secrets stored individually in their own files is a good primitive to build off of. It’s easy for users to manage by hand, and a lot of systems provide an interface like this (e.g. nix-sops, Kubernetes, etc.) If that’s acceptable, then the problem of plumbing just becomes a problem of getting these individual secrets files into a place where the services can access them.
systemd offers LoadCredential which is an ideal way to plumb these files from the system into the services. Since it happens as root, at least by default, it can work with services that use DynamicUser without any awkward hacks. A service that supports switching DynamicUser on and off or providing configuration knobs for privileges wouldn’t need to worry about the secrets being inaccessible.
I’m envisioning that secrets could be bundled up automatically, such that it could be expressed like this:
{
myService.secrets = {
password = ${config.sops.secrets."myservice/Password".path}";
};
}
{ config, lib, pkgs, ... }:
let
json = pkgs.formats.jsonWithSubstitutions { };
mkConfigFile = json.generate config.myConfig;
in
{
options = {
myConfig = lib.mkOption {
type = json.type;
};
myService.secrets = lib.mkOption {
type = lib.types.systemdCredentials; # not sure what to call it/where to put it
};
};
config = {
myConfig = {
username = "hello";
password = config.myService.secrets.password.substitution;
};
systemd.services.myService = {
serviceConfig = {
LoadCredential = config.myService.secrets.loadCredential;
ExecStartPre = "${mkConfigFile} $RUNTIME_DIRECTORY/config.json";
ExecStart = "${pkgs.myService}/bin/myService --config=$RUNTIME_DIRECTORY/config.json"
};
};
};
}
One property that I am trying to maintain here is making it relatively easy for the user to define additional secrets and substitutions. This is nice because in many cases disagreements over what constitutes a “secret” necessitates a lot of unnecessary complexity. It’ll never be ideal, but making it relatively easy and self-contained for users to add more secrets seems like a good idea in general.
For complex configurations, something like Envoy where the shape of the configuration is arbitrary, this sort of capability may be strictly necessary for it to be useful.
Thoughts?
This isn’t really a well-thought-out proposal, more like some thoughts that have been floating around in my head. I’m not really convinced it is necessarily a great idea, but I’m hopeful that with refinement, we can improve how NixOS modules deal with secrets.
Something I didn’t mention is unifying all of the other secrets options. Today in NixOS, some modules provide some secrets management capabilities, but they are not all uniform. Either they forward what the upstream program offers, or they add something on top, e.g. along the lines of envsubst. It might be worthwhile to enumerate the different ways this pans out for complex software and seeing if there’s anything that can be done to unify these interfaces to all work similarly.