Managing Secrets in NixOS

Help
1

I’ve been looking at ways to manage secrets in Nix, and have not yet chosen one.

Being new to Nix, I setup a Raspberry Pi as an exercise, with a few services, that each consume secrets from their config files.

As a first step, I chose the easiest approach I could think of: to separate my secrets from the configuration.nix file, by simply importing a separate “./secrets.nix” file from configuration.nix.

Once I got things working, I created a git repo, with configuration.nix, flake.nix, etc, and put the ./secrets.nix in .gitignore, and was surprised that nixos-rebuild was now searching for the secrets.nix in the nix store !

I then learned that nixos-rebuild requires all config to be in git (nixos-rebuild assumes that if the file is not in git, it’s in the nix store). I can understand why, but I’m not sure if I like this, and I tend to agree with this reddit poster Reddit - The heart of the internet

I don’t like the idea of being forced to put secrets in source control. I think there are valid reasons to do so (put encrypted secrets in git), but I don’t like the idea of forcing that choice.

I’m about to try out agenix, and I’m hopping it will allow me to keep secrets out of source control.

I’m mainly interested of secrets management for systemd services, and from what I notice, there is quite a bit of “diversity” in the way a nix service package decides to load it’s secrets, but from the services I have configured so far, they all generate a config file with their secrets inside.

The diversity so far is the location of config files :

-r--r--r-- 1 root root 751 Jan  1  1970 /nix/store/iflsvvfjgvyc81wrs21vpv33v5yp3kdl-zigbee2mqtt.yaml
-rw------- 1 mpd mpd 561 Nov 24 14:05 /var/run/mpd/mpd.conf
lrwxrwxrwx 1 root root 45 Nov 25 15:10 /etc/home-assistant/configuration.yaml -> /etc/static/home-assistant/configuration.yaml

One can observe “guidelines”, are not necessarily followed by whoever wrote the nix packages (secrets in /nix/store, permissions on /etc/home-assistant/configuration.yaml), and I’m sure each service I’ll install will bring new “diversity”.

I’m not a purist, so my only concern is that a package “consumer” (or user) has a way to compensate for whatever he finds unacceptable.

For example, I’m fine with /nix/store/iflsvvfjgvyc81wrs21vpv33v5yp3kdl-zigbee2mqtt.yaml ending up in /nix/store, because it’s owned by root. Perhaps there are reasons why I should still care, I’d like to know !

For /etc/home-assistant/configuration.yaml → /etc/static/home-assistant/configuration.yaml, I suppose there is an easy way to do a chmod from the nix config.

I looked at nixops, and thought that their solution had some elegance: each secret gets a systemd service that is responsible for decrypting the secret. Unfortunately the project is dead, not sure if nixops4 follows this idea.

It seems that secrets management falls in one of the two approaches :

  1. secrets are decrypted at build time, stored in a config file (with hopefully correct permissions and location)

  2. secrets are decrypted at serice lauch time and never saved to disk unencrypted (but a “master” secret may be stored unencrypted, for the machine to be bootable without intervention)

Ideally a secret management tool, should be able to support both.

2

Aaaand you’ve fallen for one of the easiest footguns. Everything that goes into a .nix file ends up in the nix store (especially with flakes, but we’ll get to that in a moment). The nix store has world-readable permissions. Any process on your system can read those files (including many sandboxed ones, because the nix store cannot be selectively filtered).

So by putting your secrets in there, you’re potentially opening yourself up to privilege escalation exploits.

This is even worse if you’re doing remote builds, which you may or may not. Simply don’t put secrets in .nix files (and don’t builtins.readFile or import or even refer to them with path literals).

Note that I do mean secrets here - things like your email address are sensitive data, not secrets. Sensitive data can be treated quite differently, if all you care about is not publishing it on GitHub or whatever.

It doesn’t actually. This is only true for flakes. The reasons for it are arguably stupid; it’s only true because it’s a heuristic for nix to not copy everything in the directory into the store.

… which, by the way, if you put secrets in a non-git nix flake project, or forgot to .gitignore them for a bit, congratulations, you just PWNED yourself. Yeah, I don’t like flakes, we need a solution to this mess.

Anyway, if not being able to exclude specific files is a problem for you, you can simply not use flakes. This does not mean you should put secrets in .nix files.

This is fair, especially if you don’t need to share those secrets! Both sops-nix and agenix feature the ability to not place those secrets in source control, but to source them from files in a random place on the host.

Incidentally, you don’t actually need sops-nix or agenix. You can also just create a directory on your host by hand, and manually put unencrypted files in there, and then point the various secretFile options at the paths for those files.

That is a much lower-tech solution, and what I’d personally recommend to newbies.

This is probably partly that, yes, but also many services don’t follow “guidelines”. There are many projects with hard-coded config locations and other issues, and few upstreams like NixOS folks coming along and telling them they’re doing config locations wrong.

In general, though, a good bit of thought goes into why those files are where they are. You should not attempt to move them by hand, if you think you can improve the situation, contribute to the upstream module.

The generally accepted “best practice” in the nix world is to place configuration files in /nix/store, and to use either binary wrappers, environment variables or CLI args to set the location of the configuration file.

Sadly, application code doesn’t always make this possible. Software is messy, and developers often don’t care. Did you know it took mozilla 17 years to become compliant with the XDG dirs spec?

If you like this, you’ll love systemd creds. This allows doing exactly what I described above, but in a standardized fashion, and it additionally allows encryption with a master password (which can even be passed on through firmware) if you want encryption at rest. I’ve not looked much at nixops, but I can imagine they’re ultimately just using those.

I think systemd is the ultimate secrets management tool. Systemd creds are fully usable in NixOS - we just don’t currently have a culture of doing so by default in modules. I’m aware of a few modules that do, though.

Unfortunately service upstreams are also largely unaware of systemd creds, so very few services support them natively. Often you need to do some gnarly logic in your module to enable that, and I imagine that’s a lil’ too cumbersome for many module authors (if they are even aware of systemd creds in the first place).

Even if you do that logic, many of the benefits go away since services end up doing silly things with the secrets, like just holding them in memory indefinitely. Hell, very few services support storing credentials in separate files so you don’t need to mix config and secrets, and many still use CLI args or env variables, both of which are poor security practice. Our hands are a little tied here; the problem is less NixOS and more that a lot of software is kinda bad.

That said, I’d love to see a drive to start using systemd creds extensively in NixOS modules. It’d really help - we could simply tell newbies to put their secrets in files in /var/secret or whatever and everything would work, no sops or agenix required. But that kind of initiative would take a good bit of work; I should maybe try to start an RFC sometime.

Until then, NixOS makes modifying module services pretty easy. You can add your own systemd credential reading code downstream, and hey, maybe even contribute some of it upstream.

5 Likes
3

I fell into this footgun as well. I’ve looked at sops and agenix and realized both are not an ideal solution. Even encrypting secrets into the nix store still feels very bad. Though, a good way I’ve found for this to work is by using a Yubikey to auth into a HashiCorp Vault that is self hosted, that then propagates keys into a directory and should do just fine. There’s no secrets in the nix store and it requires the presence of the Yubikey to auth.

4

I said so in my really long post, but for the tl;dr: sops-nix and agenix do not require this.

A stringly-typed absolute path will work. It means nix won’t deploy your encrypted secrets file for you, but deploying a single file by other mechanisms (e.g. nixos-anywhere’s file copying feature) isn’t too difficult.

I still think that sops-nix and agenix are overkill for many use cases (as is self-hosting hashicorp tbf), and we should have some kind of PSA or guide to managing secrets in an unencrypted fashion, but they can still be used quite sensibly and do give you encryption at rest by default.

1 Like
5

To follow your advise (contributing to the upstream module), the first level upstream would be the nix package, There is quite a bit of Nix magic I need to understand first.

I have a hard time to decipher how this code :

along with my config (see below), manages to get this yaml file to appear in this directory :

/nix/store/iflsvvfjgvyc81wrs21vpv33v5yp3kdl-zigbee2mqtt.yaml

Note that the service itself loads the conf file in /var/lib/zigbee2mqtt/configuration.yaml

but the service has a “pre start” command :

ExecStartPre=/nix/store/w7q69dnrmhg6p5sdgvwsxa5waadd829g-unit-script-zigbee2mqtt-pre-start/bin/zigbee2mqtt-pre-start

that essentially does this :

cp --no-preserve=mode /nix/store/iflsvvfjgvyc81wrs21vpv33v5yp3kdl-zigbee2mqtt.yaml "/var/lib/zigbee2mqtt/configuration.yaml"

so the conf file is first generated in the store by some magic in nix, I suppose encoded in one of these lines :

 nativeBuildInputs = [
    nodejs
    npmHooks.npmInstallHook
    pnpm.configHook
  ];
  buildInputs = lib.optionals withSystemd [
    systemdMinimal
  ];
  buildPhase = ''
    runHook preBuild

    pnpm run build

    runHook postBuild
  '';
  passthru.updateScript = nix-update-script { };

If I manage to even find how to get the conf file not to end up in the store, or if I wanted to ask a question on the zigbee2mqtt nix package, where is the proper place to ask ? Is the nixpkgs the right place, given that it contains the whole universe of packages ?

Monorepos have some advantages, but posting a question in a repo with zillions of users and issues is not one of them. I wonder if a hybrid model could work: small projects (like a nix package) would have a dedicated repo, that would get mirrored in the monorepo…

    settings = {
      serial = {
         port =  "/dev/serial/by-id/usb-Silicon_Labs_Sonoff_Zigbee_3.0_USB_Dongle_Plus_0001-if00-port0";
         adapter = "zstack";
      };
      mqtt.server = "mqtt://127.0.0.1:1883";
      frontend = true; # Enables Z2M web app on port 8080

      advanced = {
          channel = 25;
          # journalctl  -u zigbee2mqtt
          # Configuration is not consistent with adapter...
          # encode python :
          # "["+ ' '.join([str(int(hex_string[i:i + 2], 16)) for i in range(0, len(hex_string), 2)]) + "]"
          #network_key = #"GENERATE";
          pan_id = 63459; #18569; #"GENERATE";
          ext_pan_id = [183 173 145 92 246 197 243 123]; # "GENERATE";
          last_seen = "ISO_8601_local";
          log_symlink_current = true;
      };
      availability = {
          enabled = true;
          active = {
             # Time after which an active device will be marked as offline in minutes (default: 10 minutes)
             timeout = 10;
             # Maximum jitter (in msec) allowed on timeout to avoid availability pings trying to trigger around the same time (default: 30000, min: 1000)
             max_jitter = 30000;
             # Enable timeout backoff on failed availability pings (default: true)
             # Pattern used: x1.5, x3, x6, x12... (with default timeout of 10min: 10, 15, 30, 60, 120...)
             backoff = true;
             # Pause availability pings when backoff reaches over this limit until a new Zigbee message is received from the device. (default: 0, min: 0)
             # A value of zero disables pausing, else see `backoff` pattern above.
             pause_on_backoff_gt = 0;
           };
          passive = {
             # Time after which a passive device will be marked as offline in minutes (default: 1500 minutes aka 25 hours)
             timeout = 1500;
          };
      };
    };
  };
6

The config file is created in this line: nixpkgs/nixos/modules/services/home-automation/zigbee2mqtt.nix at 1c8ba8d3f7634acac4a2094eef7c32ad9106532c · NixOS/nixpkgs · GitHub

And the copying is done in those lines: nixpkgs/nixos/modules/services/home-automation/zigbee2mqtt.nix at 1c8ba8d3f7634acac4a2094eef7c32ad9106532c · NixOS/nixpkgs · GitHub

As the pre start command is basically a bash script running before the service you could just add a line that inserts a secret when running the script.

7

I developed my own module for this. It is a simplistic approach that does not solve the full problem, but it works well for me

8

Well, as I was saying, the best practice is to read things from the store instead of /var/lib. Things nix deploys must always be in the store, so you can never get around having your config files in there - the best you can do is try to make sure the files aren’t also somewhere else, but that’s difficult because not all services support that.

Presumably that cp happens because zigbee doesn’t allow changing the config file location and is silly about symlinks. Or maybe it refuses to start if it cannot edit its configuration file at runtime.

This discourse is a pretty good place, as is evident from the replies you’re getting. That, or the matrix rooms if it’s something better handled by slightly more synchronous communication.

Maybe, every few months the talk of splitting nixpkgs apart comes up again, but a gaggle of mini-projects also has disadvantages. Honestly, better visibility of questions probably isn’t gonna be the result of that suggestion, you’d just have hundreds of thousands of repos with no maintainers so there’s zero chance you’d get an answer rather than the current state where at least a maintainer who’s never even tried running the software looks at you confused :smiley:

These are just the realities of contributing to large projects, but I promise people here are generally helpful, give it a shot and see how things go!

9

Interesting, now that you pointed me where the magic resides, it seems obvious, I wonder how I missed it !

I have two questions regarding this code snippet :

  cfg = config.services.zigbee2mqtt;

  format = pkgs.formats.yaml { };

  configFile = format.generate "zigbee2mqtt.yaml" cfg.settings;

  1. Where is the doc for the function “pkgs.formats.yaml {}”, more generaly, when I see z.y.z, is there an unamiguous way to turn something like "pkgs.formats.yaml (or x.y.z) into a url that will contain the doc, or some site with a search engine that will point me the doc of the code, or at least the source where the function is defined ?

  2. I’m guessing that a configuration.nix is simply an expression that evaluates to a full structure representing the entire config, that is then “realized”.

When I learn a language, I start with experimenting small expression, to test various hypothesis to understand how things work.

With Nix, even the smallest config ends up evaluating to something quite huge, It would be nice to have a tutorial with minimalist configs that evaluate to tiny structures, not real OS or machine config, just a structure for the sake of understanding the language.

By looking at my config, I can infer a few things, for example in the following snippet

  cfg = config.services.zigbee2mqtt;

  format = pkgs.formats.yaml { };

  configFile = format.generate "zigbee2mqtt.yaml" cfg.settings;

I’m guessing that cfg, is assigned a fully evaluated structure, that is constructed by “merging” my the default config (in zigbee2mqtt.nix), with my own config :

The zigbee2mqtt default config :

  config = lib.mkIf (cfg.enable) {
    # preset config values
     services.zigbee2mqtt.settings = {
        homeassistant = lib.mkDefault config.services.home-assistant.enable;
  ...
  ...

My own config

  services.zigbee2mqtt = {
    enable = true;

    settings = {
      serial = {
 ...
 ...

Some mysteries remain as how the code is evaluated, how does my own config get “merged” with the default, does configuration.nix get evaluated in a first pass, does it resolve to a function that is then given in parameter to another function, etc… I’d love to get some pointers to articles or blog posts that explains this.

10

Re docs and links, have a look at https://noogle.dev/ and NixOS Search. They cover most things; sometimes you need to grep though.

Not if you stick to nix, you’re talking about NixOS. See here for a more basic tutorial: Nix language basics — nix.dev documentation

It also has docs on the module system, which is the rest you’re asking about: Module system — nix.dev documentation

The NixOS manual is also helpful as far as understanding modules goes: NixOS Manual

11

Thanks @TLATER that’s exactly the links I was looking for.

One mystery persists however, I was not able to find any doc on this function :

 format = pkgs.formats.yaml { };

From what I understand this code snippet instantiates a function and assigns it to the variable “format”.

Search engines are kind of useless because each of the 3 words (pkgs, formats, yaml ), even AIs were not that useful in pointing me to the code, let alone documentation.

I think I’ve found the code of the function : nixpkgs/pkgs/pkgs-lib/formats.nix at bc0da6f230efdc7772bac7f1e6f925da75145728 · NixOS/nixpkgs · GitHub

But I’m not 100% sure. The translation from the function instantiating code, to the doc, of the source code, seems far from straight forward.

The fact that search engines like https://noogle.dev/ and myNixos are not able to find the definition tells me that it could be a language weakness. I’m hoping it’s not. In theory, if the nix interpreter can instantiate the function, then a doc generation tool should be capable, but in practice a lot of things can make this hard, or even impossible.

I was interested to investigate this particular function, because it generates a file in the store, and there is no indication of this from looking at the code :

configFile = format.generate "zigbee2mqtt.yaml" cfg.settings;

It seems that some kind of implicit parameter (or global variable) determins the path of the file generated (/nix/store/iflsvvfjgvyc81wrs21vpv33v5yp3kdl-zigbee2mqtt.yaml) looking at the code below, it looks like they could even be env variables : $valuePath $out

  yaml_1_1 =
    { }:
    {
      generate =
        name: value:
        pkgs.callPackage (
          { runCommand, remarshal_0_17 }:
          runCommand name
            {
              nativeBuildInputs = [ remarshal_0_17 ];
              value = builtins.toJSON value;
              passAsFile = [ "value" ];
              preferLocalBuild = true;
            }
            ''
              json2yaml "$valuePath" "$out"
            ''
        ) { };

      type = mkStructuredType { typeName = "YAML 1.1"; };
    };

I have a positive bias towards Nix, I’ve been wanting to use if for years, I’m convinced that “package and configuration management” can absolutely benefit from functional programming principles. All these years my “inner pragmatist” told me to wait, and in the mean time the distro I’ve been using got worst, so I finally tried it.

So far, my impressions are mostly positive, but I think that the main weakness is the “principe of least surprise” (POLS) criteria.

POLS is a huge factor in overall user experience, and it not easy to get right, it requires not only dog fooding (developers using their own stuff), but it also requires that the developers doing the dog fooding make an effort at “forgetting” what they know about what they wrote, while attempting to put themselves in the mind of an outsider.

12

That is indeed the correct function, but you’ve clearly not read or understood the tutorials; you’ll want to learn about derivations before this will make sense to you. Work through the basic nix tutorials, build some packages, and maybe have a look at the nix pills. Your understanding of nix is still way too limited for you to be able to reason about what’s happening here. I’d try to explain, but I’d just end up rewriting the nix pills.

Folks here are aware of the principle of least surprise, but nix comes from a different tradition than the ones you are likely familiar with, and you’re looking at it with bias from prior experiences that clearly pull your understanding in a certain direction. It’s much less confusing to some groups than others, building intuition about new ecosystems takes some time.

Anyway, I think you’re approaching the ecosystem from the wrong angle jumping straight away into the deep end (especially since you don’t seem to have much experience with functional programming). NixOS is an incredibly complex use case of nix, since it’s not just nix, but a whole framework and almost-CSL written in nix; starting here is hard, you’re actually getting surprisingly far.

I really recommend trying the tutorials first, reading the nix pills, learning about at least partial application, and maybe some of the other functional primitives nix projects use (e.g. lib.fix), and then trying to grok these library functions again.

tl;dr, though: You cannot implement a version of pkgs.format.yaml that creates a path outside /nix/store. The idea of nix’ derivations is fundamentally incompatible with paths outside the nix store. It’s how nix solves the diamond dependency problem, and in fact its entire raison d’etre; without it, nix cannot give you any of the guarantees it gives you.

You could build a distro that uses nix-built binaries, and uses a different deployment system (ansible?) for configuration files, I suppose, but that would not be NixOS.

13

Anyway, I think you’re approaching the ecosystem from the wrong angle jumping straight away into the deep end (especially since you don’t seem to have much experience with functional programming).

I’m actually quite familiar with functional programming, I’ve spent a few years in Clojure and Scala. FP is the main reason I’ve been wanting to adopt Nix, I was instantly sold on the concept of FP applied to package and configuration management.

I considered trying Guix, for the Scheme language, my intuition is that scheme is a safer bet than inventing a language for a package manager. In fact inventing a language for a project seems like a liability more than anything, but I want to keep an open mind.

The reason I tried Nix before Guix is the community size and “livelyness”.

You’re right that I need to familiarize with the nix language a bit for things to make sense. I hope my critiques did not sound like arrogance, I was mainly thinking out loud of what went through my mind while trying to understand how things worked.

I’ll looke at the nix pills, it will surely help. Things are already making more sense thanks to your links and answers !

14

Hm, fair, I might have misunderstood your confusion, too, just seems like you’re struggling to follow the code itself, but I’m probably reading it wrong.

Good luck, I swear the ecosystem makes quite a bit of sense if you tough out the first week or two and are ready to throw everything you know about Linux systems out of the window :wink: