Secrets in Nix Suck (and how to fix them)

Development Nix
1

Premise

Currently it’s really really easy to accidentally put your secret keys in to the nix-store, there’s plenty of bad advise around, especially for resource acquisition, nix is great until you can’t use a fetcher, and then there’s a billion ways to mess up manually acquiring resources, It’s also just a pain in the ass (coughheavilymoddedfactorioserver), nix can fetch resources, and it has access to every application that one might use to download, the issue is that you cannot cleanly, nicely use them inside the sandbox.

If you want to see a package that does everything wrong, see the Factorio Downloader that writes secrets in to the nix-store while trying avoid transient issues.

The entire point of nix to me is that I can type nix build and it builds an entire workable package, or full system, I should not need to wrangle all sorts of weird things, that otherwise could be just a prompt.

A solution should not leak secrets to arbitrary users via nix-build, it should not need raised privileged, and it should not need me to do non-nix things, or manage a reverse-proxy, or accidentally write your secrets to bash history.

Current solutions include:

  • NIX_PATH

    • probably leaks to nix-store

  • netrc

    • only works with basic http authentication. (RIP circa 2004)
    • probably not safe on a multi-user system due to it being available to all builders.

  • impureEvars

    • only supported on single user installs, doesn’t actually work on NixOS!

  • ad-hoc scripting to load the files in to store manually

    • Not compatible with nixpkgs.
    • undermines the point of using nix.

  • Setup a http proxy injecting headers or full custom backend

    • requires a running, error-prone, system already setup (not newbie friendly)
    • only works with http style payloads (steamcmd no workie)

  • providing secrets over http/udp/tcp

    • technically works, but http/udp/tcp is not secure

  • extra-sandbox-paths

Actual solution

Nix really needs a way so that with minimal setup nix build can work, providing a prompt to the user if possible, and then passing these secrets from the front end, to the builders safely and securely.

It seems like a MVP could be constructed using pre-build-hook and post-build-hook, and some per-derivation sandbox paths, Still not exactly sure how to authenticate which derivation gets which secrets. but it seems trivial to implement.

But this would need to be added to nix itself, so that nixpkgs could start accepting fetchers of this sort, I assume any packages that use these fetchers would be no different to the requireFile, except you could wrap your package in a “secret provider” override for nixos-config, it could be easily documented in nixpkgs, the fetchers that use this functionality can be audited by nixpkg maintainers in one location instead of the many different ad-hoc solutions inside nixpkgs or outside it.

11 Likes
2

There is no solution. The entire premise of binary caching is that all aspects of a build are public knowledge. There’s no way around this. Secrets is a stateful problem, fundamentally unsuitable for Nix to solve.

6 Likes
3

This. If your secrets don’t need to be read at build time, put them where you put the rest of your application state (/var), not where you put your immutable, world-readable software. If your software really expects those secrets to be in a particular store location, symlink the /var location into the store. If your software requires the secrets to live in a configuration file along with a bunch of other things that aren’t secret, petition your maintainer to fix that or patch it yourself, because that’s just not great design in general.

The catch is if you’re using secrets to sign software; that’s when extra-sandbox-paths or secret servers can be appropriate.

5 Likes
4

As the author of the mentioned bad advise, I primarily meant setting root’s and user’s passwords, which finally end up in Nix Store in hashed form and symlinked from /etc – the existing NixOS design.

builtins.readFile let to put the secrets separately from the *.nix codebase, into git-crypt’ed directories, etc; so, it is about how to manage the server’s secrets on the workstation we develop server’s software on, not about how to deploy the secrets onto the server and not about how to avoid having plain-text secrets in world-readable .drv files: there are at least 3 separate issues with the secrets.

5

Related RFC:

And implementation PR:

8 Likes
6

Just to mention another thing to take into consideration: with most of the options and the solution, the secret is available inside a build environment, which can thus leak to the build output on accident.

That isn’t the case for netrc with builtins.fetchurl. The secret never touches the build environment. Nixpkgs fetchers are based on being a derivation, which does depend on having the secret in a build environment.

That said, I do like your suggestion. It would indeed be really nice if the Nix frontend would be able to pass through secrets. Better multi-user experience and isolation. It helps the UX quite a bit as well.

1 Like
7

Security is more important than 100% hermetic build systems, and none of this matters for FODs.

And you’re missing the forrest for the trees, nix is meant to be public yes, so we should embbed as much public facts about our build process as possible, requireFile is a UX nightmare because it doesn’t document enough about the build process, it explicitly requires extra knowledge to be passed via a side channel. This is stupid.

Factorio mod downloads needs secret passed in to a http custom header, nix should describe this fact and handle it, it should just work with tools like nixos-rebuild build --target-host, and nixops and other “nix deployment tools”.

An impure side channel for secrets, on the file system, is just as bad for reproducibility as http-proxy injection, and netrc for inputs that require secrets, that are currently allowed in FOD’s, the issue is that they’re inflexible and end up with hacky broken solutions that put secrets in to the build system by accident (Which also breaks reproducibility).

1 Like
8

This does sound good, especially for stuff like nixos-config files (like icecast password inside xml), I’m very much in favour of anything that is transient remaining as transient as possible, ideally we should avoid ever storing passwords / secrets on disk in the nix-store.

9

readFile is read in to nix var, and all nix vars are unsafe, and end up being written to your nix-store. And NixOS does support directly specifying hash/KDF’s, which should be better here.

I usually don’t consider low security level users as “trusted” these days.

10

My entire OP is about secrets in builders, especially fetchers…if we throw our hands up in the air and say “can’t be improved” then people will not use nix/NixOS because if you’re build process requires nix and non-nix tooling, then you have the worst of both worlds, the answer will be “don’t use nix, use something that is easy to reason about from a security perspective”.

11

(I agree with you rhendric on that.)
I would say, there’s only one exception to all of this.

Take for example: https://packages.debian.org/sid/grub-efi-amd64-signed – this is signed in the build environment of Debian, a random user cannot “reproduce” this package.

We don’t have similar packages yet in nixpkgs, but in the future, it’s possible to envision that hydra.nixos.org or similar can have extra-sandbox-paths to some control socket of some HSM or signing server to sign artifacts that are de-facto impossible to reproduce.

But @mschwaig provided some interestings leads regarding this problem: https://www.youtube.com/watch?v=-CUa3yVTK5U&list=PLgknCdxP89ReD6gxl755B6G_CI65z4J2e&index=20.

FYI, nixpkgs has strict compatibility constraints w.r.t. Nix. Even if it was added to Nix, it would take a long time to use it.

12

FWIW you can use builtins.fetchGit to fetch your private code using your local Git auths, such as SSH keys.

Of course, the private code itself lands in the nix store. I don’t know what’s worse. But what else could it do? :man_shrugging:

1 Like
13

On servers sops-nix does the job for me.

On desktops, I found a trick by having only secrets encrypted with gpg in the nix-store, and using some shell wrappers to decrypt them with a yubikey. It doesn’t fit for all usages, but it works well for me for wireguard and other stuff, the secrets are decrypted/read on demand and only used in memory. That’s the most secure solution I found, but it’s pretty hacky :exploding_head:

1 Like
14

builtin fetchers block the nix evaluator, which is very bad for eval performance (and would mean that you can not evaluate nix expressions without being able to authenticate to your build resources).

1 Like
15

I think there is a narrower case here which would still prove extremely useful without as many pitfalls: allow secret passing only in fixed-output-derivations.

FoDs already suffer from non-reproducibility. There is no guarantee that the net will give back the same bytes. We accept this because the fixed-output guarantees that we will detect when we fail to produce the expected output.

So allowing secret passing to a FoD wouldn’t make it any less pure than it already is(n’t). It would provide tremendous utility without significantly devaluing Nix in use cases that don’t require secrets (i.e. it shouldn’t cause harm to nixpkgs).

1 Like
16

We have another use case for secrets in the sandbox: Custom NixOS VM tests.

Our software relies heavily on third party APIs, and we’d like to run full end-to-end tests, from the OS up. Internet access can be enabled in the sandbox fairly easily, but there’s no straightforward way to use API keys.

Output “reproducability” is not really a concern here, because no-one’s using the output of the test derivations (aside to check the occasional screenshot). All we care about is whether the test passes (so the derivation builds successfully) or fails. Like a FOD, we don’t really care how we got there (what secrets we used) if it passes.

17

To be fair, that’d be a very flaky set of tests. Usually the solution to this kind of thing is mocking.

18

Understood, and we have many tests with mocks as well. But at some point we do need to test the system as a whole before release.

19

That doesn’t have to be part of the build process though, does it? I would make that a separate derivation that produces a test executable or script that is run entirely outside of the sandbox.

20

We could, but that becomes harder to integrate with our existing CI systems (Hydra + GitLab integration) that are built on derivations.

It’d also make it harder to take advantage of the existing NixOS VM testing infrastructure. We are building an embedded device, so testing the whole OS is appropriate.