Avoiding information leakage in multi-host deployments

Help
1

Hi, I’m playing around with a flake-based multi-host repository (a repository containing multiple NixOS configurations for different machines) as I’ve seen many people do. I think that this approach has some benefits because it allows one to share common modules, packages, and other code.

Though I’m wondering about the security implications of such an approach. I noticed that the whole source of the repository is copied into the nix store of a deployed machine, even the parts which shouldn’t be evaluated for the deployed NixOS configuration (I was surprised because I thought lazy evaluation takes care of that). I think this could cause some (severe) information leakage, e.g., system/network configs, user names, etc. I know that managing secrets like passwords, ssh keys is an issue in NixOS and should be done with special consideration. But even if secrets are encrypted the leakage of their mere existence could be problematic (Imagine that you are an IT service provider and manage multiple, independent clients and you leak one client’s information to another).

I would be curious if anyone else has had thoughts about how to approach managing mutli-host setups from a security & maintainability perspective (with standard NixOS setups, without relying on too much on 3rd party tools as those might introduce additional problems).

1 Like
2

Hm… a possible solution would be to limit visibility of files during evaluation, e.g. by creating a temporary repo where only the relevant files are copied. Tools like Haumea might then help to avoid evaluation errors due to missing files.

1 Like
3

Yeah this is a bit of a problem, depending on how you deploy.

If you have the repo on your machine and use nixos-rebuild switch --target-host=$target, then you will only copy the final store paths, not the sources as far as I know, so that would somewhat circumvent the issue you’re describing. Using terranix might make this a bit easier, depending on how big your infrastructure is.

Lazy evaluation is part of the evaluation semantics of the Nix language. This means that it can prevent files from being included only when the files are imported during evaluation time. However, with flakes, this is not the case right now. As you discovered, the whole flake is copied to the store for it to be evaluated. It doesn’t necessarily have to work that way. A flake could be evaluated without copying, only the inputs must be in the store, and there is an open draft PR to implement this, but it still has issues.

There’s also an RFC for adding ACLs to the nix store, which could somewhat solve this problem.

As for how to actually solve this issue right now: you could store the client-specific configurations in their own flakes as default and put those in your repo separately. For the common configuration values, you can add another “common” flake that the other ones just use as an input. This is a little annoying because you have to always run nix flake update for every client when the “common” flake is changed, but you can script your way out of that.

For the secrets, a simple solution is to have a well-known location outside of the Nix Store where you store a script that the environment setup from your config sources on startup. This could be populated from a key-value secret store, or you store these files with git-crypt in a separate directory in your repo that is not visible to the client configuration flakes.

Thinking about it, you could potentially store an encrypted file containing all secrets for every client in your flake for every single client, and only deploy the decryption key separately. The only “information leak” in that case would be that there are secrets, and the file size can indicate how many, but that’s it.

1 Like
4

I think if you can verify this is actually true it’s probably the best option. I believe it is as well, but I wouldn’t be too surprised if some information could still leak.

If you’re doing this with a large-ish commercial deployment you’ll likely want to do something better than manually running nixos-rebuild on each target anyway, there’s a whole ecosystem of nix deploy tools that all roughly do the same thing to consider for this.

Note that this necessitates a push-based approach, unless there’s some pull-based approach to get activation scripts without evaluating nix files. This trades off scaleability for secrecy.

That said, it’s worth considering whether you want everything in one nix project if the systems are dissimilar enough that configuration details can’t simply be guessed.

For the secrets, a simple solution is to have a well-known location outside of the Nix Store where you store a script that the environment setup from your config sources on startup. This could be populated from a key-value secret store, or you store these files with git-crypt in a separate directory in your repo that is not visible to the client configuration flakes.

Git-crypt is a bad solution to this problem, since the secrets end up world-readable in the nix store. Especially when caches get involved this can be bad news.

Don’t use git-crypt for secrets, @iFreilicht

This is an ok solution to keeping part of a public repository invisible, but at evaluation time the full sources will need to be visible anyway, so it doesn’t help with this configuration secrecy problem.

This is roughly what sops does, and there’s sops-nix for NixOS integration.

Personally I think sops-nix, agenix and the nixops features are the only viable secret deployment strategies for NixOS currently (sadly systemd credentials are a bit fiddly).

Alas, they only help with secret deployment. Full-configuration secrecy is much more tricky.

2 Likes
5

I’m not sure if it’s problematic in the solution I was proposing. My idea was a directory structure like this:

.
├── common
│   └── flake.nix
├── kiosk
│   └── flake.nix
├── laptop
│   └── flake.nix
├── secrets
│   ├── kiosk
│   ├── laptop
│   ├── server1
│   └── server2
├── server1
│   └── flake.nix
└── server2
    └── flake.nix

Every file in secrets would be an encrypted shell script that is decrypted on the admin’s machine and copied with scp or similar to a well-known location like /etc/deploy/secrets with mode 0640. It’s not in the nix-store, so not world-readable, and is only present on the admin and client machine.

1 Like
6

Thanks for all the input.

Ok, makes sense. I assumed that the flake is evaluated on the host invoking nixos-rebuild and the build instructions are sent to the build-host. Of course if this is not the case then the flake must be copied at some point to the build-host. I haven’t tried yet to just use a target-host and to build locally (that wasn’t my intention) so I can’t confirm whether this avoids sending the flake’s whole source to the target-host. I that case it would be manageable though.

Oh having an implementation of this would be really great. It would not help with the information leakage which I described as root could still access information belonging to unrelated systems, but for multi-user environments I’d say it’s a must have. I’m still surprised about the many drawbacks of nix’s world-readable store model in multi-user setups. By now I’ve given up on trying to finagle some isolation between users on the same system—there will always be some leakage which makes defense-in-depth of NixOS systems almost an impossibility I think.

Were multi-user setups not part of the original use cases & requirements for nix? If the original intent were single service machines/containers only, one could live w/o isolation of the store—probably. Though a successful attacker who’s just able to exfiltrate information with the permissions of the compromised (unpriviledged) service could still use the store to inspect system configs (which could otherwise be protected on a typical *nix system by permissions, ACLs, etc.) and find additional attack vectors to gain full access to the system/network.

I agree with @TLATER on this point. Encryption won’t cut it if the data (not secrets per se, here an out-of-bands methods should be used anyway) needs to be readable by the whole system later on. I’d even go so far to say that true, non-rotatable secrets (e.g. personal data) shouldn’t ever be pushed to a public git repository—even in encrypted form. The encryption might protect you now, but what about 10y down the line? Will the algorithms (& implementations!) still be unbroken? (esp. asymmetric encryption schemes—which probably won’t be used to protect data at rest) Depending on what you are trying to protect and for what time-period, public access to encrypted information can still become an issue.

7

Well they kind-of were, the security aspects were even mentioned in the original thesis from 2006, though the initial focus was on reproducibility, and this is only possible with a public, world-readable store. I believe containers weren’t much of a thing at that point yet, so the scenarios for service deployments were actual servers that you as an admin had control over, and in the initial development stage of linux, multi-user setups didn’t exist at all.

The current multi-user setup with the Nix Daemon and build users was already planned, but took a while to materialize, I think it only became the recommended installation method a few years ago.

Right now the focus (AFAICT) is on implementing content addressed store paths to at least allow for safe sharing of build artifacts between users.

That’s a good point.

Ideally, you’d only deploy a single access key (separately from Nix, of course) and manage the secrets in a separate service like Hashicorp Vault or Bitwarden Secrets Manager.