QMENU, Secrets, SOPS and NixOS

Help
1

A side project of mine is deployed with NixOS and it uses nix-sops for handling secrets. The app running inside of NixOS uses a combination of LoadCredential and Environment, two systemd features, to make SOPS secrets available to the application.

In production all of this works, but I’d like to run as much of my NixOS config as possible locally in a VM. You can imagine it as a sort of staging environment.

But I don’t know how to get my secrets into the VM. I could share my host PGP key so that I can use the same SOPS solution in the VM as in production. I don’t know how I would go about sharing the key though, without it ending up in the Nix store.

I could also pass environment variables to QEMU. I use an .envrc file anyway so secrets are already part of my local developer environment. But I don’t know what I’d do with environment variables inside a VM running NixOS.

Or maybe there are other solutions that I’m not thinking of right now.

Also I’m using Flakes for all of this so I can’t use --arg or --argstr :frowning: for nix build .#vm and getEnv also doesn’t work since it just returns an empty string.

So far the only thing that works is building the QEMU VM with --impure

2

I’m using a NixOS VM and pass-through some content from my host filesystem to the guest VM via the 9p filesystem. Maybe something similar would work for you to get some secrets or decryption keys into your VM.

...
  virtualisation.qemu = {
    # mapping directories to same path in vm
    options = [
      "-virtfs local,path=${builtins.getEnv "HOME"},security_model=none,mount_tag=home"
    ];
  };
...
  # overide default vm filesystems behavior
  fileSystems = pkgs.lib.mkOverride 0 ({
    "/".device = vmConfig.bootDevice;
    ${if vmConfig.writableStore then "/nix/.ro-store" else "/nix/store"} = {
      device = "store";
      fsType = "9p";
      options = [ "trans=virtio" "version=9p2000.L" "cache=loose" ];
      neededForBoot = true;
    };
    "/mnt/home" = {
      device = "home";
      fsType = "9p";
      options = [ "trans=virtio" "version=9p2000.L" ];
    };
  });
...
3

Thanks for the elaborate reply. I was hoping for something less low level though :slight_smile: I’ll have to check if this is something I’m comfortable doing.

4

You could use another layer of nix-sops to deploy your PGP key to your staging VM.

5

That sounds like a recursive problem now because how would my VM be able to unlock THAT pgp_sops_layer.yaml secrets file? The source is here https://github.com/cideM/lions-backend/blob/31dda1963477dcdf0603c8db2b741e5ed6044082/flake.nix by the way, it’s now in a shape where it can be shared

6

It sounds like you’re sharing your secret set between test/prod, otherwise you could just setup a new set of test credentials, a throwaway gpg/ssh-host-key key and could just use that for testing, no?

You could still do something like that, just adding another key to your sops list of keys, re-encrypting, and sharing that between test/prod, but now you have a much weaker link protecting [every secret in your sops payload].

I see AWS stuff in there, are you using the Sops Amazon KMS integration? If you were, you could maybe run a local-kms mocking the AWS server and then get even closer to testing prod.

1 Like
7

No it’s just some AWS access keys to be able to access SES from my server.

could just setup a new set of test credentials, a throwaway gpg/ssh-host-key key and could just use that for testing, no

But that still leaves the question how my VM gets those credentials so it can use SOPS to decrypt the secrets file. If it were Docker I’d just mount my host keys or share them via an environment variable at runtime or something.

8

You can create a separate “staging” nixosConfiguration that imports the configuration from the normal host, and uses a different set of secrets supplied directly with services.openssh.hostKeys on top of the normal host config. Then build it with nixos-rebuild build-vm. #staging.

Assuming the secret isn’t actually secret in staging, of course, otherwise you’ll end up with a plaintext secret in git.

Alternatively you can share files with the VM as you would in dockerland (and possibly use the above options to set some key imported from the host to the host key). The VM by default includes files from $TMPDIR/xchg.

9

That last part is interesting thanks I wasn’t aware of this. I’ll report back!

10

Reporting back, better late than never.

I’m currently working on using SOPS in the VM and I’m fairly optimistic that my current approach will work. I am not planning on mounting any files from host to VM, since the “virtfs” feature seems to be pretty wonky on MacOS.

Instead, I generated a GPG key, which will be added to the repository, and used to encrypt a secrets file which contains only fake secrets. The files are added to the VM through Nix, so environment.etc."pgpstuff/whatever".source = ./local/path.

That way I should be able to reuse my SOPS setup, with only the defaultSopsFile and sops.gnupgHome needing to be adjusted. My server will then read the normal secrets, but also include a switch so it doesn’t try to send actual emails in a test environment. That last part can be controlled through an environment variable.

Hopefully this will minimize the configuration that is specific to local vs. production. Ideally, it will let me include SOPS secrets into an end-to-end. Of course I can still mess up the PGP keys on the actual server or make configuration errors, but that’s always the case.

I’ll report back, and link to the final solution, once it’s up and running.

1 Like
11

It’s now working and you can see the code here: https://github.com/cideM/lions-backend/blob/37a6bdd3efb59349cbbf7096be2583c1b499318a/nix/vm.nix

The gist is that I created a separate secrets file which is encrypted with a key that’s part of the repository as well. That key, plus some additional files, is copied into the VM so that SOPS can function normally inside the VM. The key was generated with sops-init-gpg-key from GitHub - Mic92/sops-nix: Atomic secret provisioning for NixOS based on sops

That way my VM can exercise the whole SOPS infrastructure which should hopefully alert me, during tests, of configuration issues.

1 Like