Sops-nix trying to set the user password

kagehisa · January 30, 2024, 1:30pm

Hi I’m having some trouble setting up sops-nix to setup the user password for my user.
I’m using pgp to encrypt my secrets and as far as I understood sops-nix will automatically generate a private pgp key based on the target hosts ssh key. I followed the documentation as close as possible.
But whenever I try to activate my configuration via:
sudo nixos-rebuild -v test --show-trace --flake /etc/nixos#nixvm

I get this:

activating the configuration...
sops-install-secrets: Imported /etc/ssh/ssh_host_rsa_key as GPG key with fingerprint fc5624a466cdf7729a4be7e41d407079412e01bc
sops-install-secrets: Imported /etc/ssh/ssh_host_ed25519_key as age key with fingerprint age1mhlx8c8xwkg4ae4awq6tmhhae4evwqwq5j7hwt4cqfp483cyz5hser9ef6
/nix/store/6kpwdjb1wa46cj9pc0ipvg7xd82vpy3j-sops-install-secrets-0.0.1/bin/sops-install-secrets: Failed to decrypt '/nix/store/fk9imaal42irdmbslvan3ddv2m3qzhx6-default.yaml': Error getting data key: 0 successful groups required, got 0
Activation script snippet 'setupSecretsForUsers' failed (1)
warning: password file ‘/run/secrets-for-users/password_manji’ does not exist
setting up /etc...
reloading user units for manji...
setting up tmpfiles
warning: error(s) occurred while switching to the new configuration

My .sops.yaml looks like this:

keys:
  - &admin_manji 5C3960F507DD4ACC468EE93FE6C10F242BF4141B
  - &nixvm fc5624a466cdf7729a4be7e41d407079412e01bc
creation_rules:
  - path_regex: secrets/[^/]+\.(yaml|json|env|ini)$
    key_groups:
    - pgp:
      - *admin_manji
      - *nixvm

My Git layout is like this:

.
├── flake.lock
├── flake.nix
├── home_pw
├── programs
│   └── shell.nix
├── README.md
├── secrets
│   ├── default.yaml
│   └── keys
│       ├── systems
│       │   └── nixvm.asc
│       └── user
│           └── manji.asc
├── shell.nix
├── ssh
│   └── authorized_keys
├── systems
│   ├── common.nix
│   ├── nixbook
│   │   ├── configuration.nix
│   │   └── disko-config.nix
│   └── nixvm
│       ├── configuration.nix
│       ├── disko-config.nix
│       └── hardware-configuration.nix
└── users
    └── manji
        ├── default.nix
        ├── home
        │   └── default.nix

I incuded the sops-nix inputs to my flake and the module import to the modules section for each system configuration.

Inside my configuration the secret is referenced like this:
In systems/common.nix

  #sops integration
  sops.defaultSopsFile = ../secrets/default.yaml;

In users/manji/default.nix

let
  inherit (inputs) ssh-keys;
in
{
  sops.secrets.password_manji.neededForUsers = true;

  # Define a user account. Don't forget to set a password with ‘passwd’.
  users.users.manji = {
    isNormalUser = true;
    description = "manji";
    hashedPasswordFile = config.sops.secrets.password_manji.path;
    openssh.authorizedKeys.keyFiles = [ ssh-keys.outPath ];
    extraGroups = [ "networkmanager" "wheel" ];
    packages = with pkgs; [];
  };

}

The secret in unencrypted but censored form (secrets/default.yaml):

password_manji: <somehashvalue>

So if I use the provided shell.nix from the documentation and manually generate the pgp private-key from the host key and import it manually with gpg --import I am able to decrypt the secret. But as far as I can see in the nixos-switch output, sops-nix did create and import the pgp key but the usage for decryption seems to fail.

Is there a way to debug this further? Or has someone an idea where I messed up, or as to why the sops generated and imported pgp key is not available for decryption?

I’m completely lost atm.

TLATER · January 30, 2024, 2:03pm

Not quite, you need to do this manually. Did you see step 3 in the usage example?

kagehisa · January 30, 2024, 2:41pm

Yes I did this. I did not name ist server01.asc but nixvm.asc (located in secrets/keys/systems). But this just generates the public key for the system and gives you the fingerprint (which I included in my .sops.yaml).

If I understand the code correctly both keys private and public are generated based on the host ssh key.
This go funtion should to exactly that. The message during ‘nixos-rebuild’ tells me that this step was successfull.

func SSHPrivateKeyToPGP(sshPrivateKey []byte) (*openpgp.Entity, error) {
	key, err := parsePrivateKey(sshPrivateKey)
	if err != nil {
		return nil, fmt.Errorf("failed to parse private ssh key: %w", err)
	}

	// Let's make keys reproducible
	timeNull := time.Unix(0, 0)

	gpgKey := &openpgp.Entity{
		PrimaryKey: packet.NewRSAPublicKey(timeNull, &key.PublicKey),
		PrivateKey: packet.NewRSAPrivateKey(timeNull, key),
		Identities: make(map[string]*openpgp.Identity),
	}
	uid := packet.NewUserId("root", "Imported from SSH", "root@localhost")
	isPrimaryID := true
	gpgKey.Identities[uid.Id] = &openpgp.Identity{
		Name:   uid.Id,
		UserId: uid,
		SelfSignature: &packet.Signature{
			CreationTime:              timeNull,
			SigType:                   packet.SigTypePositiveCert,
			PubKeyAlgo:                packet.PubKeyAlgoRSA,
			Hash:                      crypto.SHA256,
			IsPrimaryId:               &isPrimaryID,
			FlagsValid:                true,
			FlagSign:                  true,
			FlagCertify:               true,
			FlagEncryptStorage:        true,
			FlagEncryptCommunications: true,
			IssuerKeyId:               &gpgKey.PrimaryKey.KeyId,
		},
	}
	err = gpgKey.Identities[uid.Id].SelfSignature.SignUserId(uid.Id, gpgKey.PrimaryKey, gpgKey.PrivateKey, nil)
	if err != nil {
		return nil, err
	}

	return gpgKey, nil
}

Then what is sops-nix doing with the above function? If I read the output I have at hand correctly it seems that sops-nix creates said pgp secret-key and identity, imports it but then it fails to find it… Or am I interpreting this all wrong? Where does the target system gets the secret key to decode the secret from, if not from sops-nix?

olmokramer · January 30, 2024, 5:08pm

Sops-nix does not generate a key. Sops can only use keys in the age and pgp formats, so to be able to use the host’s ssh key sops-nix needs to be able to convert the key from the ssh represenatation to the pgp representation (or to the age representation if it is not an RSA key). But it is still the same key.

This line in the output of nixos-rebuild

/nix/store/6kpwdjb1wa46cj9pc0ipvg7xd82vpy3j-sops-install-secrets-0.0.1/bin/sops-install-secrets: Failed to decrypt '/nix/store/fk9imaal42irdmbslvan3ddv2m3qzhx6-default.yaml': Error getting data key: 0 successful groups required, got 0

indicates that the file could not be decrypted. Specifically that 0 groups are required to decrypt the file which seems odd. If you inspect secrets/default.yaml (the encrypted file), does it list any recipients under the pgp key?

kagehisa · January 30, 2024, 5:24pm

Ok, generating a new key was wrong terminology on my side. But it converts the systems ssh key automatically to a private/public pgp and age key. So somehow importing the pgp secret key manually should be not needed. Except one uses pgp keys not based on the systems ssh host key.

Yes I get two recipients for my two keys. The fingerprints match each other.

password_manji: ENC[AES256_GCM,data:LLwLGlfC0XmRS+XV6rvRZdoc4IJWSQrYgP1HuU97MnfBarDOuJMfM7JKvmldE2vt3Qfmupq5FM99FZjF0q3ZB03+EfoOwMoQTw==,iv:vuPgqJgnAXLq4a0yxsfTjz2shdJJX6ZlHTVV8jmMhdw=,tag:z6vp3DQdb2eFgKqSiCRABQ==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2024-01-28T19:24:57Z"
    mac: ENC[AES256_GCM,data:s+mp3xXYuYGjvbD/rOhEWjY2gcxZrihGKf5AvEr8YiHmvllvGs8eDbB2B9ookgoX9wOumCBDwPieaqynVSXm5kc5Qk9+XyzuZd74ZqXU/iNl927Hv93il10iTgello/LnlzMYWOMs0bzHpHx906BqN5NHjhKbPskA9CWL2jWf9s=,iv:PzDApVNC7MIvNq4Wi1fbj9DYVyRNM4Oml189/omMxxY=,tag:lIvveM1sTpz8GteMQjCW5g==,type:str]
    pgp:
        - created_at: "2024-01-28T14:56:38Z"
          enc: |-
            -----BEGIN PGP MESSAGE-----

            hQIMAwAAAAAAAAAAAQ//d1IMjv/9sWflMDGm3bwjUKFRSy74vv/sb/RI82AOr3/9
            WQ7m6ReZ9j8H8GmMKdkugeksondUSSmm91E/JYV6ok1HNwjKC8OkmgnXwHV+G4T1
            hkd7DiMP9isfjH1xF0fM+9yFOAmvy9p6QDEa+EBV3yg2WxI7IPCfWHWIEVT8Mawt
            b6RxElSeiVlb1Sq27SWcQxZE+Ttw7ZJZu0TzJnTvsnUEY0lQxnWypE+1xtvfqsjC
            fmMJ1nw+Iy0CCym+JbjeJeGgoIaPyrb7/GzB3rgQ5IUGBP4VM1FHSHqqtR/phu3I
            RgsauzpOav+4QJTqzSPv/wT5xU4RjYm5+HL1vsVPDLSNWErshEic8QCe9XgCrnq3
            E5i9LkpzTncmveKwhdlBxOdFZREWy/YlFRAIOKHs2Zy8nk0c+QXA87CmqON8PirQ
            8byFUvAfBxqmRaJme49jPR/keT7my2szBWtpzc4JHAeHTqPA9ecuzownGBBl9LkI
            RoBQx2uziJNKEgH7dKpFZdAY4iifK2vAu/yjxgXlbXxd1G1Tk2KR+LUlmVQcAIzu
            HVobA7CX7yfGbAsL7Mw9dP1ZYpziWYsUzBWvzYAbdaxVwxKBc+jDTL9b6oTXvh63
            pdTpBjvYBYHgSY3ioYbmIQBX2+LSQmVOFSAk6e9NesOLSFZvNPeOuS3wdN4szg7S
            XgHnZ7Oe89/BAXeD+njcUa0ljs2QLNtg+buFrKa2hX/lhWNqeRWZpUmLdvgOstVe
            Ui9PyabYPgJSFccMM7Bx7X3s1VrbnA237KTVAGjy2ouLEbyOeFz2k3p0Rb7ckYY=
            =IOuO
            -----END PGP MESSAGE-----
          fp: 5C3960F507DD4ACC468EE93FE6C10F242BF4141B
        - created_at: "2024-01-28T14:56:38Z"
          enc: |-
            -----BEGIN PGP MESSAGE-----

            hQIMAwAAAAAAAAAAAQ/8DOhem7QLejIo2MuHPlea66TluiTxjo2USOeGiGjCjxan
            d1DpHdOMb/sYgFLiYE+G+hQfe2bcvrR5ziLYkTfT6QPafV4RcPSVp/O014yk2oaO
            zbsOiVEbrhofFStcuMlUAk9IqnTdEMkXppqiMOEWYw6tqGnOYa/bvNP7EGKl8SLg
            HCVYnhXxXwAWFrtAfmALnYzgPvVLwQJX1GPXIV7n/TI+ibpiAhZh54JDGx3obBPK
            WwV9MPPA312MKOkw5upypGkWWPx2OUtG4j9NhvErUmh/TSrGLIvkFSt72Ts+EZyn
            ESBNKKs9dlVwmqmJG0HeNiwOrgiWCjcDrwyT6oPlwXDvzudZiuaGXsPOVvbOyyUt
            3p7ahp3wxEA07Lm2fJtSAc0YZsS3wZ9qBUT67rVP7Lt72tjkZWe6uVNZgWz8IAnQ
            vl+Eg1x4irNfW/1vBBD0Iv4jNpfrO15J71e1hTYnFZtuq02EMhALdzZWjyqXLj9h
            O7Bf94t9AEB2GHW18OK+vKSrS+5qPQb2HiAgy5MhdgeW1eWOdXJBmB3N7yecc831
            0l+jQvmMrpE1olpZPzu+a3AGCPmIjHJQ1GmdlfgDBiF0iu3LsFCaeQxIgiJS5/jJ
            A5HWWU+I69lyIaypQ3yzlNCQyTA9zoSA5ryQxurpyk4WXqm/R4YF2da6VNRpiibS
            UAG/l+QWiQC/6voSi8wPB3YqALHiUTpbNVyCDAvam+vgFH6WcM9y6Wt4V/neIuGd
            LI5CDcJXhZn9c9IHigQtk/8yTp0Xw4CzaTtoTWZNEARD
            =LSAZ
            -----END PGP MESSAGE-----
          fp: fc5624a466cdf7729a4be7e41d407079412e01bc
    unencrypted_suffix: _unencrypted
    version: 3.8.1

When I create shell with nix-shell -p sops gnupg ssh-to-pgp inside my git folder and create the private key manually with sudo ssh-to-pgp -private-key -i /etc/ssh/ssh_host_rsa_key -o secrets/nixvm_private.asc and import that key to gnupg (gpg --import /etc/nixos/secrets/nixvm_private.asc), I can decrypt the secret with sops.
So it seems there is either a config option for sops-nix missing on my side oder sops-nix has a problem after importing the pgp key it converted from the systems ssh key.

Is there a way to get more debug output or log entries from sops-nix?

olmokramer · January 30, 2024, 5:37pm

You shouldn’t need to convert the private key. You encrypt the secrets with every recipient’s public key. Then each recipient can decrypt it with their private key. So use

ssh-to-pgp -i /etc/ssh/ssh_host_rsa_key.pub -o secrets/nixvm_public.asc
gpg --import secrets/nixvm_public.asc

Then copy the fingerprint printed by ssh-to-pgp into your .sops.yaml and run the following command to re-encrypt it now also using the public key of your nixvm host.

sops updatekeys secrets/default.yaml

kagehisa · January 30, 2024, 5:47pm

Ah sorry I guess I should have mentioned that I did the following on the target host (nixvm) in order to see if there was something wrong with the keys or sops iteself:

The puplic key for the target host was generated as described in the documentation (ssh to the host, run ssh-to-pgp and then import it on the admin machines gpg). The encrypted secret file also shows that the secret has two recipients. The fingerprints of the pgp key for the admin machine and the nixvm.

olmokramer · January 30, 2024, 6:09pm

I’ve never done any setup on the target host, always just used ssh-keyscan to get its public ssh key. Then I passed that key into ssh-to-pgp, imported the generated .asc file so that I could encrypt with the target host as a recipient, and added the fingerprint into .sops.yaml.

I cannot find a flaw in your setup, though, so I don’t know what to try next, because as you say you are able to decrypt the secrets using the hosts private ssh key… Except for maybe a reboot and pray it works (don’t forget to nixos-rebuild boot before you do)

kagehisa · January 30, 2024, 7:59pm

I"m not really doing a setup. I just execute a shell with the shell-hook provided by the sops-nix documentation in order to see where the problem lies.

I really hoped it was something simple as a messed up cfg parameter for sops-nix. Praying didn’t work… still not working, but thanks for sticking with me so far. Since I don’t even know if sops-nix produces something like log entries I’m totally out of options, which is why I created an issue at the sops-nix github project. Maybe it really is an issue with the key import.

TLATER · January 30, 2024, 8:33pm

It does, but they’re hard to spot since it’s so early during boot. journalctl -e + /secret<RET> usually gives me the line it prints.

It will likely just give you:

kagehisa · January 31, 2024, 8:09am

Sadly yes…No real further information…
This is just me guessing., but the flake for sops-nix still has inputs.nixpkgs-stable.url = "github:NixOS/nixpkgs/release-23.05";. My VM is a fresh build system based on version 23.11. Is there a way I can test if thats part of the issue?

And just to make sure, since I use flakes and have sops-nix.nixosModules.sops in my modules section of my flake.nix the part imports = [ <sops-nix/modules/sops> ]; in my configuration.nix as described in the documentation should not be neccessary. Or do I have the error here?

kagehisa · January 31, 2024, 9:30am

Ok, the error is definitely in the gpg implementation of sops-nix. I just switched my setup to age and it works. And since pgp fingerprints and keys match and I can perform the decryption manually with sops, sops-nix seems to be having issues with importing the ssh derived secret key.
I hope some of the devs responds to my issue, I would be able to handle debugging a go application but not in a mix with nix (atm).

TLATER · January 31, 2024, 3:27pm

Yep, that’s correct. You could pass through the module to that import if you wanted to, but that would be a cosmetic change.

I’ve made mistakes setting up pgp that took months to debug myself before, I suspect it’s more likely this is a small error somewhere in setting up the sops config. Maybe how you call the hook, or a mistake with which key is used. Ultimately, pgp is just not very user friendly.

kagehisa · January 31, 2024, 4:42pm

I hope thats the case. But I don’t see where this could be wrong.

Sops-nix derives the correct fingerprint from the ssh key
Sops works when I manually add the ssh derived secret pgp key
I use PGP for some time now, including using it with yubikeys. I know what you mean by pgp not being user friendly. But it worked for me for years now.

Since sops itself works I doubt it is the config. What do you mean by how I call the hook?

kagehisa · January 31, 2024, 7:04pm

Ok now I know for certain that my pgp setup works and the problem lies with the automated import step of the ssh derived pgp key from sops-nix.

I altered my setup to create gpghome manually on the target host. I did the following:

Create a shell with sudo nix-shell -p sops gnupg ssh-to-pgp
Creating the GNUPGHOME, create private pgp key for host based on ssh key, import the private and the public keys

mkdir -p /var/lib/sops
chmod -R640
ssh-to-pgp -private-key -i /etc/ssh/ssh_host_rsa_key -o secrets/nixvm_private.asc 
gpg --homedir /var/lib/sops --import secrets/nixvm_private.asc
gpg --homedir /var/lib/sops --import secrets/keys/systems/nixvm.asc
gpg --homedir /var/lib/sops --import secrets/keys/user/manji.asc
GNUPGHOME=/var/lib/sops sops secrets/default.yaml

setting the config parameters sops.gnupg.home = "/var/lib/sops"; and sops.gnupg.sshKeyPaths = [];

Since my debugging capabilities are limited regarding to nix I’m now completely stuck.

kagehisa · February 4, 2024, 11:28am

I found the problem.
After some digging I disabled the removal of the GPGHOME dir from sops-install-secrets.

func (k *keyring) Remove() {
	//os.RemoveAll(k.path)
	os.Unsetenv("GNUPGHOME")
}

This left the generated directory with the generated secring.gpg in the /run/secrets.d directory after the failed nixos-switch attempt.

Then I created a nix-shell just like before with gnupg, sops, ssh-to-pgp and set the environment variable GNUPGHOME to the gpg directory in /run/secrets.d.
And with this sops is able to derypt the secret file. No issues.

Which only leaves the call to "go.mozilla.org/sops/v3/decrypt" as the culprit.

plain, err := decrypt.File(s.SopsFile, string(s.Format))
if err != nil {
   return fmt.Errorf("Failed to decrypt '%s': %w", s.SopsFile, err)
}

It seems it is unable to use the generated secring.gpg. Sops itself triggers a conversion of the generated secring.gpg, into the newer keybox format and in general to the now used form of GNUPGHOME through a call to gnupg itself and is therefore able to make use of the sops-nix generated secring. The call to decrypt.File() however does nothing like this.