/run/keys and NixOS examples

I ran into an issue where I lost my freshly generated keys after a reboot, because I put them into /run/keys, which is a tmpfs by default.

There are many examples for keyfiles in the config options though that have them located in /run/keys.

Is this intended or was there a change making /run/keys a tmpfs?

That’s usually done with NixOps, not NixOS itself. NixOps has a send-keys command, and copies the keys after every restart done with the nixops tooling. So you’re not in much danger putting your keys in a tmpfs (and usually desirable to prevent theft of your secrets along with server hardware, for example :wink:

[Edit]: Sorry, I should clarify.

Putting keys in that directory is usually done with NixOps. NixOS itself defaults that directory to a ramfs. I just checked again for usages in modules, and it’s apparently all over the place!
We should probably do some cleanup to make people either aware that this will be wiped on reboot, or that they should definitely have them backed up. Most of the usage right now is as examples in modules, but it seems like there are also some that default to it. If one is unaware that it’s a ramfs, it might cause quite some headaches.

I don’t understand why /run/keys is the default with NixOps and those NixOS services manveru referred to. Isn’t it tricky to maintain a system that loses its secrets every time it reboots or crashes? Do the people advocating this approach have a cluster of machines, so they can tolerate a few of them not having secrets? It seems like an advanced case, not a default.

I’m also unsure. The people who manage my on premise cloud were very clear that machines could reboot at 3am under various problem scenarios. If a web server reboots at 3am a 30 second downtime isn’t a big deal… but having secrets wiped on reboot would obviously cause problems there.

1 Like

I also have my secrets under /run/keys but I link them there via an activationScript. The original keys are under /var/src/secrets, like krops has it as default, and the copies under /run have the correct ownership so that not everyone can read them.


what is the solution to this, should it wiped keys every reboot? seems to me like deployment paranoia…or maybe not depending on how paranoid you want to be?

Are you asking how you can avoid your secrets being wiped out? Don’t store them in /run.

1 Like