This week I got a new machine, and for variety of reasons, setup NixOS-WSL on Windows Subsystem for Linux 2. It has been working well enough with my development environment all setup.
However, yesterday I setup agenix on it. I was already using it on previous machine with NixOS. Agenix has a bug (Update: As described in solution, the bug was actually in my NixOS config, and not agenix) where it randomly fails to decrypt files on boot. I haven’t observed any pattern, but when it does fail, on NixOS, it just prints a log during boot and proceeds to rest of the system. However, on WSL it prints the log, but subsequently refuses to get to shell:
[agenix] symlinking new secrets to /run/agenix (generation 1)...
[agenix] decrypting root secrets...
[agenix] decrypting non-root secrets...
decrypting '/nix/store/7va5sd50rymgkh2h4bj7yrb9dxfjzb02-source/hosts/enterprise/secrets/enterprise-nix-cache-key-sec.age' to '/run/agenix.d/1/enterprise-nix-cache-key-sec'...
[agenix] WARNING: encrypted file /nix/store/7va5sd50rymgkh2h4bj7yrb9dxfjzb02-source/hosts/enterprise/secrets/enterprise-nix-cache-key-sec.age does not exist!
Error: No such file or directory (os error 2)
[ Did rage not do what you expected? Could an error be more useful? ]
[ Tell us: https://str4d.xyz/rage/report ]
chmod: cannot access '/run/agenix.d/1/enterprise-nix-cache-key-sec.tmp': No such file or directory
chown: cannot access '/run/agenix.d/1/enterprise-nix-cache-key-sec.tmp': No such file or directory
mv: cannot stat '/run/agenix.d/1/enterprise-nix-cache-key-sec.tmp': No such file or directory
decrypting '/nix/store/7va5sd50rymgkh2h4bj7yrb9dxfjzb02-source/hosts/enterprise/secrets/secret__1.age' to '/run/agenix.d/1/secret__1'...
[agenix] WARNING: encrypted file /nix/store/7va5sd50rymgkh2h4bj7yrb9dxfjzb02-source/hosts/enterprise/secrets/secret__1.age does not exist!
Error: No such file or directory (os error 2)
[ Did rage not do what you expected? Could an error be more useful? ]
[ Tell us: https://str4d.xyz/rage/report ]
chmod: cannot access '/run/agenix.d/1/secret__1.tmp': No such file or directory
chown: cannot access '/run/agenix.d/1/secret__1.tmp': No such file or directory
mv: cannot stat '/run/agenix.d/1/secret__1.tmp': No such file or directory
decrypting '/nix/store/7va5sd50rymgkh2h4bj7yrb9dxfjzb02-source/hosts/enterprise/secrets/secret__2.age' to '/run/agenix.d/1/secret__2'...
[agenix] WARNING: encrypted file /nix/store/7va5sd50rymgkh2h4bj7yrb9dxfjzb02-source/hosts/enterprise/secrets/secret__2.age does not exist!
Error: No such file or directory (os error 2)
[ Did rage not do what you expected? Could an error be more useful? ]
[ Tell us: https://str4d.xyz/rage/report ]
chmod: cannot access '/run/agenix.d/1/secret__2.tmp': No such file or directory
chown: cannot access '/run/agenix.d/1/secret__2.tmp': No such file or directory
mv: cannot stat '/run/agenix.d/1/secret__2.tmp': No such file or directory
decrypting '/nix/store/7va5sd50rymgkh2h4bj7yrb9dxfjzb02-source/hosts/enterprise/secrets/secret__3.age' to '/run/agenix.d/1/secret__3'...
[agenix] WARNING: encrypted file /nix/store/7va5sd50rymgkh2h4bj7yrb9dxfjzb02-source/hosts/enterprise/secrets/secret__3.age does not exist!
Error: No such file or directory (os error 2)
[ Did rage not do what you expected? Could an error be more useful? ]
[ Tell us: https://str4d.xyz/rage/report ]
chmod: cannot access '/run/agenix.d/1/secret__3.tmp': No such file or directory
chown: cannot access '/run/agenix.d/1/secret__3.tmp': No such file or directory
mv: cannot stat '/run/agenix.d/1/secret__3.tmp': No such file or directory
decrypting '/nix/store/7va5sd50rymgkh2h4bj7yrb9dxfjzb02-source/hosts/enterprise/secrets/secret__4.age' to '/run/agenix.d/1/secret__4'...
[agenix] WARNING: encrypted file /nix/store/7va5sd50rymgkh2h4bj7yrb9dxfjzb02-source/hosts/enterprise/secrets/secret__4.age does not exist!
Error: No such file or directory (os error 2)
[ Did rage not do what you expected? Could an error be more useful? ]
[ Tell us: https://str4d.xyz/rage/report ]
chmod: cannot access '/run/agenix.d/1/secret__4.tmp': No such file or directory
chown: cannot access '/run/agenix.d/1/secret__4.tmp': No such file or directory
mv: cannot stat '/run/agenix.d/1/secret__4.tmp': No such file or directory
decrypting '/nix/store/7va5sd50rymgkh2h4bj7yrb9dxfjzb02-source/hosts/enterprise/secrets/secret__5.age' to '/run/agenix.d/1/secret__5'...
[agenix] WARNING: encrypted file /nix/store/7va5sd50rymgkh2h4bj7yrb9dxfjzb02-source/hosts/enterprise/secrets/secret__5.age does not exist!
Error: No such file or directory (os error 2)
[ Did rage not do what you expected? Could an error be more useful? ]
[ Tell us: https://str4d.xyz/rage/report ]
chmod: cannot access '/run/agenix.d/1/secret__5.tmp': No such file or directory
chown: cannot access '/run/agenix.d/1/secret__5.tmp': No such file or directory
mv: cannot stat '/run/agenix.d/1/secret__5.tmp': No such file or directory
decrypting '/nix/store/7va5sd50rymgkh2h4bj7yrb9dxfjzb02-source/hosts/enterprise/secrets/secret__6.age' to '/run/agenix.d/1/secret__6'...
[agenix] WARNING: encrypted file /nix/store/7va5sd50rymgkh2h4bj7yrb9dxfjzb02-source/hosts/enterprise/secrets/secret__6.age does not exist!
Error: No such file or directory (os error 2)
[ Did rage not do what you expected? Could an error be more useful? ]
[ Tell us: https://str4d.xyz/rage/report ]
chmod: cannot access '/run/agenix.d/1/secret__6.tmp': No such file or directory
chown: cannot access '/run/agenix.d/1/secret__6.tmp': No such file or directory
mv: cannot stat '/run/agenix.d/1/secret__6.tmp': No such file or directory
Activation script snippet 'agenix' failed (1)
Copying /usr/share/applications
Copying /usr/share/icons
setting up /etc...
[process exited with code 1 (0x00000001)]
The bug was already reported and apparently it is not something agenix’s fault. I am thinking about moving to sops-nix, but haven’t had time to actually implement it.
On NixOS if shell is unreachable the simplest way to go is by booting into older generation. But here, there is no bootloader. Because of few things in my setup requiring imperative actions (I’m still n00b at NixOS), so any help in fixing this beyond new VM setup is really appreciated.
WARNING: encrypted file /nix/store/7va5sd50rymgkh2h4bj7yrb9dxfjzb02-source/hosts/enterprise/secrets/enterprise-nix-cache-key-sec.age does not exist!
so it seems that somehow the nix store doesn’t have that file. Are you able to boot some other way where you could check for the file’s presence? Could you share your configuration?
Before reboot, I did nixos-rebuild switch successfully, and even used the correctly decrypted and deployed secrets. The issue only shows up on activation at reboot, but not on every reboot. I was kinda hoping it would be some idiosyncrasy of my old machine, but maybe not.
Huh. I accessed VM drive from Windows and this path /nix/store/7va5sd50rymgkh2h4bj7yrb9dxfjzb02-source doesn’t exist. The Nix store exists with plenty of other stuff, but this one doesn’t. Now I am 100% certain this config built and worked with switch before. So I see 2 problems:
Nix (or something, but considering the store is supposed to be read-only, probably Nix) is eating the stuff from store.
On NixOS this error means a little logger, secrets don’t deploy and system boots. On WSL no boot.
What more information do we need for taking any further action? In fact, since this is all WSL2 on Windows 11, anybody with access to Windows 11 machine can replace the key and secrets from my config and test out. Might even find a pattern there.
Could you check that your system .drv file depends on /nix/store/7va5sd50rymgkh2h4bj7yrb9dxfjzb02-source/hosts/enterprise/secrets/enterprise-nix-cache-key-sec.age? I’m slightly concerned the way you are referencing file paths is not getting them into the system closure properly, but I don’t know enough about the inner workings of strings in Nix to be totally sure.
Unfortunately no. There is no bootloader on WSL. It can probably be done since the store probably just needs a different symlink, but I don’t have enough know-how for that.
Wait, Nix stores this info in sqlite db right? Can I copy it over and run some query for this? I have no idea how Nix works so please consider this an example of Dunning-Kruger effect, but Nix might be doing that internally. Can we do it manually?
I think this explains the problem you’re having. If your system closure does not depend on the encrypted file, the decryption depends on the file happening to be there instead of being guaranteed to be there. For example, the nix garbage collector might delete the encrypted file if nothing else depends on it.
Generally it is tricky to get strings right in Nix and I would recommend just writing all the secret files you want to include as a list, but you really want to automate it, maybe something like this will work:
with builtins;
with lib;
let
secretsDir = ../hosts/enterprise/secrets;
secretsFile = "${secretsDir}/secrets.nix";
payas = "payas";
in
{
# imports = [ agenix.nixosModules.age ];
# TODO: Find a way to make agenix available in the runtime NixOS evaluation
# environment.systemPackages = [ agenix.defaultPackage.x86_64-linux ];
age = {
secrets =
if pathExists secretsFile
then
mapAttrs'
(n: _: nameValuePair (removeSuffix ".age" n)
{
file = "${secretsDir}/${n}";
owner = payas;
})
(import secretsFile)
else
{ };
identityPaths = lib.mkForce
[
"/home/payas/.ssh/age"
];
};
}
However, be careful because I expect this to cause the ENTIRE secretsDir to be a dependency of your system closure, which may not be what you want from a security perspective.
Even though it seems that you have fixed your problem, in case you break the VM in the future:
You can run wsl -d NixOS -e sh to get a shell outside of the systemd namespace. From there running source /etc/set-environment will give you access to everything in path, including nixos-rebuild (switch will fail to restart the services, but boot works just fine). You can also use this shell to start /nix/var/nix/profiles/system/activate manually to troubleshoot issues with any activation scripts
UPDATE: Oh I see, the key point is to use types.path instead of types.str.
Thanks for the inspiring explanation. Sorry for asking a possibly silly question, but is the key change of the example code being explicitly specifying the secretsDir and secretsFile? Why this would make nix evaluate the secrets into a standalone path under /nix/store instead being a subfolder under source?