Broke /etc/nix/nix.conf after running nix store gc && nix store optimise

Help
1

Hey folks,

After setting secret-key-files from a nix flake, then running nix gc && nix store optimise, somehow the key has been deleted from the Nix store.

Now I can’t build because /etc/nix/nix.conf refers to the now deleted key in the store. But because /etc/nix/nix.conf is read-only, I can’t reset the secret-key-files line to blank, so I can run nixos-rebuild again.

I’ve tried running nix store verify --repair --all, but it didn’t help or produce any error.

Any thoughts on how to go about fixing this issue?

nixos-rebuild build fails because of missing file in /nix/store

Link

/etc/nix/nix.conf references this file 
john@saturn:~/.config/nix/ > cat /etc/nix/nix.conf         
# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix.  Do not edit it!
allowed-users = *
auto-optimise-store = false
cores = 0
experimental-features = nix-command flakes
max-jobs = auto
require-sigs = true
sandbox = true
sandbox-fallback = false
secret-key-files = /nix/store/zwibnvq1c72qaw6bx1hw7im9shdgb1ar-source/secrets/sun/nix-bin-cache-priv-key.pem
substituters = https://hyprland.cachix.org ssh-ng://sun https://cache.nixos.org/
system-features = nixos-test benchmark big-parallel kvm
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.lix.systems:aBnZUw8zA7H35Cz2RyKFVs3H4PlGTLawyY5KRbvJR8o= nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs= hyprland.cachix.org-1:a7pgxzMz7+chwVL3/pzj6jIBMioiJM7ypFP8PwtkuGc= sun:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
trusted-substituters = 
trusted-users = root root johnhamelink john @wheel
extra-sandbox-paths = 
extra-substituters = https://cache.lix.systems https://nix-community.cachix.org
builders-use-substitutes = true

john@saturn:~/.config/nix/ >
Can't edit /etc/nix/nix.conf 
john@saturn:~/.config/nix/ > readlink -f /etc/nix/nix.conf 
/nix/store/jwsll18j0grpcnshyjxvlmaccgvhwxvd-nix.conf

root@saturn:/home/john/.config/nix/ > echo " " | tee -a /nix/store/jwsll18j0grpcnshyjxvlmaccgvhwxvd-nix.conf
tee: /nix/store/jwsll18j0grpcnshyjxvlmaccgvhwxvd-nix.conf: Read-only file system
Key doesn't exist 
john@saturn:~/.config/nix/ > ls -al /nix/store/zwibnvq1c72qaw6bx1hw7im9shdgb1ar-source/secrets/sun/nix-bin-cache-priv-key.pem
ls: cannot access '/nix/store/zwibnvq1c72qaw6bx1hw7im9shdgb1ar-source/secrets/sun/nix-bin-cache-priv-key.pem': No such file or directory
Nix doctor passes 
john@saturn:~/.config/nix/ > nix doctor                              
[PASS] PATH contains only one nix version.
[PASS] All profiles are gcroots.
[PASS] Client protocol matches store protocol.
[INFO] You are trusted by store uri: daemon

Thanks!
JH

2

Ahh, I was able to fix my install! The answer is to use NIX_CONFIG='secret-key-files =' to override secret-key-files, then --builders '' to disable distributed builds.

NIX_CONFIG='secret-key-files =' nixos-rebuild build --verbose --builders '' --flake .# |& nom
3

Looks like there’s a couple of things going on here. Your flake’s path in the nix store is not protected from the GC. There’s nothing to keep it live. You can fix this by somehow making it a dependency of your system config, e.g. with environment.etc."nix/self".source = inputs.self.outPath;

Now, the reason it wasn’t kept alive despite nix.conf having secret-key-files pointing to that file is because of where the nix evaluation got that path string from. It looks like values in nix.settings get toString’d. There’s a difference between "${x}" and toString x for path literals. "${./foo}" will copy foo to the store and create a “string with context” that depends on that new store path, meaning including this string in any derivations gives them a properly tracked nix dependency on that path. But toString ./foo just gives you a string literal pointing to the original file location. So toString /home/will/foo returns the string "/home/will/foo". And because this isn’t copied to the store and can point anywhere, this means the mere fact that the path you chose is in your flakes store path doesn’t create a dependency on that store path.

TL;DR: You can fix this by replacing ./nix-bin-cache-priv-key.pem with "${./nix-bin-cache-priv-key.pem}".

However, I’m noticing an unrelated problem here. You’re putting your secret file in the nix store. This is a big nono. The store has read permissions for all users on all the files it contains. You’re allowing any user on the system to see this secret file, meaning any user can sign any NAR they create, even if it’s fraudulent. agenix or sops-nix can make this sort of thing more tolerable, but the easiest thing to do is to set secret-key-files to a hard coded string pointing at a path that you’re in control of. Like nix.settings.secret-key-files = "/var/lib/secrets/my-key.pem";, and then make sure the file exists there.

Actually, the mere fact that you’re storing this secret in your flake means that when your flake is copied to the store for evaluation, it’s being exposed this way.

1 Like
4

Thank you for this detailed reply. I was aware of the mistake of putting the key in the store, and I’m actually using sops for other things, but I found myself painted into a corner once I realised what I’d done!

What I didn’t know beforehand however why this had happened, and now I understand. I’ve moved the key into sops and I’m referencing it using the config.sops.secret.* syntax so that I can’t accidentally break things.

Thanks again!