Broken configuration include in `nix.extraOptions` made the system unbuildable

Help
1

Hi,

I have a flake-based setup and I was trying to add my Github PAT to nix.extraOptionslike this:

nix = let
    flakeInputs = lib.filterAttrs (_: lib.isType "flake") inputs;
  in {
    settings = {
      <...>
      
    };
    
    extraOptions = ''
      !include ${config.sops.secrets."tokens/nix/auth_tokens".path}
      '';
    
    <...>
  };

…but I stupidly typoed the sops file…

github.com=github_pat_<redacted>

…and now I cannot rebuild, roll back or do, well, anything, really…

❯ sudo nixos-rebuild switch --rollback --flake .#leviathan
error: syntax error in configuration line 'github.com=github_pat_<redacted>' in "/run/secrets/tokens/nix/auth_tokens"
Try 'nix --help' for more information.
Command 'nix --extra-experimental-features 'nix-command flakes' build --print-out-paths '.#nixosConfigurations."leviathan".config.system.build.nixos-rebuild' --no-link' returned non-zero exit status 1.

❯ sudo nix-store --verify --check-contents --repair
error: syntax error in configuration line 'github.com=github_pat_<redacted>' in "/run/secrets/tokens/nix/auth_tokens"
Try 'nix-store --help' for more information.

❯ nix-env --rollback
error: syntax error in configuration line 'github.com=github_pat_<redacted>' in "/run/secrets/tokens/nix/auth_tokens"
Try 'nix-env --help' for more information.

❯ nix flake metadata
error: syntax error in configuration line 'github.com=github_pat_<redacted>' in "/run/secrets/tokens/nix/auth_tokens"
Try 'nix --help' for more information.

So. It seems I’d need to find the current configuration.nix and poke it manually so that I can access Nix tooling again. I tried rooting around in the Nix store but didn’t get anywhere. And, as you can see, nix flake metadata is not available.

I guess I could try rolling back by rebooting to an earlier generation, but I don’t dare try that.

What can I do?

EDIT: The PAT got exposed in all the copy-paste. Don’t worry, it’s already revoked.

2

I managed to find the file! strace pointed me to /etc/nix/nix.conf, but it’s a read-only file system. I guess rebooting is the way to go?

3

Secure way: Use a rescuse disk and rebuild from there using nixos-enter + nixos-rebuild boot or nixos-install

More risky way: rm the clear file from /run/secrets.

4

What happens if you:

/nix/var/nix/profiles/system-###-link/bin/switch-to-configuration switch

? (replace the ### with whatever generation you want to rollback to). It’s probably obvious but

ls /nix/var/nix/profiles/

if you aren’t sure what generations you have

5

Nope.

❯ sudo /nix/var/nix/profiles/system-157-link/bin/switch-to-configuration switch
Checking switch inhibitors... done
Skipping "/boot/EFI/systemd/systemd-bootx64.efi", newer boot loader version in place already.
Skipping "/boot/EFI/BOOT/BOOTX64.EFI", newer boot loader version in place already.
error: syntax error in configuration line 'github.com=github_pat_<redacted>' in "/run/secrets/tokens/nix/auth_tokens"
Try '/nix/store/2gkq62yzbw9kplg5yb6yhf57ckhrhrsd-nix-2.34.7/bin/nix-env --help' for more information.
Traceback (most recent call last):
  File "/nix/store/zjrvv2np38fk7awx51p77ipk2h0pw10b-systemd-boot/bin/systemd-boot", line 688, in <module>
    main()
    ~~~~^^
  File "/nix/store/zjrvv2np38fk7awx51p77ipk2h0pw10b-systemd-boot/bin/systemd-boot", line 666, in main
    install_bootloader(args)
    ~~~~~~~~~~~~~~~~~~^^^^^^
  File "/nix/store/zjrvv2np38fk7awx51p77ipk2h0pw10b-systemd-boot/bin/systemd-boot", line 541, in install_bootloader
    gens = get_generations()
  File "/nix/store/zjrvv2np38fk7awx51p77ipk2h0pw10b-systemd-boot/bin/systemd-boot", line 435, in get_generations
    gen_list = run(
               ~~~^
        [
        ^
    ...<6 lines>...
        stdout=subprocess.PIPE,
        ^^^^^^^^^^^^^^^^^^^^^^^
    ).stdout
    ^
  File "/nix/store/zjrvv2np38fk7awx51p77ipk2h0pw10b-systemd-boot/bin/systemd-boot", line 243, in run
    return subprocess.run(cmd, check=True, text=True, stdout=stdout, stderr=sys.stderr)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/l9k0anq0z7zz81zcwy035jfwap9ga6rl-python3-3.13.13/lib/python3.13/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/nix/store/2gkq62yzbw9kplg5yb6yhf57ckhrhrsd-nix-2.34.7/bin/nix-env', '--list-generations', '-p', '/nix/var/nix/profiles/system']' returned non-zero exit status 1.
Failed to install bootloader
6

Yeah, I thought as much. I’ll wait a bit in case someone shows up with an ingenious solution (probably involving chroot or somesuch wizardry) and then I’ll just try rebooting. I’m pretty sure deleting the secret isn’t an option since it’s already hard-coded in the nix.conf and breaks any attempts at using the tooling.

7

You could try the direct activation as proposed above, but actually elevated (aka use sudo)

8

You might be able to do something like

NIX_CONF_DIR= NIX_USER_CONF_FILES= NIX_CONFIG= /nix/var/nix/profiles/system-157-link/bin/switch-to-configuration switch

You might also be able to setup your NIX_CONFIG in such a way to “fix” the whole thing? See man nix.conf for details on these env variables.

And yes, sudo or run as root, although I see you did already.

EDIT: you could also use test instead of switch as just an operation that does generally less and specifically doesn’t try to mess with the bootloader

9

Oh, I totally misread the error… Just saw the “failed to install bootloader” and implied a permission error…

10

And actually, I don’t know much about /run… but can you just edit /run/secrets/tokens/nix/auth_tokens directly with vim/emacs/nano as root? (and/or link a different file or tree at some point along this path?)

11

Thanks everyone! NIX_CONF_DIR= NIX_USER_CONF_FILES= NIX_CONFIG= sudo /nix/var/nix/profiles/system-157-link/bin/switch-to-configuration test got me into a good state where I was able to rebuild the system.

1 Like