Nix keeps trying removed substituter

Hi there,

I’m experiencing something weird and can’t find the setting / solution. I (on NixOS) played with a custom binary cache locally (using attic), just out of curiosity. A ran attic use test-cache and it edited my ~/.config/nix/nix.conf adding the new substituter. I ran attic via nix shell github:zhaofengli/attic. After a while I removed it from my config file and reverted it back to its original state. However, nix keeps trying this local substituter, making many builds fail. I have to always specify --substituters "" in order to make my commands work now.

I checked all config files on my system that I know, but can’t find this entry anywhere.

Where can I remove this (I assume cached entry) once and for all? And what was my mistake that resulted in this :slight_smile: ?

Thanks a lot

1 Like

I am not aware of any cache for configuration. Check that it’s also removed from /etc/nix/nix.conf. You can find out about configuration locations that nix will use here nix.conf - Nix Reference Manual.
nix show-config substituters will print the current configured value.

Thank you so much! So this is strange. I checked /etc/nix/nix.conf and it is the generated one (no other substituters defined). The substituter is not in the list of known ones:

❯ nix show-config substituters
https://cache.nixos.org/

When i run nix build it sometimes fails due to trying this binary cache. But sometimes not. So now I realized that it could be related to projects where I have direnv in use. I’ll try this route. Currently I have to pause due to github rate limit for fetching nixpkgs.

You could also check the value of nix show-config accept-flake-config, if it was true it could IMHO be one of the flakes has the substituter entry.

Thanks for the suggestion! The output of nix show-config accept-flake-config is false

I think I also falsified that direnv is related. It happens usually when I build packages of my flake. With direnv enabled projects or not doesn’t make a difference. When doing nix build nixpkgs#hello or ...#rustc etc it doesn’t seem to use the local cache entry. But for some flake evaluation it seems to always apply

❯ nix build github:eikek/confnix#msgconvert
warning: error: unable to download 'http://localhost:8080/test/2bx2zy0r4zmy5qaj8hqw90lirybx2lxi.narinfo': Couldn't connect to server (7); retrying in 265 ms
...

Some other packages from the same flake work fine though.

For reference, this is my /etc/nix/nix.conf (I removed the one in my home dir)

# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix.  Do not edit it!
allowed-users = *
auto-optimise-store = false
builders = 
cores = 0
extra-sandbox-paths = 
max-jobs = auto
require-sigs = true
sandbox = true
sandbox-fallback = false
substituters = https://cache.nixos.org/
system-features = nixos-test benchmark big-parallel kvm
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
trusted-substituters = 
trusted-users = root
experimental-features = nix-command flakes

Could it be that nix tries this cache with packages I used it for when I was playing around? I think it happens to packages that I tried to push to the cache at tinkering time.

Does the flake in question has a line like this?

No, unfortunately not. It is this flake: https://github.com/eikek/sharry

it seems to happen for flakes I evaluated while the binary cache was active. I first thought i didn’t remove it properly when I reverted my nix.conf.

Mhm. Can you try to remove $HOME/.cache/nix?

Oh I totally missed this directory! But, hard to believe, I still get the same error…

sharry on  nix-build:master [$⇡] via ☕ v19.0.2 via 🆂 v2.13.12
❯ nix build
warning: error: unable to download 'http://localhost:8080/test/bp4sabqjd3c0sdg77yg325bw151xc8kh.narinfo': Couldn't connect to server (7); retrying in 252 ms
warning: error: unable to download 'http://localhost:8080/test/bp4sabqjd3c0sdg77yg325bw151xc8kh.narinfo': Couldn't connect to server (7); retrying in 659 ms
...

I could make a little progress. I removed .cache/nix from my own user and root. I also restarted nix-daemon via systemctl restart nix-daemon. Removing from /root made the difference, I think. Now, nix will still try this local cache for some reason, but it won’t fail the build at least.

1 Like

This is so frustrating. I also had attic installed and using a custom binary cache, and I now face the same issue. Clearing the cache (and following other suggestions above) does not help.

It keeps stubbornly attempting to use the, now deleted, cache. Have you made any further progress on your issue?

It could be the narinfo cache which for positive results (found on cache) remains for 30 days. You can disable this by using --option narinfo-cache-positive-ttl 0 or setting the narinfo-cache-positive-ttl option to zero in nix.conf, see nix.conf - Nix Reference Manual for more information.

nix config show, are you sure it’s not in there somewhere? Checked both /etc/nix/nix.conf and $XDG_CONFIG_HOME/nix/nix.conf?

Not sure how narinfo-cache-positive-ttl is supposed to work but running as below I get:

 sudo nixos-rebuild switch --option narinfo-cache-positive-ttl 0 --flake /home/stratos/ultragear/#iocaine
building the system configuration...
downloading 'https://attic.gerakakis.net/ultracache/nix-cache-info'^CTraceback (most recent call last):
  File "/run/current-system/sw/bin/ultragear", line 878, in <module>
    main()
  File "/run/current-system/sw/bin/ultragear", line 874, in main
    fire.Fire(name="ultragear", component=UG)
  File "/nix/store/df4jmlp6bbll0w4l7va39x82j35pk1sw-python3-3.12.6-env/lib/python3.12/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/df4jmlp6bbll0w4l7va39x82j35pk1sw-python3-3.12.6-env/lib/python3.12/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/df4jmlp6bbll0w4l7va39x82j35pk1sw-python3-3.12.6-env/lib/python3.12/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/run/current-system/sw/bin/ultragear", line 755, in update
    run(f"sudo nixos-rebuild switch {cacheOpt}--flake {ULTRAGEAR}/#{currentHost}")
  File "/run/current-system/sw/bin/ultragear", line 125, in run
    out = subprocess.run(
          ^^^^^^^^^^^^^^^
  File "/nix/store/wfbjq35kxs6x83c3ncpfxdyl5gbhdx4h-python3-3.12.6/lib/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/wfbjq35kxs6x83c3ncpfxdyl5gbhdx4h-python3-3.12.6/lib/python3.12/subprocess.py", line 1196, in communicate
    stdout = self.stdout.read()
             ^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

I have to CTRL+C my way out of there because the https://attic.gerakakis.net binary cache (based on atticd, obviously) has been removed. There is NO mention of attic nowhere in my flake or nix files.

That’s so weird!!!

I have completely removed my ~/.config/nix/nix.conf and /etc/nix/nix/conf looks “clean”:

✗  cat ~/.config/nix/nix.conf
cat: /home/stratos/.config/nix/nix.conf: No such file or directory

~ via  
✗  cat /etc/nix/nix.conf     
# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix.  Do not edit it!
allowed-users = root stratos
auto-optimise-store = true
builders = 
cores = 0
experimental-features = nix-command flakes
http-connections = 50
keep-derivations = true
keep-outputs = true
log-lines = 50
max-jobs = auto
require-sigs = true
sandbox = relaxed
sandbox-fallback = false
substituters = https://cache.nixos.org https://nixoscache.gerakakis.net https://ultragear.cachix.org
system-features = nixos-test benchmark big-parallel kvm
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWyp6ZPMQFGspcDShjY= ultragear.cachix.org-1:fIBH3wFNWTVWBcn4ESODB/t6boV52KavU=
trusted-substituters = 
trusted-users = root stratos
warn-dirty = false
extra-platforms = aarch64-linux i686-linux
extra-sandbox-paths = /run/binfmt /nix/store/hlf1icv45d8dsgfhrbvhinpby68z0qz5-qemu-aarch64-binfmt-P-x86_64-unknown-linux-musl
extra-experimental-features = nix-command flakes

What about /root/.config/nix/nix.conf as you are running nixos-rebuild as root?

1 Like
[root@iocaine:~/.config/nix]# cat /root/.config/nix/nix.conf 
# substituters = https://attic.gerakakis.net/ultragear https://cache.nixos.org
substituters = https://attic.gerakakis.net/ultracache https://cache.nixos.org
trusted-public-keys = ultracache:MIG6o2MscTvt7Q3IHLj4uUrXnJEWCnjM/VZV04hcM7U= ultracache:z1QarlQKaU84jtir4EQWZ/pgm7tCW0CcrOvDrv4D0Cc= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
netrc-file = /root/.config/nix/netrc

You’re a life saver!!! No idea how I failed to look under that file!

Here’s a virtual :beer:

Thanks so much! :slight_smile:

1 Like