Broken state of the system after 25.11

Easiest would probably be to run dmesg -w and then to run ping, grab whatever turns up.

Since networking works outside the chroot, this isn’t a hardware or kernel issue, it’s entirely in your networking stack. I’m hoping whatever that is pops up in the kernel log somehow, but something is royally screwed.

duh, if it affects all previous configurations as well then it’s truly FUBAR

erm… ruh roh… dmesg -w doesn’t budge

That means it never hits the kernel :slight_smile: Not completely unexpected, given that networking does work outside the chroot.

Have you checked for store corruption with nix store repair & co.? I think it’s unlikely, since you seem to be able to ping on your intranet, but well… I’m at a loss here otherwise.

You could try booting up in a VM like @erolm_a did, if you use the nixos-rebuild build-vm feature to do that it will reuse your store paths, which would allow you to confirm whether it’s state on your system or your store contents. Though tbh if nix store repair doesn’t fix this it’s definitely system state. If it’s state on your system, time to reinstall.

If you end up reinstalling, it’d be great if you could keep a (perhaps compressed) disk image, both immediately before wiping and immediately after nixos-install; that way you can diff later and we can do some postmortem to avoid this in the future.

Also worth considering an erase-your-darlings setup with preservation or impermanence if you’re reinstalling anyway; that’d completely prevent things like this.

:sob: okay… i hoped that this wouldn’t be that big of a problem but I will see what I can do.

I will keep this thread up to date on the progress

1 Like

If you want to avoid the reinstall and continue debugging, you can also create a diff with a system you produce in a disk image with nixos-install, I suppose. But well, the fact that it isn’t reproduced in a VM makes this pretty difficult to debug remotely.

that requires internet access under nixos-enter :

[root@nixos:/home/nika/.dotfiles]# nix store repair
warning: Git tree '/home/nika/.dotfiles' is dirty
warning: error: unable to download 'https://cache.nixos.org/2bcv91i8fahqghn8dmyr791iaycbsjdd.narinfo': Could not resolve hostname (6) Could not resolve host: cache.nixos.org (Could not contact DNS servers); retrying in 299 ms
warning: error: unable to download 'https://cache.nixos.org/2bcv91i8fahqghn8dmyr791iaycbsjdd.narinfo': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 605 ms
warning: error: unable to download 'https://cache.nixos.org/2bcv91i8fahqghn8dmyr791iaycbsjdd.narinfo': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 1024 ms
warning: error: unable to download 'https://cache.nixos.org/2bcv91i8fahqghn8dmyr791iaycbsjdd.narinfo': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 2501 ms
error: unable to download 'https://cache.nixos.org/2bcv91i8fahqghn8dmyr791iaycbsjdd.narinfo': Could not resolve hostname (6) Could not resolve host: cache.nixos.org

To test the migration theory, maybe you could try renaming the entirety of .config into eg. .config.bak. And also delete .cache. You will lose your kde settings and (maybe, not sure) your network manager settings.

I can live with that as long as my other user data doesn’t get destroyed

you mean the .config and .cache in my home dir, or…?

yes the ones in your home

I doubt that will help. If ping doesn’t work that’s stuff in /. Perhaps some manual mucking with /etc/hosts or something.

Ok, sorry, I meant nix store verify - used to be repair, too under nix-store.

Anyway, point is, verify checks validity (which can be done offline), repair repairs (which needs connectivity). We can resolve the latter part of this when we know if your store is ok.

Remember to use --check-contents or whatever the flags for that are these days.

same thing once again:

[root@nixos:/home/nika/.dotfiles]# nix store verify
warning: Git tree '/home/nika/.dotfiles' is dirty
warning: error: unable to download 'https://cache.nixos.org/2bcv91i8fahqghn8dmyr791iaycbsjdd.narinfo': Could not resolve hostname (6) Could not resolve host: cache.nixos.org (Could not contact DNS servers); retrying in 270 ms
warning: error: unable to download 'https://cache.nixos.org/2bcv91i8fahqghn8dmyr791iaycbsjdd.narinfo': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 646 ms
warning: error: unable to download 'https://cache.nixos.org/2bcv91i8fahqghn8dmyr791iaycbsjdd.narinfo': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 1130 ms
warning: error: unable to download 'https://cache.nixos.org/2bcv91i8fahqghn8dmyr791iaycbsjdd.narinfo': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 2563 ms
error: unable to download 'https://cache.nixos.org/2bcv91i8fahqghn8dmyr791iaycbsjdd.narinfo': Could not resolve hostname (6) Could not resolve host: cache.nixos.org

contents of my hosts :

127.0.0.1 localhost
::1 localhost
127.0.0.2 nixos

nix store verify --offline /run/current-system should be what you want.

[root@nixos:/home/nika/.dotfiles]# nix store verify --offline /run/current-system

[root@nixos:/home/nika/.dotfiles]# nix store verify --offline /run/current-system


no output

That means all the paths used for nixos-enter are uncorrupt - assuming you used this inside nixos-enter.

what about /etc/resolv.conf ?

uh huh… pinging one of the IP addresses that cache.nixos.org is under does give me responses

could very likely be something with DNS..?

1 Like

Yes. This might be your DHCP server pushing a broken DNS config, then. Could you try force-setting your DNS to something sensible, ideally with DoH to prevent MiTM from a DHCP server that thinks it knows better? Like, say, one.one.one.one? services.dnscrypt-proxy is supported by NixOS for that purpose.

You can install into your system with nixos-install, even if that isn’t the cleanest solution.

Wouldn’t explain why it’s working outside the chroot, but…

okay, sure can do but I’m not the most technical person regarding such topics. Yes, I understand more than most but…

how do I go about this? thank you

Hm, well, the simplest thing to try first is:

  networking.networkmanager.insertNameservers = [
    "1.1.1.1"
  ];

… assuming you don’t actually have collisions with systemd-networkd.