Oddly specific Networking issues with NetworkManager

For a while now I had problems with Ethernet, that after a certain amount of uptime (i would say between 10 minutes and a few hours), Ethernet will just stop to work entirely.
The only fix I could find was rebooting.
I tried to restart all kinds of systemd services NetworkManager wpa_supplicant firewall, even pipewire wireplumber etc., nothing helps.
My network related config is this:

{
  # Networking
  networking.networkmanager.enable = true;
  networking.firewall.enable = true;
  # Adblock Hosts
  networking.stevenblack = {
    enable = true;
    block = ["fakenews" "gambling"];
  };
}

Has someone had a similar problem or has any hints on what could cause this?
If I find a way to fix it without rebooting, I might be able to find the cause myself, fix it and if not, “fix” it by automating the fix without rebooting.
It’s generally really annoying to be interrupted in my work to reboot my computer just to get Network Connection again, because I don’t have Wi-Fi available right now.

My whole system config is available here, the laptop host is the machine this problem is occuring on. The attributes of dotconfig are the dotconfig.nix in /dotconfig/<name>, so you know how to navigate the modules.