Sporadic network issues

shnaw · June 23, 2023, 2:59pm

Hello all. I hope I am posting in the correct place.

I have recently installed nixos on my laptop, to try it out. This means that my knowledge about the system is shallow, to say the least. Everything was going great, until I tried to install the editor I use, doom emacs. I tried installing it as I would on any other linux os.

In the installation section, there are two commands that need to be ran. It consistently fails, though on different stages, and the problem seems to be caused by failing to connect to github. Below are 3 examples of outputs.

fatal: unable to access 'https://github.com/doomemacs/doomemacs/': Could not resolve host: github.com

Cloning into '/home/canos/.config/emacs'...
remote: Enumerating objects: 1157, done.
remote: Counting objects: 100% (1157/1157), done.
remote: Compressing objects: 100% (1045/1045), done.
remote: Total 1157 (delta 85), reused 505 (delta 69), pack-reused 0
Receiving objects: 100% (1157/1157), 1.13 MiB | 1.15 MiB/s, done.
Resolving deltas: 100% (85/85), done.
Installing Doom Emacs!

- Skipping ~/.config/doom/ (already exists)
  - Skipping init.el (already exists)
  - Skipping config.el (already exists)
  - Skipping packages.el (already exists)
Generate an envvar file? (see `doom help env` for details) (y or n) y
> Generating envvars file
  ✓ Generated ~/.config/emacs/.local/env
Installing plugins
> Installing straight...
x There was an unexpected runtime error
  Message: File is missing
  Details: ("Cannot open load file" "No such file or directory" "/home/canos/.config/emacs/.local/straight/repos/straight.el/straight.el")
  Backtrace:
    (require straight "/home/canos/.config/emacs/.local/straight/repos/straight.el/straight.el")
    (let ((repo-dir (doom-path straight-base-dir "straight/repos/straight.el")) (repo-url (concat "http" (if gnutls-verify-error "s") "://github.com/" (or (plist-get recipe :repo) "radian-software/straight.el"))) (branch (or (plist-...
    (let ((process-environment (copy-sequence process-environment))) (setenv "GIT_CONFIG" nil) (setenv "GIT_CONFIG_NOSYSTEM" "1") (setenv "GIT_CONFIG_GLOBAL" (or (getenv "DOOMGITCONFIG") "/dev/null")) (let ((repo-dir (doom-path stra...
    (doom--ensure-straight (:host github :repo "radian-software/straight.el" :branch "develop" :local-repo "straight.el" :files ("straight*.el")) "56a8b6c8a46fb78655d512402ad7840d85167225")
    (let* ((--cl-rest-- (alist-get 'straight packages)) (recipe (car (cdr (plist-member --cl-rest-- ':recipe)))) (pin (car (cdr (plist-member --cl-rest-- ':pin))))) (doom--ensure-straight recipe pin))
    (let ((packages (doom-package-list '((:core))))) (let* ((--cl-rest-- (alist-get 'straight packages)) (recipe (car (cdr (plist-member --cl-rest-- ':recipe)))) (pin (car (cdr (plist-member --cl-rest-- ':pin))))) (doom--ensure-stra...
    (progn (if doom-inhibit-log nil (doom--log "Initializing straight")) (let ((packages (doom-package-list '((:core))))) (let* ((--cl-rest-- (alist-get 'straight packages)) (recipe (car (cdr (plist-member --cl-rest-- ':recipe)))) (...
    (if (or force-p (null (and (boundp 'straight-recipe-repositories) straight-recipe-repositories))) (progn (if doom-inhibit-log nil (doom--log "Initializing straight")) (let ((packages (doom-package-list '((:core))))) (let* ((--cl...
    (doom-initialize-core-packages nil)
    (doom-initialize-packages)
    (doom-packages-install)
    (if (eq install\? :no) (doom-print (doom-print--format (doom-print--class 'warn "Not installing plugins, as requested"))) (doom-print (doom-print--format "Installing plugins")) (doom-packages-install))
  ! Wrote extended backtrace to ~/.config/emacs/.local/state/logs/cli.doom.230623150243.15801.error

and

Cloning into '/home/canos/.config/emacs'...
^[[Aremote: Enumerating objects: 1157, done.
remote: Counting objects: 100% (1157/1157), done.
remote: Compressing objects: 100% (1048/1048), done.
remote: Total 1157 (delta 85), reused 502 (delta 66), pack-reused 0
Receiving objects: 100% (1157/1157), 1.13 MiB | 2.02 MiB/s, done.
Resolving deltas: 100% (85/85), done.
Installing Doom Emacs!

- Skipping ~/.config/doom/ (already exists)
  - Skipping init.el (already exists)
  - Skipping config.el (already exists)
  - Skipping packages.el (already exists)
Generate an envvar file? (see `doom help env` for details) (y or n) y
> Generating envvars file
  ✓ Generated ~/.config/emacs/.local/env
Installing plugins
> Installing straight...
x The package manager threw an error
x Last 16 lines of straight's error log:
  $ cd /home/canos/.config/emacs/.local/straight/repos/
  $ git clone --origin origin --no-checkout https\://github.com/melpa/melpa.git /home/canos/.config/emacs/.local/straight/repos/melpa/ --depth 1 --single-branch --no-tags
  Cloning into '/home/canos/.config/emacs/.local/straight/repos/melpa'...
  fatal: unable to access 'https://github.com/melpa/melpa.git/': Could not resolve host: github.com
  [Return code: 128]
  $ cd /home/canos/.config/emacs/.local/straight/repos/
  $ git clone --origin origin --no-checkout https\://github.com/melpa/melpa.git /home/canos/.config/emacs/.local/straight/repos/melpa/ --no-single-branch
  Cloning into '/home/canos/.config/emacs/.local/straight/repos/melpa'...
  fatal: unable to access 'https://github.com/melpa/melpa.git/': Could not resolve host: github.com
  [Return code: 128]
  $ cd /home/canos/.config/emacs/.local/straight/repos/melpa/
  $ git branch -r
  [File error while Setting current directory]
Opening output file: No such file or directory, /home/canos/.config/emacs/.local/state/logs/cli.doom.230623151350.16683.error

It’s pretty obvious this is caused by some kind of connection issue to github.

Here are the most important points:

The same thing does not happen on my main pc with fedora, but it did happen when I used the nixos iso with only the command nix-env -iA nixos.emacs and ran the commands.

The same thing does not happen on vanilla os, which I dual boot in the laptop

I have restarted my pc several times, and tried to run the commands under my friends’ wifi, which didn’t work either

It shouldn’t have anything to do with the changes I did to my configuration since the issue is reproducible from the nixos iso

It happens in the browser too, and not just with github: There were several times where it refused to connect to reddit, for instance. However, there are also some sites where it never fails, I have yet to have a failed nixos update. It is recurrent with github though.

The issue is consistent: There hasn’t been one time where the doom install script lasted more than 10 seconds installing when it usually takes several minutes to install

As much as I hate to admit it, I have been unable to debug the issue myself, and I haven’t tried much aside from basic stuff such as disabling the firewall, especially since my knowledge of both networking and nixos are extremely shallow. I haven’t even got any clue I could investigate from. I’d like to know if anyone can replicate this issue, or has any idea of what could be causing it.

Thank you in advance

EDIT: Oddly enough, using mobile data, the process managed to run for quite a few seconds without any error. I didn’t go the whole way though, since I have a limited amount of data. Though this means that replicating the issue will be hard, I will not delete this post since this only happens with nixos. If anyone has any idea what could be happening, any and all advice would be greatly appreciated. Thanks.

ericgundrum · June 25, 2023, 6:31pm

Are you installing doom-emacs the way their instructions say for nixOS?

When I have had difficulties with updating my doom-emacs, they were caused by stale config compilations or by some incompatibility between doom’s master branch and the plugins that got pulled. For an initial installation, I’d encourage you to use a tagged release of doom-emacs. If you are copying .config/emacs from another system, be sure to clear all the compiled config.

I install doom-emacs dependencies using a flake specifying them like this

      doomemacs = buildEnv {
        name = "doomemacs-pkgs";
        paths = [
          emacs28NativeComp
          ripgrep
          fd
          shellcheck
          nixfmt
          editorconfig-core-c
          libvterm
          (aspellWithDicts (dicts: with dicts; [ en en-computers en-science ]))
        ];
        pathsToLink = [ "/share" "/bin" ];
        extraOutputsToInstall = [ "man" "doc" ];
      };

I do not use HomeManager, which might manage those dependencies differently. I do not see why missing any of those dependencies would cause the install to fail.

If you are truly having network problems, there may be clues about them in the system journal. Immediately after getting the installation failure, try something like sudo journalctl -ex and page backwards looking for network problems.

You also might try manually cloning the failing repos to see if you can reproduce the problem outside of emacs.

Could not resolve host: github.com suggests to a possible problem with your nixos DNS config. Maybe you can uncover differences from your working systems.

I don’t know what DNS tools are in a default nixos installation. I use resolvectl. You can find others among nixpkgs.

Hopefully something here will help unstick your debugging.

ericgundrum · June 25, 2023, 10:37pm

Let’s suppose that your ISP DNS service is flaky. This could explain why your mobile connection does not suffer the problem. And, likely your other OSes are configured to overcome it.

One way to overcome it is to bypass your ISP DNS by configuring your device networking to use well-known public DNS servers such as 1.1.1.1, 8.8.8.8, 9.9.9.9 and their IPv6 equivalents. Some network providers try to prevent bypassing their DNS (sometimes at the request of the government). There are ways around that, but it gets complicated.

From your recent post I noticed you use NetworkManager. You can run nmcli to get a summary of how your network interfaces are configured; DNS appears at the end of that output. For instance, mine contains

DNS configuration:
        servers: 10.0.45.1
        domains: lan
        interface: wlp170s0

This indicates that my WiFi connection (wlp170s0) uses server 10.0.45.1 for DNS (among others), which happens to be my home router (and DHCP server). What yours reports will depend on how the various parts of your Internet access are configured.

If your ISP controls your home router (as often is the case) then likely they will have it configured to use their DNS servers. You can use NetworkManager (gui or cli) to override the DNS config and specify a more reliable server. See NetworkManager - ArchWiki.

Much of the ArchWiki info on networking is very deep. Maybe you can find helpful bits by focusing on what talks about client configuration, particularly for DNS and DHCP. You also may find good, desktop-oriented info among the Ubuntu docs.

Assuming your system is configured for DHCP network addressing (very much most common), you may overcome the problem with a simple nixos config change such as adding this to your configuration.nix

networking.networkmanager.insertNameservers = [ "1.1.1.1" "8.8.8.8"];

If you are using IPv6 addressing, then those addresses would be different. I think possibly ["2606:4700:4700::1111" "2606:4700:4700::1001"] would work for IPv6.

An alternative way to deal with a flaky DNS server is to run a local DNS caching service, such as resolved. That is where the resolvectl comes from. It is part of systemd, but many parts of systemd are outside the default OS installation.

Configuring to use resolved is a little more involved. If you want to pursue that path, search for info here in discourse and in the nixos wiki.

shnaw · June 26, 2023, 2:32pm

Here’s what I did:
I woke up, ran sudo resolvectl flush-caches, added networking.networkmanager.insertNameservers = [ "1.1.1.1" "8.8.8.8"]; and boom, it somehow started working. It was probably one of these two changes that did the trick, since last night I am sure it wasn’t working and if I boot into the live iso it doesn’t work anymore. However, I can’t really test which, since it is impossible to run sudo nixos-rebuild switch on the live iso without getting an error, as far as I have tested. and especially because even if I remove the line I added to the configuration.nix it doesn’t stop working… I can’t make it stop working now!

I would just like to add that, in order to not look as stupid as I might look now since the second change was easily findable online, it appears I enabled the wrong option when I tried to do that…

and it apparently didn’t work. Though I only used the 8.8.8.8 one at that time, which could have had something to do with it, maybe? Regardless, I thank you very much for the time you took to answer this issue, and for the extremely helpful tips.

Cheers!

PS: I deleted my last post because I discovered you had to enable a setting to use resolvectl right after posting it, and went to sleep before testing anything. I apologize for that

ericgundrum · June 26, 2023, 3:31pm

You shouldn’t feel bad; networking gets complicated fast as soon as you stray from the default path. I’ve been doing network configuration for many years, and I still need to lookup details for just about everything. My experience just helps me know where to look and how to place things in context.

For instance, I do not recall if NetworkManager tries DNS servers one at a time (in order) or in parallel. The existence of before and after config implies that it does lookups serially, which makes insertNameservers the better option for your use case.

networking.nameservers is not a NetworkManager option. I’d have to dig deeper to understand how it interacts with NetworkManager and if it is compatible. It could be a way to remove your ISP DNS from the list, but this could cause problems when you connect to captive-portal network such as a cafe wifi.

There are many public nameservers you could use. CloudFlare, 1.1.1.1, and Google, 8.8.8.8, are just two. But listing more than you need could make diagnosing future problems more difficult.

Having services.resolved working is great. It means your system will look for a cached DNS answer before going to the network. This should save some time with repeat connections to Internet hosts – especially if your ISP network is slow.

Welcome to the world of nixos. The journey can be hard at times, but it is well worth it for what we learn along the way and for the results we get.