Connected to Mullvadvpn but no internet connection

Hello, I just updated to the unstable channel and got this error, although it works just fine in stable channel. I’m connected to Mullvadvpn but no internet connection, but when I disconnect Mullvad, I can use internet just fine.
This is my nixos-info:

$ nix-info -m
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.62, NixOS, 23.11 (Tapir), 23.11pre550515.e4ad989506ec`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - channels(root): `"nixos"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

when i do mullvad status, this is the output:

$ mullvad status
Connected to us-nyc-wg-502 in New York, NY, USA

This definitely means that I am in fact, connected. But I have no internet connection. This is my configuration for mullvad in my configuration.nix:

  # Enable mullvad service
  networking.firewall.checkReversePath = "loose";
  networking.wireguard.enable = true;
  services.mullvad-vpn.enable = true;
  networking.iproute2.enable = true;

I also tried open the ports like the solution for the similar issue:

  # Open ports in the firewall.
   networking.firewall.allowedTCPPorts = [53 80 443 1401];
   networking.firewall.allowedUDPPorts = [53 1194 1195 1196 1197 1300 1301 1302 1303 1400];

Nothing works, I’m desperate, yet again.

1 Like

When you say you “have no internet connection”, could you be a little more specific? Can you, for example, ping some IP address, such as 1.1 or 8.8.8.8?

I tried it, but no luck. I have to roll back to the stable channel and mullvad just works wonders. No idea why tho

Edit: I rolled back to the unstable channel and try pinging 8.8.8.8, works fine, but literally no idea why it can’t connect to websites on browsers. And also installing packages won’t work.
I thought it has something to do with the dns server, so i added another dns server to MullvadVPN, also didn’t work

1 Like

I’m having the same issue. It seems to only be affecting WireGuard servers, though. OpenVPN ones are working fine.

If you can ping ip but not reach servers, it’s a dns-related issue. People mentionned that wireguard dns issues are usually coming from the fact that the client did not allow the ip of the DNS server : https://www.reddit.com/r/WireGuard/comments/ejjppr/wireguard_not_sending_dns_requests_through_tunnel/

Maybe check which dns you use and if their are allowed. Sometimes the DNS is a local IP given by the routeur so I guess this fails with wireguard, so maybe try to force a different DNS like 8.8.8.8? This can be done in network manager and seems like you can also put it in the config client of wireguard. See e.g. WireGuard DNS Configuration for Systemd | Pro Custodibus

For what it’s worth, I’ve recently had similar connection issues with mullvad client on macos. I would be connected but no traffic would go through. The problems with dns are way more common but I had cases when on some servers (don’t remember if wireguard or openvpn) I couldn’t even ping 8.8.8.8.

1 Like

Mullvad may anyway intercept DNS traffic and substitute its own servers to prevent DNS leaks.

Check with dig to see what’s actually going on:

dig google.com
dig @8.8.8.8 google.com
1 Like

I think the issue could be caused by this commit to the Mullvad module.

This is the message I got from the daemon when I connected to a WireGuard server. This is the one when I connect to an OpenVPN server. It seems to be a DNS resolution issue, and it appears that the fix (at least for me) is to enable systemd-resolved in your configuration.nix with services.resolved.enable = true

2 Likes

Nice! Can you then maybe create either a PR fixing it, or at least an issue to report it to the maintainers?

(I was the one who PR’d that commit, apologies if it broke anything!)

If it did break something, it must be DNS. The lines that stands out to me is

Nov 23 19:34:59 nanotwerp mullvad-daemon[2217]: [talpid_dbus::network_manager][DEBUG] /etc/resolv.conf differs from reference resolv.conf, therefore NM is not managing DNS
Nov 23 19:34:59 nanotwerp mullvad-daemon[2217]: [talpid_core::dns::imp][DEBUG] Managing DNS via resolvconf

So I’m assuming you’re in a situation where ping 8.8.8.8 works but DNS doesn’t. This can happen if mullvad is connected (mullvad status says “Connected to…”) but the configured DNS doesn’t match its wireguard ip (10.64.0.1). Since mullvad uses firewalls (?) to prevent DNS leaks, it won’t allow DNS to be resolved by any other nameserver, for example, running

[nix-shell:~]$ dig @10.64.0.1 google.com

; <<>> DiG 9.18.19 <<>> @10.64.0.1 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28999
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;google.com.			IN	A

;; ANSWER SECTION:
google.com.		228	IN	A	142.251.163.102
google.com.		228	IN	A	142.251.163.113
google.com.		228	IN	A	142.251.163.100
google.com.		228	IN	A	142.251.163.101
google.com.		228	IN	A	142.251.163.138
google.com.		228	IN	A	142.251.163.139

;; Query time: 11 msec
;; SERVER: 10.64.0.1#53(10.64.0.1) (UDP)
;; WHEN: Fri Nov 24 12:41:48 EST 2023
;; MSG SIZE  rcvd: 135

resolves verses using 8.8.8.8 as the dns nameserver.

[nix-shell:~]$ dig @8.8.8.8 google.com
;; communications error to 8.8.8.8#53: connection refused
;; communications error to 8.8.8.8#53: connection refused
;; communications error to 8.8.8.8#53: connection refused

; <<>> DiG 9.18.19 <<>> @8.8.8.8 google.com
; (1 server found)
;; global options: +cmd
;; no servers could be reached

So I’m assuming what happened is mullavd uses resolvconf to update /etc/resolv, but since there’s some NetworkManager misconfiguration, NetworkManager either doesn’t respect it or updates it itself shortly after. This is also why openvpn works but not wireguard, because wireguard is more strict about DNS leaks. Perhaps you can check the contents of /etc/resolv.conf to make sure it only has 10.64.0.1 and /etc/resolvconf.conf (if it exists)?

It shouldn’t be necessary to set services.resolved.enable = true. I don’t use NetworkManager and so I haven’t tested this, but based on my reading of the arch wiki, the gnome documentation, and the NetworkManager man page it should be enough to set in /etc/NetworkManager/conf.d/

[main]
dns=default
rc-manager=resolvconf

I think (again, untested) this can be done with networking.networkmanager.dns = "default" and

networking.networkmanager.extraConfig = ''
  [main]
  rc-manager=resolvconf
'';

Alternatively, services.resolved.enable = true works because the default setting dns=default

If the key is unspecified, default is used, unless /etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf, /run/systemd/resolve/resolv.conf, /lib/systemd/resolv.conf or /usr/lib/systemd/resolv.conf. In that case, systemd-resolved is chosen automatically.

and rc-manager=auto

auto: if systemd-resolved plugin is configured via the dns setting or if it gets detected as main DNS plugin, NetworkManager will update systemd-resolved without touching /etc/resolv.conf. Alternatively, if resolvconf or netconfig are enabled at compile time and the respective binary is found, NetworkManager will automatically use it. Note that if you install or uninstall these binaries, you need to reload the rc-manager setting with SIGHUP or systemctl reload NetworkManager. As last fallback it uses the symlink option (see next).

both support systemd-resolved.

My personal configuration using iwd looks something like

{
  networking = {
    wireless.iwd = {
      enable = true;
      settings = {
        General.EnableNetworkConfiguration = true;
        Network.NameResolvingService = "resolvconf";
      };
    };
  };
  services.mullvad-vpn.enable = true;
}

and I can connect to wireguard fine and it properly updates resolvconf (I also use the dns server unbound).

1 Like

update: I played around with NetworkManager myself.

tl;dr set networking.resolvconf.enable = false. Indeed, this exactly reverts the PR.

If you want to use resolvconf/systemd-resolved, then set services.resolved.enable = true.

For those interested in the details, let me try to explain why I think the PR is correct and why the breakage is ultimately caused by upstream mullvad. First, let me explain the motivation behind the PR.

Mullvad has four different network managers that it dispatches to, picking the first that doesn’t error.

  • systemd-resolved
  • networkmanger
  • resolvconf
  • static

where the first three are pretty obvious and static on connect modifies /etc/resolv.conf to contain the nameserver 10.64.0.1, copying the old contents to /etc/resolv.conf.mullvadbackup, and restores the old resolv.conf on disconnect, ignoring resolvconf entirely.

Before the PR, even if networking.resolvconf.enable = true, mullvad would still use static.

[talpid_core::dns][INFO] Setting DNS servers to 10.64.0.1
[talpid_core::dns::imp::static_resolv_conf][DEBUG] No DNS state backup to restore
[talpid_core::dns::imp][DEBUG] Managing DNS via /etc/resolv.conf

This is because mullvad directly checks the path for resolvconf, and the resolvconf wasn’t exposed on the path. So all the PR did was put resolvconf on the path if resolvconf was enabled.

path = [
  pkgs.iproute2
  # Needed for ping
  "/run/wrappers"
  # See https://github.com/NixOS/nixpkgs/issues/262681
] ++ (lib.optional config.networking.resolvconf.enable
  config.networking.resolvconf.package);

And now mullvad correctly uses resolvconf to manage DNS rather than overwriting it.

[talpid_core::dns][INFO] Setting DNS servers to 10.64.0.1
[talpid_core::dns::imp][DEBUG] Managing DNS via resolvconf

Now that we understand what the PR did, why does this break NetworkManager? Since NetworkManager is higher up on the list than both resolvconf and static, shouldn’t mullvad use the NetworkManager backend? Unfortunately, there are multiple reasons why it wouldn’t.

Before we talk about why, let’s see why the PR breaks what used to work. Since mullvad isn’t using NetworkManager, it checks whether resolvconf works. Before the PR, resolvconf never works, so it ends up at static. This allows mullvad to modify /etc/resolv.conf directly.

But after the PR, it goes through resolvconf which merges all the DNS nameservers it knows, namely,

  • default nameservers (like Google’s 8.8.8.8)
  • those discovered through DHCP (like from your ISP)
  • and mullvad’s nameservers (for wireguard, 10.64.0.1)

so now /etc/resolv.conf looks something like

nameserver ...
nameserver 8.8.8.8
nameserver ...
nameserver 10.64.0.1

where ... are IP addresses discovered with DHCP as opposed to

nameserver 10.64.0.1

when the manager was static and ignored the resolvconf configuration.

The fix is simple. Don’t use resolvconf so it falls back to the static codepath. This can be done with
networking.resolvconf.enable = false and DNS is properly managed whether mullvad is connected or not. Alternatively, enabling services.resolved.enable = true makes it so mullvad uses the systemd-resolved codepath which just works, ignoring both resolvconf and static codepaths.

But ideally mullvad would use the networkmanager codepath, and now we’ll see why that’s not the case. First, let’s go back to the warning

[talpid_dbus::network_manager][DEBUG] /etc/resolv.conf differs from reference resolv.conf, therefore NM is not managing DNS
[talpid_core::dns::imp][DEBUG] Managing DNS via resolvconf

It’s pretty straightforward why this is the case. /run/NetworkManager/resolv.conf looks like

# Generated by NetworkManager
nameserver ...
nameserver 8.8.8.8
nameserver ...

while /etc/resolv.conf might look like

# Generated by resolvconf
nameserver ...
nameserver 8.8.8.8
nameserver ...
options edns0

if it’s generated by resolvconf. To ensure NetworkManager is the only one touching resolv.conf, use

{
  networking = {
    networkmanager = {
      enable = true;
      dns = "default";
      extraConfig = ''
        [main]
        rc-manager=symlink
      '';
    };
    resolvconf.enable = false;
  };
}

where the option symlink or file instructs NetworkManager to directly write to /etc/resolv.conf.

symlink: If /etc/resolv.conf is a regular file or does not exist, NetworkManager will write the file directly. If /etc/resolv.conf is instead a symlink, NetworkManager will leave it alone. Unless the symlink points to the internal file /run/NetworkManager/resolv.conf, in which case the symlink will be updated to emit an inotify notification. This allows the user to conveniently instruct NetworkManager not to manage /etc/resolv.conf by replacing it with a symlink.
file: NetworkManager will write /etc/resolv.conf as regular file. If it finds a symlink to an existing target, it will follow the symlink and update the target instead. In no case will an existing symlink be replaced by a file. Note that older versions of NetworkManager behaved differently and would replace dangling symlinks with a plain file.

To check NetworkManager is behaving as expected, start it with

sudo NetworkManager --debug --log-level=TRACE --log-domains=DNS

Now /etc/resolv.conf should start with # Generated by NetworkManager and mullvad shouldn’t complain about it differing from the reference. But mullvad still refuses to use the NetworkManager codepath, saying Managing DNS via resolvconf.

If I’m tracing the code right (Rust isn’t my specialty), it calls ensure_can_be_used_to_manage_dns which consists of three checks

  • ensure_resolv_conf_is_managed
  • ensure_network_manager_exists
  • nm_version_dns_works

For the first check, it first checks the rc-manager option, then the dns option, and a familiar error.

if !verify_etc_resolv_conf_contents() {
    log::debug!("/etc/resolv.conf differs from reference resolv.conf, therefore NM is not managing DNS");
    return Err(Error::NetworkManagerNotManagingDns);
}

The second check just makes sure the version can be queried over dbus.

Finally, the third check makes sure the NetworkManager version is old enough. This is where it fails as the current version on unstable is 1.44.0 and the supported minor version has to between 16 and 26.

The dbus version query is

sudo busctl get-property org.freedesktop.NetworkManager /org/freedesktop/NetworkManager org.freedesktop.NetworkManager Version

and the dbus queries can be viewed with

mullavd disconnect
sudo busctl monitor org.freedesktop.NetworkManager
mullvad connect

which shows

‣ Type=method_call  Endian=l  Flags=0  Version=1 Cookie=16  Timestamp="Fri 2023-11-24 21:49:59.512038 UTC"
  Sender=:1.877  Destination=org.freedesktop.NetworkManager  Path=/org/freedesktop/NetworkManager/DnsManager  Interface=org.freedesktop.DBus.Properties  Member=Get
  UniqueName=:1.877
  MESSAGE "ss" {
          STRING "org.freedesktop.NetworkManager.DnsManager";
          STRING "RcManager";
  };

‣ Type=method_return  Endian=l  Flags=1  Version=1 Cookie=271  ReplyCookie=16  Timestamp="Fri 2023-11-24 21:49:59.512591 UTC"
  Sender=:1.879  Destination=:1.877
  UniqueName=:1.879
  MESSAGE "v" {
          VARIANT "s" {
                  STRING "symlink";
          };
  };
‣ Type=method_call  Endian=l  Flags=0  Version=1 Cookie=17  Timestamp="Fri 2023-11-24 21:49:59.512757 UTC"
  Sender=:1.877  Destination=org.freedesktop.NetworkManager  Path=/org/freedesktop/NetworkManager/DnsManager  Interface=org.freedesktop.DBus.Properties  Member=Get
  UniqueName=:1.877
  MESSAGE "ss" {
          STRING "org.freedesktop.NetworkManager.DnsManager";
          STRING "Mode";
  };

‣ Type=method_return  Endian=l  Flags=1  Version=1 Cookie=272  ReplyCookie=17  Timestamp="Fri 2023-11-24 21:49:59.513083 UTC"
  Sender=:1.879  Destination=:1.877
  UniqueName=:1.879
  MESSAGE "v" {
          VARIANT "s" {
                  STRING "default";
          };
  };

‣ Type=method_call  Endian=l  Flags=0  Version=1 Cookie=18  Timestamp="Fri 2023-11-24 21:49:59.513312 UTC"
  Sender=:1.877  Destination=org.freedesktop.NetworkManager  Path=/org/freedesktop/NetworkManager  Interface=org.freedesktop.DBus.Properties  Member=Get
  UniqueName=:1.877
  MESSAGE "ss" {
          STRING "org.freedesktop.NetworkManager";
          STRING "Version";
  };

‣ Type=method_return  Endian=l  Flags=1  Version=1 Cookie=273  ReplyCookie=18  Timestamp="Fri 2023-11-24 21:49:59.513594 UTC"
  Sender=:1.879  Destination=:1.877
  UniqueName=:1.879
  MESSAGE "v" {
          VARIANT "s" {
                  STRING "1.44.0";
          };
  };

‣ Type=method_call  Endian=l  Flags=0  Version=1 Cookie=19  Timestamp="Fri 2023-11-24 21:49:59.513756 UTC"
  Sender=:1.877  Destination=org.freedesktop.NetworkManager  Path=/org/freedesktop/NetworkManager  Interface=org.freedesktop.DBus.Properties  Member=Get
  UniqueName=:1.877
  MESSAGE "ss" {
          STRING "org.freedesktop.NetworkManager";
          STRING "Version";
  };

‣ Type=method_return  Endian=l  Flags=1  Version=1 Cookie=274  ReplyCookie=19  Timestamp="Fri 2023-11-24 21:49:59.513979 UTC"
  Sender=:1.879  Destination=:1.877
  UniqueName=:1.879
  MESSAGE "v" {
          VARIANT "s" {
                  STRING "1.44.0";
          };
  };

exactly as expected (a RcManager and Mode query, a Version query, and another Version query).

If someone wants to file an issue with mullvad, go ahead, as 1.26.0 is three years old at this point.

But I think the PR is correct — if resolvconf is enabled, it should be used.

In terms of a fix to the mullvad nixos module in nixpkgs, perhaps we could emit a warning if both networking.networkmanager.enable and networking.resolvconf.enable are true and services.resolved.enable is false. These are default settings so it covers most cases, but not all as if systemd-resolved is enabled and rc-manager is not the default of auto, there can still be issues.

6 Likes

apologies for the spam, but I’ve raised the issues

both in mullvad/mullvadvpn-app. I don’t think the mullavd nixos module needs to be modified.

Resolving the first issue should fix the problem for all NetworkManager configurations, with no modifications to your existing setup whatsoever (no need to set networking.resolvconf.enable = false or services.resolved.enable = true anymore).

I might have slightly overcomplicated the previous analyses. The real issue is that resolvconf loads
the mullvad nameserver 10.64.0.1 too late, i.e. it appears after NetworkManager nameservers.

nameserver ...
nameserver 8.8.8.8
nameserver ...
nameserver 10.64.0.1

As I describe in the issue, this is because the interface name is wg-mullvad.mullvad while openresolv’s dynamic_order which controls which order the interfaces are loaded in is

dynamic_order="tap[0-9]* tun[0-9]* vpn vpn[0-9]* wg[0-9]* ppp[0-9]* ippp[0-9]*"

which doesn’t match (since wg[0-9] expects a digit, like wg0-mullavd.mullvad, e.g.). This means it’s loaded lexographically, and NetworkManager comes before wg-mullvad.mullvad. This is also why openvpn works but not wireguard, since openvpn’s interface name is tun0.mullvad (at least on my computer) which is properly matched.

Whether this is considered a mullvad bug or a openresolv bug is a matter of taste. I chose to file the issue with mullvad since it’s easy to change and I trust openresolv is doing the “right thing”. Before a fix lands, one simple solution is to change dynamic_order to match wg-mullvad.mullvad,

networking.resolvconf.extraConfig = ''
  dynamic_order='tap[0-9]* tun[0-9]* vpn vpn[0-9]* wg* wg[0-9]* ppp[0-9]* ippp[0-9]*'
'';

i.e., introduce wg* into dynamic_order to match wg-mullvad.mullvad.

The second issue doesn’t fix any major problems but I figured I might as well. It turns out the NetworkManager version check is basically unnecessary and mullvad works perfectly fine on NetworkManager versions newer than 1.26. I say it’s unnecessary because as I’ve elaborated on length before, mullvad has four different dns managers (systemd-resolved, networkmanger, resolvconf, and static). If the networkmanger dns manager doesn’t work, then either the resolvconf manager (which should work after the first issue is fixed) or the static manager will be used and either should work fine.

Again, apologies for breaking people’s configurations and posting so many times. Hopefully mullvad will change its default interface name, if they don’t, then we can see if openresolv is wiling to change their default dynamic_order, and if they aren’t, the mullvad nixos module should set dynamic_order in resolvconf.extraConfig. I hope the issue, solution, and next steps are clear now!

3 Likes

I don’t regard this as spam, i regard this a deep drive into VPN and DNS technology.

There is a wealth of great information here.

Your analysis seems detailed and methodical.

Your an asset to any team that needs a 3rd line troubleshooter…

, either in a volunteer or paid basis.

DNS on it’s own it hard to get right, add a VPN , things get complex , quickly.

There also seems to be a bit of arugment who should be managing the resolver …

Being a systemd ULTRA , i believe it should be systemd.

Hi @vazae I just updated the flakes and the mullvad-vpn service is blocking all my connections. (github:NixOS/nixpkgs/9b19f5e77dd906cb52dade0b7bd280339d2a1f3d)

Any idea? Thanks!

There haven’t been any nontrivial commits to the nixos module recently as far as I can tell.

Mullvad was recently bumped from 2023.5 to 2023.6; I think they updated the setting format?

When I updated I removed the files in /etc/mullvad-vpn and restored my settings manually, e.g.

mullvad account login
mullvad auto-connect set on
mullvad relay set tunnel-protocol wireguard
mullvad relay set location ...

Hey. I just wanted to let you know that your recommended solution in the post that I am replying to was the only way I could get my mullvad VPN functional. Thank you very much for that! I can only hope a fix (from whichever side) comes out soon enough so that we could stop relying on such solutions.

1 Like