update: I played around with NetworkManager myself.
tl;dr set networking.resolvconf.enable = false. Indeed, this exactly reverts the PR.
If you want to use resolvconf/systemd-resolved, then set services.resolved.enable = true.
For those interested in the details, let me try to explain why I think the PR is correct and why the breakage is ultimately caused by upstream mullvad. First, let me explain the motivation behind the PR.
Mullvad has four different network managers that it dispatches to, picking the first that doesn’t error.
- systemd-resolved
- networkmanger
- resolvconf
- static
where the first three are pretty obvious and static on connect modifies /etc/resolv.conf to contain the nameserver 10.64.0.1, copying the old contents to /etc/resolv.conf.mullvadbackup, and restores the old resolv.conf on disconnect, ignoring resolvconf entirely.
Before the PR, even if networking.resolvconf.enable = true, mullvad would still use static.
[talpid_core::dns][INFO] Setting DNS servers to 10.64.0.1
[talpid_core::dns::imp::static_resolv_conf][DEBUG] No DNS state backup to restore
[talpid_core::dns::imp][DEBUG] Managing DNS via /etc/resolv.conf
This is because mullvad directly checks the path for resolvconf, and the resolvconf wasn’t exposed on the path. So all the PR did was put resolvconf on the path if resolvconf was enabled.
path = [
pkgs.iproute2
# Needed for ping
"/run/wrappers"
# See https://github.com/NixOS/nixpkgs/issues/262681
] ++ (lib.optional config.networking.resolvconf.enable
config.networking.resolvconf.package);
And now mullvad correctly uses resolvconf to manage DNS rather than overwriting it.
[talpid_core::dns][INFO] Setting DNS servers to 10.64.0.1
[talpid_core::dns::imp][DEBUG] Managing DNS via resolvconf
Now that we understand what the PR did, why does this break NetworkManager? Since NetworkManager is higher up on the list than both resolvconf and static, shouldn’t mullvad use the NetworkManager backend? Unfortunately, there are multiple reasons why it wouldn’t.
Before we talk about why, let’s see why the PR breaks what used to work. Since mullvad isn’t using NetworkManager, it checks whether resolvconf works. Before the PR, resolvconf never works, so it ends up at static. This allows mullvad to modify /etc/resolv.conf directly.
But after the PR, it goes through resolvconf which merges all the DNS nameservers it knows, namely,
- default nameservers (like Google’s
8.8.8.8)
- those discovered through DHCP (like from your ISP)
- and mullvad’s nameservers (for wireguard,
10.64.0.1)
so now /etc/resolv.conf looks something like
nameserver ...
nameserver 8.8.8.8
nameserver ...
nameserver 10.64.0.1
where ... are IP addresses discovered with DHCP as opposed to
nameserver 10.64.0.1
when the manager was static and ignored the resolvconf configuration.
The fix is simple. Don’t use resolvconf so it falls back to the static codepath. This can be done with
networking.resolvconf.enable = false and DNS is properly managed whether mullvad is connected or not. Alternatively, enabling services.resolved.enable = true makes it so mullvad uses the systemd-resolved codepath which just works, ignoring both resolvconf and static codepaths.
But ideally mullvad would use the networkmanager codepath, and now we’ll see why that’s not the case. First, let’s go back to the warning
[talpid_dbus::network_manager][DEBUG] /etc/resolv.conf differs from reference resolv.conf, therefore NM is not managing DNS
[talpid_core::dns::imp][DEBUG] Managing DNS via resolvconf
It’s pretty straightforward why this is the case. /run/NetworkManager/resolv.conf looks like
# Generated by NetworkManager
nameserver ...
nameserver 8.8.8.8
nameserver ...
while /etc/resolv.conf might look like
# Generated by resolvconf
nameserver ...
nameserver 8.8.8.8
nameserver ...
options edns0
if it’s generated by resolvconf. To ensure NetworkManager is the only one touching resolv.conf, use
{
networking = {
networkmanager = {
enable = true;
dns = "default";
extraConfig = ''
[main]
rc-manager=symlink
'';
};
resolvconf.enable = false;
};
}
where the option symlink or file instructs NetworkManager to directly write to /etc/resolv.conf.
symlink: If /etc/resolv.conf is a regular file or does not exist, NetworkManager will write the file directly. If /etc/resolv.conf is instead a symlink, NetworkManager will leave it alone. Unless the symlink points to the internal file /run/NetworkManager/resolv.conf, in which case the symlink will be updated to emit an inotify notification. This allows the user to conveniently instruct NetworkManager not to manage /etc/resolv.conf by replacing it with a symlink.
file: NetworkManager will write /etc/resolv.conf as regular file. If it finds a symlink to an existing target, it will follow the symlink and update the target instead. In no case will an existing symlink be replaced by a file. Note that older versions of NetworkManager behaved differently and would replace dangling symlinks with a plain file.
To check NetworkManager is behaving as expected, start it with
sudo NetworkManager --debug --log-level=TRACE --log-domains=DNS
Now /etc/resolv.conf should start with # Generated by NetworkManager and mullvad shouldn’t complain about it differing from the reference. But mullvad still refuses to use the NetworkManager codepath, saying Managing DNS via resolvconf.
If I’m tracing the code right (Rust isn’t my specialty), it calls ensure_can_be_used_to_manage_dns which consists of three checks
ensure_resolv_conf_is_managedensure_network_manager_existsnm_version_dns_works
For the first check, it first checks the rc-manager option, then the dns option, and a familiar error.
if !verify_etc_resolv_conf_contents() {
log::debug!("/etc/resolv.conf differs from reference resolv.conf, therefore NM is not managing DNS");
return Err(Error::NetworkManagerNotManagingDns);
}
The second check just makes sure the version can be queried over dbus.
Finally, the third check makes sure the NetworkManager version is old enough. This is where it fails as the current version on unstable is 1.44.0 and the supported minor version has to between 16 and 26.
The dbus version query is
sudo busctl get-property org.freedesktop.NetworkManager /org/freedesktop/NetworkManager org.freedesktop.NetworkManager Version
and the dbus queries can be viewed with
mullavd disconnect
sudo busctl monitor org.freedesktop.NetworkManager
mullvad connect
which shows
‣ Type=method_call Endian=l Flags=0 Version=1 Cookie=16 Timestamp="Fri 2023-11-24 21:49:59.512038 UTC"
Sender=:1.877 Destination=org.freedesktop.NetworkManager Path=/org/freedesktop/NetworkManager/DnsManager Interface=org.freedesktop.DBus.Properties Member=Get
UniqueName=:1.877
MESSAGE "ss" {
STRING "org.freedesktop.NetworkManager.DnsManager";
STRING "RcManager";
};
‣ Type=method_return Endian=l Flags=1 Version=1 Cookie=271 ReplyCookie=16 Timestamp="Fri 2023-11-24 21:49:59.512591 UTC"
Sender=:1.879 Destination=:1.877
UniqueName=:1.879
MESSAGE "v" {
VARIANT "s" {
STRING "symlink";
};
};
‣ Type=method_call Endian=l Flags=0 Version=1 Cookie=17 Timestamp="Fri 2023-11-24 21:49:59.512757 UTC"
Sender=:1.877 Destination=org.freedesktop.NetworkManager Path=/org/freedesktop/NetworkManager/DnsManager Interface=org.freedesktop.DBus.Properties Member=Get
UniqueName=:1.877
MESSAGE "ss" {
STRING "org.freedesktop.NetworkManager.DnsManager";
STRING "Mode";
};
‣ Type=method_return Endian=l Flags=1 Version=1 Cookie=272 ReplyCookie=17 Timestamp="Fri 2023-11-24 21:49:59.513083 UTC"
Sender=:1.879 Destination=:1.877
UniqueName=:1.879
MESSAGE "v" {
VARIANT "s" {
STRING "default";
};
};
‣ Type=method_call Endian=l Flags=0 Version=1 Cookie=18 Timestamp="Fri 2023-11-24 21:49:59.513312 UTC"
Sender=:1.877 Destination=org.freedesktop.NetworkManager Path=/org/freedesktop/NetworkManager Interface=org.freedesktop.DBus.Properties Member=Get
UniqueName=:1.877
MESSAGE "ss" {
STRING "org.freedesktop.NetworkManager";
STRING "Version";
};
‣ Type=method_return Endian=l Flags=1 Version=1 Cookie=273 ReplyCookie=18 Timestamp="Fri 2023-11-24 21:49:59.513594 UTC"
Sender=:1.879 Destination=:1.877
UniqueName=:1.879
MESSAGE "v" {
VARIANT "s" {
STRING "1.44.0";
};
};
‣ Type=method_call Endian=l Flags=0 Version=1 Cookie=19 Timestamp="Fri 2023-11-24 21:49:59.513756 UTC"
Sender=:1.877 Destination=org.freedesktop.NetworkManager Path=/org/freedesktop/NetworkManager Interface=org.freedesktop.DBus.Properties Member=Get
UniqueName=:1.877
MESSAGE "ss" {
STRING "org.freedesktop.NetworkManager";
STRING "Version";
};
‣ Type=method_return Endian=l Flags=1 Version=1 Cookie=274 ReplyCookie=19 Timestamp="Fri 2023-11-24 21:49:59.513979 UTC"
Sender=:1.879 Destination=:1.877
UniqueName=:1.879
MESSAGE "v" {
VARIANT "s" {
STRING "1.44.0";
};
};
exactly as expected (a RcManager and Mode query, a Version query, and another Version query).
If someone wants to file an issue with mullvad, go ahead, as 1.26.0 is three years old at this point.
But I think the PR is correct — if resolvconf is enabled, it should be used.
In terms of a fix to the mullvad nixos module in nixpkgs, perhaps we could emit a warning if both networking.networkmanager.enable and networking.resolvconf.enable are true and services.resolved.enable is false. These are default settings so it covers most cases, but not all as if systemd-resolved is enabled and rc-manager is not the default of auto, there can still be issues.