No DNS in initrd. 1.1.1.1: resolve call failed: All attempts to contact name servers or networks failed

I’m trying to setup remote disk unlocking and am running into some networking issues in the initrd. I have internet, but no DNS.
Config at codeberg.org/jackr/nixos. (unfortunately down at time of posting)

My network setup:
-bash-5.3# resolvectl

Global
         Protocols: +LLMNR +mDNS +DNSOverTLS DNSSEC=yes/supported
  resolv.conf mode: stub
Current DNS Server: 194.242.2.4#base.dns.mullvad.net
       DNS Servers: 194.242.2.3#adblock.dns.mullvad.net 194.242.2.4#base.dns.mullvad.net 194.242.2.2#dns.mullvad.net
        DNS Domain: ~.

Link 2 (enp5s0)
    Current Scopes: LLMNR/IPv4 LLMNR/IPv6
         Protocols: -DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=yes/supported
     Default Route: no

-bash-5.3# networkctl status

● Interfaces: 1, 2
       State: routable                           
Online state: online                             
     Address: 192.168.2.234 on enp5s0
              fe80::1549:bc73:7c42:a3df on enp5s0
              fe80::da43:aeff:fe65:ffee on enp5s0
     Gateway: 192.168.2.1 on enp5s0

I can ping just fine:
-bash-5.3# ping 8.8.8.8

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=5.41 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=119 time=5.80 ms

and there are no obvious errors that jump out to me in the debug logs…
-bash-5.3# SYSTEMD_LOG_LEVEL=debug resolvectl query 1.1.1.1

Bus n/a: changing state UNSET → OPENING
sd-bus: starting bus by connecting to /run/dbus/system_bus_socket...
Bus n/a: changing state OPENING → AUTHENTICATING
Resolving 1.1.1.1.
Bus n/a: changing state AUTHENTICATING → HELLO
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a
Got message type=method_return sender=org.freedesktop.DBus destination=:1.21 path=n/a interface=n/a member=n/a  cookie=1 reply_cookie=1 signature=s error-name=n/a error-message=n/a
Bus n/a: changing state HELLO → RUNNING
Sent message type=method_call sender=n/a destination=org.freedesktop.resolve1 path=/org/freedesktop/resolve1 interface=org.freedesktop.resolve1.Manager member=ResolveAddress cookie=2 reply_cookie=0 signature=iiayt error-name=n/a error-message=n/a
Got message type=error sender=:1.0 destination=:1.21 path=n/a interface=n/a member=n/a  cookie=326 reply_cookie=2 signature=s error-name=org.freedesktop.DBus.Error.Timeout error-message=All attempts to contact name servers or networks failed
1.1.1.1: resolve call failed: All attempts to contact name servers or networks failed
Bus n/a: changing state RUNNING → CLOSED

happy to share more if needed :slight_smile:

That’s only setting the log level for resolvectl not systemd-resolved. The correct way to set the log level for systemd-resolved is to run

systemctl service-log-level systemd-resolved debug

in the initrd.

Also, what’s in /etc/ssl/certs?

1 Like

debug logs:

/etc/ssl/certs does not exist…

for @MagicRB:

  • ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether d8:43:ae:65:ff:ee brd ff:ff:ff:ff:ff:ff
    altname enxd843ae65ffee
    inet 192.168.2.234/24 brd 192.168.2.255 scope global enp5s0
       valid_lft forever preferred_lft forever
    inet6 fe80::1549:bc73:7c42:a3df/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::da43:aeff:fe65:ffee/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
  • Content of /etc/resolv.conf
nameserver 127.0.0.53
options edns0 trust-ad
search .
  • resolvectl
    above ^

Did you try to resolve an address after restarting systemd-resolved?

yes, nothing in the logs

It appears that restarting a service resets the log level.

To get debug-level logging from startup for a daemon implemented by the systemd project, add an override file for the service unit (e.g. with systemctl edit <daemon>.service) containing

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

and then restart the service. (With immutable /etc you can still use systemctl edit --runtime to make changes that will persist until reboot.)

2 Likes

ah, looks like it’s an ssl error:

systemd-resolved[301]: No static record files changed, not re-reading.
systemd-resolved[301]: Hook /run/systemd/resolve.hook/io.systemd.Network does not match question, skipping.
systemd-resolved[301]: Cache miss for 1.1.1.1.in-addr.arpa IN PTR
systemd-resolved[301]: Firing regular transaction 11191 for <1.1.1.1.in-addr.arpa IN PTR> scope dns on */* (validate=yes).
systemd-resolved[301]: Using feature level TLS+EDNS0+DO for transaction 11191.
systemd-resolved[301]: Using DNS server 194.242.2.3#adblock.dns.mullvad.net for transaction 11191.
systemd-resolved[301]: Sending query via TCP since UDP isn't supported or DNS-over-TLS is selected.
systemd-resolved[301]: Using feature level TLS+EDNS0+DO for transaction 11191.
systemd-resolved[301]: Announcing packet size 1472 in egress EDNS(0) packet.
systemd-resolved[301]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.0 path=n/a interface=n/a member=n/a  cookie=39 reply_cookie=168 signature=n/a error-name=n/a error-message=n/a
systemd-resolved[301]: Match type='signal',sender='org.freedesktop.DBus',path='/org/freedesktop/DBus',interface='org.freedesktop.DBus',member='NameOwnerChanged',arg0=':1.5' successfully installed.
systemd-resolved[301]: Failed to invoke SSL_do_handshake: error:0A000086:SSL routines::certificate verify failed
systemd-resolved[301]: Connection failure for DNS TCP stream: Connection refused
systemd-resolved[301]: Retrying transaction 11191, after switching servers.
systemd-resolved[301]: Switching to system DNS server 194.242.2.4#base.dns.mullvad.net.
systemd-resolved[301]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=170 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a

It appears that you might need to add TLS certificates to the initrd, this can be done with

{ config, ... }: {
  boot.initrd.systemd.contents = {
    "/etc/ssl/certs/ca-certificates.crt".source = config.security.pki.caBundle;
    "/etc/ssl/certs/ca-bundle.crt".source = config.security.pki.caBundle;
    "/etc/pki/tls/certs/ca-bundle.crt".source = config.security.pki.caBundle;
  };
}
1 Like

Looks like this was it, thanks a bunch!