Declarative containers lose ability to resolve host names over time. Any tips for troubleshooting?

[boxofrox]

I set up a few declarative containers on a NixOS 20.09 server I manage, and I find that over time, they lose the ability to resolve host names.

If I check their /etc/resolv.conf shortly after a restart, I find a nameserver x.x.x.x entry, and ping google.com works.

e.g.

(host) # machinectl shell root@certs
(certs) # cat /etc/resolv.conf
# Generated by resolvconf
domain example.local
nameserver 192.168.70.1
options edns0
options edns0

If I check back a month or two later, and review the /etc/resolv.conf, then I find the nameserver entry missing, and ping google.com fails to resolve the host name.

e.g.

(host) # machinectl shell root@certs
(certs) # cat /etc/resolv.conf
# Generated by resolvconf
options edns0
options edns0

I can echo 'nameserver 192.168.70.1' >> /etc/resolv.conf and restore DNS resolution inside the container, but I’d like to identify the cause that is breaking my resolv.conf.

One declarative container I use is:

let
  options = { /* Redacted */ };

in
  containers.certs = {
    autoStart = true;

    bindMounts = {
      "${options.path}" = {
        hostPath = "/srv/certs";
        isReadOnly = false;
      };
    };

    config = { pkgs, ... }: {
      networking.firewall.enable = false;

      environment.systemPackages = [ pkgs.lego ];

      systemd = {
        services.certs = {
          description = "Renew certificates for hosts on the network.";

          after = [ "network-online.target" ];
          requires = [ "network-online.target" ];

          environment = envVars;

          path = [ pkgs.lego ];

          serviceConfig = {
            Type = "oneshot";
            User = "caddy";
            Group = "caddy";
          };

          script = concatStringsSep "\n" (map (cert: renewCert cert.domains) options.certs);

          preStart = ''
            if [[ ! -d "${options.path}/certificates" ]]; then
              mkdir -p "${options.path}/certificates"
              chmod 750 "${options.path}/certificates"
            fi

            ${concatStringsSep "\n" (map (cert: installCert cert.domains) options.certs)}
          '';
        };

        timers.certs = {
          description = "Try to renew certificates daily.";

          after = [ "network-online.target" ];
          requires = [ "network-online.target" ];
          wantedBy = [ "timers.target" ];

          timerConfig = {
            OnCalendar = "daily";
            Persistent = "true";
          };
        };
      };

      users.users.caddy = {
        uid = hostCfg.ids.uids.caddy;
        group = "caddy";
        createHome = false;
      };

      users.groups.caddy.gid = hostCfg.ids.gids.caddy;

    };
  };
...

This basically just checks the certificates once a day, and when they are within 30 days of expiring, then it connects to LetsEncrypt to renew the certificates. So for 60 days, this container doesn’t really touch the network.

So far I’ve tried:

1. Review journalctl -eb0 and journalctl -eb-1
   Only failures logged are related to my certificate service unable to resolve host names.
2. Review systemctl status and systemctl failed
   System is degraded.
   Only certs.service failed.
3. Identify resolve service(s).
   I have resolvconf.service. Status active (exited).
   I have nscd.service. Status active (running) (thawing). I’m unsure why it’s thawing…
4. In a separate container also afflicted with bad-resolv-syndrome, I manually restarted resolvconf.service and nscd.service, and neither updated the /etc/resolve.conf with a nameserver entry.

Does anyone have any recommendations where I should look further to identify the culprit?

[nixinator]

this rings a bell, I’m sure i fixed this problem at some stage…for now i can only give you this link for idea’s…

DNS in declarative container

I was able to fix this using hte bindMounts container option, like: containers.flx_web = { autoStart = true; privateNetwork = true; hostAddress = "192.168.100.10"; localAddress = "192.168.100.11"; bindMounts = { "/etc/resolv.conf" = { hostPath = "/etc/resolv.conf"; isReadOnly = true; }; };

this could also be systemd trying to resolve stuff, there is a bit of a bit of battle going on between the systemd resolver, and normal dns resolutions libraries on the machine.

like i said, just idea’s…

are you containers restarting at any time? could be systemd start up ordering within the container…something to watch for.

Make ACME renew systemd-service depend on DNS/nss-lookup?

I think adding nss-lookup.target here (in after and/or wants?) should work: https://github.com/NixOS/nixpkgs/blob/fbf553ea2b02cef3962480ed1929573de4429b56/nixos/modules/security/acme.nix#L319

[boxofrox]

Thank for the tips, I’ll give them a try.

@nixinator

this could also be systemd trying to resolve stuff, there is a bit of a bit of battle going on between the systemd resolver, and normal dns resolutions libraries on the machine.

Are you talking about systemd-resolved? Because I looked for that service, and it’s not used in the containers or the host.

@nixinator

are you containers restarting at any time? could be systemd start up ordering within the container…something to watch for.

When the name resolution fails, no, there’s no sign of containers restarting in the logs between when name resolution works and stops working.

If I manually restart a container after name resolution stops working, then the container’s /etc/resolv.conf has the appropriate nameserver entry and name resolution works (100% success for the dozen or so times I’ve tried).

It seems to me as if something in the container is managing the resolv.conf, gets triggered some time later, and forgets there’s a name server.

I’ll read up on these, try them, and report back. Thanks again!

[boxofrox]

Update:

I decided to bind mount the host’s resolv.conf. Adding the nss-lookup.target doesn’t seem like it’ll have an effect in my case. The containers I have that fail run repeating daily “cron” services, so I don’t care if they fail when the container starts, because eventually the nss-lookup.target will be met, and the service will have nss-lookup.target at its next “cron” run.

It’s probably going to take upwards of a month to know whether bind mouting resolv.conf will work. I applied the bind mount fix to my containers, and set up two containers to monitor/test name resolution over time and report to my desktop.

The basic setup is this.

The test containers will use ping to test name resolution against a LAN server (my router resolves my domains to local IPs internally) every minute. A pass/fail message is then sent with netcat via UDP to my desktop. My desktop filters the UDP messages through a few scripts to reduce spam in my terminals, and I have a “dashboard” that lets me monitor if/when name resolution drops out for either test container.

Here are the two test containers on my NixOS server.

  containers = let
    testConfig = containerName: alertPort: { pkgs, ... }: {
      networking.firewall.enable = false;

      environment.systemPackages = with pkgs; [
        netcat
        unixtools.ping
      ];

      systemd = {
        services.test = {
          description = "Test dns name resolution.";

          after = [ "network-online.target" ];
          requires = [ "network-online.target" ];

          path = with pkgs; [
            netcat
            unixtools.ping
          ];

          serviceConfig = {
            Type = "oneshot";
          };

          script = ''
            if ping -c 1 -w 1 my-server.com; then
              nc -Nu -w 1 my-desktop-ip ${toString alertPort} <<<"${containerName} Pass: Ping jitsi.lcent.com"
            else
              nc -Nu -w 1 my-desktop-ip ${toString alertPort} <<<"${containerName} Fail: Ping jitsi.lcent.com"
            fi
          '';
        };

        timers.test = {
          description = "Run test service periodically.";

          after = [ "network-online.target" ];
          requires = [ "network-online.target" ];
          wantedBy = [ "timers.target" ];

          timerConfig = {
            OnCalendar = "*-*-* *:00/1";
          };
        };
      };
    };
  in {
    test-dns-broken-config = {
      autoStart = true;

      config = testConfig "test-dns-broken-config" 9000;
    };

    test-dns-use-host-resolve = {
      autoStart = true;

      bindMounts = {
        "/etc/resolv.conf" = {
          hostPath = "/etc/resolv.conf";
          isReadOnly = true;
        };
      };

      config = testConfig "test-dns-use-host-resolve" 9001;
    };
  };

On my desktop, I use a shell script and awk script to add timestamps, log to file, and consolidate duplicate lines before displaying.

# monitor
#!/bin/bash

PORT=${1:-9000}

# netcat only connected to one host.  Use socat to allow multiple hosts to connect.
socat -u udp-recv:$PORT,reuseport stdout \
  | while read LINE; do \
    date -Is | tr -d "\n"; \
    echo -n " "; \
    echo $LINE; \
  done \
  | tee logs/port-$PORT.log \
  | awk -f filter-uniq.awk

# filter-uniq.awk
function rtrim(s) { sub(/[ \t\r\n]+$/, "", s); return s }

BEGIN {
  last="xxx"; count=0;
}

{
  line = $0;
  regex = last"$";

  if (line ~ regex) {
    count = count + 1;
    printf("\x1b[K%s %d\r", line, count);
  } else {
    count = 1;
    last = rtrim(sprintf("%s %s %s %s", $2, $3, $4, $5));
    printf("\n%s\r", line);
  }
}

With this, I can set up two shells in tmux to monitor ports 9000 and 9001 for the following live report.

┌───────────────────────────────────────────────────────────────────────────────────┐
│ ./monitor 9000                                                                    │
│                                                                                   │
│ 2021-04-01T02:17:09-04:00 test-dns-broken-config Pass: Ping my-server.com 192     │
│ 2021-04-01T02:20:10-04:00 test-dns-broken-config Fail: Ping my-server.com 3       │
│ 2021-04-02T04:47:09-04:00 test-dns-broken-config Pass: Ping my-server.com 1587    │
│ 2021-04-02T04:52:10-04:00 test-dns-broken-config Fail: Ping my-server.com 5       │
│ 2021-04-04T12:22:09-04:00 test-dns-broken-config Pass: Ping my-server.com 3330    │
├───────────────────────────────────────────────────────────────────────────────────┤
│                                                                                   │
│ ./monitor 9001                                                                    │
│                                                                                   │
│ 2021-04-01T02:16:08-04:00 test-dns-use-host-resolve Pass: Ping my-server.com 156  │
│ 2021-04-01T02:19:09-04:00 test-dns-use-host-resolve Fail: Ping my-server.com 3    │
│ 2021-04-02T04:47:35-04:00 test-dns-use-host-resolve Pass: Ping my-server.com 1588 │
│ 2021-04-02T04:52:09-04:00 test-dns-use-host-resolve Fail: Ping my-server.com 5    │
│ 2021-04-04T12:24:08-04:00 test-dns-use-host-resolve Pass: Ping my-server.com 3332 │
└───────────────────────────────────────────────────────────────────────────────────┘

A few spurious failures appear here and there, but I think those can be attributed to the “wait 1 second” option I applied to ping (ping -w 1) not waiting long enough for slow dns responses.

We’ll see how this turns out.

Cheers.

[boxofrox]

I do wonder though, if constantly pinging will cause the dns records to be cached and delay the point of failure in the monitor report.

I should look into flushing the dns cache after each ping…

[boxofrox]

The results are in. On 2021-04-28 at 15:51, the test-dns-broken-config container lost its ability to resolve domain names 5426 consecutive times. The other test container that bind mounted the hosts /etc/resolv.conf continues to resolve domain names. I’ll keep the second going to see if it ever drops out (it did have a blackout of 567 minutes before recovering) permanently.

Thanks for providing the fix, @nixinator.

[nixinator]

thanks for providing good debug, and doing some tests, but most of all, thanks for feeding back the results… i’m sure this will help others in the future…