ACME with Route53 DNS Challenge Stopped Working

I setup ACME previously with this configuration:

  # LetsEncrypt
  security.acme = {
    acceptTerms = true;
    defaults.email = "letsencrypt@example.com";
    certs."${config.networking.hostName}.example.com" = {
      dnsProvider = "route53";
      credentialsFile = "/example/dns.txt";
    };
  };

All of this was working fine including periodic renewal. Recently I got an email from LetsEncrypt saying my certificate will expire soon. Nothing was changed or updated on the NixOS side of things so I’m not sure what would have caused it to stop working. I also know the same AWS credentials are working for the DNS challenge on other machines in my same local network. It also looks like the actual interaction with AWS is working but the challenge is still failing overall.

I tried running the request process manually with sudo systemctl restart acme-host.example.com and see these errors in journalctl -xeu acme-host.example.com.service:

Jul 03 08:51:47 host acme-host.example.com-start[907294]: 2024/07/03 08:51:47 [INFO] [host.example.com] acme: use dns-01 solver
Jul 03 08:51:47 host acme-host.example.com-start[907294]: 2024/07/03 08:51:47 [INFO] [host.example.com] acme: Preparing to solve DNS-01
Jul 03 08:51:47 host acme-host.example.com-start[907294]: 2024/07/03 08:51:47 [INFO] Wait for route53 [timeout: 2m0s, interval: 4s]
Jul 03 08:52:20 host acme-host.example.com-start[907294]: 2024/07/03 08:52:20 [INFO] [host.example.com] acme: Trying to solve DNS-01
Jul 03 08:52:20 host acme-host.example.com-start[907294]: 2024/07/03 08:52:20 [INFO] [host.example.com] acme: Checking DNS record propagation. [nameservers=127.0.0.53:53]
Jul 03 08:52:24 host acme-host.example.com-start[907294]: 2024/07/03 08:52:24 [INFO] Wait for propagation [timeout: 2m0s, interval: 4s]
Jul 03 08:52:24 host acme-host.example.com-start[907294]: 2024/07/03 08:52:24 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:52:28 host acme-host.example.com-start[907294]: 2024/07/03 08:52:28 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:52:32 host acme-host.example.com-start[907294]: 2024/07/03 08:52:32 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:52:36 host acme-host.example.com-start[907294]: 2024/07/03 08:52:36 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:52:40 host acme-host.example.com-start[907294]: 2024/07/03 08:52:40 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:52:44 host acme-host.example.com-start[907294]: 2024/07/03 08:52:44 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:52:48 host acme-host.example.com-start[907294]: 2024/07/03 08:52:48 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:52:52 host acme-host.example.com-start[907294]: 2024/07/03 08:52:52 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:52:56 host acme-host.example.com-start[907294]: 2024/07/03 08:52:56 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:00 host acme-host.example.com-start[907294]: 2024/07/03 08:53:00 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:04 host acme-host.example.com-start[907294]: 2024/07/03 08:53:04 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:08 host acme-host.example.com-start[907294]: 2024/07/03 08:53:08 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:12 host acme-host.example.com-start[907294]: 2024/07/03 08:53:12 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:16 host acme-host.example.com-start[907294]: 2024/07/03 08:53:16 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:20 host acme-host.example.com-start[907294]: 2024/07/03 08:53:20 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:24 host acme-host.example.com-start[907294]: 2024/07/03 08:53:24 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:28 host acme-host.example.com-start[907294]: 2024/07/03 08:53:28 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:32 host acme-host.example.com-start[907294]: 2024/07/03 08:53:32 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:36 host acme-host.example.com-start[907294]: 2024/07/03 08:53:36 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:40 host acme-host.example.com-start[907294]: 2024/07/03 08:53:40 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:44 host acme-host.example.com-start[907294]: 2024/07/03 08:53:44 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:48 host acme-host.example.com-start[907294]: 2024/07/03 08:53:48 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:52 host acme-host.example.com-start[907294]: 2024/07/03 08:53:52 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:56 host acme-host.example.com-start[907294]: 2024/07/03 08:53:56 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:54:00 host acme-host.example.com-start[907294]: 2024/07/03 08:54:00 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:54:04 host acme-host.example.com-start[907294]: 2024/07/03 08:54:04 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:54:08 host acme-host.example.com-start[907294]: 2024/07/03 08:54:08 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:54:12 host acme-host.example.com-start[907294]: 2024/07/03 08:54:12 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:54:16 host acme-host.example.com-start[907294]: 2024/07/03 08:54:16 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:54:20 host acme-host.example.com-start[907294]: 2024/07/03 08:54:20 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:54:24 host acme-host.example.com-start[907294]: 2024/07/03 08:54:24 [INFO] [host.example.com] acme: Cleaning DNS-01 challenge
Jul 03 08:54:24 host acme-host.example.com-start[907294]: 2024/07/03 08:54:24 [INFO] Wait for route53 [timeout: 2m0s, interval: 4s]
Jul 03 08:54:57 host acme-host.example.com-start[907294]: 2024/07/03 08:54:57 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/372167621707
Jul 03 08:54:57 host acme-host.example.com-start[907294]: 2024/07/03 08:54:57 Could not obtain certificates:
Jul 03 08:54:57 host acme-host.example.com-start[907294]:         error: one or more domains had a problem:
Jul 03 08:54:57 host acme-host.example.com-start[907294]: [host.example.com] propagation: time limit exceeded: last error: DNS call error: read udp [::1]:40418->[::1]:53: read: connection refused [ns=localhost.:53, question='_acme-challenge.host.example.com. IN  TXT']
Jul 03 08:54:57 host acme-host.example.com-start[907291]: + echo Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Jul 03 08:54:57 host acme-host.example.com-start[907291]: Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Jul 03 08:54:57 host acme-host.example.com-start[907291]: + exit 10

Does anyone know what could cause this or how to fix it?

Looks like the script is querying a DNS server at addess 127.0.0.53:53 and can’t for some reason: read: connection refused [ns=localhost.:53, question='_acme-challenge.host.example.com. IN TXT'].

Maybe you can try something like dig -t txt _acme-challenge.host.example.com @127.0.0.53 (or just use some generic DNS requests since it seems to be unable to connect to your resolver). These should succeed or there might be something wrong with your nameserver configuration.

1 Like

Thanks, that helped me figure it out. I’m not sure why the “connection refused” error was happening, it seems like it was just a DNS issue. I think it had to do with my LAN router handling DNS locally for “example.com” to give the local LAN IP addresses, and the DNS queries not getting the actual DNS TXT records from Amazon. I don’t think any of the other processes I’m using for DNS challenges work exactly like this so they were not affected. I ended up fixing it by setting security.acme.defaults.dnsResolver to an external public DNS just for this check, and now it works normally.

1 Like