ACME with Route53 DNS Challenge Stopped Working

Help
1

I setup ACME previously with this configuration:

  # LetsEncrypt
  security.acme = {
    acceptTerms = true;
    defaults.email = "letsencrypt@example.com";
    certs."${config.networking.hostName}.example.com" = {
      dnsProvider = "route53";
      credentialsFile = "/example/dns.txt";
    };
  };

All of this was working fine including periodic renewal. Recently I got an email from LetsEncrypt saying my certificate will expire soon. Nothing was changed or updated on the NixOS side of things so I’m not sure what would have caused it to stop working. I also know the same AWS credentials are working for the DNS challenge on other machines in my same local network. It also looks like the actual interaction with AWS is working but the challenge is still failing overall.

I tried running the request process manually with sudo systemctl restart acme-host.example.com and see these errors in journalctl -xeu acme-host.example.com.service:

Jul 03 08:51:47 host acme-host.example.com-start[907294]: 2024/07/03 08:51:47 [INFO] [host.example.com] acme: use dns-01 solver
Jul 03 08:51:47 host acme-host.example.com-start[907294]: 2024/07/03 08:51:47 [INFO] [host.example.com] acme: Preparing to solve DNS-01
Jul 03 08:51:47 host acme-host.example.com-start[907294]: 2024/07/03 08:51:47 [INFO] Wait for route53 [timeout: 2m0s, interval: 4s]
Jul 03 08:52:20 host acme-host.example.com-start[907294]: 2024/07/03 08:52:20 [INFO] [host.example.com] acme: Trying to solve DNS-01
Jul 03 08:52:20 host acme-host.example.com-start[907294]: 2024/07/03 08:52:20 [INFO] [host.example.com] acme: Checking DNS record propagation. [nameservers=127.0.0.53:53]
Jul 03 08:52:24 host acme-host.example.com-start[907294]: 2024/07/03 08:52:24 [INFO] Wait for propagation [timeout: 2m0s, interval: 4s]
Jul 03 08:52:24 host acme-host.example.com-start[907294]: 2024/07/03 08:52:24 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:52:28 host acme-host.example.com-start[907294]: 2024/07/03 08:52:28 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:52:32 host acme-host.example.com-start[907294]: 2024/07/03 08:52:32 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:52:36 host acme-host.example.com-start[907294]: 2024/07/03 08:52:36 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:52:40 host acme-host.example.com-start[907294]: 2024/07/03 08:52:40 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:52:44 host acme-host.example.com-start[907294]: 2024/07/03 08:52:44 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:52:48 host acme-host.example.com-start[907294]: 2024/07/03 08:52:48 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:52:52 host acme-host.example.com-start[907294]: 2024/07/03 08:52:52 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:52:56 host acme-host.example.com-start[907294]: 2024/07/03 08:52:56 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:00 host acme-host.example.com-start[907294]: 2024/07/03 08:53:00 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:04 host acme-host.example.com-start[907294]: 2024/07/03 08:53:04 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:08 host acme-host.example.com-start[907294]: 2024/07/03 08:53:08 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:12 host acme-host.example.com-start[907294]: 2024/07/03 08:53:12 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:16 host acme-host.example.com-start[907294]: 2024/07/03 08:53:16 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:20 host acme-host.example.com-start[907294]: 2024/07/03 08:53:20 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:24 host acme-host.example.com-start[907294]: 2024/07/03 08:53:24 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:28 host acme-host.example.com-start[907294]: 2024/07/03 08:53:28 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:32 host acme-host.example.com-start[907294]: 2024/07/03 08:53:32 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:36 host acme-host.example.com-start[907294]: 2024/07/03 08:53:36 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:40 host acme-host.example.com-start[907294]: 2024/07/03 08:53:40 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:44 host acme-host.example.com-start[907294]: 2024/07/03 08:53:44 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:48 host acme-host.example.com-start[907294]: 2024/07/03 08:53:48 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:52 host acme-host.example.com-start[907294]: 2024/07/03 08:53:52 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:53:56 host acme-host.example.com-start[907294]: 2024/07/03 08:53:56 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:54:00 host acme-host.example.com-start[907294]: 2024/07/03 08:54:00 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:54:04 host acme-host.example.com-start[907294]: 2024/07/03 08:54:04 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:54:08 host acme-host.example.com-start[907294]: 2024/07/03 08:54:08 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:54:12 host acme-host.example.com-start[907294]: 2024/07/03 08:54:12 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:54:16 host acme-host.example.com-start[907294]: 2024/07/03 08:54:16 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:54:20 host acme-host.example.com-start[907294]: 2024/07/03 08:54:20 [INFO] [host.example.com] acme: Waiting for DNS record propagation.
Jul 03 08:54:24 host acme-host.example.com-start[907294]: 2024/07/03 08:54:24 [INFO] [host.example.com] acme: Cleaning DNS-01 challenge
Jul 03 08:54:24 host acme-host.example.com-start[907294]: 2024/07/03 08:54:24 [INFO] Wait for route53 [timeout: 2m0s, interval: 4s]
Jul 03 08:54:57 host acme-host.example.com-start[907294]: 2024/07/03 08:54:57 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/372167621707
Jul 03 08:54:57 host acme-host.example.com-start[907294]: 2024/07/03 08:54:57 Could not obtain certificates:
Jul 03 08:54:57 host acme-host.example.com-start[907294]:         error: one or more domains had a problem:
Jul 03 08:54:57 host acme-host.example.com-start[907294]: [host.example.com] propagation: time limit exceeded: last error: DNS call error: read udp [::1]:40418->[::1]:53: read: connection refused [ns=localhost.:53, question='_acme-challenge.host.example.com. IN  TXT']
Jul 03 08:54:57 host acme-host.example.com-start[907291]: + echo Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Jul 03 08:54:57 host acme-host.example.com-start[907291]: Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Jul 03 08:54:57 host acme-host.example.com-start[907291]: + exit 10

Does anyone know what could cause this or how to fix it?

2

Looks like the script is querying a DNS server at addess 127.0.0.53:53 and can’t for some reason: read: connection refused [ns=localhost.:53, question='_acme-challenge.host.example.com. IN TXT'].

Maybe you can try something like dig -t txt _acme-challenge.host.example.com @127.0.0.53 (or just use some generic DNS requests since it seems to be unable to connect to your resolver). These should succeed or there might be something wrong with your nameserver configuration.

1 Like
3

Thanks, that helped me figure it out. I’m not sure why the “connection refused” error was happening, it seems like it was just a DNS issue. I think it had to do with my LAN router handling DNS locally for “example.com” to give the local LAN IP addresses, and the DNS queries not getting the actual DNS TXT records from Amazon. I don’t think any of the other processes I’m using for DNS challenges work exactly like this so they were not affected. I ended up fixing it by setting security.acme.defaults.dnsResolver to an external public DNS just for this check, and now it works normally.

2 Likes
4

I had this same thing happening to me today so thank you for the post. My path to solution was a bit different though and I thought I’d share in case someone else is having the same issue.

In my case, the DNS lookup for the _acme-challenge subdomain was getting routed to a local network DNS server because of a split horizon setup. Obviously, the local DNS isn’t helpful for DNS challenge verifications. That created a bit of a puzzle though. I need the resolver for the acme cert to go to a public DNS where the txt record is being created but all the other server traffic to follow the configured resolutions for the local network. Turns out there’s a configuration option for this specifically.

This is pretty simple solve that doesn’t require you to alter the DNS configurations for he LAN.

  security.acme = {
    acceptTerms = true;
    defaults.email = "sum-rock@example.com";
    defaults = {
      dnsProvider = "namecheap";
      dnsResolver = "8.8.8.8:53"; # Solution here. Specify the DNS resolver for txt lookups
      credentialsFile = "/my/cert/path/certs.secret";
      dnsPropagationCheck = true;
    };
  };

P.S. I’m reasonably confident that this was not required prior to June, 2024. I’m unaware of anything that changed in the LAN that would make this stop working. Of course, sometimes networking is a dark art so maybe I’m just blind to the thing that caused this to break. But I do know that my proxy was configured without this option for over a year and certs were renewing just fine.