ACME and NGINX not working as expected

I am attempting to serve a static site from a NixOS box. Between the manual, wiki, etc I have compiled the following config:

{ ... }: {
  networking.firewall.allowedTCPPorts = [ 80 443 ];

  services.nginx = {
    enable = true;
    logError = "stderr info";
    recommendedGzipSettings = true;
    recommendedOptimisation = true;
    recommendedProxySettings = true;
    recommendedTlsSettings = true;
    virtualHosts."bitbop.io" = {
      default = true;
      enableACME = true;
      forceSSL = true;
      listen = [
        { addr = "0.0.0.0"; port = 443; ssl = true; }
        { addr = "[::]"; port = 443; ssl = true; }
      ];
      root = "/var/www/bitbop.io";
    };
  };
  security.acme = {
    acceptTerms = true;
    defaults.email = "my-email+acme@gmail.com";
  };
}

However this does not work:

❯ sudo nixos-rebuild switch -I nixos-config=./nixos/configuration.nix
...
warning: the following units failed: acme-bitbop.io.service

× acme-bitbop.io.service - Renew ACME certificate for bitbop.io
     Loaded: loaded (/etc/systemd/system/acme-bitbop.io.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Sat 2024-03-02 01:24:12 UTC; 212ms ago
TriggeredBy: ● acme-bitbop.io.timer
    Process: 1558708 ExecStart=/nix/store/1xkvpxyllx3yj3ijwkglj9jv74dvbv3n-unit-script-acme-bitbop.io-start/bin/acme-bitbop.io-start (code=exited, status=10)
   Main PID: 1558708 (code=exited, status=10)
         IP: 17.2K in, 7.2K out
        CPU: 103ms

Mar 02 01:24:12 bitbop-proxy-arm acme-bitbop.io-start[1558743]: 2024/03/02 01:24:12 Could not obtain certificates:
Mar 02 01:24:12 bitbop-proxy-arm acme-bitbop.io-start[1558743]:         error: one or more domains had a problem:
Mar 02 01:24:12 bitbop-proxy-arm acme-bitbop.io-start[1558743]: [bitbop.io] acme: error: 400 :: urn:ietf:params:acme:error:connection :: 54.201.111.94: Fetching http://bitbop.io/.well-known/acme-challenge/Ka9b-Aflt0Bl-p1dL4yArbxeUUecBLsnMgUutEkO1tQ: Connection refused
Mar 02 01:24:12 bitbop-proxy-arm acme-bitbop.io-start[1558708]: + echo Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Mar 02 01:24:12 bitbop-proxy-arm acme-bitbop.io-start[1558708]: Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Mar 02 01:24:12 bitbop-proxy-arm acme-bitbop.io-start[1558708]: + exit 10
Mar 02 01:24:12 bitbop-proxy-arm systemd[1]: acme-bitbop.io.service: Main process exited, code=exited, status=10/n/a
Mar 02 01:24:12 bitbop-proxy-arm systemd[1]: acme-bitbop.io.service: Failed with result 'exit-code'.
Mar 02 01:24:12 bitbop-proxy-arm systemd[1]: Failed to start Renew ACME certificate for bitbop.io.
Mar 02 01:24:12 bitbop-proxy-arm systemd[1]: acme-bitbop.io.service: Consumed 103ms CPU time, received 17.1K IP traffic, sent 7.2K IP traffic.
warning: error(s) occurred while switching to the new configuration

Some info:

  • I have confirmed that 80 and 443 traffic is making it through all the necessary firewalls: I am able to see nginx log spam when pinging the address.
  • I have A and AAAA records correctly configured for IPv4 and v6 traffic respectively. I know that these records are correct since I am able to successfully connect to other services running on other ports.

My current guess is that this is bug related to IPv6: perhaps the NixOS module doesn’t listen for ACME challenges on IPv6 addresses?

Anyone have any ideas?

I ran into this the other week and apparently it’s been a known failure. The service will fail to run properly on the first try, but if you nixos-rebuild switch and then restart the service it will succeed on future runs. Not sure why it happened…

do we have a github issue tracking this?