Renewing certificates fails after update from simp_le to lego

Help
1

First of all thanks to the maintainers for their hard work on this new release.
I tried upgrading on my server yesterday and everything worked fine except for
the upgrade from simp_le to lego. As recommended by the release notes, I
added the acceptTerms option and move email out of the domain entry in the
security.acme module. Here is an excerpt from my
/etc/nixos/configuration.nix. I hope it is not shortened too much.

{ config, pkgs, options, ... }:

let
  acmeRoot = "/var/lib/acme/acme-challenge";
in
{
  security.acme = {
    acceptTerms = true;
    email = "henrimenke@gmail.com";
    certs = {
      "henrimenke.com" = {
        webroot = acmeRoot;
        extraDomains = {
          "www.henrimenke.com" = null;
          "git.henrimenke.com" = null;
          "git-private.henrimenke.com" = null;
        };
        postRun = "systemctl reload nginx.service";
      };
    };
  };

  services.nginx = {
    virtualHosts = {
      "henrimenke.com" = {
        forceSSL = true;
        enableACME = true;
        acmeRoot = acmeRoot;
        http2 = true;

        serverName = "henrimenke.com";
        serverAliases = [ "www.henrimenke.com" ];
      };

      "git.henrimenke.com" = {
        forceSSL = true;
        useACMEHost = "henrimenke.com";
        http2 = true;

        serverName = "git.henrimenke.com";
      };

      "git-private.henrimenke.com" = {
        forceSSL = true;
        useACMEHost = "henrimenke.com";
        http2 = true;

        serverName = "git-private.henrimenke.com";
      };
    };
  };
}

Here is the systemd journal log from the failed renewal:

Apr 22 04:28:49 henrimenke.com systemd[1]: Starting Renew ACME Certificate for henrimenke.com...
Apr 22 04:28:50 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:28:50 Error while loading the certificate for domain henrimenke.com
Apr 22 04:28:50 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]:         open certificates/henrimenke.com.crt: no such file or directory
Apr 22 04:28:50 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:28:50 [INFO] [henrimenke.com, git-private.henrimenke.com, git.henrimenke.com, www.henrimenke.com] acme: Obtaining bundled SAN certificate
Apr 22 04:28:52 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:28:52 [INFO] [git-private.henrimenke.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4088935047
Apr 22 04:28:52 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:28:52 [INFO] [git.henrimenke.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4088935049
Apr 22 04:28:52 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:28:52 [INFO] [henrimenke.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4104783141
Apr 22 04:28:52 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:28:52 [INFO] [www.henrimenke.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4104783142
Apr 22 04:28:52 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:28:52 [INFO] [henrimenke.com] acme: authorization already valid; skipping challenge
Apr 22 04:28:52 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:28:52 [INFO] [www.henrimenke.com] acme: authorization already valid; skipping challenge
Apr 22 04:28:52 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:28:52 [INFO] [git-private.henrimenke.com] acme: Could not find solver for: tls-alpn-01
Apr 22 04:28:52 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:28:52 [INFO] [git-private.henrimenke.com] acme: use http-01 solver
Apr 22 04:28:52 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:28:52 [INFO] [git.henrimenke.com] acme: Could not find solver for: tls-alpn-01
Apr 22 04:28:52 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:28:52 [INFO] [git.henrimenke.com] acme: use http-01 solver
Apr 22 04:28:52 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:28:52 [INFO] [git-private.henrimenke.com] acme: Trying to solve HTTP-01
Apr 22 04:29:09 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:29:09 [INFO] [git.henrimenke.com] acme: Trying to solve HTTP-01
Apr 22 04:29:23 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:29:23 [INFO] Skipping deactivating of valid auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4088935047
Apr 22 04:29:23 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:29:23 [INFO] Skipping deactivating of valid auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4088935049
Apr 22 04:29:23 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:29:23 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4104783141
Apr 22 04:29:24 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:29:24 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4104783141
Apr 22 04:29:24 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:29:24 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4104783142
Apr 22 04:29:24 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:29:24 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4104783142
Apr 22 04:29:24 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: 2020/04/22 04:29:24 Could not obtain certificates:
Apr 22 04:29:24 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]:         acme: Error -> One or more domains had a problem:
Apr 22 04:29:24 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: [git-private.henrimenke.com] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from http://git-private.henrimenke.com/.well-known/acme-challenge/a4g3X5U1lH7yckNjZDNtPfeGzXj0wG6Vb4X7vF6npMY [94.16.117.117]: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx</center>\r\n", url:
Apr 22 04:29:24 henrimenke.com 0pn5921n948jvghxp2k1z2gm2dy7f809-acme-start[11940]: [git.henrimenke.com] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from http://git.henrimenke.com/.well-known/acme-challenge/R_SQ4hSo863lIuNoJYCMDZ-8zyY4O5IdH8rc7yiBQ5E [94.16.117.117]: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx</center>\r\n", url:
Apr 22 04:29:24 henrimenke.com systemd[1]: acme-henrimenke.com.service: Main process exited, code=exited, status=1/FAILURE
Apr 22 04:29:24 henrimenke.com systemd[1]: acme-henrimenke.com.service: Failed with result 'exit-code'.
Apr 22 04:29:24 henrimenke.com systemd[1]: Failed to start Renew ACME Certificate for henrimenke.com.
Apr 22 04:29:24 henrimenke.com systemd[1]: acme-henrimenke.com.service: Consumed 691ms CPU time, received 57.8K IP traffic, sent 23.6K IP traffic.
2

Hello,

If you look at the failed authorizations you can see that nginx responded with 404 for both git vhosts.

According to the nginx module, the useACMEHost option is adding the acme-challenge location block in the nginx config.
But can you check in the generated nginx config if that’s the case and if the acmeRoot is the same for each vhost ?

Maybe you used a custom path for acmeRoot ? If that’s the case you need the acmeRoot = acmeRoot; line for both your git vhosts, else it will use the default.

On another note you can use inherit acmeRoot instead of acmeRoot = acmeRoot

1 Like
3

Copying the acmeRoot into the other two vHosts worked, thanks! It looks like the default path for acmeRoot has changed and in 19.09 I just hit the default path by accident. Maybe that should be added to the release notes.

There is still one error remaining in the journal, however, that does not cause the unit to fail.

Apr 22 12:24:44 henrimenke.com vl1nqbw4piirfpm14ckki19q8wx2knnr-acme-start[19868]: 2020/04/22 12:24:44 Could not create client: get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get "https://acme-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-v02.api.letsencrypt.org: no such host
4

Hm. This new lego seems to have some more issues. I just rebooted my machine and directly after reboot lego again failed, probably because the network wasn’t up yet.

Apr 22 12:33:07 henrimenke.com systemd[1]: Starting Renew ACME Certificate for henrimenke.com...
Apr 22 12:33:08 henrimenke.com vl1nqbw4piirfpm14ckki19q8wx2knnr-acme-start[751]: 2020/04/22 12:33:08 Could not create client: get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get "https://acme-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-v02.api.letsencrypt.org: no such host
Apr 22 12:33:08 henrimenke.com vl1nqbw4piirfpm14ckki19q8wx2knnr-acme-start[751]: 2020/04/22 12:33:08 Could not create client: get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get "https://acme-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-v02.api.letsencrypt.org: no such host
Apr 22 12:33:08 henrimenke.com systemd[1]: acme-henrimenke.com.service: Main process exited, code=exited, status=1/FAILURE
Apr 22 12:33:08 henrimenke.com systemd[1]: acme-henrimenke.com.service: Failed with result 'exit-code'.
Apr 22 12:33:08 henrimenke.com systemd[1]: Failed to start Renew ACME Certificate for henrimenke.com.

Restarting it manually made it pass.

Apr 22 12:35:30 henrimenke.com systemd[1]: Starting Renew ACME Certificate for henrimenke.com...
Apr 22 12:35:31 henrimenke.com vl1nqbw4piirfpm14ckki19q8wx2knnr-acme-start[1207]: 2020/04/22 12:35:31 [henrimenke.com] The certificate expires in 89 days, the number of days defined to perform the renewal is 30: no renewal.
Apr 22 12:35:31 henrimenke.com systemd[1]: acme-henrimenke.com.service: Succeeded.
Apr 22 12:35:31 henrimenke.com systemd[1]: Started Renew ACME Certificate for henrimenke.com.
Apr 22 12:35:31 henrimenke.com systemd[1]: acme-henrimenke.com.service: Consumed 152ms CPU time, received 4.8K IP traffic, sent 1.2K IP traffic.
5

The services are depending on Wants=network-online.target so it depends on how your network is configured but it is possible services will start before they really could access the internet.

It could be because you have multiple network interfaces and the one connecting to internet is slower to get DHCP or something like that.

More information here : NetworkTarget

1 Like
6

Since this is a server the interfaces are statically configured and are therefore by definition always online. However, I think the problem here might be that the acme service runs before nginx is ready. There should probably be something like this

before = map (vhostConfig: "acme-${vhostConfig.serverName}.service") acmeEnabledVhosts;

in the nginx service unit.