Hi! I have a homeserver which exposes a few services to the internet and uses ACME/certbot for the certificates. Today, I updated the system and it basically broke every certificate for my services. This is the journalctl of my acme config:
ov 15 22:04:37 muffinman acme-example.duckdns.org-start[771346]: Waiting to acquire lock /run/acme/1.lock
Nov 15 22:04:37 muffinman acme-example.duckdns.org-start[771346]: Acquired lock /run/acme/1.lock
Nov 15 22:04:37 muffinman acme-example.duckdns.org-start[771346]: + set -euo pipefail
Nov 15 22:04:37 muffinman acme-example.duckdns.org-start[771348]: + mkdir -p /var/lib/acme/acme-challenge//.well-known/acme-challenge
Nov 15 22:04:37 muffinman acme-example.duckdns.org-start[771348]: + chgrp acme /var/lib/acme/acme-challenge//.well-known/acme-challenge
Nov 15 22:04:37 muffinman acme-example.duckdns.org-start[771346]: + echo dfe3cc6be433255f4654
Nov 15 22:04:37 muffinman acme-example.duckdns.org-start[771346]: + cmp -s domainhash.txt certificates/domainhash.txt
Nov 15 22:04:37 muffinman acme-example.duckdns.org-start[771346]: + lego --accept-tos --path . -d example.duckdns.org --email admin@example.duckdns.org --key-type ec256 --http --http.webroot /var/lib/acme/acme-challenge/ --server https://acme-staging-v02.api.letsencrypt.org/directory -d nextcloud.example.duckdns.org -d jellyfin.example.duckdns.org -d forge.example.duckdns.org run
Nov 15 22:04:37 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:04:37 [INFO] [example.duckdns.org, nextcloud.example.duckdns.org, jellyfin.example.duckdns.org, forge.example.duckdns.org] acme: Obtaining bundled SAN certificate
Nov 15 22:04:38 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:04:38 [INFO] [forge.example.duckdns.org] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14930951253
Nov 15 22:04:38 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:04:38 [INFO] [jellyfin.example.duckdns.org] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14930951263
Nov 15 22:04:38 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:04:38 [INFO] [nextcloud.example.duckdns.org] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14930951273
Nov 15 22:04:38 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:04:38 [INFO] [example.duckdns.org] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14930951283
Nov 15 22:04:38 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:04:38 [INFO] [forge.example.duckdns.org] acme: Could not find solver for: tls-alpn-01
Nov 15 22:04:38 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:04:38 [INFO] [forge.example.duckdns.org] acme: use http-01 solver
Nov 15 22:04:38 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:04:38 [INFO] [jellyfin.example.duckdns.org] acme: Could not find solver for: tls-alpn-01
Nov 15 22:04:38 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:04:38 [INFO] [jellyfin.example.duckdns.org] acme: use http-01 solver
Nov 15 22:04:38 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:04:38 [INFO] [nextcloud.example.duckdns.org] acme: Could not find solver for: tls-alpn-01
Nov 15 22:04:38 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:04:38 [INFO] [nextcloud.example.duckdns.org] acme: use http-01 solver
Nov 15 22:04:38 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:04:38 [INFO] [example.duckdns.org] acme: Could not find solver for: tls-alpn-01
Nov 15 22:04:38 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:04:38 [INFO] [example.duckdns.org] acme: use http-01 solver
Nov 15 22:04:38 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:04:38 [INFO] [forge.example.duckdns.org] acme: Trying to solve HTTP-01
Nov 15 22:05:05 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:05:05 [INFO] [forge.example.duckdns.org] The server validated our request
Nov 15 22:05:05 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:05:05 [INFO] [jellyfin.example.duckdns.org] acme: Trying to solve HTTP-01
Nov 15 22:05:19 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:05:19 [INFO] [nextcloud.example.duckdns.org] acme: Trying to solve HTTP-01
Nov 15 22:05:33 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:05:33 [INFO] [example.duckdns.org] acme: Trying to solve HTTP-01
Nov 15 22:05:39 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:05:39 [INFO] Skipping deactivating of valid auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14930951253
Nov 15 22:05:39 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:05:39 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14930951263
Nov 15 22:05:40 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:05:40 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14930951273
Nov 15 22:05:40 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:05:40 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14930951283
Nov 15 22:05:40 muffinman acme-example.duckdns.org-start[771351]: 2024/11/15 22:05:40 Could not obtain certificates:
Nov 15 22:05:40 muffinman acme-example.duckdns.org-start[771351]: error: one or more domains had a problem:
Nov 15 22:05:40 muffinman acme-example.duckdns.org-start[771351]: [jellyfin.example.duckdns.org] acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: server failure at resolver looking up CAA for example.duckdns.org
Nov 15 22:05:40 muffinman acme-example.duckdns.org-start[771351]: [nextcloud.example.duckdns.org] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: 142.250.185.174: Invalid response from http://nextcloud.example.duckdns.org/.well-known/acme-challenge/-KOtmhVhHN6iBYLZY3iibJ-gsoqQV70TILedG46Sk5E: 404
Nov 15 22:05:40 muffinman acme-example.duckdns.org-start[771351]: [example.duckdns.org] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: 142.250.185.174: Invalid response from http://example.duckdns.org/.well-known/acme-challenge/oamdYr6gqN0VL7erKvdnwYPWbuBjBYhK4thISjn_pOA: 404
Nov 15 22:05:40 muffinman acme-example.duckdns.org-start[771346]: + echo Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Nov 15 22:05:40 muffinman acme-example.duckdns.org-start[771346]: Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Nov 15 22:05:40 muffinman acme-example.duckdns.org-start[771346]: + exit 10
Nov 15 22:05:40 muffinman systemd[1]: acme-example.duckdns.org.service: Main process exited, code=exited, status=10/n/a
This is my acme module:
lib, config, ...}: let
localConfig = config.myNixOS.acme;
topdomain = localConfig.topdomain;
in {
options.myNixOS.acme = {
topdomain = lib.mkOption {
type = lib.types.str;
example = "example.org";
};
subdomains = lib.mkOption {
type = with lib.types; listOf str;
default = [];
};
};
security.acme = {
acceptTerms = true;
defaults = {
email = "admin@${topdomain}";
};
certs."${topdomain}" = {
webroot = "/var/lib/acme/acme-challenge/";
domain = topdomain;
# for testing
server = "https://acme-staging-v02.api.letsencrypt.org/directory";
extraDomainNames =
builtins.map
(subdomain: "${subdomain}.${topdomain}")
localConfig.subdomains;
};
};
users.users.nginx.extraGroups = [ "acme" ];
}
When adding a service to ACME, I put it in a container and register the nginx virtualHost like this:
{config, ...}: let
subdomain = "exampleservice"
virtualHostDomain = "${subdomain}.config.myNixOS.acme.topdomain}";
in {
services.nginx.virtualHosts = {
"${virtualHostDomain}" = {
onlySSL = true; # can also be addSSL
useACMEHost = "${globalConfig.myNixOS.acme.topdomain}";
locations."/".proxyPass = "http://${localCfg.containerAddress}";
};
myNixOS.acme.subdomains = [ subdomain ];
}
I have no idea, why it’s not working anymore. Am I overlooking some new option? I hope the info I gave is sufficient.