Hi everyone,
I’ve found quite a few issues on Github and right here on the Discourse and have tried various fixes to make it work (Such as the drastic back up/removal of /var/lib/acme to ‘start fresh’ so to speak, as well as making sure my users/groups are configured correctly, a workaround involving using e-mail aliases such as znc+myemail@gmail.com for each cert), but I’m still just getting the same errors every time for this.
I already have (or had) two Lets Encrypt certs however now that one of them is up for renewal it’s been erroring out on me, even weirder is that now both certs are failing even though one of them doesn’t actually need a renewal so I’m guessing something’s gone seriously wrong with my configuration.
So I have two certs - one is for a Nextcloud instance (under cloud.mydomain.com) and one is for a ZNC server, under znc.mydomain.com (domain changed as it’s one I pretty much just use for myself running on a Pi at my house, don’t need it bombarded!) - the ‘znc’ one has a post-run line that moves the certificate to my actual Znc directory so it can be used by this daemon.
running nixos-option security.acme.certs shows this:
Value:
{
cloud.mydomain.com = {
allowKeysForGroup = "_mkRemovedOptionModule";
credentialsFile = «error: The option `security.acme.certs.cloud.mydomain.com.credentialsFile' is used but not defined.»;
directory = "/var/lib/acme/cloud.mydomain.com";
dnsPropagationCheck = true;
dnsProvider = null;
domain = "cloud.mydomain.com";
email = "myemail@gmail.com";
extraDomainNames = [ ];
extraDomains = "_mkMergedOptionModule";
extraLegoFlags = [ ];
extraLegoRenewFlags = [ ];
extraLegoRunFlags = [ ];
group = "nginx";
keyType = "ec256";
ocspMustStaple = false;
postRun = "";
server = null;
user = "_mkRemovedOptionModule";
webroot = "/var/lib/acme/acme-challenge";
};
znc.piecemaker.rocks = {
allowKeysForGroup = "_mkRemovedOptionModule";
credentialsFile = «error: The option `security.acme.certs.znc.piecemaker.rocks.credentialsFile' is used but not defined.»;
directory = "/var/lib/acme/znc.mydomain.com";
dnsPropagationCheck = true;
dnsProvider = null;
domain = "znc.mydomain.com";
email = "myemail@gmail.com";
extraDomainNames = [ ];
extraDomains = "_mkMergedOptionModule";
extraLegoFlags = [ ];
extraLegoRenewFlags = [ ];
extraLegoRunFlags = [ ];
group = "acme";
keyType = "ec256";
ocspMustStaple = false;
postRun = ''
cp full.pem "/var/lib/znc//znc.pem"
'';
server = null;
user = "_mkRemovedOptionModule";
webroot = "/var/lib/acme/acme-challenge";
};
}
Default:
{ }
Type:
"attribute set of submodules"
Example:
{
text = ''
{
"example.com" = {
webroot = "/var/www/challenges/";
email = "foo@example.com";
extraDomainNames = [ "www.example.com" "foo.example.com" ];
};
"bar.example.com" = {
webroot = "/var/www/challenges/";
email = "bar@example.com";
};
}
'';
}
Description:
''
Attribute set of certificates to get signed and renewed. Creates
<literal>acme-''${cert}.{service,timer}</literal> systemd units for
each certificate defined here. Other services can add dependencies
to those units if they rely on the certificates being present,
or trigger restarts of the service if certificates get renewed.
''
Declared by:
[ "/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/security/acme.nix" ]
Defined by:
[
"/etc/nixos/znc.nix"
"/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/services/web-servers/nginx/default.nix"
]
And nixos-option services.nginx.virtualHosts gives me:
Value:
{
cloud.mydomain.com = {
acmeFallbackHost = null;
acmeRoot = "/var/lib/acme/acme-challenge";
addSSL = false;
basicAuth = { };
basicAuthFile = null;
default = false;
enableACME = true;
enableSSL = false;
extraConfig = ''
index index.php index.html /index.php$request_uri;
expires 1m;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header X-Robots-Tag none;
add_header X-Download-Options noopen;
add_header X-Permitted-Cross-Domain-Policies none;
add_header X-Frame-Options sameorigin;
add_header Referrer-Policy no-referrer;
add_header Strict-Transport-Security "max-age=15552000; includeSubDomains" always;
client_max_body_size 512M;
fastcgi_buffers 64 4K;
fastcgi_hide_header X-Powered-By;
gzip on;
gzip_vary on;
gzip_comp_level 4;
gzip_min_length 256;
gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;
rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json last;
rewrite ^/.well-known/webfinger /public.php?service=webfinger last; '';
forceSSL = true;
globalRedirect = null;
http2 = true;
listen = [ ];
locations = {
/ = {
alias = null;
extraConfig = "rewrite ^ /index.php;";
index = null;
priority = 900;
proxyPass = null;
proxyWebsockets = false;
return = null;
root = null;
tryFiles = null;
};
= /robots.txt = {
alias = null;
extraConfig = ''
allow all;
log_not_found off;
access_log off;
'';
index = null;
priority = 100;
proxyPass = null;
proxyWebsockets = false;
return = null;
root = null;
tryFiles = null;
};
^~ /.well-known = {
alias = null;
extraConfig = ''
location = /.well-known/carddav {
return 301 $scheme://$host/remote.php/dav;
}
location = /.well-known/caldav {
return 301 $scheme://$host/remote.php/dav;
}
try_files $uri $uri/ =404;
'';
index = null;
priority = 210;
proxyPass = null;
proxyWebsockets = false;
return = null;
root = null;
tryFiles = null;
};
~ \.(?:css|js|woff2?|svg|gif|map)$ = {
alias = null;
extraConfig = ''
try_files $uri /index.php$request_uri;
expires 6M;
access_log off;
'';
index = null;
priority = 1000;
proxyPass = null;
proxyWebsockets = false;
return = null;
root = null;
tryFiles = null;
};
~ \.(?:png|html|ttf|ico|jpg|jpeg|bcmap|mp4|webm)$ = {
alias = null;
extraConfig = ''
try_files $uri /index.php$request_uri;
access_log off;
'';
index = null;
priority = 1000;
proxyPass = null;
proxyWebsockets = false;
return = null;
root = null;
tryFiles = null;
};
~ ^/(?:\.|autotest|occ|issue|indie|db_|console) = {
alias = null;
extraConfig = ''
return 404;
'';
index = null;
priority = 1000;
proxyPass = null;
proxyWebsockets = false;
return = null;
root = null;
tryFiles = null;
};
~ ^/(?:build|tests|config|lib|3rdparty|templates|data)(?:$|/) = {
alias = null;
extraConfig = ''
return 404;
'';
index = null;
priority = 1000;
proxyPass = null;
proxyWebsockets = false;
return = null;
root = null;
tryFiles = null;
};
~ ^/store-apps = {
alias = null;
extraConfig = "root /var/lib/nextcloud;";
index = null;
priority = 201;
proxyPass = null;
proxyWebsockets = false;
return = null;
root = null;
tryFiles = null;
};
~ ^\/(?:index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+|.+\/richdocumentscode\/proxy)\.php(?:$|\/) = {
alias = null;
extraConfig = ''
include /nix/store/nyc6rcgpqp9ihd12kix1p8ls2b75n29m-nginx-1.18.0/conf/fastcgi.conf;
fastcgi_split_path_info ^(.+?\.php)(\\/.*)$;
set $path_info $fastcgi_path_info;
try_files $fastcgi_script_name =404;
fastcgi_param PATH_INFO $path_info;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param HTTPS on;
fastcgi_param modHeadersAvailable true;
fastcgi_param front_controller_active true;
fastcgi_pass unix:/run/phpfpm/nextcloud.sock;
fastcgi_intercept_errors on;
fastcgi_request_buffering off;
fastcgi_read_timeout 120s;
'';
index = null;
priority = 500;
proxyPass = null;
proxyWebsockets = false;
return = null;
root = null;
tryFiles = null;
};
~ ^\/(?:updater|ocs-provider|ocm-provider)(?:$|\/) = {
alias = null;
extraConfig = ''
try_files $uri/ =404;
index index.php;
'';
index = null;
priority = 1000;
proxyPass = null;
proxyWebsockets = false;
return = null;
root = null;
tryFiles = null;
};
};
onlySSL = false;
root = «derivation /nix/store/n6kxazah9jc0zqja2dcyvpx8fhg1z7ss-nextcloud-19.0.4.drv»;
serverAliases = [ ];
serverName = null;
sslCertificate = «error: The option `services.nginx.virtualHosts.cloud.mydomain.com.sslCertificate' is used but not defined.»;
sslCertificateKey = «error: The option `services.nginx.virtualHosts.cloud.mydomain.com.sslCertificateKey' is used but not defined.»;
sslTrustedCertificate = null;
useACMEHost = null;
};
znc.mydomain.com = {
acmeFallbackHost = null;
acmeRoot = "/var/lib/acme/acme-challenge";
addSSL = false;
basicAuth = { };
basicAuthFile = null;
default = false;
enableACME = true;
enableSSL = false;
extraConfig = "";
forceSSL = false;
globalRedirect = null;
http2 = true;
listen = [ ];
locations = {
/ = {
alias = null;
extraConfig = "";
index = null;
priority = 1000;
proxyPass = null;
proxyWebsockets = false;
return = "301 https://$host$request_uri";
root = null;
tryFiles = null;
};
/.well-known/acme-challenge = {
alias = null;
extraConfig = "";
index = null;
priority = 1000;
proxyPass = null;
proxyWebsockets = false;
return = null;
root = "/var/lib/acme/acme-challenge";
tryFiles = null;
};
};
onlySSL = false;
root = null;
serverAliases = [ ];
serverName = null;
sslCertificate = «error: The option `services.nginx.virtualHosts.znc.mydomain.com.sslCertificate' is used but not defined.»;
sslCertificateKey = «error: The option `services.nginx.virtualHosts.znc.mydomain.com.sslCertificateKey' is used but not defined.»;
sslTrustedCertificate = null;
useACMEHost = null;
};
}
Default:
{ localhost = { }; }
Type:
"attribute set of submodules"
Example:
{
text = ''
{
"hydra.example.com" = {
forceSSL = true;
enableACME = true;
locations."/" = {
proxyPass = "http://localhost:3000";
};
};
};
'';
}
Description:
"Declarative vhost config"
Declared by:
[ "/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/services/web-servers/nginx/default.nix" ]
Defined by:
[
"/etc/nixos/nextcloud.nix"
"/etc/nixos/web.nix"
"/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/services/web-apps/nextcloud.nix"
]
I seem to get one of two errors from this which makes it even weirder.
systemctl status acme-znc.mydomain.com.service gives me:
● acme-znc.mydomain.com.service - Renew ACME certificate for znc.mydomain.com
Loaded: loaded (/nix/store/lxsavpzz8yc7ppkbipigbhvd47ai9byx-unit-acme-znc.mydomain.com.service/acme-znc.mydomain.com.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2021-01-10 14:12:49 GMT; 2h 33min ago
TriggeredBy: ● acme-znc.mydomain.com.timer
Process: 10313 ExecStart=/nix/store/byprr9xbdikwhaasq6kivnj9p1zzmvfb-unit-script-acme-znc.mydomain.com-start/bin/acme-znc.mydomain.com-start (code=exited, status=1/FAILURE)
Main PID: 10313 (code=exited, status=1/FAILURE)
IP: 13.6K in, 6.5K out
CPU: 575ms
Jan 10 14:12:43 NixPi acme-znc.mydomain.com-start[10314]: 2021/01/10 14:12:43 [INFO] [znc.mydomain.com] acme: Trying to solve HTTP-01
Jan 10 14:12:49 NixPi acme-znc.mydomain.com-start[10314]: 2021/01/10 14:12:49 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/9970293082
Jan 10 14:12:49 NixPi acme-znc.mydomain.com-start[10314]: 2021/01/10 14:12:49 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/9970293082
Jan 10 14:12:49 NixPi acme-znc.mydomain.com-start[10314]: 2021/01/10 14:12:49 Could not obtain certificates:
Jan 10 14:12:49 NixPi acme-znc.mydomain.com-start[10314]: error: one or more domains had a problem:
Jan 10 14:12:49 NixPi acme-znc.mydomain.com-start[10314]: [znc.mydomain.com] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from http://znc.mydomain.com/.well-known/acme-challenge/OYKpe5Ache0PW2XRzL9D7E3r9fEvHmcqqqQJu4Aav8o [86.5.144.176]: "<html>\r\n<head><title>403 Forbidden</title></head>\r\n<body>\r\n<center><h1>403 Forbidden</h1></center>\r\n<hr><center>nginx</center>\r\n", url:
Jan 10 14:12:49 NixPi systemd[1]: acme-znc.mydomain.com.service: Main process exited, code=exited, status=1/FAILURE
Jan 10 14:12:49 NixPi systemd[1]: acme-znc.mydomain.com.service: Failed with result 'exit-code'.
Jan 10 14:12:49 NixPi systemd[1]: Failed to start Renew ACME certificate for znc.mydomain.com.
Jan 10 14:12:49 NixPi systemd[1]: acme-znc.mydomain.com.service: Consumed 575ms CPU time, received 13.5K IP traffic, sent 6.5K IP traffic.
Whereas systemctl status acme-cloud.mydomain.com.service -l gives me:
● acme-cloud.mydomain.com.service - Renew ACME certificate for cloud.mydomain.com
Loaded: loaded (/nix/store/qchpxfcll1h0hrnhnrhg99bhcg34akg9-unit-acme-cloud.mydomain.com.service/acme-cloud.mydomain.com.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2021-01-15 23:14:20 GMT; 16min ago
TriggeredBy: ● acme-cloud.mydomain.com.timer
Process: 22434 ExecStart=/nix/store/51zvbdms4xaq4s0s9dgcp7rlybak1dri-unit-script-acme-cloud.mydomain.com-start/bin/acme-cloud.mydomain.com-start (code=exited, status=1/FAILURE)
Main PID: 22434 (code=exited, status=1/FAILURE)
IP: 13.0K in, 6.5K out
CPU: 592ms
Jan 15 23:14:14 NixPi acme-cloud.mydomain.com-start[22435]: 2021/01/15 23:14:14 [INFO] [cloud.mydomain.com] acme: Trying to solve HTTP-01
Jan 15 23:14:20 NixPi acme-cloud.mydomain.com-start[22435]: 2021/01/15 23:14:20 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/10107201795
Jan 15 23:14:20 NixPi acme-cloud.mydomain.com-start[22435]: 2021/01/15 23:14:20 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/10107201795
Jan 15 23:14:20 NixPi acme-cloud.mydomain.com-start[22435]: 2021/01/15 23:14:20 Could not obtain certificates:
Jan 15 23:14:20 NixPi acme-cloud.mydomain.com-start[22435]: error: one or more domains had a problem:
Jan 15 23:14:20 NixPi acme-cloud.mydomain.com-start[22435]: [cloud.mydomain.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://cloud.mydomain.com/.well-known/acme-challenge/meJ-x9NYbu6tsGaOXcskGw0HpKkDQCfl-sbScOR4wsk: Connection refused, url:
Jan 15 23:14:20 NixPi systemd[1]: acme-cloud.mydomain.com.service: Main process exited, code=exited, status=1/FAILURE
Jan 15 23:14:20 NixPi systemd[1]: acme-cloud.mydomain.com.service: Failed with result 'exit-code'.
Jan 15 23:14:20 NixPi systemd[1]: Failed to start Renew ACME certificate for cloud.mydomain.com.
Jan 15 23:14:20 NixPi systemd[1]: acme-cloud.mydomain.com.service: Consumed 592ms CPU time, received 12.9K IP traffic, sent 6.5K IP traffic.
Has anyone got this setup to work correctly? It must have worked at some point as my certs have only just begun to expire and I’m not sure if I’ve changed anything serious since then. Thanks a lot!
