Unable to start Renew ACME Certificates when trying to use Nextcloud or Jitsi

Help
1

I am trying to run Jitsi and Nextcloud on my NixOS server and am having some issues. First I put this in my configuration.nix:

security.acme = {
    acceptTerms = true;
    # Replace the email here!
    email = "myemail@myemail.com";
};

If I just do this and then run it, there are no issues. Then I made a nextcloud.nix file and put in the following code which I found here:

{config, pkgs, ...}:
{
services.nginx = {
   enable = true;

 # Use recommended settings
    recommendedGzipSettings = true;
    recommendedOptimisation = true;
    recommendedProxySettings = true;
    recommendedTlsSettings = true;

 # Only allow PFS-enabled ciphers with AES256
    sslCiphers = "AES256+EECDH:AES256+EDH:!aNULL";

# Setup Nextcloud virtual host to listen on ports
 virtualHosts = {

     "nextcloud.example.com" = {
       ## Force HTTP redirect to HTTPS
       forceSSL = true;
       ## LetsEncrypt
       enableACME = true;
    };
  };
};
services.nextcloud = {
    enable = true;
    hostName = "nextcloud.example.com";
    # Enable built-in virtual host management
    # Takes care of somewhat complicated setup
    # See here: https://github.com/NixOS/nixpkgs/blob/dab87a5bac0459886d1ab68fa52f71bcc42c396a/nixos/modules/services/web-apps/nextcloud.nix#L529
    nginx.enable = true;

    # Use HTTPS for links
    https = true;
    
    # Auto-update Nextcloud Apps
    autoUpdateApps.enable = true;
    # Set what time makes sense for you
    autoUpdateApps.startAt = "05:00:00";

    config = {
      # Further forces Nextcloud to use HTTPS
      overwriteProtocol = "https";

      # Nextcloud PostegreSQL database configuration, recommended over using SQLite
      dbtype = "pgsql";
      dbuser = "nextcloud";
      dbhost = "/run/postgresql"; # nextcloud will add /.s.PGSQL.5432 by itself
      dbname = "nextcloud";
      dbpassFile = "/var/nextcloud-db-pass";

      adminpassFile = "/var/nextcloud-admin-pass";
      adminuser = "admin";
 };
};
services.postgresql = {
    enable = true;

    # Ensure the database, user, and permissions always exist
    ensureDatabases = [ "nextcloud" ];
    ensureUsers = [
     { name = "nextcloud";
       ensurePermissions."DATABASE nextcloud" = "ALL PRIVILEGES";
     }
    ];
};
systemd.services."nextcloud-setup" = {
    requires = ["postgresql.service"];
    after = ["postgresql.service"];
};

}

And then imported nextcloud.nix in my configuration.nix.

The only things I changed were the virtual hosts and the hostname to “nc.mywebsite.com” and I removed services.nextcloud.nginx.enable=true which was apparently deprecated.

Then on my DNS provider, I added a Type A DNS record named “nc” with the content being my server’s external IP address. I also opened up port 80. As far as I know, this should be all the things I need to change outside of NixOS.

I then created and filled /var/nextcloud-db-pass and /var/nextcloud-admin-pass and changed the file permissions to nextcloud:nextcloud.

Then after running sudo nixos-rebuild swtich, I got the following error:

the following new units were started: acme-nc.mywebsite.dev.timer, session-1698.scope
warning: the following units failed: acme-nc.mywebsite.dev.service

â—Ź acme-nc.mywebsite.dev.service - Renew ACME certificate for nc.mywebsite.dev
     Loaded: loaded (/nix/store/gwrn0lqjzlsjdgihgc8krj7z389wn5b0-unit-acme-nc.mywebsite.dev.service/acme-nc.mywebsite.dev.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2021-06-13 16:11:50 EDT; 355ms ago
TriggeredBy: â—Ź acme-nc.mywebsite.dev.timer
    Process: 1477695 ExecStart=/nix/store/0757c2ydlkgag3w9k202dipf8dpb1zv5-unit-script-acme-nc.mywebsite.dev-start/bin/acme-nc.mywebsite.dev-start (code=exited, status=1/FAILURE)
   Main PID: 1477695 (code=exited, status=1/FAILURE)
         IP: 19.1K in, 8.7K out
        CPU: 310ms

Jun 13 16:11:05 nixos acme-nc.mywebsite.dev-start[1477704]: 2021/06/13 16:11:05 [INFO] [nc.mywebsite.dev] acme: Trying to solve HTTP-01
Jun 13 16:11:50 nixos acme-nc.mywebsite.dev-start[1477704]: 2021/06/13 16:11:50 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/13952639864
Jun 13 16:11:50 nixos acme-nc.mywebsite.dev-start[1477704]: 2021/06/13 16:11:50 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/13952639864
Jun 13 16:11:50 nixos acme-nc.mywebsite.dev-start[1477704]: 2021/06/13 16:11:50 Could not obtain certificates:
Jun 13 16:11:50 nixos acme-nc.mywebsite.dev-start[1477704]:         error: one or more domains had a problem:
Jun 13 16:11:50 nixos acme-nc.mywebsite.dev-start[1477704]: [nc.mywebsite.dev] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from http://nc.mywebsite.dev/.well-known/acme-challenge/92i_mHWVNCGJtftMKW7GT9Z54FWA0IVc24dtSuukuvU [2606:4700:3036::ac43:a114]: "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js "...
Jun 13 16:11:50 nixos systemd[1]: acme-nc.mywebsite.dev.service: Main process exited, code=exited, status=1/FAILURE
Jun 13 16:11:50 nixos systemd[1]: acme-nc.mywebsite.dev.service: Failed with result 'exit-code'.
Jun 13 16:11:50 nixos systemd[1]: Failed to start Renew ACME certificate for nc.mywebsite.dev.
Jun 13 16:11:50 nixos systemd[1]: acme-nc.mywebsite.dev.service: Consumed 310ms CPU time, received 19.0K IP traffic, sent 8.7K IP traffic.
Hint: Some lines were ellipsized, use -l to show in full.
warning: error(s) occurred while switching to the new configuration

I then tried the same thing with Jitsi meet, finding the configuration here: NixOS - NixOS 21.05 manual. And had nearly the exact same issue.

What can I do here? Is there something I am missing or doing wrong?

Thank you in advance.

2 Likes
2 
Jun 13 16:11:50 nixos acme-nc.mywebsite.dev-start[1477704]: [nc.mywebsite.dev] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from http://nc.mywebsite.dev/.well-known/acme-challenge/92i_mHWVNCGJtftMKW7GT9Z54FWA0IVc24dtSuukuvU [2606:4700:3036::ac43:a114]: "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js "...

It looks like your Nginx does not serve the challenge expected by Let’s Encrypt at that URL and serves some HTML document instead. I would start by checking the Nginx configuration (specifically the .well-known location that should be set up by enabling ACME for the vhost), then maybe putting something in your acmeRoot directory and doing some curl calls for it.

3

Sorry, can you elaborate on this? I see the issue you are mentioning, but am not sure what to do to fix this.

The only occurrence of Nginx in my configurations was mentioned on the post.

services.nginx = {
   enable = true;

 # Use recommended settings
    recommendedGzipSettings = true;
    recommendedOptimisation = true;
    recommendedProxySettings = true;
    recommendedTlsSettings = true;

 # Only allow PFS-enabled ciphers with AES256
    sslCiphers = "AES256+EECDH:AES256+EDH:!aNULL";

# Setup Nextcloud virtual host to listen on ports
 virtualHosts = {

     "nextcloud.example.com" = {
       ## Force HTTP redirect to HTTPS
       forceSSL = true;
       ## LetsEncrypt
       enableACME = true;
    };
  };
};

I would start by checking the Nginx configuration (specifically the .well-known location that should be set up by enabling ACME for the vhost), then maybe putting something in your acmeRoot directory and doing some curl calls for it.

You mean services.nginx.virtualHosts.<name>.acmeRoot? It is saying that the default value for that is /var/lib/acme/acme-challenges. It looks like that directory contains a directory called .well-known which contains an empty directory called acme-challenge.

Do you think that could be the issue? The directories in acme-challenge being empty?

4

What I meant was to first check the Nginx configuration file that is generated by the nginx NixOS module. You can find the configuration file by either systemctl cat nginx.service or ps ax | grep nginx, it will show up as an argument of the nginx command/process, e.g.:

# ps ax | grep 'nginx '
 1556 ?        Ss     0:00 nginx: master process /nix/store/k3mvi6xc9z30552w0am05g52wlknv4vd-nginx-1.16.1/bin/nginx -c /nix/store/4cgri80w5yv901pyllahbjb3k906mfrv-nginx.conf -p /var/spool/nginx
10235 pts/0    R+     0:00 grep nginx

^^ Here the /nix/store/4cgri80w5yv901pyllahbjb3k906mfrv-nginx.conf is the Nginx configuration file.

In this file there must be a location section that is meant to handle the .well-known path for the relevant FQDN. I would check the root setting of the location, then I would create a file in that directory and I’d try to request that file via curl (also watching the logs while doing this).

According to the error you are getting it looks like Let’s Encrypt tries to read its challenge back from your Nginx instance, but instead gets an HTML document back and hence fails to issue a certificate.

2 Likes
5

Yeah, I see my nginx.conf.


âžś  ~ cat /nix/store/cli37gkds2sabyd0hlxycljcjx38mphm-nginx.conf
pid /run/nginx/nginx.pid;
error_log stderr;
daemon off;
events {
}
http {
	# The mime type definitions included with nginx are very incomplete, so
	# we use a list of mime types from the mailcap package, which is also
	# used by most other Linux distributions by default.
	include /nix/store/rya2s79k5ncqbas09rwhq43lrbr4s3m0-mailcap-2.1.52/etc/nginx/mime.types;
	include /nix/store/8nyvpwp7c4slmcqq987n3pyb3kz9mfa2-nginx-1.20.1/conf/fastcgi.conf;
	include /nix/store/8nyvpwp7c4slmcqq987n3pyb3kz9mfa2-nginx-1.20.1/conf/uwsgi_params;
	default_type application/octet-stream;
	# optimisation
	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;
	keepalive_timeout 65;
	types_hash_max_size 4096;
	ssl_protocols TLSv1.2 TLSv1.3;
	ssl_ciphers AES256+EECDH:AES256+EDH:!aNULL;
	# Keep in sync with https://ssl-config.mozilla.org/#server=nginx&config=intermediate
	ssl_session_timeout 1d;
	ssl_session_cache shared:SSL:10m;
	# Breaks forward secrecy: https://github.com/mozilla/server-side-tls/issues/135
	ssl_session_tickets off;
	# We don't enable insecure ciphers by default, so this allows
	# clients to pick the most performant, per https://github.com/mozilla/server-side-tls/issues/260
	ssl_prefer_server_ciphers off;
	# OCSP stapling
	ssl_stapling on;
	ssl_stapling_verify on;
	gzip on;
	gzip_proxied any;
	gzip_comp_level 5;
	gzip_types
	application/atom+xml
	application/javascript
	application/json
	application/xml
	application/xml+rss
	image/svg+xml
	text/css
	text/javascript
	text/plain
	text/xml;
	gzip_vary on;
	proxy_redirect          off;
	proxy_connect_timeout   60s;
	proxy_send_timeout      60s;
	proxy_read_timeout      60s;
	proxy_http_version      1.1;
	include /nix/store/zqdr08b9qfbqa78cl62ixcmkgdfrzdlm-nginx-recommended-proxy-headers.conf;
	# $connection_upgrade is used for websocket proxying
	map $http_upgrade $connection_upgrade {
		default upgrade;
		''      close;
	}
	client_max_body_size 10m;
	server_tokens off;
	server {
		listen 0.0.0.0:80 ;
		listen [::]:80 ;
		server_name nc.mywebsite.dev ;
		location /.well-known/acme-challenge {
			root /var/lib/acme/acme-challenge;
			auth_basic off;
		}
		location / {
			return 301 https://$host$request_uri;
		}
	}
	server {
		listen 0.0.0.0:443 ssl http2 ;
		listen [::]:443 ssl http2 ;
		server_name nc.mywebsite.dev ;
		location /.well-known/acme-challenge {
			root /var/lib/acme/acme-challenge;
			auth_basic off;
		}
		ssl_certificate /var/lib/acme/nc.mywebsite.dev/fullchain.pem;
		ssl_certificate_key /var/lib/acme/nc.mywebsite.dev/key.pem;
		ssl_trusted_certificate /var/lib/acme/nc.mywebsite.dev/chain.pem;
	}
}

So the relevant portion is:

		location /.well-known/acme-challenge {
			root /var/lib/acme/acme-challenge;
			auth_basic off;
		}
		location / {
			return 301 https://$host$request_uri;
		}

and

	server {
		listen 0.0.0.0:443 ssl http2 ;
		listen [::]:443 ssl http2 ;
		server_name nc.mywebsite.dev ;
		location /.well-known/acme-challenge {
			root /var/lib/acme/acme-challenge;
			auth_basic off;
		}

Right? The current acme-challenge directory only has a .well-known path. Then the .well-known directory only has an empty acme-challenge directory.

So just any file in /var/lib/acme/acme-challenge would be sufficient?

Could you please elaborate on this? Do you just mean curl file:/var/lib/acme/acme-challenge/test.txt?

6

I was having the same problem with TT-RSS and email server.
ACME renew suddenly stopped working.

The way I tested this was as follow:
Create a test file on the server:

# echo "TEST" > /var/lib/acme/acme-challenge/.well-known/acme-challenge/test
# chown acme /var/lib/acme/acme-challenge/.well-known/acme-challenge/test
# chgrp nginx /var/lib/acme/acme-challenge/.well-known/acme-challenge/test

Try to download it on a client:
$ curl http://news.rinsa.eu/.well-known/acme-challenge/test
It would say 403 Forbidden , so I check the nginx log on the server:
# journalctl -u nginx --since '5 minutes ago'
which said:
nginx[71033]: 2021/06/20 11:16:59 [error] 71033#71033: *322 open() "/var/lib/acme/acme-challenge/.well-known/acme-challenge/test" failed (13: Permission denied)
Indicating the there was a file permission problem.

Turn out the directory: /var/lib/acme/acme-challenge/ had both owner and group set to acme, and didn’t have executable bit set for others, so nginx couldn’t browse this directory.

Running the following command fixed it:

# chmod o+x /var/lib/acme/acme-challenge/

I have no idea what changed that caused it to stop working, or if this is the “proper” fix, but things do seem to work now.