Nginx acme - what am I missing?

The acme verification always fails with

Aug 11 22:25:48 rpi acme-biscotty.online-start[72484]: + exit 10
Aug 11 22:25:48 rpi systemd[1]: acme-biscotty.online.service: Main process exited, code=exited, status=10/n/a
Aug 11 22:25:48 rpi systemd[1]: acme-biscotty.online.service: Failed with result 'exit-code'.
Aug 11 22:25:48 rpi systemd[1]: Failed to start Renew ACME certificate for biscotty.online.
Aug 11 22:25:48 rpi systemd[1]: acme-biscotty.online.service: Consumed 340ms CPU time, received 21.3K IP traffic, sent 11.0K IP traffic.

my configuration.nix

  # Open ports in the firewall.
  networking.firewall.allowedTCPPorts = [ 3000 80 443 ];
  networking.firewall.allowedUDPPorts = [ 3000 80 443 ];
  # Or disable the firewall altogether.
  # networking.firewall.enable = false;

  services.nginx.package = pkgs.nginxStable.override { openssl = pkgs.libressl; };
  services.nginx.enable = true;
  services.nginx.virtualHosts."biscotty.online" = {
    addSSL = true;
    enableACME = true;
    root = "/home/biscotty/www/biscotty.online";
  };
   security.acme = {
    acceptTerms = true;
    defaults.email = "biscotty666@gmail.com";
  };

What am I missing?

PS. I’ll post another about this but if it’s a quick answer: what are the permissions needed for root www directory? I was a little surprised that this directory wasn’t auto-created, no biggie, I’m getting 403 Forbidden from nginx when trying to access the main page (via http with curl).

I’m still stuck on this. I have a new domain name which is active and points to my modem. The modem passes 80 and 443 to my server. Ports 80 and 443 are open on the server. I can curl the domain name and get a response from nginx.

So what could be wrong?

Your permissions; the acme service needs sufficient permissions to create some temporary files in that directory, and nginx needs permission to at least read them. This is also likely why the directory isn’t created by default; a random system user can’t just rummage in your home dir.

I’d suggest not creating this in your home directory, but sticking to the default in /var/lib. Failing that, you’ll need to figure out the correct user/group ownership for these files (and likely make them inaccessible by your real user).

I had originally tried it in /var/lib/www/biscotty.online and it didn’t work. Just tried again and no joy. I can’t find the config file ( looked in /run/current-system/etc ) so I can’t confirm the nginx user but I assume it is ‘nginx’. No /var/lib/www/biscotty.me directory was created so I created it and chown’ed the directories to ‘nginx’.

No matter what I still get the same error.

The default for acme is /var/lib/acme/<domain>. The directories are owned by acme:nginx, and have 750 permissions for me. Can you share the output of journalctl -xe --unit acme-biscotty.online.service?

You can confirm the user of the nginx process with ps aux | grep nginx, and checking which username is on the very left; it appears to indeed be nginx.

I don’t have any manually installed html files, but for nextcloud the permissions are a+r, since it’s in the nix store. The acme directories should be created automatically, but I think the nginx ones won’t be indeed.

Can you run:

chown -R nginx:nginx /var/lib/www/biscotty.online
find -type d /var/lib/www/biscotty.online -exec chmod 750 {} +
find -type f /var/lib/www/biscotty.online -exec chmod 640 {} + 

Just to ensure the permissions are set correctly and recursively?

I do believe we’re getting somewhere. Using the default location seems to have done the trick for the permission problems and I can curl http now. Thanks for that.

Indeed there was an entry in /var/lib/acme for my domain name. I learned something there, too. But it still doesn’t work. Here is the journalctl output, which just says it failed, at least to me. (BTW the ownership of the acme directory is acme:nginx).

journalctl -xe --unit acme-biscotty.online.service
Aug 14 04:22:38 rpi acme-biscotty.online-start[83081]: [biscotty.online] acme: error: 4>
Aug 14 04:22:38 rpi acme-biscotty.online-start[83076]: + echo Failed to fetch certifica>
Aug 14 04:22:38 rpi acme-biscotty.online-start[83076]: Failed to fetch certificates. Th>
Aug 14 04:22:38 rpi acme-biscotty.online-start[83076]: + exit 10
Aug 14 04:22:38 rpi systemd[1]: acme-biscotty.online.service: Main process exited, code>
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ An ExecStart= process belonging to unit acme-biscotty.online.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 10.
Aug 14 04:22:38 rpi systemd[1]: acme-biscotty.online.service: Failed with result 'exit->
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit acme-biscotty.online.service has entered the 'failed' state with result 'ex>
Aug 14 04:22:38 rpi systemd[1]: Failed to start Renew ACME certificate for biscotty.onl>
░░ Subject: A start job for unit acme-biscotty.online.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit acme-biscotty.online.service has finished with a failure.
░░
░░ The job identifier is 36860 and the job result is failed.
Aug 14 04:22:38 rpi systemd[1]: acme-biscotty.online.service: Consumed 318ms CPU time, >
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit acme-biscotty.online.service completed and consumed the indicated resources.
lines 347-378/378 (END)

I did manage to find nginx.conf in the /nix/store. It doesn’t appear to be listening on 443, only 80. Perhaps if I manually add it? Do modifications made in this file persist across rebuilds? I’m used to multiple config files including a conf.d style one when working with nginx. Where would I find such files/directories or where should I create them?

So many questions. Thanks.

You’ll find that anything in /nix/store is read-only, and you won’t be able to edit the file even as root. If you ever bypass the read-only bit on that anyway you risk borking your system by making the nix store inconsistent. Don’t touch things in /nix/store beyond some reading, it’s not worth it.

You won’t. In the NixOS world, services are configured using the NixOS module system, i.e. in your configuration.nix. If you want to manually add things to your nginx config you use services.nginx.appendConfig.

The multi-files stuff isn’t really necessary, it’s just done for organization. You can use the NixOS module system (and especially the cool virtualHost options) instead for organization.

Adding it manually shouldn’t be required, you’ll notice multiple “server” sections, one of which should be listening on :443 for that domain (if not, your virtualHost may be missing some options, but I’m almost certain you just overlooked it).

You cut off the actual error message. You should be able to scroll to the right with the arrow keys while in the journalctl UI. Dump the log into a file with > if you want to share it.

For the acme question, indeed I see now this:

Aug 11 22:16:05 rpi acme-biscotty.online-start[70606]: 2023/08/11 22:16:05 Could not obtain certificates:
Aug 11 22:16:05 rpi acme-biscotty.online-start[70606]:         error: one or more domains had a problem:
Aug 11 22:16:05 rpi acme-biscotty.online-start[70606]: [biscotty.online] acme: error: 400 :: urn:ietf:params:acme:error:connection :: 76.127.10.154: Fetching http://biscotty.online/.well-known/acme-challenge/RnDJEmRoI92nTqmTWuRAW2ydG2Tq2DkVI5airH_V8-E: Timeout after connect (your server may be slow or overloaded)
Aug 11 22:16:06 rpi acme-biscotty.online-start[70598]: + echo Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Aug 11 22:16:06 rpi acme-biscotty.online-start[70598]: Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and deacme.txt

but the domain uses the default nameservers of the domain name provider (name.com), and in actual practice the domain resolves correctly.

You won’t. In the NixOS world, services are configured using the NixOS module system, i.e. in your configuration.nix. If you want to manually add things to your nginx config you use services.nginx.appendConfig … You can use the NixOS module system (and especially the cool virtualHost options) instead for organization.

I thought the answer would be like this. (Hoping, actually.) Where can I find human-readable documentation on nginx.appendConfig and using virtualHost options? Going to the link reminds me of the first time I looked at a man page lol. Is there a place where I could see an example or 2?

Adding it manually shouldn’t be required, you’ll notice multiple “server” sections, one of which should be listening on :443 for that domain (if not, your virtualHost may be missing some options, but I’m almost certain you just overlooked it).

Indeed netstat shows nginx listening on 443, though there really is only 1 server block in the config file.

pid /run/nginx/nginx.pid;
error_log stderr;
daemon off;
events {
}
http {
	include /nix/store/80fh5a5lh89bg60xs9a6a2230i0rnc5c-mailcap-2.1.53/etc/nginx/mime.types;
	types_hash_max_size 4096;
	include /nix/store/rmjzvb9qa511szqri9rm8256ashk5rhs-nginx-1.24.0/conf/fastcgi.conf;
	include /nix/store/rmjzvb9qa511szqri9rm8256ashk5rhs-nginx-1.24.0/conf/uwsgi_params;
	default_type application/octet-stream;
	ssl_protocols TLSv1.2 TLSv1.3;
	ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
	# $connection_upgrade is used for websocket proxying
	map $http_upgrade $connection_upgrade {
		default upgrade;
		''      close;
	}
	client_max_body_size 10m;
	server_tokens off;
	server {
		listen 0.0.0.0:80 ;
		listen [::0]:80 ;
		server_name biscotty.me ;
		root /var/www/biscotty.me;
	}
}

In configuration.nix I have.

  services.nginx.virtualHosts."biscotty.online" = {
    addSSL = true;
    enableACME = true;
    root = "/var/lib/www/biscotty.online";
  };

It must be the addSSL line that makes nginx listen on 443? And the second that creates the acme user and associated directories, yes?

Right, I figure that’s an older instance of the config file. Old instances won’t go away until you nix-collect-garbage.

To see what the current instance is, try systemctl status nginx.service, and then look at the CLI args to see what config file it’s using.

In theory, the NixOS manual should have such, but there is none for nginx.

The NixOS wiki can sometimes help, but it’s notoriously full of bad practices and outdated information.

NixOS has a smaller userbase than the big four, and most of our users tend to be heavy programmers, who mostly do their work upstream - this results in a poorly maintained wiki.

Your next best option are blogs and other unofficial sources, @solene usually covers topics like this, but their nginx-focused articles are focused around openbsd, so no luck there…

Yes, correct.

Hrm. This comes down to networking, and DNS configuration. A quick duck gives me this: Timeout after connect (your server may be slow or overloaded) · Issue #827 · win-acme/win-acme · GitHub

Any chance you’re having IPv6 issues?

On the NixOS end everything seems to be working now, nginx is responding and the acme bot is running - the error comes from their servers.

1 Like

I thought I would give an update on this. nginx is running fine with a self-signed certificate. Running certbot manually fails, so I’ve posed the question on letsencrypt’s site. I’ll let you know what they say.

Really it’s not a big deal at this point for me because I can just pay a couple of bucks to get one from my domain name provider. I’d just like to get this to work out of principle.

1 Like

I ended up getting a cert from my domain name provider so I’m not pursuing this further.

But I believe I was close to a solution, so I wanted to document it here in case someone comes across the same problem.

In working with the folks at letsencrypt it transpired that the nginx set-up per the wiki does not listen to port 80, which is required by acme/letsencrypt. I opened it with listen = [ { addr = "0.0.0.0"; port = 80; } { addr = "0.0.0.0"; port = 443; } ];. I could then see (netstat) that it was listening. It still failed to get the cert though, and since I went another route I didn’t pursue it further. But opening the port was/is a necessary step.

I sincerely doubt it is, I’ve been running with the acme options for years and have not had to manually screw with the nginx config besides adding the HSTS header: tlaternet-server/webserver.nix at master - tlaternet-server - Forgejo: Beyond coding. We Forge.

Also this is where the module figures out whether to add 443 to the listen block (which it should with the above config if you resolve the conditions): https://github.com/NixOS/nixpkgs/blob/bfd953b2c6de4f550f75461bcc5768b6f966be10/nixos/modules/services/web-servers/nginx/default.nix#L316

I suspect your configuration there is a no-op at least, and not actively harmful, as the server will already listen to 443.

You clearly also don’t need to add 443 to firewall.allowedTCPPorts, since the other cert is working… I really think that there’s an issue with either your or letsencrypt’s networking (maybe DNS has not yet updated on their end?). But well, not a problem if you already have a certificate I suppose.

My post was perhaps misleading. Port 443 wasn’t the problem and shouldn’t be in that line. It was port 80 that had no listener. I should also say that one idea floated was that my ISP is somehow interfering.

I think that my post is related

Maybe add some more infos

Sorry, but I never resolved the issue. I just got a certificate from my name provider and used that. I think it may be related to my ISP (xfinity), but never took the time to pursue it. I do hope someone can help.

Hey, sorry for chiming in late to the party. But that line sparked my interest!
I’m currently facing the same issue biscotty was.

Could not obtain certificates:
acme: error: 500 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:serverInternal :: Error creating new order
+ echo Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
+ exit 10

resolving my domain works fine (the AAAA record is correct an i can reach my server that way) BUT my isp uses CGNAT and therefor the A record my domain resolves to will not reach my server. I suppose thats the problem?

Is there a way i can tell acme to use ipv6 only or would telling dynv6.com (my dyndns provider) to only answer with AAAA records help (i dont know how i would do that so I#d prefer the former solution)?

update:
seems like security.acme.certs.<name>.dnsResolver could help if i put an ipv6 resolver there and then supply the certificate that was created that way to my nginx virtualhost by replacing .enableACME with the corresponding .useACMEHost

will try that later

edit: yes, that worked like a charm ( i just use the ipv6 address of one.one.one.one (cloudflare) as the resolver)!
P:S.: I also had to open up port 80 ofc

2 Likes

The ACME TLS challenge might also help. My PR here adds those options to NixOS:

This was the only way in my case to get Let’s Encrypt working with port 80 blocked and me not controlling the DNS.

2 Likes

Awesome. This has been a long-standing problem for me even before NixOS. I will try it when I get a new domain name. Thank you.

Would this work without the security.acme.certs setting described above?