Make ACME renew systemd-service depend on DNS/nss-lookup?

kvtb · May 29, 2020, 1:52pm

I wonder if the ACME renew service should depend on nss-lookup.target or not.
I have unbound as local DNS, when doing nixos-rebuild, the ACME renew services cannot resolve letsencrypt hosts because the local DNS (unbound) is not yet started.

Unbound systemd sets it should start before nss-lookup.

The most logical thing should be that the ACME renew services should execute after nss-lookup.target

I can fix this in my own configuration with some hacks, but I wonder if this shouldn;t be the default in NixOS?

fricklerhandwerk · May 30, 2020, 8:34am

Chances are high you are one of very few users of this setup, and it‘s awesome that you put your idea into the open instead of being satisfied with your hack. Why not submit a PR and include your reasoning? Seems like ideally it is just one line of code. I‘ll be happy to review if you reference the PR in this thread. That would be yet another little thing that Just Works™, and your contribution will make NixOS and the world a better place.

kvtb · July 5, 2020, 3:33pm

I think adding nss-lookup.target here (in after and/or wants?) should work:

github.com

NixOS/nixpkgs/blob/fbf553ea2b02cef3962480ed1929573de4429b56/nixos/modules/security/acme.nix#L319


          ++ concatLists (mapAttrsToList (name: root: [ "-d" name ]) data.extraDomains)
          ++ (if data.dnsProvider != null then [ "--dns" data.dnsProvider ] else [ "--http" "--http.webroot" data.webroot ])
          ++ optionals (cfg.server != null || data.server != null) ["--server" (if data.server == null then cfg.server else data.server)];
certOpts = optionals data.ocspMustStaple [ "--must-staple" ];
runOpts = escapeShellArgs (globalOpts ++ [ "run" ] ++ certOpts);
renewOpts = escapeShellArgs (globalOpts ++
  [ "renew" "--days" (toString cfg.validMinDays) ] ++
  certOpts ++ data.extraLegoRenewFlags);
acmeService = {
  description = "Renew ACME Certificate for ${cert}";
  after = [ "network.target" "network-online.target" ];
  wants = [ "network-online.target" ];
  wantedBy = mkIf (!config.boot.isContainer) [ "multi-user.target" ];
  serviceConfig = {
    Type = "oneshot";
    User = data.user;
    Group = data.group;
    PrivateTmp = true;
    StateDirectory = "acme/.lego/${cert} acme/.lego/accounts ${lpath}";
    StateDirectoryMode = if data.allowKeysForGroup then "750" else "700";
    WorkingDirectory = spath;

hmenke · August 10, 2020, 6:25am

I’ve followed your suggestion of adding nss-lookup.target to the after dependencies of the ACME renewal job:

{
  systemd.services."acme-henrimenke.com".after = lib.mkAfter [ "nss-lookup.target" ];
}

However, this still doesn’t fly for me. In the journal at boot, I can clearly see that the ACME renewal is started after Unbound, but it still fails:

Aug 10 04:42:29 henrimenke.com systemd[1]: Started Unbound recursive Domain Name Server.
Aug 10 04:42:29 henrimenke.com systemd[1]: Reached target Host and Network Name Lookups.
Aug 10 04:42:29 henrimenke.com systemd[1]: Starting Renew ACME Certificate for henrimenke.com...
Aug 10 04:42:29 henrimenke.com yz6frbp9w9s150pq449sxva06iyk95fz-acme-start[790]: 2020/08/10 04:42:29 Could not create client: get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get "https://acme-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-v02.api.letsencrypt.org: no such host
Aug 10 04:42:29 henrimenke.com yz6frbp9w9s150pq449sxva06iyk95fz-acme-start[790]: 2020/08/10 04:42:29 Could not create client: get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get "https://acme-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-v02.api.letsencrypt.org: no such host
Aug 10 04:42:30 henrimenke.com systemd[1]: acme-henrimenke.com.service: Main process exited, code=exited, status=1/FAILURE
Aug 10 04:42:30 henrimenke.com systemd[1]: acme-henrimenke.com.service: Failed with result 'exit-code'.
Aug 10 04:42:30 henrimenke.com systemd[1]: Failed to start Renew ACME Certificate for henrimenke.com.

kvtb · August 11, 2020, 7:31pm

I’m puzzled, I’m afraid I don’t know enough about both systemd and Nix to understand why this does not work

kvtb · September 18, 2020, 7:03pm

it seems the following (closed) GitHub issue is related:

nh2 · October 20, 2020, 7:23pm

Can you check if 20.09 fixes it for you?

It does for me: https://github.com/NixOS/nixpkgs/issues/85794#issuecomment-713085557

kvtb · November 1, 2020, 1:20pm

@nh2 unfortunately it doesn’t, it still tries to run ACME renew before unbound is started

hopefully this will solve it:

kvtb · May 1, 2021, 6:57pm

@andir does your refactor also (as a side effect) resolves the problem described in the first post?

andir · May 2, 2021, 5:31pm

Unfortunately it can’t fix that outside of system start. The way that NixOS is currently changing generations doesn’t allow enforcing service dependencies in targets that have previously been reached. It is a limitation with systemctl. There was some talk about rewriting our activation script with that limitation in mind but nobody has put in a lot of effort yet. @arianvp is one of those that I discussed it with in #nixos-systemd (on Freenode).

kvtb · May 11, 2021, 6:20pm

OK, thanks for the clarification! I now understand why this is a difficult issue.

nh2 · June 13, 2021, 12:03am

ACME renew fails if there is a local DNS server configured as the system resolver

opened 01:13AM - 14 Dec 20 UTC

m1cr0man

0.kind: bug 6.topic: nixos

**Describe the bug** During a `nixos-rebuild`, ACME renewal can fail because th…e enabled local DNS server (confirmed with bind and dnsmasq) is not ready to serve requests. **To Reproduce** Steps to reproduce the behavior: 1. Enable bind and an acme cert on your server 2. Perform a nixos-rebuild 3. The acme renew service usually fails **Expected behavior** The acme renew service should not fail. The solution here would require a generalised way to reliably wait on the DNS server to be online. Changes need to be made in the mentioned DNS server modules more than the ACME module, despite it appearing as an ACME failure. **Screenshots** See [this renew service output](https://github.com/NixOS/nixpkgs/issues/103121#issuecomment-744100208) **Notify maintainers** @nixos/acme **Metadata** ``` - system: `"x86_64-linux"` - host os: `Linux 5.4.83, NixOS, 21.03.git.16f80f8221cM (Okapi)` - multi-user?: `yes` - sandbox: `yes` - version: `nix-env (Nix) 2.3.9` - channels(root): `"nixos-21.03pre257780.e9158eca70a"` - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos` ``` Maintainer information: ```yaml # a list of nixpkgs attributes affected by the problem attribute: lego # a list of nixos modules affected by the problem module: security.acme ```