Snakeoil ACME Server in NixOS Test

I’m writing a test using pkgs.testers.runNixOSTest for my own server where I have a fixture ACME server to provide certificates. I use the certificates in Caddy like so:

{
  services.caddy.virtualHosts.${cfg.domain} = ''
        tls "${config.security.acme.certs.${cfg.domain}.directory}/fullchain.pem" "${config.security.acme.certs.${cfg.domain}.directory}/key.pem"
        reverse_proxy unix/${some-socket}
        handle /.well-known/acme-challenge* {
          root * ${cfg.webroot}
          file_server
        }
  '';
    security.acme.certs.${cfg.domain} = {
      inherit (cfg) domain webroot;
      inherit group;
      reloadServices = ["caddy.service"];
    };
}

In order for the testing server to be issued certificates, I added a fake ACME and fake DNS servers, and the "${inputs.nixpkgs}/nixos/tests/common/acme/client" module to the server so it trusts the root certificate coming from acme:

# nodes
let
  dnsServerIP = nodes: nodes.dnsserver.networking.primaryIPAddress;
in
{
      dnsserver = {nodes, ...}: {
      networking = {
        firewall.allowedTCPPorts = [
          8055
          53
        ];
        firewall.allowedUDPPorts = [53];

        # nixos/lib/testing/network.nix will provide name resolution via /etc/hosts
        # for all nodes based on their host names and domain
        hostName = "dnsserver";
        domain = "test";
      };
      systemd.services.pebble-challtestsrv = {
        enable = true;
        description = "Pebble ACME challenge test server";
        wantedBy = ["network.target"];
        serviceConfig = {
          ExecStart = "${pkgs.pebble}/bin/pebble-challtestsrv -dns01 ':53' -defaultIPv6 '' -defaultIPv4 '${nodes.server.networking.primaryIPAddress}'";
          # Required to bind on privileged ports.
          AmbientCapabilities = ["CAP_NET_BIND_SERVICE"];
        };
      };
    };
    acme = {
      nodes,
      config,
      modulesPath,
      pkgs,
      lib,
      inputs,
      ...
    }: {
      imports = [
        "${inputs.nixpkgs}/nixos/tests/common/acme/server"
        (modulesPath + "/profiles/qemu-guest.nix")
      ];
      networking.nameservers = lib.mkForce [(dnsServerIP nodes)];
    };
    # Web interface node
    server = {
      nodes,
      modulesPath,
      config,
      pkgs,
      lib,
      inputs,
      ...
    }: {
      imports = [
        "${inputs.nixpkgs}/nixos/tests/common/acme/client"
        (modulesPath + "/profiles/qemu-guest.nix")
      ];
      networking.nameservers = lib.mkForce [(dnsServerIP nodes)];
      system.production = lib.mkForce true;
      services = {
        caddy.acmeCA = config.security.acme.defaults.server;
      };
    };
  };
}

After server starts running, I ran openssl s_client -showcerts -connect subdomain.example.com:443 to check that the snakeoil ACME server produced trusted certificates, but the result is

CONNECTED(00000003)
---
Certificate chain
 0 s:
   i:CN=Pebble Intermediate CA 1c7953
   a:PKEY: EC, (prime256v1); sigalg: sha256WithRSAEncryption
   v:NotBefore: Mar 15 07:00:28 2026 GMT; NotAfter: Mar 15 07:00:27 2031 GMT
-----BEGIN CERTIFICATE-----
MIICr..... (deleted because its too long)
-----END CERTIFICATE-----
 1 s:CN=Pebble Intermediate CA 1c7953
   i:CN=Pebble Root CA 4b1bb2
   a:PKEY: RSA, 2048 (bit); sigalg: sha256WithRSAEncryption
   v:NotBefore: Mar 15 07:00:00 2026 GMT; NotAfter: Mar 15 07:00:00 2056 GMT
-----BEGIN CERTIFICATE-----
MIIDU... (deleted because its too long)
-----END CERTIFICATE-----
---
Server certificate
subject=
issuer=CN=Pebble Intermediate CA 1c7953
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ecdsa_secp256r1_sha256
Negotiated TLS1.3 group: X25519MLKEM768
---
SSL handshake has read 2994 bytes and written 1606 bytes
Verification error: unable to get local issuer certificate

The most notable thing here is that no certificate in this chain of certificates matches the snakeoil in nixos/tests/common/acme/server. All acme-order-renew-... services on server exited gracefully, which would not happen had acme not exist.

server # [   26.760672] systemd[1]: acme-order-renew-subdomain.example.com.service: Deactivated successfully.

Why does the certificate chain not contain the trusted snakeoil?

See here for a MWE.

On letsencrypt discourse, a commenter said that pebble generates a new cert every time it starts up. This poses a problem for my tests since I have another service that validates the SSL certificate generated by an ACME server.

2 Likes

The moving parts here aren’t trivial to use together, so it’s no surprise you’re getting confused. My best advice is to look at nixos/tests/acme/caddy.nix, and its dependency nixos/tests/acme/python-utils.py, for pointers on how you should set up your tests.

But basically, Pebble isn’t going to use the minica-generated certificates for issuing; it will generate its own, and then the test script has to download the Pebble-generated root CA and use it. (This is just how Pebble works; NixOS can’t do anything about it except use a different ACME implementation for testing.) python-utils.py makes this a bit cleaner to do (see download_ca_certs and check_connection), but you may need to pick apart its guts if you’re doing something more complex.

You shouldn’t have to run your own Pebble instance if you’re importing nixos/tests/common/acme/server. Again, see nixos/tests/acme/caddy.nix. (Though don’t cargo-cult everything from that file; there’s a lot of extra complexity due to wanting to test alternate subdomains and such. Study it until you understand it and then take only what you need.)

1 Like

The tests in nixos/tests/acme can use python-utils.py since there is no downstream service that checks certificates. In my use case, I have another service which depends on caddy that requires the caddy certificates to be valid. For this reason I created a service that starts before caddy, but somehow this manual replacement of certificates does not work either.

#!/bin/sh

curl https://acme.test:15000/roots/0 > /tmp/pebble-ca.pem
cat $(readlink -f /etc/ssl/certs/ca-certificates.crt) /tmp/pebble-ca.pem > /tmp/ca-bundle.crt
rm -f /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-bundle.crt
cp /tmp/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt
cp /tmp/ca-bundle.crt /etc/ssl/certs/ca-bundle.crt
      systemd.services.pebble-cert = {
        enable = true;
        description = "Add Pebble certificate to root";
        after = ["network.target"];
        wantedBy = ["caddy.service"];
        serviceConfig = {
          ExecStart = "${./replace-cert.sh}";
        };
      };

What is the other service? I expect you’d have more luck if you can set SSL_CERT_FILE or something for that service’s environment, instead of trying to hack up the symlinks NixOS creates in /etc.

The other services are openbao and continuous integration services that rely on the openbao secret vault

Hm. I don’t know anything about OpenBao, but I would try experimenting with something like systemd.services.openbao.environment.SSL_CERT_FILE = "/var/some/path/pebble-ca.pem"; and seeing if that gets you anywhere. (You might also need to concatenate the downloaded cert with the system bundle.)

does that mean I will have to set this environment variable for every service that uses openssl to access a host with certificate issued by pebble?

Or you can use systemd.globalEnvironment if you’d prefer, but I recommend going service by service — making changes in smaller scopes, one at a time, makes it easier to keep track of the things on which this trick works and what needs a different solution. (It sounds like you’re trying to build a very complicated test, so who knows what might break if you set this?)

Rather than doing this, is there any reason to use pebble compared to other ACME implementations like step-ca?

No idea, but obviously if you do that you won’t be able to reuse the NixOS ACME testing infrastructure.

I ended up using step-ca since in which case I can provision the root cert. Test file: From git-bugreport(1) in that changeset, not just the files on the Unix domain.

Strangely, the test does not work if I remove the intermediate cert from trusted certs.

      security.pki.certificateFiles = lib.mkForce [
        "${test-certificates}/root_ca.crt"
        "${test-certificates}/intermediate_ca.crt"
      ];

If you get things working, I’d really appreciate a more detailed Guides - this is a really useful thing to know how to set up for tests and staging environments.

Currently I just hard-code pre-generated root certs, overriding the acme settings, but being able to test everything e2e is a longer-term goal I have.

I’ll write an e2e testing guide. The reason I could not use pre-generated certs here is because I want to test a secret engine with continuous integration. They authenticate with each other using TLS client certificates.

1 Like

I wrote a guide here for using ACME fixtures. I use this to e2e test openbao and Concourse CI without them complaining about insecure TLS.

3 Likes