Tailscale + caddy certificates not working

ccalhoun · March 6, 2025, 12:36pm

I was following along with the below link to try and setup matrix, and wanted to use tailscale to expose the matrix server to my tailnet. I’ve heard it should “just work”, and it wasn’t, so I tried adding in caddy to the mix which also “should just work”, but doesn’t. I can ping the server, but it seems like https certs aren’t being handled? I’ve set tailscale.service.permitCertUid = "caddy" however it still doesn’t automatically handle certs for me. Am I missing something here? I can connect to it fine, but I don’t get an https connection, and when trying to connect using the element android app, it gets an ssl error.

config:

let # Change these values
  domain_name = "machine.magicdns.ts.net"; #actual name used here
  user_name = "test";
  secure_token = "test";
in

{
 networking = {
    firewall = {
      enable = true;
      allowedTCPPorts = [ 22 80 443 8448 ];
      allowedUDPPorts = [ 53 ];
    };
  };
  services = {
    caddy = {
      enable = true;
      virtualHosts = {
        "${domain_name}" = {
          extraConfig = ''
          # reverse_proxy /_matrix/* 127.0.0.1:6167 #I tried this as well
          reverse_proxy /_matrix/* 0.0.0.0:6167
          '';
        };
      };
    };
    matrix-conduit = {
      enable = true;
      settings.global = {
        # address = "127.0.0.1"; 
        address = "0.0.0.0";
        allow_registration = true;
        registration_token = "${secure_token}";
        database_backend = "rocksdb";
        port = 6167;
        server_name = "${domain_name}";
      };
    };

gist.github.com

https://gist.github.com/nat-418/1ba7609d42e27999759eb4b9cd9810cb

conduit-heisenberg-caddy-nixos.md

# Self-hosting Matrix with an IRC bridge on NixOS

## Warning

I initially wrote this after getting everything described set up. Since then I have learned the hard way that Matrix is actually broken in several important aspects and I do not recommend self-hosting it. I am leaving the Gist up for
reference and in the off-chance it may help others.

## Introduction

There are two main free and open-source technologies today for real-time

This file has been truncated. show original

configuration.nix

{ pkgs, ... }:

let # Change these values
  domain_name = "example.com";
  user_name = "alice";
  secure_token = "opensesame";
in

{
  config = {

This file has been truncated. show original

sebohe · March 6, 2025, 2:44pm

Hey, I did something using tailscale-nginx-auth and the certificates worked well with caddy.

This is a nixos module that I wrote:

{ config
, lib
, ...
}:
let
  cfg = config.ts-sso;
  inherit (lib) mkOption types mkIf;
in
{
  options.ts-sso = {
    enable = mkOption {
      type = types.bool;
      default = true;
    };
    group = mkOption {
      type = types.str;
      default = "tailnet";
    };
    tld = mkOption {
      type = types.str;
      default = "";
    };
    portToForward = mkOption {
      type = types.port;
      default = 80;
    };
    interface = mkOption {
      type = types.str;
      default = "eth0";
    };
  };
  config = mkIf (cfg.enable) {
    users.extraGroups.${cfg.group}.members = [
      "caddy"
      "tailscale"
      "tailscale-nginx-auth"
    ];

    services.tailscale.permitCertUid = "caddy";
    services.tailscaleAuth = {
      enable = true;
      socketPath = "/run/ts.sock";
      group = cfg.group;
    };

    networking.firewall.interfaces."${cfg.interface}".allowedTCPPorts = [
      80
      443
    ];
    services.caddy = {
      group = "tailnet";
      enable = true;
      logFormat = ''
        level WARN
      '';
      virtualHosts."${config.networking.hostName}.${cfg.tld}" = {
        listenAddresses = [
          "0.0.0.0"
        ];
        extraConfig = ''
          forward_auth unix/${config.services.tailscaleAuth.socketPath} {
           uri /auth
           header_up Remote-Addr {remote_host}
           header_up Remote-Port {remote_port}
           header_up Original-URI {uri}
           copy_headers {
             Tailscale-User>X-Webauth-User
             Tailscale-Name>X-Webauth-Name
             Tailscale-Login>X-Webauth-Login
             Tailscale-Tailnet>X-Webauth-Tailnet
             Tailscale-Profile-Picture>X-Webauth-Profile-Picture
           }
          }
          reverse_proxy :${builtins.toString cfg.portToForward}
        '';
      };
    };
  };
}

ccalhoun · March 11, 2025, 7:34am

Thanks for the config! I’ve got it building but still can’t get a secure connection to the server.

I put my tailnet ***.ts.net into the tld option, and tried connecting to port 80, 443, no port, adding conduit (6167) to the config and connecting to that, but none of them can get a secure connection.

sebohe · March 17, 2025, 10:18am

I believe that caddy obtains the certificates from LetsEncrypt and doesn’t allow direct access to port 80, since after the certificate generation is complete, it automatically forwards all requests to 443. Also you can only access the port through the domain: ..ts.net.

You need to modify the caddy (Caddyfile) config a bit more to not automatically forward/upgrade the ports you want to access.

nolan · March 17, 2025, 9:36pm

Sorry if this isn’t what you’re after–there are a few ways to solve this after all–but I took the route of installing caddy-tailscale as a Caddy plugin. This lets me do something like:

    services.caddy.virtualHosts."https://audiobookshelf.tailcwtfever.ts.net".extraConfig = ''
      bind tailscale/audiobookshelf
      reverse_proxy localhost:8000
    '';

This automatically creates a Tailscale machine/device called audiobookshelf and proxies that to localhost:8080.

{
  inputs = {
    nixpkgs.url = "nixpkgs";
    flake-utils.url = "github:numtide/flake-utils";
    caddy.url = "github:vincentbernat/caddy-nix";
  };
  outputs =
    {
      self,
      nixpkgs,
      flake-utils,
      caddy,
    }:
    flake-utils.lib.eachDefaultSystem (
      system:
      let
        pkgs = import nixpkgs {
          inherit system;
          overlays = [ caddy.overlays.default ];
        };
      in
      {
        packages = {
          default = pkgs.caddy.withPlugins {
            plugins = [ "github.com/tailscale/caddy-tailscale@f21c01b660c896bdd6bacc37178dc00d9af282b4" ];
            hash = "sha256-WCyobNu2We2q/wP8H3C3pwxmXQ4cqybsNKL3nOSHrFo=";
          };
        };
      }
    );
}

One problem I have with this setup is that the caddy systemd unit sometimes hangs and times out. I haven’t fully figured out if this only happens when provisioning a new device, but either way, ctrl-c’ing out of the rebuild and systemctl restart caddy seems to get things unstuck, including finishing the rebuild I thought I’d killed. If you figure out what’s up with this, I’d certainly appreciate the fix. Could be it’s fixed in newer Caddy/plugin versions but I haven’t tried bumping those yet. I don’t much like killing rebuilds but they seem to pick up like nothing went wrong when caddy is restarted manually.

Otherwise, no, I’m not sure how to proxy from Tailscale magic domain names to something local without explicitly using either the tailscale binary or a plugin/integration of some sort.

ccalhoun · March 18, 2025, 12:54am

I’m still working this out (I suck at nixlang so its always a nightmare figuring out how to get stuff where it needs to be), but it seems like you no longer need that flake as it was merged into nixpkgs: https://github.com/NixOS/nixpkgs/pull/358586

ccalhoun · March 18, 2025, 1:20am

So that sort of works, but the caddy service fails to start.

reloading the following units: caddy.service
Failed to reload caddy.service
warning: the following units failed: caddy.service
● caddy.service - Caddy
     Loaded: loaded (/etc/systemd/system/caddy.service; enabled; preset: ignored)
    Drop-In: /nix/store/07kss5a0fys2bgjc3lv20m3wpvp80cw6-system-units/caddy.service.d
             └─overrides.conf
     Active: activating (auto-restart) (Result: timeout) since Tue 2025-03-18 10:14:11 JST; 261ms ago
 Invocation: 3bcfda84cc2f4a8889520ebc7644036f
       Docs: https://caddyserver.com/docs/
    Process: 450223 ExecStart=/nix/store/38f10xbjg129haa24gxbiz6m7lkcsic4-caddy-2.9.1/bin/caddy run --config /etc/caddy/caddy_config --adapter caddyfile (code=killed, signal=KILL)
   Main PID: 450223 (code=killed, signal=KILL)
     Status: "Stopped; run 'tailscale up' to log in"
         IP: 14.2K in, 22.3K out
         IO: 0B read, 32K written
   Mem peak: 19.6M
        CPU: 142ms

Is there some sort of authentication I need to setup manually? I thought it would just piggy-back off the host to work.

{
  pkgs,
  ...
}:
let # Change these values
  domain_name = "aaa.aaa.ts.net";
  secure_token = "test";
in
{
  services = {
    caddy = {
      enable = true;
      package = pkgs.caddy.withPlugins {
        plugins = [ "github.com/tailscale/caddy-tailscale@v0.0.0-20250207163903-69a970c84556" ];
        hash = "sha256-UR9CG/zIslkXHDj1fDWmhx8hJZ8VLvZzOTGvGqqx1Ls=";
      };
      virtualHosts."https://matrix.aaa.ts.net".extraConfig = ''
        bind tailscale/matrix
        reverse_proxy localhost:6167
      '';
    };
    matrix-conduit = {
      enable = true;
      settings.global = {
        # address = "127.0.0.1"; 
        address = "0.0.0.0";
        # Change this to `false` after the first user (admin) is registered,
        # and then run `$ nixos-rebuild switch`.
        allow_registration = true;
        registration_token = "${secure_token}";
        database_backend = "rocksdb";
        port = 6167;
        server_name = "${domain_name}";
      };
    };
  };
}

ccalhoun · March 18, 2025, 4:39am

I also tried with audiobookshelf on port 8000 just to see if I could get anything to work, but it doesn’t. I can confirm caddy is working using their hello world example, but all the tailscale plugin seems to do is cause timeouts when starting the caddy service. The status is “stopped; run ‘tailscale up’ to log in”

ccalhoun · March 18, 2025, 5:17am

Ah, I see, it requires an auth key in order to make the nodes. Making one with agenix gets this working. Still figuring out https though.

ccalhoun · March 18, 2025, 5:22am

https is working! The authkey was the missing piece! Thanks @nolan