Traefik OCI Container on Port 80

Help
1

For the life of me I cannot put my finger on why I am unable to cURL my traefik instance running via oci-containers. I am brand new to NixOS, so I am assuming I am missing something obvious, but alas, I have been unable to find it.

Here is the configuration for my container

traefik = {
        image = "traefik:v2.9";
        autoStart = true;
        extraOptions = ["--network=traefik"];
        environmentFiles = ["/etc/traefik/.env"];
        ports = [
          "80:80"
          "443:443"
          "8080:8080"
        ];
        volumes = [
          "/var/run/docker.sock:/var/run/docker.sock"
        ];
        cmd = [
          "--api.insecure=true"
          "--log.level=DEBUG"
          "--accesslog=true"
          "--providers.docker=true"
          "--providers.docker.exposedbydefault=false"
          "--providers.docker.swarmMode=false"
          "--providers.docker.network=traefik"
          "--entrypoints.web.address=:80"
          "--entrypoints.websecure.address=:443"
          "--entrypoints.websecure.forwardedHeaders.insecure=true"
          "--certificatesresolvers.letsencrypt.acme.dnsChallenge=true"
          "--certificatesresolvers.letsencrypt.acme.dnsChallenge.provider=cloudflare"
          "--certificatesresolvers.letsencrypt.acme.email=test@email.com"
          "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
        ];
        labels = {
          "traefik.enable" = "true";
          "traefik.http.routers.traefik.rule" = "Host(`traefik.test.test`)";
          "traefik.http.routers.traefik.entrypoints" = "websecure";
          "traefik.http.routers.traefik.tls.certresolver" = "letsencrypt";
          "traefik.http.routers.traefik.middlewares" = "authelia@docker";
          "traefik.http.services.traefik.loadbalancer.server.port" = "8080";
        };
      };

I do also have the ports open in the firewall, though I read that may not be strictly necessary.

When I cURL traefik from a container within the traefik network, I see the request appear in the access log. However, what I cannot figure out, is that when I cURL from my local machine (curl http://0.0.0.0) I get a 404 with no matching access log.

To me, it very much looks like something besides my traefik instance is listening on port 80 and responding with these 404s, but netstat disagrees, when I terminate my traefik instance, it shows no listeners.

Here is my curl output WITHOUT traefik running:

curl http://0.0.0.0
404 page not found

With traefik it is much the same

curl http://0.0.0.0
404 page not found

But again, there are no access logs in traefik to corroborate these requests. I think this is what has me dumbfounded, I cannot figure out where these requests are going and why they are not hitting my traefik instance.

If anyone can shed some light on the matter I will be extremely grateful.

EDIT: After some additional testing, I have some interesting results. I spun my oci-container config into a standard docker-compose file for a traefik deployment. When I spun that up listening on port 80, again I get no access logs, BUT, when I stand up the container listening on port 8081, I do get access logs when I curl locally.

It really looks like traefik isn’t actually listening to port 80, but the only proof I can find is the lack of access logs during the request.

1 Like
2

Okay, I finally got to the bottom of what was happening, but am struggling with why.

Turns out, k3s is a messy thing to install on nixos.

I have been playing with migrating some of my docker services to k8s, and had seen that k3s was in the nix wiki (K3s - NixOS Wiki). Unfortunately, it was not clear to me that running that service would bind to 80 and 443, nor was it clear that removing the package and service would not be enough to remove the lingering components.

It boils down to this issue: k3s: package k3s-killall.sh script · Issue #98090 · NixOS/nixpkgs · GitHub

While my current quandary is resolved, I would appreciate any insights into why I was unable to find the container listening to port 80 using netstat.

Best,
Tom

3

sudo ss -tlnp lists the processes listening to tcp ports, also does sudo netstat -tpln, you need the flag -p and also root privileges. A pid belonging to a component of k3s should have been listed.

EDIT: AFAIK k3s shouldn’t bind to 80 or 443 by default :confused:

4

Strange, as soon as I ran the k3s killall script, my system behaved exactly as I expected it to. It does appear that k3s stands up a traefik instance by default, I think this stack overflow issue is related: kubernetes - How to move k3s' ingress to another port - Stack Overflow

As for the listeners, I think you are spot on, I hadn’t been running those commands with sudo, which may explain why I did not see k3s components.

It does strike me as odd that I was able to start traefik “bound” to 80 and 443 from oci-containers if those ports were already in use.