Traefik OCI Container on Port 80

Help
1

For the life of me I cannot put my finger on why I am unable to cURL my traefik instance running via oci-containers. I am brand new to NixOS, so I am assuming I am missing something obvious, but alas, I have been unable to find it.

Here is the configuration for my container

traefik = {
        image = "traefik:v2.9";
        autoStart = true;
        extraOptions = ["--network=traefik"];
        environmentFiles = ["/etc/traefik/.env"];
        ports = [
          "80:80"
          "443:443"
          "8080:8080"
        ];
        volumes = [
          "/var/run/docker.sock:/var/run/docker.sock"
        ];
        cmd = [
          "--api.insecure=true"
          "--log.level=DEBUG"
          "--accesslog=true"
          "--providers.docker=true"
          "--providers.docker.exposedbydefault=false"
          "--providers.docker.swarmMode=false"
          "--providers.docker.network=traefik"
          "--entrypoints.web.address=:80"
          "--entrypoints.websecure.address=:443"
          "--entrypoints.websecure.forwardedHeaders.insecure=true"
          "--certificatesresolvers.letsencrypt.acme.dnsChallenge=true"
          "--certificatesresolvers.letsencrypt.acme.dnsChallenge.provider=cloudflare"
          "--certificatesresolvers.letsencrypt.acme.email=test@email.com"
          "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
        ];
        labels = {
          "traefik.enable" = "true";
          "traefik.http.routers.traefik.rule" = "Host(`traefik.test.test`)";
          "traefik.http.routers.traefik.entrypoints" = "websecure";
          "traefik.http.routers.traefik.tls.certresolver" = "letsencrypt";
          "traefik.http.routers.traefik.middlewares" = "authelia@docker";
          "traefik.http.services.traefik.loadbalancer.server.port" = "8080";
        };
      };

I do also have the ports open in the firewall, though I read that may not be strictly necessary.

When I cURL traefik from a container within the traefik network, I see the request appear in the access log. However, what I cannot figure out, is that when I cURL from my local machine (curl http://0.0.0.0) I get a 404 with no matching access log.

To me, it very much looks like something besides my traefik instance is listening on port 80 and responding with these 404s, but netstat disagrees, when I terminate my traefik instance, it shows no listeners.

Here is my curl output WITHOUT traefik running:

curl http://0.0.0.0
404 page not found

With traefik it is much the same

curl http://0.0.0.0
404 page not found

But again, there are no access logs in traefik to corroborate these requests. I think this is what has me dumbfounded, I cannot figure out where these requests are going and why they are not hitting my traefik instance.

If anyone can shed some light on the matter I will be extremely grateful.

EDIT: After some additional testing, I have some interesting results. I spun my oci-container config into a standard docker-compose file for a traefik deployment. When I spun that up listening on port 80, again I get no access logs, BUT, when I stand up the container listening on port 8081, I do get access logs when I curl locally.

It really looks like traefik isn’t actually listening to port 80, but the only proof I can find is the lack of access logs during the request.

1 Like
2

Okay, I finally got to the bottom of what was happening, but am struggling with why.

Turns out, k3s is a messy thing to install on nixos.

I have been playing with migrating some of my docker services to k8s, and had seen that k3s was in the nix wiki (K3s - NixOS Wiki). Unfortunately, it was not clear to me that running that service would bind to 80 and 443, nor was it clear that removing the package and service would not be enough to remove the lingering components.

It boils down to this issue: k3s: package k3s-killall.sh script · Issue #98090 · NixOS/nixpkgs · GitHub

While my current quandary is resolved, I would appreciate any insights into why I was unable to find the container listening to port 80 using netstat.

Best,
Tom

3

sudo ss -tlnp lists the processes listening to tcp ports, also does sudo netstat -tpln, you need the flag -p and also root privileges. A pid belonging to a component of k3s should have been listed.

EDIT: AFAIK k3s shouldn’t bind to 80 or 443 by default :confused:

4

Strange, as soon as I ran the k3s killall script, my system behaved exactly as I expected it to. It does appear that k3s stands up a traefik instance by default, I think this stack overflow issue is related: kubernetes - How to move k3s' ingress to another port - Stack Overflow

As for the listeners, I think you are spot on, I hadn’t been running those commands with sudo, which may explain why I did not see k3s components.

It does strike me as odd that I was able to start traefik “bound” to 80 and 443 from oci-containers if those ports were already in use.

5

I had exactly the Same Issue.
Traefik Returned 404, even when the Service was available otherwise.

I wanted to manually deploy a Traefik Container via Podman, while having k3s Installed.
The k3s Traefik Instance snatched the Requests, resulting in a 404 for the Service behind the subdomain.domain Call. But the Service on its Exposed Port was available.
So the Problem was definitely within Traefik.
The Issue was driving me Insane, as I couldn’t figure out why not even the Hello World Example provided by Traefik would run on my NixOs Setup.

Unfortunately, I Personally don’t see K3S / Traefik listening to Port 80, or 443, with the Commands proposed by @aorith.

(But I’m still getting into this stuff.)

Temporarily removing k3s from my NixOs Config Resolved the Problem (for now).
And made the Service on subdomain.domain available.

I don’t think this Behavior can be justified with User Error alone.
There should be At least a warning somewhere, in my Opinion.

Where would you suggest would be the Right Place to Write a Bug Report be?
K3s in itself? The NixOs Repository for the K3s Package?
Or is it a Problem from the OCI Container Software itself not Warning the User about a Dual Used Port?

Additonal Note: I didn’t have to use the killall Script, the NixOs rebuild boot (without k3s) & a Reboot did the Trick for me.

6

I had to tinker with a Traefik container with a docker compose image once and had a similar problem.
I knew there was a server there AND running, but the site flatly refused to reply to anything unless it was formatted in a specific way.

I fail to recall many of the details, but the short version is the exposed service from traefik can end up mutated to only accept calls or GET requests from a certain domain or address.
My issue was like yours, i knew it was there, could find the container but couldnt connect to it.
Turns out it would only allow connections from the configured domain name, it would not accept a connection from a local address or direct ip connection.

You need to set traefik to explicitly allow traffic from sources before it will actually pass it through. Its really picky.
The module i was working with was prebaked into the container i forked so i never could dig much into how it was connected, but it was set as an env variable in the compose.yml and that was passed to traefik to expose the domain only, disallowing any and all other connections.
Had to do some redirect wizardry to make it work without editing the source directly.

You may find this discussion helpful: How to access traefik via ip address (both local and public)? - Traefik v2 - Traefik Labs Community Forum

7

You don’t see anything listening on ports 80 & 443 because the Klipper Service Load Balancer sets up iptables rules to redirect traffic destined to these ports (see here). You can see the klipper-lb pods with a svclb-traefik- prefix in the kube-system namespace.

The svclb pods can be disabled by setting --disable=servicelb on every server node, but this also disables routing traffic for ports 80 & 443 to the cluster.

See also this k3s discussion: What does ServiceLB (Klipper LB) actually do ? · k3s-io/k3s · Discussion #9927 · GitHub

1 Like