Why did pihole container stop working?

Help
1

I have a Raspberry Pi running couple of services including pihole on local network. This morning I noticed pihole was not working. podman ps returned empty. This is my config:

{ config, pkgs, lib, ... }:

let
  serverIP = "0.0.0.0";
in
{
  virtualisation.oci-containers.containers.pihole = {
    image = "pihole/pihole:latest";
    ports = [
      "${serverIP}:53:53/tcp"
      "${serverIP}:53:53/udp"
      "3080:80"
      "30443:443"
    ];
    volumes = [
      "/var/lib/pihole/:/etc/pihole/"
      "/var/lib/dnsmasq.d:/etc/dnsmasq.d/"
    ];
    environment = {
      ServerIP = serverIP;
      TZ = "Asia/Kolkata";
    };
    extraOptions = [
      "--cap-add=NET_ADMIN"
      "--dns=127.0.0.1"
      "--dns=1.1.1.1"
    ];
    workdir = "/var/lib/pihole/";
  };
}

It was taken from here with minor updates and has been working nicely.

Manually doing below seems to be working:

sudo podman run -dt -p 53:53/tcp -p 53:53/udp -p 3080:80/tcp docker.io/pihole/pihole:latest

Why would declarative configuration not work and how can it be fixed?

2

Please check logs with command journalctl -u podman-pihole.service.

1 Like
3

Huh, this is weird:

May 10 07:25:38 host systemd[1]: Started podman-pihole.service.
May 10 07:25:38 host podman-pihole-start[1108]: Resolving "pihole/pihole" using unqualified-search registries (/etc/containers/registries.conf)
May 10 07:25:38 host podman-pihole-start[1108]: Trying to pull docker.io/pihole/pihole:latest...
May 10 07:25:38 host podman-pihole-start[1108]: Trying to pull quay.io/pihole/pihole:latest...
May 10 07:25:38 host podman-pihole-start[1108]: Error: 2 errors occurred while pulling:
May 10 07:25:38 host podman-pihole-start[1108]:  * initializing source docker://pihole/pihole:latest: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2>
May 10 07:25:38 host podman-pihole-start[1108]:  * initializing source docker://quay.io/pihole/pihole:latest: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp: look>
May 10 07:25:38 host systemd[1]: podman-pihole.service: Main process exited, code=exited, status=125/n/a
May 10 07:25:38 host podman-pihole-post-stop[1120]: Error: no container with name or ID "pihole" found: no such container
May 10 07:25:38 host systemd[1]: podman-pihole.service: Failed with result 'exit-code'.

So if I’m understanding correctly, simple ‘pihole/pihole’ is not fully qualified, understandably, so it looks up on docker.io. But that lookup fails. But same search via manual command works as mentioned in OP. Not sure what to make of this…

4

Have you set your pihole as the only one DNS provider for your RPI? It might be you’re having some DNS issues.

5

Yes, only DNS provider. After it stopped working this morning, I restored the ISP DNS, ran ‘nix-collect-garbage’ and rebuilt the system, but getting same error.

6

Does command sudo podman pull pihole/pihole work for you? If it does not work then check your /etc/resolve.conf. If its nameserver is 127.0.0.1 you might change it to Cloudflare’s 1.1.1.1 or Quad9’s 9.9.9.9 and try that command again.

7

I rebuilt the system again, manually restarted the service and got a new one this time:

podman-pihole-start[7289]: Error: workdir "/var/lib/pihole/" does not exist on container 0b2fb492fc4982814b6f8db53df4767527756fb3decb3bb45efce115481caaec

I’m assuming this means some the workdir from config is not matching with that of the container. I removed the line from config, rebuilt and pihole seems to be working fine. No idea of the significance of that directory, though it seems to have few files important for pihole operation like hosts list and whatnot.