Nixos kubernetes localhost:8080 was refused

Help
1

Following the guide on the kubernetes page to a tee I get this issue with my setup:

sudo kubectl cluster-info dump                                         
The connection to the server localhost:8080 was refused - did you specify the right host or port?

sudo kubectl cluster-info     
E0223 20:32:29.693102   38190 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused"
E0223 20:32:29.694741   38190 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused"
E0223 20:32:29.696175   38190 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused"
E0223 20:32:29.697626   38190 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused"
E0223 20:32:29.699242   38190 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused"

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server localhost:8080 was refused - did you specify the right host or port?

I have firewall disabled, and I am using systemd.network if that matters:

  networking.hostName = "hostname"; # Define your hostname.
  networking.networkmanager.enable = false;
  systemd.network.enable = true;
  networking.useNetworkd = true;
  networking.extraHosts = "${kubeMasterIP} ${kubeMasterHostname}";
systemd.network.networks."10-wan" = {
    matchConfig.Name = "enp40s0";
    address = [
        "10.3.0.53/24"
    ];
    domains = [
      "internal.lan"
    ];
    routes = [
      { Gateway = "10.3.0.1"; }
    ];
    linkConfig.RequiredForOnline = "routable";
  };

I just want to create a simple server kubernetes node for learning purposes… I tried k3s and that worked great, but I want to go the next step up.

2

Hi Caret,

I’m not an expert with the default Kubernetes distro anymore, but I’ll try to help.
In the wiki ( It’s best to use Kubernetes - NixOS Wiki instead of the one you referenced because that is considered obsolete) the API server port is set with kubeMasterAPIServerPort = 6443; and that’s where kubectl should connect. Probably you have some stale kubeconfig at ~/.kube/config or where a KUBECONFIG env variable is pointing.

I see that the wiki page advises you to do:

ln -s /etc/kubernetes/cluster-admin.kubeconfig ~/.kube/config

did you do that? Also, you shouldn’t run kubectl with sudo, it just opens client tcp connections.

1 Like
3

I hit this issue too. I believe the NixOS wiki example is expecting you to be logged in as root. When I sudo su and run kubectl api-resources and kubectl cluster-info, it works as expected. The main issue seems to be a non-root user not having correct permission to read the PEM certs in /var/lib/kubelet directory. I haven’t had time to amend my config to work around this issue but I think the simplest way is to run the systemd services under a dedicated kubernetes group, give that group permissions to the PEM directory and also add your standard user to the group. Alternatively, I might just run it under a VM, rather than make my laptop a node.

1 Like
4

Please check if you have a file or a link as the path /root/.kube/config or if you have defined an environment variable KUBECONFIG for the root user.

All the authentication data needed for administering the cluster are in the YAML file at path /etc/kubernetes/cluster-admin.kubeconfig it’s self contained and doesn’t reference any external file. Any user wanting to administer the cluster just needs an environment variable KUBECONFIG that points to it (or the link as stated before). That user just needs to be able to read that file and nothing else . No access to other paths on the local filesystem are necessary. For sure no access to the private var location for the kubelet is necessary. That file can be also copied to another system and it will work without issues after updating the server field with an URL for the apiserver that is accessible externally.

1 Like