Grafana not getting data from Prometheus

Help
1

I’ve been following How to Setup Prometheus, Grafana and Loki on NixOS to set up monitoring of a single host (for now):

  1. Gather stats with Prometheus
  2. Create TLS certificate using ACME
  3. Expose Prometheus website - I can see stats here just fine
  4. Retrieve Prometheus stats in Grafana - this is not working
  5. Expose Grafana website - the site works fine, clicking “Test” in the Prometheus data source configuration page reports “Data source is working”, but when I try to click “Explore” and click “Select metric” it says “No options found”

The relevant configuration.nix content:

{
  services = {
    grafana = {
      enable = true;
      provision = {
        datasources.settings.datasources = [
          {
            name = "prometheus ${config.networking.hostName}";
            type = "prometheus";
            url = "http://127.0.0.1:${toString config.services.prometheus.exporters.node.port}";
          }
        ];
        enable = true;
      };
      settings.server = {
        domain = "${config.networking.hostName}.${config.networking.domain}";
        root_url = "%(protocol)s://%(domain)s:%(http_port)s/grafana/";
        serve_from_sub_path = true;
      };
    };
    nginx = {
      enable = true;
      recommendedGzipSettings = true;
      recommendedOptimisation = true;
      recommendedProxySettings = true;
      recommendedTlsSettings = true;
      virtualHosts."${config.services.grafana.settings.server.domain}" = {
        forceSSL = true;
        locations."/grafana/" = {
          proxyPass = "http://127.0.0.1:${toString config.services.grafana.settings.server.http_port}";
          proxyWebsockets = true;
        };
        locations."/prometheus/" = {
          proxyPass = "http://127.0.0.1:${toString config.services.prometheus.port}";
        };
        useACMEHost = config.networking.domain;
      };
    };
    prometheus = {
      enable = true;
      exporters.node = {
        enable = true;
        enabledCollectors = ["systemd"];
      };
      scrapeConfigs = [
        {
          job_name = config.networking.hostName;
          static_configs = [
            {
              targets = ["127.0.0.1:${toString config.services.prometheus.exporters.node.port}"];
            }
          ];
        }
      ];
      webExternalUrl = "https://${config.networking.hostName}.${config.networking.domain}/prometheus/";
    };
  };
}

The only error in the Grafana logs is this:

level=error msg=“Failed to read plugin provisioning files from directory” path=/nix/store/scf0y7j13i1jnv6wy4w1m25l02cbqmb2-grafana-provisioning/plugins error=“open /nix/store/scf0y7j13i1jnv6wy4w1m25l02cbqmb2-grafana-provisioning/plugins: no such file or directory”

I’ve also tried url = "http://127.0.0.1:${toString config.services.prometheus.port}"; (same as the original article), but then I get a “404 page not found” error when trying to select a Prometheus metric in Grafana. The accompanying log message:

level=info msg=“Request Completed” method=GET path=/api/datasources/uid/PF6B2D90455FD0ED6/resources/api/v1/label/name/values status=404 remote_addr=192.168.50.66 time_ms=1 duration=1.430507ms size=19 referer=“https://[…]/grafana/api/datasources/uid/PF6B2D90455FD0ED6/resources/api/v1/label/name/values?end=1682844471&start=1682840871” handler=/api/datasources/uid/:uid/resources/*

1 Like
2

Found it :smiley:! Turns out setting webExternalUrl breaks the Grafana data source. Presumably webExternalUrl disables “server” access to the data, or means the data source has to connect to that URL.

1 Like