When I then go into Grafana though it says “NoData” for everything. I think I’m missing something obvious, but I’ve not found any documentation that helped me understand what I’m doing wrong.
If you’re on 24.11, you need to use the Prometheus data source for Grafana, or at least that’s how I had to have it configured. In the last two or so months, Victoria Metrics paid the ransom (The VM dev told me it was like $70K USD) to get the VM plugin in the Grafana plugin store (or whatever its called), and while I see it in Grafana, I haven’t been able to install it. I don’t see that you’ve included any plugins in your Grafana config, you might want to try that was well, see “Declarative Configurations” here: Grafana - NixOS Wiki
I forgot to mention that I had the VictoriaMetrics plugin added non-declaratively, though now I’ve added it declaratively like @TLATER showed.
The Grafana datasources were adapted from the plugin page here under “Configuration via file”.
The reason the scrape_config looks like it does is because in the Victoria Metrics docs it configured them like that.
In the logs it seems like it discovers the targets, but they won’t show up on VictoriaMetrics’s targets list, so I’m assuming the issue comes from vmagent not communicating properly with the VictoriaMetrics instance.
...
starting service discovery routines...
started 2 service discovery routines in 0.001 seconds
static_configs: added targets: 2, removed targets: 0; total targets: 2
The targets should show up under /targets on the VictoriaMetrics, but sadly they don’t:
At journalctl -xeu victoriametrics.service I can’t find anything that seems off as well - except that the targets are never found of course.
Sorry that I forgot to bring up why everything’s configured stuff like that earlier. Thanks for the help though, glad I could add the plugin declaratively.
If anybody is interested in replicating this and has any questions, let me know. Also if anybody sees any issues I’d be happy to see other suggestions!