Nixos-telemetry flake

Announcements
21

You are right, mem should be part of the always on inputs.

(is added now)

1 Like
22

I documented your recommend approach. Thank you for the input

1 Like
23

If you want some inspiration about the OTEL and observability stack, I do have a monitoring module too with declarative LDAP and SSO integrations and declarative dashboard provisioning. It has some quite extensive NixOS VM testing going on to ensure LDAP and SSO integrations do not break. selfhostblocks/test/blocks/monitoring.nix at c3ae7bfe0c703cf34012801c6ed9e1acb3018c5e · ibizaman/selfhostblocks · GitHub Coincidentally I just wrote a blog post explaining how those work if you’re curious Testing LDAP and SSO Integrations in NixOS with Playwright

4 Likes
24

Hey, thanks for the input. Grafana is not the focus of this project. Telemtry collection and transport is. The grafana configuration in this project will always be an adhoc version, and not fit production use cases. Loki and prometheus are too, at the moment, but I want to make them easy to configure with a production ready defaults (but it’s not there yet).

At the moment in scratching my head on giving good defaults but also make it easy to wire upstream module configuration with opentelemetry without making the interface (the nixos options) explode. Also adding more telemetry services.

25

I totally resonate with how you think about allowing to wire with multiple upstreams. I suppose you take as granted that upstream must support the OpenTelemetry protocol and you’re wondering how to wire things up in nix. Is that a valid assumption?

26

Not quite.

at the moment config.telemetry.netdata.enable = true must be set to wire netdata with opentelemetry-collector. But I would like to condition on config.services.netdata.enable to wire netdata with the opentelemetry-collector, because I have the feeling that people want to configure the collection of metrics configure them self, but don’t want to take care of the shippment (opentelemetry collector). I’m thinking about how to introduce sane auto-wireing, but keep it opt-in. ..

The opentelemetry collector is the core component of this project. Everything ends up there and will be shipped to consumers like loki or prometheus.

27

I introduced the telemetry.autowire option which is enabled by default :tada:

It handles automatic scraper configuration based on upstream nixos service configuration.
At the moment only telegraf does that but other apps will follow.

For every scraper you can disable this behavior individually.

Example

...
virtualisation.docker.enable = true;
...
telemetry.enable = true;
telemetry.telegraf.enable = true;

In this case telegraf would monitor docker.

If you don’t want telegraf to scrape information from docker you have to set telemetry.telegraf.autowire.docker.enable to false

This behavior is one of the 2 core ideas of this project. I’m not 100% happy with the interface (meaning how to disable and enable this behavior) but it’s almost there.

Again interested in your feedback.

2 Likes