Nixos-telemetry flake

Announcements
1

Hi everyone,

I’d like to share nixos-telemetry, a NixOS flake that makes setting up observability in your infrastructure easy. An OpenTelemetry collector sits at the center of each machine. Scrapers, storage, and visualization are all opt-in. You enable what you need; the collector wires the pipeline together automatically.

Why?

Wiring up telemetry in NixOS today means gluing together Telegraf, Prometheus, Loki, Grafana, Alloy, each with its own config format, ports, and inter-service dependencies. nixos-telemetry puts all of that behind a single telemetry.* option tree:

  • Turn on the system with telemetry.enable = true.
  • Every app is opt-in. Nothing starts that you didn’t ask for.
  • The collector starts automatically once a complete pipeline exists, a matching source and sink for the same signal type. No sink? It waits.
  • Forward to a remote collector, run a full local stack, or both. Fan-out to multiple sinks is supported.

What it looks like

Full local stack on one machine:

{
  telemetry.enable = true;
  telemetry.telegraf.enable = true;    # host metrics
  telemetry.alloy.enable = true;       # journald logs
  telemetry.prometheus.enable = true;  # metrics storage
  telemetry.loki.enable = true;        # logs storage
  telemetry.grafana.enable = true;      # visualization (datasources auto-provisioned)
}

Forward to a remote collector:

# machine 1
{
  telemetry.enable = true;
  telemetry.telegraf.enable = true;
  telemetry.opentelemetry.exporter.endpoints.remote = "100.64.0.1:4317";
}
# machine 2
{
  telemetry.enable = true;
  telemetry.opentelemetry.receiver.endpoint = "0.0.0.0:4317";
}

Full option reference: OPTIONS.md

Thanks!

Regarding Discourse LLM Policy : I must disclose that substantial parts of this project are llm generated. And because I coppied parts of the README in this post, substantial parts of this announcmement too.

16 Likes
2

Thanks for sharing. Will check this out. I have a bunch of soon-to-be netbooting systems that I want to log from a central host.

3

Is this too opinionated to be in Nixpkgs?

1 Like
4

Maybe, don’t know. I mean it would be possible to bring this to nixpkgs. For example by introducing an option like telemetry.enable or monitoring.enable, and if it’s true, automatic wiring against opentelemetry collector will be done if service.grafana, service.loki or service.telegraf is enabled. :person_shrugging:

5

Couldn’t this just be a single NixOS module compositionally importing your ./modules directory of NixOS modules? That would free users from needing to buy into flakes, flake-parts, & the rest of the tooling… This also makes it available to folks not using x86 or aarch limited to just Darwin & Linux.

3 Likes
6

this file maybe? you don’t have to use the flake stuff.

I need it to write tests, run formatters and can pin a version in other flakes.

If you write me how you want to include this file I can try to help, and put it in the readme for others.

7

Well that would then mean all downstream users need to accept your tooling too transitively like treefmt, flake parts, devshell (without setting flake = false; that is, but…). But the default.nix you show was exactly what I was looking for—which is a ‘super’ module importing the others & is independent of any additional tooling, flakes, systems. You should add a not eabout this to the docs as there is no mention of import like imports = [ "${inputs.nixos-telemetry}/modules" ]. It seems like this should be noted flake = false; should be the preferred method since it avoids extra tooling for downstream.

2 Likes
8

The implication of course here being that the file layout becomes a part of the API, which can be surprising if you are a flakes-first person.

I’m skeptical of the utility of something like this more broadly, I would assume most people who are arriving at the conclusion that they need an otel collector probably have their own opinions about wiring, and worse, probably has their hands tied in 7 other ways by other constraints outside of their control.

I have nothing against show-cases or sharing, so I don’t want to poo-poo this. I wouldn’t use it. I would probably read it if I were shopping about for how to get started.

EDIT:

nixos-telemetry/modules/grafana.nix at 862a85f749df22d88158bfb34991cb7eccf7da12 · mrVanDalo/nixos-telemetry · GitHub ?? I wouldn’t encourage using this.

nixos-telemetry/modules/grafana.nix at 862a85f749df22d88158bfb34991cb7eccf7da12 · mrVanDalo/nixos-telemetry · GitHub ??? Does this really default to anon being admin?

2 Likes
9

Don’t be a flakes-first person. I’ve been using flakes for development but I’ve had the insight: my projects are not consumable as flakes. Only vanilla nix. The flake is for development.

2 Likes
10

I wrote that because I’ve been bitten more that once consuming a project that had a flake at the root which kept the flake attributes stable but reorganised all of the files (or alternatively changed the attrs available on /default.nix). Not because I run flakes :slight_smile:

11

Just as much, I have been bitten by folk renaming Flake keys. The path doesn’t require experimental features, pull in transitive dependencies, & doesn’t require evaluation—just a 20-year-old, stable import "${path}" feature.

2 Likes
12

Yes grafana is a just make it work setup. The grafana configuration is meant for localhost use only, I will make sure this is part of the Description.

13

I will make this part of the README. Thanks for the input.

14

But the problem is that this is worse than the default experience already in nixpkgs

  1. key is world readable because not sourced from file
  2. instead of enforcing a “first login” password we start it open to the world. If you don’t know what shodan is, or how quickly servers get lit up on the internet these days, I recommend you take a look. Every week there is some new service full of data discovered wide open on the internet. We don’t rise to the level of our ambitions, we fail to the level of our systems.
15

a secret_key_file parameter don’t exist. i just use the legacy default.

16 
  adminPasswordFile = "${secretsRoot}/admin-password";
  secretKeyFile = "${secretsRoot}/secret_key";

  services.grafana = {
    settings = {
      security = {
        admin_user = "admin";
        admin_password = "$__file{${adminPasswordFile}}";
        secret_key = "$__file{${secretKeyFile}}";
      };
    };
  };

for instance. No world readable store values should contain secrets. Now obviously, this is not generic, it’s cribbed directly from work grafana, but it’s not a huge leap of imagination to get to a generic version that doesn’t start world-open.

2 Likes
17

I was literally just doing some research into building something like this.
Saved me a whole lotta time.

You should update the readme to highlight the minimum requirements. I had to read the source to know I had to satisfy the minimum sets:

  config = {
    telemetry.pipelines.metrics.hasSource =
      config.telemetry.telegraf.enable
      || config.telemetry.netdata.enable
      || (config.telemetry.opentelemetry.receiver.endpoint != null);

    telemetry.pipelines.metrics.hasSink =
      config.telemetry.prometheus.enable
      || (config.telemetry.opentelemetry.exporter.endpoints != { })
      || (config.telemetry.opentelemetry.exporter.debug == "metrics");

    telemetry.pipelines.logs.hasSource =
      config.telemetry.alloy.enable || (config.telemetry.opentelemetry.receiver.endpoint != null);

    telemetry.pipelines.logs.hasSink =
      config.telemetry.loki.enable
      || (config.telemetry.opentelemetry.exporter.endpoints != { })
      || (config.telemetry.opentelemetry.exporter.debug == "logs");

    telemetry.pipelines.anyComplete =
      (config.telemetry.pipelines.metrics.hasSource && config.telemetry.pipelines.metrics.hasSink)
      || (config.telemetry.pipelines.logs.hasSource && config.telemetry.pipelines.logs.hasSink);
  };
}

Notes:

  • Grafana without a secret should assert an error instead of using a hardcoded default.
  • Also Grafana defaults to 127.0.0.1 in nixos so you should expose the http_addr option
  • Telegraf is missing some inputs. It also doesn’t expose that in options
18

Thanks for you input.

In 🔒 non nix store defaults for grafana secrets · mrVanDalo/nixos-telemetry@e5cedde · GitHub the secret key will by default generated via PreExec in the grafana job. I also added an option to disable this behavior to make it easier for people to set up their own idea without mkForce everywhere.

About the admin authentication I provided 3 ways to set it up, and the default is login with admin:admin with request to change your password when you login. I want to keep grafana as default and easy to use as possible, and just add option for a more safer (and a more yolo) way.

Hope you like the solution too. Thanks again for your input.

19

Hey

This is documented in my opinion GitHub - mrVanDalo/nixos-telemetry: Provides predefined telemetry configurations for various observability tools, leveraging [OpenTelemetry](https://opentelemetry.io/docs/collector/) as the core transport layer. · GitHub says

The conditional blog obviously needs cleanup, but the sentence from the documentation should cover it.

Yes Grafana needs definitly some more love, but I had not good idea on how to dumb down the configuration for it. I don’t want to just forward every parameter. But in this case it can’t hurt to forward the http_addr parameter. (update: added this parameter)

About telegraf there are a lot of inputs which are not there yet. My plan is to configure everything based on service.<name>.enable, for example:

mkIf (config.services.nginx.enable) {
  services.telegraf.extraConfig.inputs.nginx = ...
}

What inputs are you missing (the most)?

Thanks for you input.

20

I force added mem to it. There was no memory metrics.