Telegraf configuration

Help
1

Hello,

I am having trouble configuring a processor for Telegraf,

Input/output works well, adding topic_parsing breaks

Here is my telegraf service set-up

  services.telegraf = {
    enable = true;
    extraConfig = {
      inputs = {
        mqtt_consumer = {
          servers = [ "tcp://127.0.0.1:1883" ];
          qos = 0;
          connection_timeout = "30s";
          topics =  [
            "v1/+/+/+/+"];
          data_format = "value";

         topic_parsing = {
           measurement = "measurement/_/_/_/_";
           tags = "_/site/version/device_name/_";
           fields = "_/_/_/_/field";
          };
       };
      };

      outputs = {
        influxdb_v2 = {
          urls = [ "http://localhost:8086" ];
          token = "my-super-secret-auth-token";
          organization = "my-org1";
          bucket = "my-bucket";
        };
      };
    };
  };

@Mic92 Is it possible to set up pivots and topic_parsing with nixos Telegraf package?

Thanks in advance

2

I don’t know. I would be very surprised if upstream telegraf has support and we don’t, it’s a rather self-contained package. Maybe your telegraf version is older than this feature?

1 Like
3

Thanks, no I use the latest stable version,

I could not get it to work, even with a string config :

extraConfig = {
'''
TOML CONFIG...
  [[inputs.mqtt_consumer.topic_parsing]]

'''
};

And I didn’t find any documentation or code samples
So I will use a container for now instead

4

Did you check the configuration that is created? Did you check the logs via journalctl? My experience is that finding issues with configurations like this are best solved by looking at those.

5

Hello

I tried again this morning :


services.telegraf = {
    enable = true;
    extraConfig = {
      inputs = {
        mqtt_consumer = {
          servers = [ "tcp://127.0.0.1:1883" ];
          qos = 0;
          connection_timeout = "30s";
          topics =  [
            "v1/+/+/+/+"];
          data_format = "value";

         topic_parsing = {
           topic = "v1/+/+/+/+";
           measurement = "measurement/_/_/_/_";
           tags = "_/site/version/device_name/_";
          };
       };
      };

      outputs = {
        influxdb_v2 = {
          urls = [ "http://localhost:8086" ];
          token = "my-super-secret-auth-token";
          organization = "my-org1";
          bucket = "my-bucket";
        };
      };
    };
  };
  };

Here is the error :

 error parsing mqtt_consumer, line 8: cannot unmarshal TOML table into []mqtt_consumer.TopicParsingConfig (need struct or map)

Is my syntax wrong ?

6

Can you post the generated toml config?

7

Here is the conf (working when using a container )

[[outputs.influxdb_v2]]
  urls = ["http://localhost:8086"]
  token = "my-super-secret-auth-token"
  organization = "my-org1"
  bucket = "my-bucket"

[[inputs.mqtt_consumer]]
  servers = ["tcp://0.0.0.0:1883"]
  topics = ["v1/+/+/+/+"]
  qos = 0
  connection_timeout = "30s"
  data_format = "value"

[[inputs.mqtt_consumer.topic_parsing]]
        topic = "v1/+/+/+/+"
        measurement = "measurement/_/_/_/_"
        tags = "_/place/level/room/measured"

[[processors.pivot]]
  tag_key = "measured"
  value_key = "value"

And in my previous post : there is the same configuration, written in the nix language

This is really basic, so I probably made a mistake

Wait I misunderstood , I will find the generated config

1 Like
8

Yes, that is what I meant. It makes understanding the error easier especially when you have a working config you can compare it to.

9

Here is the generated config :

systemctl status telegraf.service : 
 Process: 2859914 ExecStart=/nix/store/ddbx6bqdyknyknfijqdflrxlvcndylaz-telegraf-1.32.2/bin/
telegraf -config /nix/store/kwzayfmndmv5zjkymyjg8ix2vx36p0cv-config.toml


[root@intercom-broker:/nix/store]# cat  /nix/store/kwzayfmndmv5zjkymyjg8ix2vx36p0cv-config.toml
[inputs.mqtt_consumer]
connection_timeout = "30s"
data_format = "value"
qos = 0
servers = ["tcp://127.0.0.1:1883"]
topics = ["v1/+/+/+/+"]

[inputs.mqtt_consumer.topic_parsing]
measurement = "measurement/_/_/_/_"
tags = "_/site/version/device_name/_"
topic = "v1/+/+/+/+"

[outputs.influxdb_v2]
bucket = "my-bucket"
organization = "my-org1"
token = "my-super-secret-auth-token"
urls = ["http://localhost:8086"]
10

It seems that Telegraf expects an array of tables (telegraf/docs/TOML.md at d52ffc553c2121cb6f849c5f0679f8c4bdd30f49 · influxdata/telegraf · GitHub)

Can’t test but could be that something like this works:


inputs = {
        mqtt_consumer = [{
          servers = [ "tcp://127.0.0.1:1883" ];
          qos = 0;
          connection_timeout = "30s";
          topics =  [
            "v1/+/+/+/+"];
          data_format = "value";

         topic_parsing = {
           topic = "v1/+/+/+/+";
           measurement = "measurement/_/_/_/_";
           tags = "_/site/version/device_name/_";
          };
       }];
};
1 Like
11

Thank you for the fix and teaching,

It makes sense to go to the generated files, they should match the telegraf conf files.

Your correction works, telegrafs takes arrays for most configuration points (here : mqtt_consumer, topic_parsing, influxdb_v2, pivot)

Here is the full fixed service for anyone needing :

  services.telegraf = {
    enable = true;
    extraConfig = {
      inputs = {
        mqtt_consumer = [
          {
            servers = [ "tcp://127.0.0.1:1883" ];
            qos = 0;
            connection_timeout = "30s";
            topics = [
              "v1/+/+/+/+"
            ];
            data_format = "value";

            topic_parsing = [
              {
                topic = "v1/+/+/+/+";
                measurement = "measurement/_/_/_/_";
                tags = "_/place/level/room/measured";
              }
            ];
          }
        ];
      };

      outputs = {
        influxdb_v2 = [
          {
            urls = [ "http://localhost:8086" ];
            token = "my-super-secret-auth-token";
            organization = "my-org1";
            bucket = "my-bucket";
          }
        ];
      };
      processors = {
        pivot = [
          {
            tag_key = "measured";
            value_key = "value";
          }
        ];
      };
    };
  };
12

You are very welcome