Smartctl exporter

Help
1

Hello,
I’m stuck on a problem.
How to use the smartctl exporter? When I include it in “node” it crashes the service.

If I add it in its own exporter, it has no access to the disks
Are there any special rights to be granted?

thank you for your help!

2 
  services.prometheus.exporters.smartctl = {
    enable = true;
  };

  services.udev.extraRules = ''
    SUBSYSTEM=="nvme", KERNEL=="nvme[0-9]*", GROUP="disk"
  '';

The udev rule is to allow access to the raw nvme device to the disks group, should you have any, which is what the exporter automatically discovers.

https://github.com/nixos/nixpkgs/commit/12c26aca1fd55ab99f831bedc865a626eee39f80

3 Likes
3

interesting, unfortunately I don’t have the access authorization.

mai 15 20:53:38 cobblepot systemd[1]: Started prometheus-smartctl-exporter.service.
mai 15 20:53:38 cobblepot smartctl_exporter[3862540]: [Warning] S.M.A.R.T. output reading error: exit status 2
mai 15 20:53:38 cobblepot smartctl_exporter[3862540]: [Warning] The device error log contains records of errors.
mai 15 20:53:38 cobblepot smartctl_exporter[3862540]: [Error] Smartctl open device: /dev/nvme0n1 failed: Operation not permitted
mai 15 20:53:38 cobblepot smartctl_exporter[3862540]: [Error] smartctl returned bad data for device /dev/nvme0n1
mai 15 20:53:38 cobblepot smartctl_exporter[3862540]: [Info] Starting on 0.0.0.0:9633/metrics

Do I have to create special rights?
My disks sda,sdb etc… and my disk nvme are in right root:disk

4

I tried to change the rights of the disks or the smartctl exporter unfortunately nothing helps

“Operation not permitted”

5

Does /dev/nvme0n1 belong to the disks group? If so the exporter should be able to access it via SupplementaryGroups = [ "disk" ];

Should look like so:

brw-rw---- 1 root disk 259, 0 22. Mär 17:00 /dev/nvme0n1
6

Thank you for your help !

Unfortunately, nothing helps, I tried to use it: user:root group:disks.
The service has already supplementarygroups = [" disk "];

My disks are in:
brw-rw---- 1 root disk 259, 0 18 mai 15:40 /dev/nvme0n1

Maybe a problem to access the smartd program?

7

I don’t know if it’s the same issue but this fixed it for me:

https://github.com/NixOS/nixpkgs/pull/176553

There is also another workaround in the linked issue here:
https://github.com/NixOS/nixpkgs/issues/176524

systemd.services."prometheus-smartctl-exporter".serviceConfig.DeviceAllow = lib.mkOverride 10 [
    "block-blkext rw"
    "block-sd rw"
    "char-nvme rw"
]
8

Merged the fix just now. Apparently I forgot to follow up on the issue, sorry.

9

There is still a problem.

It seems to be impossible to read smart attributes of any disks attached to KVM VMs from the VM and the host should take care of that.

At the same time libvirtd seems to be reassigning the ownership of the main block devices attached to the guests to root:root

❯ ls -la /dev/sd?
brw-rw---- 1 root disk 8,   0 Nov  5 20:33 /dev/sda
brw-rw---- 1 root root 8,  16 Nov  5 20:33 /dev/sdb
brw-rw---- 1 root root 8,  32 Nov  5 20:33 /dev/sdc
brw-rw---- 1 root root 8,  48 Nov  5 20:33 /dev/sdd
brw-rw---- 1 root root 8,  64 Nov  5 00:03 /dev/sde
brw-rw---- 1 root root 8,  80 Nov  5 00:03 /dev/sdf
brw-rw---- 1 root root 8,  96 Nov  5 00:03 /dev/sdg
brw-rw---- 1 root disk 8, 112 Nov  5 00:03 /dev/sdh
brw-rw---- 1 root disk 8, 128 Nov  5 00:03 /dev/sdi
brw-rw---- 1 root disk 8, 144 Nov  5 00:03 /dev/sdj
brw-rw---- 1 root root 8, 160 Nov  5 20:31 /dev/sdk
brw-rw---- 1 root disk 8, 176 Nov  5 00:03 /dev/sdl
brw-rw---- 1 root root 8, 192 Nov  5 00:03 /dev/sdm
brw-rw---- 1 root disk 8, 208 Nov  5 00:03 /dev/sdn

Disks with root:disk permissions are not attached to the VMs, others are.

10

I think it would be great to add a note somewhere about it, it took me a few hours to find this post for the workaround. As security implications aren’t clear, maybe just a little note/suggested workaround on the “enable” flag?

11

Agreed. It’s an important bit to get the autodiscovery of devices going, which is pretty useful. I think we should just add the udev rule into the module for now.

12

A friendly reminder that I do not think that the udev rule has been added to the module yet. I had to manually add the udev rule in order to not get permission denied.

13

The discussion was moved upstream to systemd, as the udev rules we use are managed by them and are rolled out uniformly on many Linux distributions.

https://github.com/systemd/systemd/issues/26009

The udev rule that makes disk char devices owned by the disk group is something that you can decide to use for yourself. For nixpkgs we could ideally separate that access into a dedicated group that regulates access to the raw io interface.

https://github.com/NixOS/nixpkgs/pull/205165

1 Like