Systemd_exporter: couldn't get dbus connection: read unix @->/run/dbus/system_bus_socket: recvmsg: connection reset by peer

Help
1

Hello all. I am trying to enable services.prometheus.exporters.systemd on my NixOS 24.11 system. It is a fresh install with minimal customization.

When systemd_exporter starts, it logs the following:

May 17 00:07:07 host systemd_exporter[1163424]: ts=2025-05-17T07:07:07.177Z caller=main.go:57 level=info msg="Starting systemd_exporter" version="(version=0.6.0, branch=unknown, revision=unknown)"
May 17 00:07:07 host systemd_exporter[1163424]: ts=2025-05-17T07:07:07.177Z caller=main.go:58 level=info msg="Build context" build_context="(go=go1.23.8, platform=linux/amd64, user=nix@nixpkgs, date=unknown, tags=unknown)"
May 17 00:07:07 host systemd_exporter[1163424]: ts=2025-05-17T07:07:07.177Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9558
May 17 00:07:07 host systemd_exporter[1163424]: ts=2025-05-17T07:07:07.177Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9558
May 17 00:07:08 host systemd_exporter[1163424]: ts=2025-05-17T07:07:08.867Z caller=systemd.go:225 level=error msg="error collecting metrics" err="couldn't get dbus connection: read unix @->/run/dbus/system_bus_socket: recvmsg: connection reset by peer"
May 17 00:07:25 host systemd_exporter[1163424]: ts=2025-05-17T07:07:25.832Z caller=systemd.go:225 level=error msg="error collecting metrics" err="couldn't get dbus connection: read unix @->/run/dbus/system_bus_socket: recvmsg: connection reset by peer"
May 17 00:08:08 host systemd_exporter[1163424]: ts=2025-05-17T07:08:08.865Z caller=systemd.go:225 level=error msg="error collecting metrics" err="couldn't get dbus connection: read unix @->/run/dbus/system_bus_socket: recvmsg: connection reset by peer"
May 17 00:09:08 host systemd_exporter[1163424]: ts=2025-05-17T07:09:08.866Z caller=systemd.go:225 level=error msg="error collecting metrics" err="couldn't get dbus connection: read unix @->/run/dbus/system_bus_socket: recvmsg: connection reset by peer"
May 17 00:10:08 host systemd_exporter[1163424]: ts=2025-05-17T07:10:08.865Z caller=systemd.go:225 level=error msg="error collecting metrics" err="couldn't get dbus connection: read unix @->/run/dbus/system_bus_socket: recvmsg: connection reset by peer"
May 17 00:11:08 host systemd_exporter[1163424]: ts=2025-05-17T07:11:08.866Z caller=systemd.go:225 level=error msg="error collecting metrics" err="couldn't get dbus connection: read unix @->/run/dbus/system_bus_socket: EOF"
...

So, mostly “connection reset by peer” but occasionally “EOF”.

I tried services.dbus.implementation = "broker"; to see if this would make a difference, but it didn’t. The symptoms are the same with either implementation.

I suspect(ed) that the system user that this service runs-as does not have access to dbus, but I am not sure how to confirm or refute this. I tried setting the service’s group to “messagebus”, but that didn’t help.

Any ideas?

2

Yep, me too, this is a regression of some kind, it was working until very recently (I missed a week or two of updates due to busy weekends, though).

Nothing changed in the exporter modules, at least not anything that could affect permission to read the socket, so this must be a dbus configuration change; it’s strange that this happened on NixOS stable.

3

There is this systemd issue: DynamicUser is denied access to dbus-daemon after a recent lockup bugfix · Issue #22737 · systemd/systemd · GitHub

But that would imply that this has never worked, which evidently it did. I’m not sure what changed.

4 
services.dbus.implementation = "broker";

does fix the issue on my end. Double check with a reboot in-between.

I’m not sure what to do about this upstream. Force-enabling the broker implementation if you use this exporter seems like it might cause issues.

Edit: I’ve opened an issue: nixos/prometheus: systemd exporter fails to connect to dbus · Issue #408800 · NixOS/nixpkgs · GitHub

5

Thank you, @TLATER. I can confirm that rebooting with the “broker” DBUS implementation makes it work. Earlier I tried switching it but without a reboot, so I claimed it didn’t help. My issue is solved, though I agree that force-enabling the broker implementation can have side effects. Thanks for opening the issue.