What are people using for centralized logging/alerting of their nixos machines?
I’m looking for a stateless agent, which can export logs to a on prem or cloud log aggregator?
Anything like that available?
You’re probably aware of these, given the question, but I eventually found my way to Grafana being fed by Prometheus/Loki fed by promtail. This blog post inspired that: How to Setup Prometheus, Grafana and Loki on NixOS - Xe
Grafana does alerting based on Prometheus metrics, Loki does log aggregation, promtail plays stateless agent to export logs. It works nicely, and thanks to Grafana being surprisingly nice to configure automatically for such a graphical tool (not shown in blog post, but you can export your alerts/dashboards as JSON and set them up with the NixOS module, and services don’t have to be set up via the UI, despite the blog post instructing that), it all fits nicely within nix modules.
My only gripe with it is that a lot of these services are a little hard to harden; few of them have built-in authentication. Most need to be put behind nginx to be given authentication, which is unfortunate depending on how much you trust other things running on the hosts, especially for Loki.