Custom Prometheus Data Directory

1

I have a NAS running NixOS with a HDD ZFS mirror and an NVMe SSD with ZFS on it. The SSD is mounted at /data/apps and I’d like Prometheus to write to /data/apps/prometheus, however there doesn’t seem to be a way to overwrite the default /var/lib/prometheus2/data (I’m aware of the stateDir option, it still doesn’t do what I want) just using services.prometheus or other options. I’d rather not create a custom Prometheus service just for this, since I don’t want to lose all of the nice stuff in nixpkgs.

My question here is partially a question of semantics; should I create a ZFS dataset that mounts to /var/lib/prometheus2/data or should I create the custom service?

2

Isn’t it possible to define where data goes using services.prometheus.configText?

Personally, I would lean toward keeping the default and mounting a ZFS dataset at that path.

3

Just from looking at the source in nixpkgs it looks like the tsdb path gets passed as a flag in the systemd service and IIRC flags override text config options in Prometheus.

Thanks for your input!

4

same situation here and curious if you found a way by now or what workaround you did?

eg. did you create a ZFS dataset that mounts to /var/lib/prometheus2/data?

5

A simple workaround by using systemd.tmpfiles:

{
  # Workaround for prometheus to store data in another place
  # https://www.freedesktop.org/software/systemd/man/latest/tmpfiles.d.html#Type
  systemd.tmpfiles.rules = [
    "D /data/apps/prometheus2 0751 prometheus prometheus - -"
    "L+ /var/lib/prometheus2 - - - - /data/apps/prometheus2"
  ];

  # ...
}
1 Like