How to package software that uses periodically updated data from the internet?

Help
1

I want to package a piece of software that needs periodically updated data from the internet. The software itself doesn’t do the downloading, it expects the data to be updated periodically though an external method.

I can think of two ways to do that:

  1. package the data with the software. This means I need to manually and periodically update the packaging code and run nixos-rebuild
  2. periodically download the data at the runtime and reload the process.

#2 seems to be a more sensible approach, but is there a way to make a nix store file writable? Because when the software first runs, I would like the data to be immediately available, without having to wait for the first download. That means:

  1. The data needs to be downloaded at nix evaluation time.
  2. The data needs to be overwritable at runtime.

For #2 I can only think of copying the file over from the nix store to a writable location with system.activationScripts. I wonder is there a more elegant way?

And is the whole approach a right one to begin with?

2

Nix store cannot be written to except for by nix daemon during evaluation.

Probably the best approach would be to have the package have some base version of the data. Starting package as a service moves the data to a state directory if the data does not exist there. Second service would download additional data into first service’s state directory on a timer and issue a reload-like signal to the first service.

1 Like
3

spamassassin is an example of an existing package like this.

It has a fixed version of the rules database included in the output derivation (ends up in the nix store): nixpkgs/pkgs/servers/mail/spamassassin/default.nix at bb752745b2f44ebe6a70d5494af3101c739b3811 · NixOS/nixpkgs · GitHub

But the user can also opt to run the sa-update script (perhaps periodically in a cron job), which will fetch a newer version of the rules database into /var/lib/spamassassin. Spamassassin will use the rules database from /var/lib/spamassassin, if it exists, in preference to the packaged rules in the nix store.

1 Like
4

This helps a lot, thanks.

Spamassassin will use the rules database from /var/lib/spamassassin , if it exists, in preference to the packaged rules in the nix store.

Is this natively supported by spamassassin?

5

Yes, this fallback behaviour is a feature of Spamassassin itself. It distinguishes between packaged rules in DATADIR and downloaded rules in LOCALSTATEDIR:

1 Like