I run a few Mastodon servers on NixOS, and for this I use the ‘nixos-23.05’ channel (seems the most versatile and stable channel to me). In general the experience is pretty great: to deploy a new server costs a matter of minutes, to upgrade ‘nixos rebuild-switch --upgrade’ and everything just magically works (no need to manually run ruby upgrade scripts or restarts).
The current version available in that channel is 4.1.7 for which an update came out (4.1.8) on Tuesday with important security fixes. It got PR-ed in nixpkgs within an hour or so (or at least the same day).
I would expect that to get ‘available’ in 23.05 the day after, but it didn’t since there are ‘build problems’ in Hydra (as can be seen in https://status.nixos.org).
The result is that now (about 4 days after a critical patch came out) my NixOS based production Mastodon server is still running 4.1.7 which has a serious problem in it. This makes using NixOS for this purpose (to put it mildly) a less suitable solution.
How should I handle such a situation?
Is running NixOS the ‘stable’ way (i.e. no flakes and use nixpkgs) secure enough to run services like Mastodon?
Is there any ‘this is the way to handle this’ documentation (deemed stable or not, I just need practical advice) that I have missed?
I understand the reaction and I actually thought a bit about that, but it seems counter-productive.
Let me explain why:
If I have to keep my eyes on which packages are outdated myself that makes NixOS an unsuitable system to run publicly internet-facing software on top of.
Other distributions (like debian) have a specific channel or other way to handle security updates that do not get ‘hold up’ by the ‘normal’ updating process.
Perhaps there is some room for a specific channel that is maintained by a dedicated group of people to run ‘Fediverse Applications’. Or perhaps this is better handled by creating something that resembles such a channel using flakes.
I am trying to find a solution here, because I want the advantages of NixOS and a way to deal with the disadvantages that makes sense.
However, even if a package needs to be compiled because Hydra didn’t build it, it usually doesn’t take long - most server applications are not Chromium or Firefox or LibreOffice or something that takes a long time to build.
Thanks @xfix that is the kind of practical information that makes sense. So yes I will change those servers channels to -small.
But for now it did actually not solve my problem, since the nixos-23.05-small channel also contains just 4.1.7 (and not 4.1.8 or 4.1.9) of the mastodon package.
I also does not answer the question totally of course, since the -small channel experiences the same issue that sometimes you just cannot wait for Hydra to run through all the checks (especially if it takes 4+ days for your package to come through).
Currently nixos-23.05 has Mastodon 4.1.8, while nixos-23.05-small has Mastodon 4.1.9. Note that you need to update channels to see the newest channels with nix-channel --update (will update channels) or nixos-rebuild switch --upgrade (will upgrade the operating systems and channels) executed as root.
Also, nix-channel executed as a regular user affects channels used by an user, but not by nixos-rebuild. Each user has its own Nix channel list.
I’m at least partially responsible for the channel being blocked this time, sorry for that! The cause was this PR, which introduced a non-deterministic test failure (which is why I didn’t find out about it right away), which I only just fixed today.
Thanks to @vcunat and @hexa for watching the Hydra jobs and restarting them a couple times until the non-deterministic test happens to succeed. Because of that the 23.05 channel just got unblocked a couple minutes ago!
I can imagine a future with paid teams maintaining specific security-critical applications, with a separate Hydra channel for the fastest updates. This team would be tasked with packaging updates and fixing breakages.
To react to @Infinisil what you are suggesting is the combination of 1 and 2:
A team that maintains a ‘security’ channel that has as it’s focus to ‘patch’ security holes by pulling those PR’s that are security related into a channel quicker. This will probably also involve communication with the package managers to keep subsequent PR’s working.
If there is a reason you can not wait for the channel to update and you know that the update of package XY will not break any downstream packages you rely on, you can
use system.replaceRuntimeDependencies as detailed here with a security-hotfix for OpenSSL: OpenSSL 3.0.7 update (2022-11-01) FAQ