Github kind of died today, and, with it, nixpkgs. GH looks to be back now, so I’m not seeking help with this issue, but I am curious about Nix’s attitude to this single point of failure. Other package managers solve this with mirrors, so that if one server is down, a user can just sync their package lists with a different repo and continue as normal.
However, this looks non-trivial to apply to Nix. Even if channels were modified (somehow) to support mirrors, Flakes as they are wouldn’t cooperate too well given that inputs are supplied not by content but by address: github:nixos/nixpkgs necessarily refers to a single point (of failure) located on github.com. Are there any existing thoughts on solutions that enable the mirroring of nixpkgs and resistance to this type of outage?
If there are no existing thoughts or implementations here, here’s my proposed solution for discussion: The Nix tools would allow ‘sequential aliases’ for every domain they request. If a request for a user-specified URL doesn’t return successfully, the Nix tool consults its list of sequential aliases and dequeues an aliased URL to attempt the request to. In the nixpkgs repo, we mirror to some other hosting platform, and have Nix ship a default sequential alias list containing a mapping from github:nixos/nixpkgs[/nixos-version] to that mirror. Then, in the event of a github outage, Nix just goes down the list of mirrors like any other package manager. Thoughts?[0]
0: I’m sort of volunteering to implement this, but my time is limited by my school at the moment, and this would be slow if I’m the only one involved in the implementation. I’m also fairly certain that this would be the type of thing to balloon in size quite easily.
I think p2p protocols can probably help with the replication of repo’s and packages. The IPFS changes made in nix are interesting. I’m messing around with the hypercore protocols to the same effect.
So take a look around this, but competing alternative solutions are always welcome, even if it’s for your own systems.
As the internet collapses from a federated system, into a single ‘github.com/facebook.com/google.com/apple.com/microsoft.com/amazon.com’ eco system , its important that free independent mirrors exists, and build from source operating systems like nix/OS exist. What better way for the users of a ‘thing’ to securely donate their resources (time/cpu/disk/bandwidth) to help other users use the ‘thing’.
Same goes for the docker hub and ‘app’ stores, that are centralised points for software distribution, which on paper look good, but in practise are Technically and politically ‘disturbing’
Nix and friends are stored in git repos. Everyone has a backup of the source codes. The only thing that would be problematic to lose in this sense is the backlog - pull requests, issues etc.
There is a fundamental problem in mirroring nixpkgs. It is currently not possible to verify the authenticity of a checkout, because there is no process for signing and verifying that signature. Sure you can mirror the entire nixpkgs Git repo which provides some sort of integrity via Git hashes, but nobody wants to download the entire 1 GB repo just to perform a system update. Also if the central authority with which you’d check the Git hash (currently GitHub) is down, you’re still screwed. RFC 100 tries to address this but it started off on the wrong foot and with an ill-defined scope, so it will take a lot more time before we will see some sort of decentralized trust for nixpkgs.
Note that channel updates (the non-experimental way of getting your nixpkgs) currently are completely independent of GitHub and work via a combination of Fastly and S3 (afaik). The nixos-* branches in the GitHub repos are mostly for convenience.
So channels.nixos.org going down is the bigger problem in practice. The nice thing though is, that only updating your channel is affected until the outage is resolved — you always have at least one local version of nixpkgs available which works completely offline.
cache.nixos.org is another possible concern for mirroring, here there are mostly practical problems (the sheer size of the cache), but a subset has been mirrored before. A neat property of Nix here is that it can work without a binary cache, although compiling everything from source is not really fun of course.
So channels.nixos.org going down is the bigger problem in practice.
The nice thing though is, that only updating your channel is affected
until the outage is resolved — you always have at least one local
version of nixpkgs available which works completely offline.
Flakes don’t rely on this, do they? My understanding is that if I
request github:nixos/nixpkgs at a locked revision, Nix goes through GH
rather than any of its own infrastructure - is this correct?
cache.nixos.org is another possible concern for mirroring, here there
are mostly practical problems (the sheer size of the cache), but a
subset has been mirrored before. A neat property of Nix here is that it
can work without a binary cache, although compiling everything from
source is not really fun of course.
Yeah, I think the binary cache is another issue entirely, and would of
course be very painful (if important) to mirror.
I guess we should probably define the scope of a mirror a bit better.
The Github outage affected Flakes that specified their inputs in the
form of github:nixos/nixpkgs[/ver]. The type of nixpkgs mirror that I
would probably propose here would recognize github:nixos/nixpkgs and
sequentially redirect to some other user-defined mirrors, with at least
one official mirror supplied by default and integrated into nixpkg’s CI.
How’s this?[0][1]
0: @nixinator brought up P2P via IPFS, which looks like it could solve
some issues, but I’m not too excited about replacing one single point
of failure (github) with a protocol developed with the interests of
venture capital (especially if blockchain, et. al. are involved).
1: @hmenke brought up the issue of verifying authenticity, which is
definitely a concern. If we request a nixpkgs version by its git
hash, would that be enough to verify authenticity, or at least
equivalence with whatever was on Github at that particular time?
Yes, flakes and the channels.nixos.org are unrelated (also the channel branch is nothing special in flakes, it seems that it’s a common pitfall to use the channel instead).