Danish Road Traffic Authority switches from Microsoft to Linux

[twoolie]

@TLATER

I think AWS backing hydra is way bigger of a risk. Without hydra neither the channels nor binaries get produced, nor do we cache third party repos, and the load on hydra is even larger than GitHub.

I’ll bet a lot of us are sitting on under-utilized computing resources. Should we be looking for ways for community members to donate build services? We could have a policy where multiple systems build the package, and it is only accepted if all agree on the build output hash?

@7c6f434c

cache is what needs actual migration and has historical sources that are not always easy to find again now.

Is there a mechanism to replicate just historical sources into long-term/durable storage?

[peterc-s]

@TLATER

Just because people have been writing github: in their flakes doesn’t mean alternatives don’t exist.

I would like to know where these alternatives live and how to use them, is it documented anywhere? Is it possible to set a mirror up as a fallback?

@TLATER

but I think people are just more aware of that risk (and I think it is already mitigated significantly better).

I certainly didn’t know Hydra was backed by AWS. With all the talk about tech sovereignty recently, this does not inspire confidence. I only originally pointed out GitHub because the irony in the title.

@twoolie

Should we be looking for ways for community members to donate build services? We could have a policy where multiple systems build the package, and it is only accepted if all agree on the build output hash?

Doing majority voting on builds when I’m not certain enough people would volunteer compute to replace Hydra would probably be less than ideal. I also don’t know what Hydra costs nor how much it would cost with another provider (+ all the work associated with migration), but swapping providers would probably be the better solution.

[TLATER]

@peterc-s

I would like to know where these alternatives live and how to use them, is it documented anywhere?

You can use any of the protocols defined by the flake types. Only one of them is github.

In the context of nixpkgs, technically I would say that there currently are no official mirrors. The only official source are the channel tarballs, which live under https://channels.nixos.org/$channel/nixexprs.tar.xz.

If you use channels, you already know how to use this (and your system is set to use it by default). If you use flakes, just replace your github: URL with that https URL.

When flakes entered the scene, the github repo was used as a crutch because at the time there was no reasonable way to update tarballs. Nix got the lockable tarball protocol since, though, so now we can use the tarballs that hydra produces again.

These tarballs in fact contain a bit of extra data that you cannot get from the branches, and the branches are only moved ahead for development reasons after the tarballs are successfully published.

So, the GitHub branches have organically grown to become “mirrors” that don’t actually properly mirror the tarballs. It isn’t their intended use, they’ve just been abused for this for so long that that has been forgotten.

That said, unofficial mirrors do exist. There’s a university in China that runs one, for example (though I don’t know if it implements the lockable tarball protocol yet), and detsys’ flakehub also contains one.

IMO if our goal is to further derisk our use of GitHub as a platform, we should just aggressively update all documentation that we can reasonably update, and generally spread awareness of this mis-practice.

@sshine

It mentions half a dozen examples, nixpkgs being one

I’d also like to add that, for this precise reason, that article is inaccurate. I agree with the general premise, but the author is clearly also under the false impression that nixpkgs uses git for versioning.

There are proper, detached versions, which are published via CI jobs. For historic reasons the community has just started using an experimental feature which didn’t use the proper distribution method at one point, and instead explicitly clones the git repo.

Now doc/usage lag (note that the nixpkgs manual, against which the complaint is levied, says nothing about how to use it with flakes, with a single exception that clearly snuck in accidentally) means that few people are even aware that this isn’t the intended way to use nixpkgs.

Also, to be completely fair, nixpkgs is a Linux distro. Distro versioning is very different from project versioning. Including nixpkgs in this article just doesn’t make sense to me.

[peterc-s]

@TLATER

IMO if our goal is to further derisk our use of GitHub as a platform, we should just aggressively update all documentation that we can reasonably update, and generally spread awareness of this mis-practice.

Someone really needs to update the wiki if using the channel tarballs is the better option…

I’m sure a lot of the people installing NixOS or using Nix come from YouTube or similar. Being one of those people, I saw this in basically every video.

It might also come from the flake templates which also use github:, if it is good practice to use the tarballs, it may be worth a PR?

RE: derisk use of GitHub: how is https://channels.nixos.org/$channel/nixexprs.tar.xz hosted? Is it reliant on AWS like Hydra?

[sshine]

@TLATER

IMO if our goal is to further derisk our use of GitHub as a platform, we should just aggressively update all documentation that we can reasonably update, and generally spread awareness of this mis-practice.

@peterc-s

if it is good practice to use the tarballs, it may be worth a PR

Sounds like a well-defined documentation task. I’ll create a tracking issue.

[peterc-s]

@sshine

Sounds like a well-defined documentation task. I’ll create a tracking issue.

I will also note, the flake registry maps nixpkgs to github:nixos/nixpkgs/nixpkgs-unstable (at least according to the README, I can’t load the flake-registry.json because GitHub is borked?).

edit: cloned and checked flake-registry.json and the README is correct.

edit 2: also looks like the templates repo hasn’t been updated in a few months and there are outstanding issues from a few years ago.

[cafkafk]

@7c6f434c

It seems untenable to do any non-negligible amount of development work on Nixpkgs without a private clone of all the main branches!

Note clone != mirror in this context, as a local clone doesn’t automatically pull from GitHub - NixOS/nixpkgs: Nix Packages collection & NixOS every some minutes like forgejo repository mirrors do.

And likewise, the common pattern of having a GitHub fork of nixpkgs that automatically pulls does serve as a mirrored repository, but doesn’t provide any redundancy against GitHub!

[arianvp]

Hosted on AWS S3, Frontend by Fastly.

[7c6f434c]

@cafkafk

local clone doesn’t automatically pull from Nixpkgs every some minutes

You are right, I simply perceive «git remote update approximately daily» as a lower-quality code mirror (and in a pinch a few of those would be enough to bound the losses), but of course mirrors tracking more closely are better as mirrors.