Seeking Solutions for NixOS API Rate Limiting in a School Environment

Help
1

Over the past few years, our school, HTBLA Kaindorf in Austria, has seen a significant increase in the number of NixOS users. However, recently we’ve been facing an issue where our network is being subjected to rate limits when accessing the NixOS API. Since all of our users share the same public IP, the rate limit affects everyone on the network.

We’ve explored potential solutions, but unfortunately, our school administration does not permit the use of a NixOS cache server or similar infrastructure. I’d like to ask if there is any way to increase the rate limit for institutions or organizations like ours. Is it possible to apply for a higher rate limit, or are there alternative approaches we could consider to work around this issue?

If this forum isn’t the best place to address this concern, I would appreciate any suggestions on where else we should reach out for support.

9 Likes
2

Ok, we (the students) cant even create accounts for this forum, we are recieving the account-creation rate limit aswell :sob:

3 Likes
3

Could you explain a bit more what you mean by “NixOS API Rate Limiting” ?
Just accessing github or doing something else ? (unfortunately github does have rate limits if you do not use tokens)
You mentioned:

does not permit the use of a NixOS cache server or similar infrastructure

So I assume it is not requests to cache.nixos.org
If you can provide an example, what actions fail due to this rate limit, it could be useful too.

PS seems like Discourse has default limit for account creation by IP around 3:

which is mostly there to prevent spam

4

If it’s nix flake update that gets rate limited, and cache servers are not allowed, then the solution is indeed github access tokens. Unfortunately that is something each user has to do on their own, AFAIK github.com doesn’t provide mass-scale token provisioning.

5

nix flake update is indeed what gets rate limited. Thanks for the hint about using GitHub access tokens.

It’s not very likely that they would increase this limit for a random school setup, right?

6

If they do (and that’s a big if), it won’t be for free.
And if the school is resource-limited (as by far most schools worldwide are) then it’s unlikely they have the funds to pay for it.

7

The rate limits are there to prevent abuse and limit misbehaving programs, not for lowering costs by blocking legitimate users.

Discourse doesn’t have a way to configure registration limits per ip so you’ll just have to register with a different ip.

EDIT: discourse does have a screened ips thingy but how widely does it disable rate limits?

2 Likes
8

Not sure how limited your access to local infra, but if you have some network share available you could also try placing nixpkgs repo clone there and regularly update it.
Then all other users can point their nixpkgs inputs there.
(provided you all understand security implications of that)

9

I’ll look to get permission for setting up a local nixpkgs repository.

Given that our school specialises in networking and cybersecurity, we should at least be fine to handle the security aspects. :wink:

1 Like
10

Yep, since we (nixOS users) are not the majority in our school, there is no way that we get budget for that.

11

Huh, so this is what suffering from success looks like.
sorry I don’t have anything constructive to add to the discussion; I hope your issue is resolved feasibly.

4 Likes
12

I opened an issue about this issue 2 years ago already. I face the same problem regularly (a lot on trains. But even at home when I do flake commands in a VM). I don’t think we should require people to have a GitHub account to consume nixos Consider using CDN for serving the nixpkgs flake just as we did with nix channels? · Issue #6975 · NixOS/nix · GitHub

I think maybe this would actually work as a workaround:

inputs.nixpkgs.url = "https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.xz";

Let me know if that works. If it does we should perhaps point the default nixpkgs registry to this URI.

6 Likes
13

I just had 2 other ideas:

  1. Use a VPN
  2. Set up a proxy external to the school somewhere (eg at home if you have the hardware for it), then use that to download the updates

Both options cater to the same high level idea: to hide the school’s public IP behind something more personal (as in each student would set this up for themselves, possibly helped by a document outlining how to do it) and thus get around the rate limiting.

But getting an access token is probably less work per person.

14

A relatively easy workaround you could employ would be to use an internet proxy; commonly done via a VPN protocol these days.

There are a few reputable free ones even but tonnes of sharks, beware. Do your research.

Edit: Oh @jjpe was faster.

15

Thanks, will try that later today!

16

Can you clarify what endpoint is failing? Is the issue with cloning GitHub repositories? Or is the issue with fetching cached artifacts from cache.nixos.org?

17

RE: Discourse

I just went through the process of registering a new email (SaaS but not gmail) and a nixos discourse account with that email, all in tor-browser and I somehow didn’t even face captcha (I’m happy). This might have been lucky timing, but it worked.

Organizations with very silly NATs (read: universities) and regions with authorities that circumvent normal functioning of the Internet do exist, and I think it’s very important that “our” infra not select against people in such organizations or regions. IMO we should see your, @Flokkq, school’s situation as a test-case and look into adjusting the limits once we learn more

I also wonder how much sense IP-based rate-limiting makes considering there already is the Discourse’s “trust level” system

EDIT: I had posted a test message here with the new account, then I deleted the account on the “Preferences” page. The message was not preserved but trust me it was here :upside_down_face:

3 Likes
19

github endpoint is “failling”
❯ nix flake update
warning: error: unable to download ‘https://api.github.com/repos/Aylur/ags/commits/HEAD’: HTTP error 403
response body:

{"message":"API rate limit exceeded for *school.ip*. ("documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"};
2 Likes
20

So yes it’s indeed Github.

There isn’t much we can do from the Nix side here, this is between you, your school and Github.

21

Ah, okay I had a feeling! I ran into the same thing a while back when I was teaching a class and tried to have my students create GitHub accounts. Have you tried emailing GitHub support and seeing if they can make an exception for traffic coming from the school? This is a pretty common problem so I would be surprised if they didn’t have a quick way to make it work.

4 Likes