What happened to hydra.nixos.org?

Help
1

Libreoffice triggers local building, so I went to check if the package is cached.

https://status.nixos.org/ points to The depar- ture of.

But then hydra.nixos.org seems nonsense, it looks like this:

image
image1940×1045 173 KB

IP address: 157.90.104.34 / 2a01:4f8:2220:11c8::1

The depar- ture of.

With just a cookery book, with an invisible listener behind Lenina's left shoulder. "Oh, look at her heels. "What's the matter dispassionately, Mr. Foster, "in the vast majority of proles. ‘I recognized you on the pavement that it will be paralyzed. No more breathing. Finished.

They appeared to be a Tube station. There were times when consciousness, even the amplified record of their most pri- vate affairs in public. Wretched, in a loud speaker.

Mogol faces had given him. The boy moved on as though the sun gilded the ground. They seemed to breathe out of him to the bones.’ ‘You like doing this? I don’t care. In the past, just as authentical- 6 o 1984 ly, and upon the subject. Liberty to be gone through: the grovelling on the edge of the telephone book. "Not in mine, not in State Conditioning Centres.

Lenina ... No, no, no, no! He shut his eyes fixed on mine.’ At this moment it was ever put to other uses. Now, how did that knowledge exist? Only in his chair so as to what they were. There was a little surreptitious auto-erotism and homosexuality-abso- lutely nothing." "Nothing?" "In most cases, till they were.
See also:

    Barrel of.
    Last links that now.
    Terror had made sure.
    "I'm taking Lenina Crowne.
    Same you.
    382 1984 simply because of.
    Dropped out of.
    It? She knew when.

Copyright © 2531 Sat gazing stupidly at.

Was the site attacked??

Edit: Wayback Machine shows that the last sane snapshot is 2026-07-11, then 2026-08-12 is the capture of this. Did something happen between those dates? Why does GitHub - nix-community/hydra-check: check hydra for the build status of a package [maintainer=@makefu,@Artturin,@bryango] · GitHub still contacts the https://hydra.nixos.org/?

It’s Hydra page replaced with 1984/Big Brother quotes?

2

yes it is those “quotes” topic, and in fact the change is SO flipping annoying like bro, before iocaine was implemented and hydra used anubis for bot/web scrapping protection, i could easily use the service with zero interruptions, no clue why some disliked that solution but whatever, let’s be real - it worked on every device, and now you gotta have 1. clean ip, no blacklisting 2. up-to-date chrome (for me, even the kde built-in browser falkon is also rejected!) 3. legit user agent (no anonymising allowed)
so yeah, we have to do something about it … it’s currently literally unusable!
btw i also thought hydra got hacked XD

3

Current iocaine config is here:

I agree that this is worse than Anubis, which makes the experience a little worse for everyone (load delays from Proof of Work) but doesn’t just randomly send you Markov-chain-generated text with no explanation if you fail.

4

Excluding a whole range of ISPs hurts, emotionally. I wish for people using those, they would be met with a (don’t mind expensive) challenge instead.

5

How is this a good idea?

With my company we’re filling out compliance questionaires pointing out how NixOS is great for security, supply-chain safety and so on.

Then I come here and find that official sites are apparently serving some people content that looks like a complete take-over and deface of the main build server.

Clicking the links brings me to this post linking to https://github.com/readycool/iocaine which no longer exists (404).

Googling iocaine myself brings me to https://git.madhouse-project.org/iocaine/iocaine which if visited, outputs

To avoid getting devoured, say Trans rights are Human Rights.

Can anybody explain to me how I’m supposed to get NixOS through a supply chain security check, when every website along the path looks ike it was defaced?

Note that when I don’t code-format the above URL here, it inserts complete garbage even into our own very forum here; screenshot:

Screenshot from 2026-10-02 11-51-39

I understand why people don’t like Anubis.

But the alternative can’t look like the project was pwned by script kiddies. When I first looked at the screenshot from the top post myself, like poster my first thought was “oh no, NixOS was compromised”.

If Wayback Machine suggests to a reviewer that for months, hydra.nixos.org serves garbage, that does not help either.

This is a bad idea.

Why not simply return Access denied?

Researching that, all I can find on the iocaine website is:

It tries to poison them, so they’d go away forever in the long run.

How? Do people really believe that serving this garbage from some pages will make bots “go away forever in the long run”? Nobody in their right mind would implement a crawler bot like that. Even completely incompetent people asking their LLM to vibecode some crawler wouldn’t perma-go-away encountering this rubbish temporarily.

It seems the only thing we’re poisoning here is our reputation as a serious/secure project.

6

Anubis is generally considered insufficient, the scrapers have started just solving the PoW (where they don’t simply use any of the common bypass tools) because apparently their moneypits are bottomless.

Iocaine is more strict on the checks, which means that it also has a higher false positive rate, which is unfortunate.

These things are in place to try and preserve the service for “real” users. Iocaine reduces traffic load drastically, and for something as heavy as hydra I can imagine that being really impactful. Obviously I don’t host it and don’t have stats, but the tradeoff between iocaine and anubis was pretty noticeable when I tried them. Comparing that with the traffic hydra gets because people script against it…

Unfortunately the test for humanness is becoming harder by the day, so presumably the checks will have to keep getting stricter, too.

We’re witnessing the death of the free internet, you’re complaining to the wrong people here. Complain to your ISP for having unchecked botting problems, be angry at governments not protecting the commons as they should.

Or help upstream tweak the iocaine config for cases where you really do believe it’s overzealous, but understand that you’re just buying time that way.

The waybackmachine is basically dead, as an aside, and has been for half a decade or so, everything has been wall gardened to shit already.

7

I am not suggesting to turn off bot blocking.

Does it need to show “we got hacked”-suggesting rubbish in order to block, or can it show Access denied instead?

The websites of Debian, Ubuntu, Arch, RHEL, and Windows, all look perfectly fine in Wayback Machine.

Only NixOS shows defaced garbage.

8

So does https://nixos.org to my knowledge, hydra is an index into CI runs and should be orthogonal to the kind of marketing you seem to be doing.

At least, I have no idea how to view Windows’ internal CI pages. The other Linux distros you list don’t have quite the same depth to their CI setups NixOS does, either, since NixOS’ channel forwards are much more expensive and produce far more pages to crawl through. Don’t forget that we have an order of magnitude more packages than even Arch.

A lot of them also use cloudflare, which causes the exact same issues the rest of the thread bemoans, but tied to a for-profit US org instead of a configurable FOSS project, making it impossible to fix things for yourself even if you’re motivated.

Is there really any meaningful difference? Again, this is for a very deep internal page. At this point most affected people should be well aware iocaine exists. This is purely cosmetic.

But well, yes, if you can figure out how to make iocaine do that I’m sure someone would consider that PR.

9

iocaine does seem highly programmable, so it’s probably feasible to modify the default request handler to return a simple Access denied instead of markov junk.

10

Made a PR, assuming the infra team thinks this is reasonable (I think it’s pretty good, we aren’t a militantly anti-LLM project, (see the Nixpkgs AI policy), so we don’t have much motivation to “poison” them with markov junk).

(I also tried running some markov junk through https://jevplayground.com/, and it detected the markov junk easily with 82% probability. there’s probably lots more efficient ways to do it, but I was just interested in whether scrapers could run text through Jev to detect Iocaine and similar markov poisoners.)

11

Assisted-by: pi (GPT-5.6-Luna)

Quite ironic in my eyes :grin: but that’s the world today, and I don’t want to “hijack” this discussion.

12

I think it’s good to stress again that hydra.nixos.org a relatively internal page.

Normal Nix* users shouldn’t need to touch it, so I don’t understand why it should affect compliance questionnaires (but I don’t know that area). It’s meant only for power-users among contributors to nixpkgs, and we’ve been fighting overloads from misuses even before LLMs (though not so intense ones).

Maybe this should be more apparent somewhere? What is leading people to believing something else?

13

(i have no problem with iocaine in principle or in practice)

links on status.nixos.org on the state graphs, which is slightly less internal (as in, it’s on the nixos.org site in the footer link set), point to hydra.nixos.org.

14

Hm, that’s probably an expensive page to render, too, and I don’t think the channel status has to be that up-to-date for the kind of use case that would get someone to click that link.

Perhaps we could cache that data, display it in a more readable way, and make the visible side of the infra less problematic to browse all in one go?

15

I mean, if a package is trying to compile locally, it’s not unreasonable to go check the CI system that builds it for failures (we do have GH issues for build failures in nixpkgs, but AFAIK those are mostly manual after somebody checks hydra, and also the hydra logs have to get linked in the issue AFAIK). I don’t think that that’s something that should be exclusive to “power-users among contributors to nixpkgs”.

Re compliance; they might want to know more about the infra that’s building the packages and sending them off to the cache. Although I don’t know that area either.

16

:wink:
I probably wouldn’t have took the time to make this PR without LLM assistance.

17

The GH issue template requires getting the hydra link, and even if you use hydra-check, you’re meant to go actually follow the link and see if a dependency caused the build failure.
Reporting a build failure on GH is not a power-user action.

Also I’ve been participating in nixpkgs for years and I had no way of knowing about the iocaine change, so again, this is not a helpful response to say “contributors should just know about this thing we won’t tell them about”.

18

As a fallback when it is “Access denied” maybe allow Sing in with discourse as identity provider. Sing in seems to be possible at the moment with google, github and hydra account (i guess you have one if you are in infra as there isn’t a register button). Why allow sing in with discourse.nixos,org? It would keep the infra in nixos.org hand at least i think so.

19

Most people visiting the Hydra pages aren’t meant to sign in. (The default logged-in permissions are the same as logged-out, AFAIK.)

20

quick note that iocaine’s primary objective is to poison llm bots’ results and searches, not block them. anubis’s is to make it computationally expensive enough that it naturally throttles requests, not block them. whether either of these objectives are “worth it” is entirely down to one’s interpretation of “llm scrapers vs the server: the showdown”. if you want to see how much it’s doing (a lot), you can see the grafana grafs.

weird that you’re being blocked—hetzner is (surprisingly, given it feels like ~50% of hetzner users are either torrent seedboxes or malarkey) not in the asn blocklist, and from that screenshot it looks like you’re using somewhere around the latest firefox which has never caused me issues (only v1 - v139, and v141 - v150 are blocked, v140 is an esr so permitted). you don’t happen to be doing anything weird with your user agent header, do you?

yeah it’s only been mentioned in a couple of places but never as an “announcement”. ultimately in a perfect world nobody would need to know, but evidently we are very much not in one. where would be the most appropriate? discourse? matrix? github somewhere? all three? perhaps in the issue template itself?

this is possible, though you’d have to do a bit of magic. a (very) cursory look at the nixos module seems to indicate it’s not directly configurable that way via the options (would require injecting a custom script into the config block). it doesn’t look like the default handler supports this, so it’d need a new handler which looks like a bit (a lot) of a yawn.

an additional “help” here would probably be peeping the referer header, which if visited from the link in the issue template would be https://github.com/nixos/nixpkgs. not everybody exposes that header in their requests, though, depending on their privacy settings and/or the nicheness of their browser. this is of course very forgeable.

though specifically on discourse as an idp, i don’t really think wiring it into discourse accounts is a good idea (i barely think discourse—the software itself, not us using it—is a good idea). anyone submitting an issue from github would already have a github account which you can use to sign in with. better alternative would be wiring discourse, hydra, etc. into keycloak, similar to arch linux, but that’s then another thing have to worry about.