This cache seems to time out 99% of the time, though I’ve had weirdness happen (e.g. being able to curl the nix-cache-info file when nix tells me resolution fails) making me wonder if the problem is on my side.
OTOH, the last update about the cache is from, like, 2019, so I’m not exactly certain any of the info about it is current. It doesn’t seem like the main instance was ever switched over, and this clearly looks abandoned.
The “old” main cache “was” also hosted by fastly. This beta cache is just optimizations on top of the old setup, according to the various blog posts and announcements.
There was never an announcement that the main cache had been switched over, however, and some important issues are still marked as unresolved, so AIUI the beta cache never ended up in production.
If I’d have to guess the lack of activity is related to Graham joining detsys, the ticket dates seem to line up.
But yeah, I could run the cache tests to compare performance; I’d just like to know whether the project even is supposed to still be active before debugging deprecated infrastructure.
Anecdotally, I wonder if am also having issues with fastly. I have a nixos-rebuild that says it only needs to fetch 10GB which normally would complete within 3-5 minutes. Instead I’ve been waiting for the past half hour and it’s still just copying paths from cache.nixos.org.
Based on my previous experience of 10GB in 3-5 minutes, my previous “normal” speed with the cache would be 250-450 Mbps. I guess I had been getting pretty close to my ISP advertised speeds on fastly in the past, which as a naive evaluation seems nice and good for a CDN.
Now, with the fastly speedtest tool linked above I’m getting somewhere between 30-50 Mbps, while checking curl with specific downloads shows somewhere around 5-40 Mbps (yes, I’ve done conversions between bytes/bits so both are Mbps). It’s within order of magnitude, so I think the speed test tool vs observed cache behavior is aligned, and this overall speed aligns with the update now taking around 30-90 minutes for around 10G download. Still faster than compiling though…
I’m inferring this means that I’ve had a degradation in performance with Fastly for some reason, since it’s effectively 10x slower than my apparent previous speed (5-50 Mbps vs. 250-450 Mbps). I don’t remember having this problem when I updated a month ago, so maybe that helps the timeline.
(EDIT: also to be clear, at this point my only plan is to sit tight and hope fastly/my ISP figure something out. Unless someone else knows of another fast, reliable, trusted substitutor?)
Remember, squeaky wheel gets the oil. You’ll probably have to complain to fastly/your ISP to get anything. I’ll probably get around to asking fastly for support sometime.
There’s a Chinese university that hosts various FOSS mirrors. It hosts a nix cache on https://mirrors.tuna.tsinghua.edu.cn/nix-channels/store. I don’t think any other mirrors exist.
Its ping is pretty bad for me, but throughput seems better.
Assuming you don’t use trusted-substituters (or trusted-users or any of the other footgun options that disable signature checks), nix will confirm that the contents are in fact produced by hydra by checking package signatures, so trust shouldn’t be an issue.
Neither of those disable signature checks, trusted-substituters specifies additional substituters that unprivileged users can use and trusted-users just allows a user to perform privileged operations and set all settings (which can disable signature checks).
The documentation for trusted-public-keys (nix.conf - Nix 2.34.9 Reference Manual) makes it clear that in order for a store object to be accepted from another Nix store (such as a substituter), One of the following has to be met.
the store object has been signed using a key in the trusted keys list
If the receiving user was trusted and you either used ssh-ng: + require-sigs = falseorssh: (which implies require-sigs = false to my understanding) then yes that would probably have worked.
I’ve hit this exact issue in the past, it was the backfill from S3 → Fastly that was slow, not Fastly itself. To confirm this - if you kill the build and restart it, the downloads should be really quick, and slow to a crawl at the same point you killed it.
I think I got lucky but my personal fastly issues seems to have resolved without needing my own intervention. Fingers crossed it stays that way.
That’s really cool – for whatever reason I had assumed other caches would require new trusted-public-keys but of course if it’s really just a mirror (instead of a cache for a build farm such as the cuda cache) it doesn’t need it! Thanks again! I’ll keep this in mind.
Thanks for the thought! I had done some googling which had suggested this as a possibility. I tried multiple cached NixOS releases up to ones around 10 days old and had no better fortune so I think in my case it was not that I was hitting S3 directly instead of CDN.
But again it’s resolved now and I don’t want to dwell on it unless it happens again.
