10x slower downloads with nix's http2 default

Development
1

Sharing this here: One of the places I work from is in Germany with a “Telekom Hybrid” 5G connection.

There currently, when downloading from cache.nixos.org, I get 2 Mbit/s.

Semi-workaround: If I pass --option http2 false to my Nix commands, I get 20 Mbit/s – 10x faster.

This is with nix (Nix) 2.31.2 on NixOS 25.11.

(The connection overall can deliver much more, e.g. iperf3 -R -P20 shows 180 Mbit/s over 20 parallel TCP connections, downloading from my own iperf3 server to the work location.)

3 Likes
2

Switch the Provider or get a VPN (official Telekom statement!).

Telekom is one of the few providers worldwide who are throttling peering traffic artificially - in case of unbalanced ingress traffic - when the peer partner does not pay extra traffic.

HTTP2 is one tcp connection (throttling to 2mbit or less) with session internal multiplexing and pipelining.

Switching to HTTP1.1 does open a flood of many individual tcp connections instead. Each one is throttled to the same 2mbit or less.

Just raise the number of concurrent tcp sessions till your link is saturated.

Stateless traffic like UDP via HTTP/3 or a stateless UDP based VPN like Wireguard works as well.

The (throttling) middle box can only manipulate (unauthenticated) tcp session states.

7 Likes
3

Thanks, that sounds plausible. I was at first confused, because the fast iperf3 is from Hetzner, which previously was famously known for not paying the Telekom ransom, but apparently a couple years ago Hetzner started paying so it’s fast now, and Fastly didn’t.

Just raise the number of concurrent tcp sessions till your link is saturated.

This doesn’t work entirely though: No matter how much I increase the numbers in --option http2 false --option max-substitution-jobs 64 --option http-connections 64, I cannot get it noticeably above 20 Mbit/s (so still far away from the 180 Mbit/s I see in iperf3 -P20).

4

Please join Netzbremse - Deutsche Telekom is throttling the internet! so we can fight this illegal practise. Unfortunately Telekom is an unlawful cartel and they’re explicitly throttling fastly traffic :frowning:

There is not much we can do on our side. Go talk to your local politicians and your local consumers rights organisation

8 Likes
5

This is weird. We have a Telekom DSL line in the hackerspace and I think I would have noticed cache.nixos.org being that slow. I’m often there in the evenings from 6pm.

Can you give Lix a try? They have done lots of work on fixing HTTP2 issues with curl, and it would be good to find out whether this is an implementation or a network issue.

They also have experimental HTTP3/QUIC support (since 2.94.0) that you could try, it does work well for me. QUIC support was enabled for cache.nixos.org on 2025-11-14.

2 Likes
6

I don’t think it’s all Telekom lines everywhere that are slow.
I have access to 2 Telekom connected locations: One with Telekom Hybrid 5G (has the problem above), and a Telekom fiber connection in a different city.

I tested it right now with nix-store -r /nix/store/hqdgw5g4m4hwds79nsikncg9jglq3ajg-chromium-147.0.7727.101, simultaneously:

  • The Hybrid 5G has the problem (2 Mbit/s).
  • The Fiber in the other city is fast (250 Mbit/s).

Both run NixOS 25.11, both use no WiFi (Ethernet instead).

Also pointing out that Telekom Hybrid is usually a bonded connection between ADSL and 5G. However, I have separated the two (using my own ADSL modem + router for ADSL, and the Telekom Speedport Smart 4 router for 5G only). This allows me to measure speed for our ~200 Mbit/s 5G and our 12 MBit/s ADSL separately. Both have the problem that they pull with around 2 Mbit/s from cache.nixos.org over default nix-store -r. The ADSL has both IPv4 and IPv6; it is equally slow for both.

7

Can you give Lix a try?

Is there an easy way to use just the equivalent of nix-store -r for Lix to test just the download?

I haven’t used it yet, and tried with nix-shell -p lix to see what options I have, but get:

$ lix --help
error: unrecognised flag '--help'
'lix' is reserved for external subcommands, is your subcommand available in the PATH?

Edit: I figured now that in nix-shell -p lix, all the nix-* commands are shadowed by Lix ones.

nix-store --version
nix-store (Lix, like Nix) 2.93.3

Still need to know though: Do I need to invoke them in a special way? E.g. if I use Lix’s nix-store, do I need to run it as root so that it doesn’t make my system’s nix-daemon do the download on its behalf? Is that enough? Any clever way I can validate that it’s really Lix that’s doing the download, e.g. with some verbose logs?

8

nix-store -r <whatever> --store ~/throwaway-store should make it just do the download, no root required.

3 Likes
9

Thank you! I tried

/nix/store/j23vzyvnkfqqmabdp076ipfl3man7r3c-lix-2.93.3/bin/nix-store -r /nix/store/hqdgw5g4m4hwds79nsikncg9jglq3ajg-chromium-147.0.7727.101 --store ~/tmp/lix-throwaway-store

but it has the same low speed as Nix.

I tried --option http3 true but it does not change speed. However, bandwhich shows it as (tcp) so I think HTTP3 isn’t working.

Do I need to do something special to get HTTP3 working?

10

FWIW 2.33-2.34 have the same libcurl pausing stuff that has been an issue in the past - and I assume that’s the fixes you were referring to. Getting multi-gigabit substitution is not an issue with upstream nix even when bottlenecked by xz decompression.

1 Like
11

It turns out my lix-2.93.3 was just too old. With lix-2.94.1 seems to work

See Lix 2.94 release notes about http3.

With that I see udp for Lix’s nix-store in bandwhich.

However, Lix’s nix-store is just runing on 100% CPU for minutes now.

straceing shows that it’s busy-looping around a sendmsg() Message too long error:

[pid 1160079] sendmsg(15<socket:[6153987]>, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="B\tx`\323?\v\22\263\363\n/\206\206jZ\341P\\\366H\344\3l\4~\6\241I\246\353\334"..., iov_len=1444}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = -1 EMSGSIZE (Message too long)
[pid 1160079] sendmsg(15<socket:[6153987]>, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="B\tx`\323?\v\22\263\363\n/\206\206jZ\341P\\\366H\344\3l\4~\6\241I\246\353\334"..., iov_len=1444}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = -1 EMSGSIZE (Message too long)
[pid 1160079] sendmsg(15<socket:[6153987]>, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="B\tx`\323?\v\22\263\363\n/\206\206jZ\341P\\\366H\344\3l\4~\6\241I\246\353\334"..., iov_len=1444}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = -1 EMSGSIZE (Message too long)

Edit: That Lix uses curl-8.18.0, which has this bug fixed 3 months ago: https://github.com/curl/curl/issues/20440

Commit:

Fix only in curl >= 8.19.

Another edit: The Lix from staging-next is new enough:

/nix/store/0yih299dvjvl5nv3978sii4j6z4wplh8-lix-2.94.1/bin/nix-store -r /nix/store/hqdgw5g4m4hwds79nsikncg9jglq3ajg-chromium-147.0.7727.101 --store ~/tmp/lix-throwaway-store --option http3 true

This correctly works over HTTP3 (udp shown in bandwhich).

Speed improves from 2 - > 4-7 Mbit/s.

A bit better, but nowhere near the ~200 Mbit/s the connection can do (I confirmed it can do that also over UDP with iperf3 -R -u -b 200M, downloading from my Hetzner server).

2 Likes
12

HTTP3 just makes it harder to throttle, but not impossible.

When you have a server with full speed peering - outside of the impacted Telekom backbone: setup a Wireguard to enjoy full speed.

Wireguard is, lightweight, stateless, easy to setup and you can route simply via AllowedIP Parameter only the impacted Fastly AS networks around the problem.