This is very cool, thank you for this! I assume Harmonia is a little slower because it has to reassemble NARs from all the files in the store. I believe Attic and the nix-serve related caches behave similarly.
So, we have a couple performance “bands” involving whether they’re serving bare files or have to chase them down in the store layer, this matches a bit with intuition.
Yes. Especially zstd is expensive. But most people probably wouldn’t notice this that much. I guess if you ever run into performance issues and many clients, you could always had an LRU cache in nginx in front. But as you can see nginx rules over all.
zstd seems expensive but it also seems to amortize over more connections somewhat.
I now think it would be neat to do a nginx_nar_http_module that uses io_uring to traverse directories inorder to serve from the store… might not reach single file speed but would allow one to leave NARs unpacked and use nginx.
I don’t expect nginx with all it’s features and indirections would be much faster than harmonia. The fast majority of CPU time in uncompressed mode is already spent in the OS.
Are any of the caches built with LTO?