Serving nixpkgs with xz compression

One of the most downloaded artifacts nixpkgs is currently served with gz compression as it comes from github instead of the more effective xz compression which is used for all other artifacts coming from cache.nixos.org.

The size of the current nixpkgs unstable archive:

  • gz compressed: 26.5 MB
  • xz compressed: 18 MB

For some of you sitting behind gigabit, it might not make a difference, but for many other people time waiting for nixpkgs to finish downloading accumulates during the day.

The interesting thing is, that a xz compressed version of each nixpkgs is already available from cache.nixos.org. If one imports a remote nixpkgs and access it’s path attribute, one can observe that nix re-downloads a xz version of that nixpkgs archive.

It is there, but it seems like there is no way of finding it without first downloading the badly compressed nixpkgs.tar.gz.

Is there a way to fix that? can we somehow make a cheap API that provides us with the xz URL for a given nixpkgs revision?

11 Likes

The interesting thing is, that a xz compressed version of each nixpkgs
is already available from cache.nixos.org. If one imports a remote
nixpkgs and access it’s path attribute, one can observe that nix
re-downloads a xz version of that nixpkgs archive.

As an aside, would you be able to provide an example or otherwise say
more about that process? I’m not quite sure what you mean and how you
obtain that xz-compressed package set after downloading the tar.gz.

Other than that, xz compression, especially with the numbers you
provided, would be very welcome on my internet connection :))

1 Like

Do flakes try to substitute inputs?

I wasn’t able to reproduce nix doing the xz download by itself, but I can show that the file is there:

nix repl

nix-repl> pkgs = builtins.getFlake "nixpkgs/nixos-unstable"

nix-repl> pkgs.legacyPackages.x86_64-linux.path
/nix/store/3c249himjn7sh3cwnapmh5qgrnmd42xg-source

gives us the store hash 3c249himjn7sh3cwnapmh5qgrnmd42xg.

Ask the cache for that store path:

> curl https://cache.nixos.org/3c249himjn7sh3cwnapmh5qgrnmd42xg.narinfo
StorePath: /nix/store/3c249himjn7sh3cwnapmh5qgrnmd42xg-source
URL: nar/1gf7v3819akxna3i9ffqaffbcb72kjg15ss2qdhx2q7x4z6lpcgw.nar.xz
Compression: xz
...

Download the xz file:

> curl https://cache.nixos.org/nar/1gf7v3819akxna3i9ffqaffbcb72kjg15ss2qdhx2q7x4z6lpcgw.nar.xz > /tmp/nixpkgs.nar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 17.7M  100 17.7M    0     0  2481k      0  0:00:07  0:00:07 --:--:-- 2728k
1 Like

It seems like it. I just noticed that nixpkgs inputs locked via flake.lock will be downloaded from the cache in xz compression instead of from github. The flake.lock contains a narHash which allows nix to get the artifact from the cache.
Therefore the problem will basically be solved as soon as flakes is used.