Fetchurl with compressed files

Help
1

Hi everyone! How can I decompress files downloaded using fetchurl? I’ve figured out that postFetch hook can be used for that, but how can I rename the file in the nix store?

Example: /nix/store/xxxxx-filename.json.zst → /nix/store/xxxxx-filename.json

Thank you!

2

To decompress files you want fetchzip. If it does not deal with a particular compression format, add stuff to unpackPhase. Since usually these archive contains multiple files, i guess they will be created in a subfolder based on the pname?

3

Thank you for your suggestion. I’m dealing with regular files, not archives (e.g. “.txt.zst”). I just want to decompress them.

4

Then you can do something like:

stdenv.mkDerivation {name = "test.json"; src = ./test.json.zst; unpackPhase = ":"; nativeBuildInputs = [ zstd ]; installPhase = ''zstd -d "$src" -o "$out"'';}

Proof:

nix-repl> :b stdenv.mkDerivation {name = "test.json"; src = ./test.json.zst; unpackPhase = ":"; nativeBuildInputs = [ zstd ]; installPhase = ''zstd -d "$src" -o "$out"'';} 

This derivation produced the following outputs:
  out -> /nix/store/xrrwznjgy10pykb6ham5ch32dlgmgc0p-test.json

$ cat /nix/store/xrrwznjgy10pykb6ham5ch32dlgmgc0p-test.json
{a: 42}
5

And with the url:

stdenv.mkDerivation {
  name = "test.json";
  src = pkgs.fetchurl {
    url = "http://0.0.0.0:8000/test.json.zst";
    hash = "sha256-wNX3RotfJ043JWRvfokCF3VY+/bArkakruoUxNiyMdI=";
  };
  unpackPhase = ":";
  nativeBuildInputs = [ zstd ];
  installPhase = ''zstd -d "$src" -o "$out"'';
}

will output a derivation pointing to something like:

$ cat /nix/store/zg201kixdw2n0x9ljll3xz1dvax1wjxa-test.json
{a: 42}

Now, I’m not sure why you want to do that, but if you do some more steps, you might prefer to move the compression part to unpackPhase, and use installPhase/buildPhase to do other stuff on this json.

1 Like
6

Thank you! What does “:” in unpackPhase mean? I know that you can skip phases with “true”, but this is something I’ve never seen.

7

It is just replacing the unpackPhase with an empty phase doing nothing. Guess that you could equivalently use dontUnpack = true;

8

If I want to turn it into a function, can I use fetchzip, but skip unpacking and only decompress on “non-archives”? So, basically I guess it should be similar to your example but with unpackPhase from fetchzip?

9

I’ve seen the source code for fetchzip, can it be accomplished by using unpackPhase = "unpackFile ..."?