How to make an arbitrary command pure?

Development
#1

If I have a command that accesses the internet and downloads a bunch of files into a folder, but I know that the hash of the contents of the folder is constant, is thery anyway to convince git that it is pure?

For example:
./run.sh downloads libs/foo and libs/bar which are listed in libs.lock, from the internet.
I know that for a given libs.lock, the hash of all files in libs/* will be constant.
Is there anyway to convince Nix that this is pure? Like specifiying a sha256 argument for libs folder?

#2

Yes. This is called a fixed-output derivation (or FOD). Take a look at Advanced Attributes

3 Likes
#3

If I enable FOD, things in $out/bin are not added to $PATH. Is this expected?

#4

No that’s not normal.

You’re saying buildInputs and nativeBuildInputs are appearing in PATH when outputHash[Algo] are undefined, but not when outputHash[Algo] are defined?

If that’s what you’re seeing in your build env let us know your Nix version, Nix invocation/args, and your expression; because you might have found a bug in the new 2.10 release.

Also just to clear this up if the above isn’t the case : are you saying “a command I usually run in a devShell has a different PATH than the package builder” - because would be expected.

#5

$out is $out whether you use a FOD or not.

Though what gets changed when using a FOD is, that a FOD doesn’t get rebuild, if it already exists.

So after you built and then change something and rebuild again, nothing changes, as nix things it already has something with that hash.

You always need to update the hash.

That’s why we usually avoid FODs in the “main” derivation and try to use them only for the source or deps and copy them into place in the “main” derivation.

6 Likes
#6

@Growpotkin I’m using home-manager and adding the derivation to home.packages. I expected whatever’s in $out/bin to be symlinked to ~/.nix-profile/bin.

@NobbZ I changed the pname of the derivation to make sure that a previous version is not getting referenced.

In case it helps:

❯ nix --version
nix (Nix) 2.8.1

Here are the full .nix files that I used.

In this case, $out/bin/vue-language-server doesn’t get symlinked to ~/.nix-profile/bin:

{ stdenv, nodejs-16_x, yarn }:
stdenv.mkDerivation rec {
  pname = "volar-vue-language-server";
  version = "0.38.8";

  src = fetchTarball {
    url = "https://registry.npmjs.org/@volar/vue-language-server/-/vue-language-server-${version}.tgz";
    sha256 = "sha256:1m8k138rq5lz1hbi0zbhd0s9zn1m5idhwpwldpg2d5nndrwc6pcn";
  };

  outputHashMode = "recursive";
  outputHashAlgo = "sha256";
  outputHash = "sha256-XB9zJ1yt/E7ap4DzPLDY3bjWe43o829V7Gn9KIv5EQY=";

  buildInputs = [
    nodejs-16_x
    yarn
  ];


  buildPhase = ''
    HOME=./ yarn 
    rm -rf .cache .yarnrc
  '';

  installPhase = ''
    mkdir -p $out
    cp -r node_modules bin out $out/
    mv $out/bin/vue-language-server{.js,}
    chmod +x $out/bin/vue-language-server
  '';
}

In this case, $out/bin/volar-test does get symlinked:

{ stdenv, nodejs-16_x, yarn }:
stdenv.mkDerivation rec {
  pname = "volar-vue-language-server";
  version = "0.38.8";

  src = fetchTarball {
    url = "https://registry.npmjs.org/@volar/vue-language-server/-/vue-language-server-${version}.tgz";
    sha256 = "sha256:1m8k138rq5lz1hbi0zbhd0s9zn1m5idhwpwldpg2d5nndrwc6pcn";
  };

  # outputHashMode = "recursive";
  # outputHashAlgo = "sha256";
  # outputHash = "sha256-4vkgFB8KHHzU56nxJGvoGhUO/3hWmWLbFNsKJd/t+k0=";

  buildInputs = [
    nodejs-16_x
    yarn
  ];


  buildPhase = ''
    # HOME=./ yarn 
    # rm -rf .cache .yarnrc
  '';

  installPhase = ''
    # mkdir -p $out
    #
    # cp -r node_modules bin out $out/
    # mv $out/bin/vue-language-server{.js,}
    # chmod +x $out/bin/vue-language-server

    mkdir -p $out/bin
    echo "#!/bin/bash" >> $out/bin/volar-test
    echo "echo Hello" >> $out/bin/volar-test
    chmod +x $out/bin/volar-test
  '';

}
#7

Yarn is kind of a shit show with Nix unfortunately but you can definitely get this to work.

I see a pretty glaring difference in these builds though that will explain why bin/* isn’t showing up: did you check to see if it exists? cp -r is probably biting you.

1 Like
#8

I can’t test it currently, as I have no access to my nix machines, though does the “faulty” one indeed create something in bin as you expect it to do?

As soon as you “install” it, linking should happen.

#9

Here is a minimal repro of the behavior:

As soon as I comment out the outputHashMode, outputHashAlgo and outputHash lines, test-fod becomes available on $PATH.

{ stdenv }:
stdenv.mkDerivation rec {
  pname = "test-fod";
  version = "0.1.1";

  outputHashMode = "recursive";
  outputHashAlgo = "sha256";
  outputHash = "sha256-FhrJ5xru61YD92b5WCM9l1J9ttMU40cdS3TL0OLab1Q=";

  dontUnpack = true;

  installPhase = ''
    mkdir -p $out/bin
    echo "#!/bin/bash" >> $out/bin/test-fod
    echo "echo Hello" >> $out/bin/test-fod
    chmod +x $out/bin/test-fod
  '';
}
#10

Continuing the discussion here: FOD's `$out/bin/<binary>` is not available in `$PATH`. Is this expected?

#11

Fixed Output derivation don’t allow /nix/store paths to be referenced.

1 Like
Hosted by Flying Circus.