mkDerivation.src as list of filenames?

How do I specify a list of local filenames for the mkDerivation.src argument?

Most examples use tarballs, git repositories or folders, but what I want to do for my repo is

stdenv.mkDerivation {
  ...
  src = [
    ./Makefile
    ./Pipfile
    ./Pipfile.lock
  ];

which errors out with

do not know how to unpack source archive /nix/store/ydghjgrkj96rm3jx25rbi2djl4h2cyyq-Makefile

You can use the srcs argument instead, which is an array of paths instead of a single path. But you’re still going to run into the problem of it actually trying to unpack each source file. I see three reasonable options:

  1. Override unpackPhase to just copy each entry of $srcs into the current directory.
  2. Override unpackFile to copy the argument into a known subdirectory of the current directory (subdirectory because unpackPhase expects to calculate a source root and will error out if it can’t find one). unpackPhase will then just call this on each entry of $srcs for you.
  3. Instead of passing each path separately, you can use ./. to copy the whole directory. You’ll probably actually want to use builtins.path { name = "foobar"; path = ./.; } instead so that way the current directory name doesn’t influence the output hash (since the output hash of a path copied into the store includes the basename). You can also pass a filter argument if you want to select which files to include, which could cut down on unnecessary rebuilds when making changes to irrelevant files (such as making cosmetic changes to your default.nix).

Instead of passing each path separately, you can use ./. to copy the whole directory. You’ll probably actually want to use builtins.path { name = "foobar"; path = ./.; }

Yes, I wanted to pick explicit files to avoid unnecessary rebuilds.

I’ll take a look at the unpackFile override, thanks!

How come it’s easy to specify a folder but not so easy to specify individual files?

The default implementation of unpackFile checks if the source is a directory, and if so, it just copies the whole thing. If it’s a file, it checks the extension and tries to decompress it. If it’s an unknown extension (including no extension) it complains that it doesn’t know how to unpack it. You can define additional unpack hooks by adding them to the unpackCmdHooks shell array to teach it how to decompress new file types. The idea here is that sources are assumed to be compressed and if it doesn’t know how to decompress it, it fails because you forgot to add the appropriate build input (various build inputs define unpack commands for you).

More generally, it’s extremely unusual for the source of a package to be a single file, whereas passing a whole directory is standard practice for a default.nix bundled inside the package source.

This is what I ended up with:

  src = [
    ./Makefile
    ./Pipfile
    ./Pipfile.lock
  ];

  unpackPhase = ''
    for srcFile in $src; do
      # Copy file into build dir
      local tgt=$(echo $srcFile | cut --delimiter=- --fields=2-)
      cp $srcFile $tgt
    done
  '';

FWIW there’s a shell function stripHash which will already give you the basename back from the source file. This is what the default unpackPhase uses when copying directories.

You can also make use of nix-gitignore:

with import <nixpkgs> {};
let patterns = ''
  *
  !Makefile
  !Pipfile.*
'';
in stdenv.mkDerivation {
  name = "gitignore-test";
  src = nix-gitignore.gitignoreSourcePure patterns ./.;
  dontBuild = true;
  installPhase = "ls > $out";
};

FWIW there’s a shell function stripHash which will already give you the basename back from the source file. This is what the default unpackPhase uses when copying directories.

Perfect, thanks!

You can also make use of nix-gitignore

I probably need to give some more context. The python dependencies in the Pipfile listed take a humongous amount of time to install due to pipenv falling back to compiling from source.

So I want to setup a derivation that only has as input the pipenv dependencies that should be re-used as long as possible.

Then a second derivation which has the pipenv derivation as build input would be added which contains the actual application logic which changes far more frequently.

So whitelisting just the two Pipfile’s is really what I want to do here.

Yes, nix-gitignore should work for that use case: in the example code above, it will only import Makefile and anything beginning with Pipfile (note the first pattern, *, to ignore everything, then the two patterns prefixed with ! to include those files after all).

Gotcha. That makes sense.

It seems (to me) though that

 src = [
    ./Makefile
    ./Pipfile
    ./Pipfile.lock
  ];

  unpackPhase = ''
    for srcFile in $src; do
      cp $srcFile $(stripHash $srcFile)
    done
  '';

is a little less likely to be mis-interpreted by virtue of not mentioning gitignore or other unrelated topics.

You could also write this as

src = let
  pathsToKeep = [ ./Makefile ./Pipfile ./Pipfile.lock ]
in
  builtins.path {
    name = "foo";
    path = ./.;
    filter = (path: type: builtins.elem path pathsToKeep);
  };