mkDerivation.src as list of filenames?

How do I specify a list of local filenames for the mkDerivation.src argument?

Most examples use tarballs, git repositories or folders, but what I want to do for my repo is

stdenv.mkDerivation {
  ...
  src = [
    ./Makefile
    ./Pipfile
    ./Pipfile.lock
  ];

which errors out with

do not know how to unpack source archive /nix/store/ydghjgrkj96rm3jx25rbi2djl4h2cyyq-Makefile

2 Likes

You can use the srcs argument instead, which is an array of paths instead of a single path. But you’re still going to run into the problem of it actually trying to unpack each source file. I see three reasonable options:

  1. Override unpackPhase to just copy each entry of $srcs into the current directory.
  2. Override unpackFile to copy the argument into a known subdirectory of the current directory (subdirectory because unpackPhase expects to calculate a source root and will error out if it can’t find one). unpackPhase will then just call this on each entry of $srcs for you.
  3. Instead of passing each path separately, you can use ./. to copy the whole directory. You’ll probably actually want to use builtins.path { name = "foobar"; path = ./.; } instead so that way the current directory name doesn’t influence the output hash (since the output hash of a path copied into the store includes the basename). You can also pass a filter argument if you want to select which files to include, which could cut down on unnecessary rebuilds when making changes to irrelevant files (such as making cosmetic changes to your default.nix).
3 Likes

Instead of passing each path separately, you can use ./. to copy the whole directory. You’ll probably actually want to use builtins.path { name = "foobar"; path = ./.; }

Yes, I wanted to pick explicit files to avoid unnecessary rebuilds.

I’ll take a look at the unpackFile override, thanks!

How come it’s easy to specify a folder but not so easy to specify individual files?

The default implementation of unpackFile checks if the source is a directory, and if so, it just copies the whole thing. If it’s a file, it checks the extension and tries to decompress it. If it’s an unknown extension (including no extension) it complains that it doesn’t know how to unpack it. You can define additional unpack hooks by adding them to the unpackCmdHooks shell array to teach it how to decompress new file types. The idea here is that sources are assumed to be compressed and if it doesn’t know how to decompress it, it fails because you forgot to add the appropriate build input (various build inputs define unpack commands for you).

More generally, it’s extremely unusual for the source of a package to be a single file, whereas passing a whole directory is standard practice for a default.nix bundled inside the package source.

This is what I ended up with:

  src = [
    ./Makefile
    ./Pipfile
    ./Pipfile.lock
  ];

  unpackPhase = ''
    for srcFile in $src; do
      # Copy file into build dir
      local tgt=$(echo $srcFile | cut --delimiter=- --fields=2-)
      cp $srcFile $tgt
    done
  '';
1 Like

FWIW there’s a shell function stripHash which will already give you the basename back from the source file. This is what the default unpackPhase uses when copying directories.

1 Like

You can also make use of nix-gitignore:

with import <nixpkgs> {};
let patterns = ''
  *
  !Makefile
  !Pipfile.*
'';
in stdenv.mkDerivation {
  name = "gitignore-test";
  src = nix-gitignore.gitignoreSourcePure patterns ./.;
  dontBuild = true;
  installPhase = "ls > $out";
};
1 Like

FWIW there’s a shell function stripHash which will already give you the basename back from the source file. This is what the default unpackPhase uses when copying directories.

Perfect, thanks!

You can also make use of nix-gitignore

I probably need to give some more context. The python dependencies in the Pipfile listed take a humongous amount of time to install due to pipenv falling back to compiling from source.

So I want to setup a derivation that only has as input the pipenv dependencies that should be re-used as long as possible.

Then a second derivation which has the pipenv derivation as build input would be added which contains the actual application logic which changes far more frequently.

So whitelisting just the two Pipfile’s is really what I want to do here.

Yes, nix-gitignore should work for that use case: in the example code above, it will only import Makefile and anything beginning with Pipfile (note the first pattern, *, to ignore everything, then the two patterns prefixed with ! to include those files after all).

Gotcha. That makes sense.

It seems (to me) though that

 src = [
    ./Makefile
    ./Pipfile
    ./Pipfile.lock
  ];

  unpackPhase = ''
    for srcFile in $src; do
      cp $srcFile $(stripHash $srcFile)
    done
  '';

is a little less likely to be mis-interpreted by virtue of not mentioning gitignore or other unrelated topics.

1 Like

You could also write this as

src = let
  pathsToKeep = [ ./Makefile ./Pipfile ./Pipfile.lock ]
in
  builtins.path {
    name = "foo";
    path = ./.;
    filter = (path: type: builtins.elem path pathsToKeep);
  };
5 Likes

I did not get this to work, I think it is due to the filter receives an absolute path, while the pathsToKeep is including relative paths, I had to do a map builtins.toPath [ ... ] and as I understand it toPath is not recommended to use.

Also one thing to note is it is going to check for each file recursively, so you cannot match directories, only top level files or you have to specify the files explicitly in the pathsToKeep. It is not surprising, but just mentioning it here in case anyone else stumbles upon this.

1 Like

This is a pretty old thread, but I found myself here today while looking for something similar.

Rust workspace projects are tough, because in order to run workspace commands, cargo needs all of the entrypoints for all of the workspace packages present, e.g. the main.rs and lib.rs files usually. It does not need all of the rust files though: it only needs those for the package it’s building and any of its dependencies. Previously, I was including a ton of unneeded stuff from the repo root in the nix sources with a big, nasty filter function, which led to a lot of unnecessary rebuilds when changing unrelated files, and I was looking for a better way to do it.

I wound up combining the logic of sourceByRegex and sourceFilesBySuffices to make a function I could use to filter files by regex pattern, rather than needing to worry about making sure that each directory and nested directory was included. This allowed me to set up a standard list of regexes needed for all rust projects in the workspace, like:

  # Regular expressions for files relative to the workspace root
  rustWorkspaceSrcRegexes = [
    # Cargo.lock at the workspace root
    "^Cargo\.lock$"
    # Any cargo.toml file anywhere
    "^(.*/)?Cargo.toml$"
    # Any main.rs file anywhere
    ".*/main\.rs$"
    # Any lib.rs file anywhere
    ".*/lib\.rs$"
  ];

Which list I could expand on when generating sources for an individual package to include what it needed. Here’s the function I wound up with, almost identical to sourceByRegex except for the type == "directory" condition. This might be bad in various ways, but it’s super useful to us.

  srcByRegex = src: regexes:
    let
      isFiltered = src ? _isLibCleanSourceWith;
      origSrc = if isFiltered then src.origSrc else src;
    in
    lib.cleanSourceWith {
      filter = (path: type:
        let relPath = lib.removePrefix (toString origSrc + "/") (toString path);
        in
        type == "directory" || lib.any (re: builtins.match re relPath != null) regexes);
      inherit src;
    };

You can then use it like

# pass the path to the workspace root and regex patterns for anything you want to include
src = srcByRegex ./../.. (rustWorkspaceSrcRegexes ++ [ "regex-for-this-project's-files" ])

I actually ultimately wound up generating dependency regexes to pass to this function for local dependencies (i.e. foo = { path = "../../some-other-lib" }) automagically from the Cargo.toml, but that code is a bit hackier, so not including here.