builtins.filterSource causes rebuilds

The docs for builtins.filterSource say

However, if source-dir is a Subversion working copy, then all those annoying .svn subdirectories will also be copied to the store. Worse, the contents of those directories may change a lot, causing lots of spurious rebuilds. With filterSource you can filter out the .svn directories

That gives me the impression I can use builtins.filterSource to prevent files from being copied into the nix store and therefore prevent unnecessary rebuilds, but I am unable to make it work.

I created a simple flakes project using the filterSource example from the docs.

flake.nix

{
  description = "A very basic flake";

  outputs = { self, nixpkgs }:
  let
    pkgs = nixpkgs.legacyPackages.x86_64-linux;
  in
  {
    defaultPackage.x86_64-linux = pkgs.stdenv.mkDerivation {
      pname = "test";
      version = "dev";
      src = builtins.filterSource (path: type: type != "directory" || baseNameOf path != "data") ./.;
      buildPhase = ''
        find . > $out
      '';
      installPhase = "true";
    };
  };
}

data/exclude_me

Lorem ipsum

When I run nix build .#, the filter works as expected in the sense that $out contains

.
./flake.nix
./flake.lock

but if I make a change to the excluded file date >> date/exclude_me and re-run nix build .# the build is reexecuted despite the real src being the same (judging from $out contents).

Do I misinterpret the docs and this has never been possible, is it impossible with nix flakes, or am I just doing something wrong? Thanks in advance

If someone finds this having the same issue:

I haven’t tackled it down yet, but I suspect it is related to using flakes in a git repository, which gets fully copied to the store no matter what.

Yeah, that is the issue. filterSource is called on a different store path each time so it will result in another different set of store paths, even though their content will be the same.

Cross-link: Flakes and filterSource, `builtins.filterSource` does not work in flakes · Issue #3732 · NixOS/nix · GitHub

Interestingly, it works with builtins.path (and thus all functions in https://github.com/NixOS/nixpkgs/blob/c576998594b4b8790f291d17fa92d499d1dc5d42/lib/sources.nix).

      src = builtins.path { filter = path: type: type != "directory" || baseNameOf path != "data"; path = ./.; name = "src"; };
$ nix show-derivation . | grep src
      "/nix/store/lysamz0yzfy9m0mj653bivf5nm79dhw7-src"
      "src": "/nix/store/lysamz0yzfy9m0mj653bivf5nm79dhw7-src",
$ echo foo > data/exclude_me
$ nix show-derivation . | grep src
warning: Git tree '/tmp/foo' is dirty
      "/nix/store/lysamz0yzfy9m0mj653bivf5nm79dhw7-src"
      "src": "/nix/store/lysamz0yzfy9m0mj653bivf5nm79dhw7-src",
$ git st
 M data/exclude_me
?? result
2 Likes

I cannot reproduce it – the hash changes for me. I think flakes exclude files not added to Git. Isn’t it your case?

More generally, builtins.path is preferred over builtins.filterSource because the name argument is fixed instead of being derived from the folder name, which can vary if a user checked out the repo under a different name. But that’s not the issue here.

To better understand the core of the issue, the best thing to do is to inspect the diff of both srcs. Outside of flakes it’s possible to use nix-instanciate --eval . -A mypackage.src or nix repl . and then select mypackage.src to find out the store path. Take note, and then repeat until you have both store paths. With flakes, use nix show-derivation .#mypackage (for example) to find the inputSrcs argument. Same logic as before.

Then let us know of the diff you found.

1 Like

I forgot to say that I am working on this project, which allows more precise control on which files to add:

2 Likes

Thanks for this great suggestion. I haven’t debugged my particular issue yet, but your nix-filter does the job absolutely well.

Your issue is the same as How to make `src = ./.` in a flake.nix not change a lot?, the name of the directory created by filterSource depends on the name of the initial directory, which is an impurity. builtins.path allows specifying a name that doesn’t depend on the source directory.

@Kha uses “src” as the name in here so that’s why path is working for them. You probably didn’t specified the name explicitly and it behaves similar to filterSource in that case.

1 Like