Say I have a flake.nix with a derivation built from ./.. I would like to understand why it gets a different src when I only add a comment in flake.nix, and I filterSource a lot of files with:
let
ignoreFiles = [
# Nix files
"flake.nix" "flake.lock" "default.nix" "shell.nix"
# git commit and editing format files
".commitlintrc.yml" "package.json" ".husky" ".editorconfig"
# CI files
".cirrus.yml" "action.yml" "azure-pipelines.yml" "Dockerfile"
# Git files
".github" ".gitattributes" ".gitignore" ".git"
];
in
# If the `path` checked is found in ignoreFiles, don't add it to the source
if pkgs.lib.lists.any (p: p == (baseNameOf path)) ignoreFiles then
false
else
true
) ./.;
I checked with ls -la in a preConfigure hook that the files are filtered, and what’s peculiar is that although flake.nix is filtered, adding a comment to it, that doesn’t change evaluation, changes the hash of the source:
filterSource uses the name of the directory the filter is applied to as the store path name (the part after the hash) for the one it creates. It is not possible to specify the name yourself using filterSource.
Nix moves all the files tracked by the version control inside the self repository (i.e. directory the flake belongs to) to the Nix store. This happens even before the evaluation of the flake starts. So the flake evaluates inside a directory named <hash>-source in the Nix store, where <hash> depends on all the files tracked by the versions control.
During the evaluation, filterSource will create a directory named <hash2>-<hash>-source that includes the files that passed the filter you specified. Since <hash2> also depends on <hash>-source, we get different hashes even if the change is in a file that didn’t pass the filter.
I think this means builtins.filterSource is useless inside a flake.
Using builtins.path, which allows specifying the store path name, like below should fix the problem.
src = builtins.path {
path = ./.;
name = "sile";
filter = path: type:
let
ignoreFiles = [
# Nix files
"flake.nix"
"flake.lock"
"default.nix"
"shell.nix"
# git commit and editing format files
".commitlintrc.yml"
"package.json"
".husky"
".editorconfig"
# CI files
".cirrus.yml"
"action.yml"
"azure-pipelines.yml"
"Dockerfile"
# Git files
".github"
".gitattributes"
".gitignore"
".git"
];
in
# If the `path` checked is found in ignoreFiles, don't add it to the source
if pkgs.lib.lists.any (p: p == (baseNameOf path)) ignoreFiles then
false
else
true
;
};
Also I suggest nix-filter rather than creating the filtering boilerplate yourself in every project.
correction
I realized my previous answer was not entirely correct. It included this paragraph:
Source directories without an explicit name are not reproducible. The default name uses the basename of the current directory. Without flakes, this doesn’t change as long as you don’t rename the project directory. So it might be harder for you to discover the reproducibility issue without flakes. Also see Best practices — nix.dev documentation
The mentioned impurity is only valid for non-flake evaluations, because flake uses source as the store path name, regardless of the directory name. So saying it might be harder to discover without flakes was wrong.
another good idea is to define a function called projectPath like this:
this way projectPath "/my/folder" only depends on the NAR serialization of /my/folder and not of flake’s self (which depend on the entire repo and therefore makes the cache fragile)
we use this extensively on Makes to achieve incremental builds on big monorepos, where changing a code comment MUST only trigger the build of such code comment and leave intact everything else
Just using ./my/folder as the src value should have the same effect. It shouldn’t rebuild if a file outside that directory changes.
it works like that on nix stable,
it does not work like that on flakes,
on flakes the git repo (src = ./.) also known as. self.sourceInfo is always copied into the store for hermeticity.
On flakes ./my/folder is equivalent to: "${self.sourceInfo}/my/folder".
Given self.sourceInfo changes on any code change anywhere in the repository (because it represents the entire repository!), then ./my/folder ("${self.sourceInfo}/my/folder") also changes because of it’s invisible dependency on the entire repository (self.sourceInfo)
with the mentioned projectPath function you break that implicit dependency on self.sourceInfo
It works like that both within a flake and outside one. There is no reason to use projectPath to avoid unnecessary rebuilds because there will be none without it. Out of directory changes won’t trigger a rebuild as long as you specify a relative path (nix type) for the src value like, like ./my/folder.
I know that self repository is moved to Nix store before the evaluation, but I think this is irrelevant in this case.
This is not true. "${toString ./my/folder}"[1] is equivalent to "${self.sourceInfo}/my/folder". ./my/folder will result in nix moving that directory to a separate store path with the name <hash>-folder[2].
[1] toString prevents nix moving the directory to the store and let’s you reference the absolute path of the current directory (in a flake it’s a nix store anyway).
[2] The basename of the path is used in the store path name.
I am pretty sure changes in ./a/c don’t trigger a rebuild when you use ./a/b as the src. I created a repository to confirm this. As long as you don’t change anything inside foo/src or bar/src you won’t trigger a rebuild for foo or bar derivations, respectively. I also made them exactly the same derivation to show that, parent directory name (foo and bar) doesn’t affect the output derivation. This is true both for flake and non-flake evaluation.
Input sources of derivation foo can be checked with nix show-derivation '.#foo'| jq '.[] | .inputSrcs' for the flake evaluation and with nix show-derivation -f ./derivations/foo.nix | jq '.[] | .inputSrcs' for the non-flake evaluation.
Note that I didn’t test projectPath myself, so it is possible it has other advantages (one of them is using src as the name of the project root directory for non-flake evaluations, which removes the impurity described in here) but preventing rebuilds is not one of them.