{}
, stdenv ? pkgs.stdenv
, lib ? pkgs.lib
}:
let
inherit (lib) cleanSource cleanSourceWith;
# Remove `*/.yarn/cache' directories.
cleanYarnCacheFilter = name: type:
! ( ( type == "directory" ) &&
( ( baseNameOf name ) == "cache" ) &&
( ( baseNameOf ( dirOf name ) ) == ".yarn" ) # `dirOf' is like `dirname'
);
# Remove `*/node_modules' directories
cleanNodeModulesFilter = name: type:
! ( ( type == "directory" ) && ( ( baseNameOf name ) == "node_modules" ) );
# Clean cached Node.js artifacts, and apply Nix's default clean routine.
cleanNodeSource = src:
cleanSourceWith {
filter = cleanYarnCacheFilter;
src = cleanSourceWith {
filter = cleanNodeModulesFilter;
src = cleanSource src;
};
};
in stdenv.mkDerivation {
pname = "my-project";
version = "0.0.1";
src = cleanNodeSource ./.;
}
```
## Writing filters
Both `builtins.filterSource` and `lib.sources.cleanSourceWith` accept functions with
the prototype `filter :: Name (Path/File Name) -> Type (Filetype) -> Boolean`
where `Type ::= "directory" | "regular" | "symlink" | "unknown"`, “unknown”
being Sockets and other oddball filesystem node types.
A filter function is expected to return `true` if a file should be preserved,
and `false` if a file should be removed.
An example filter which unconditionally preserves all files would be:
```
let myFilter = name: type: true;
in builtins.filterSource myFilter ./.
# Or equivalently ( since we never reference `name' or `type' ) :
builtins.filterSource ( _: _: true ) ./.
```
The filter above could also be used with Nixpkgs `lib.cleanSourceWith`:
```
lib.cleanSourceWith { filter = _: _: true; src = ./.; }
```
### A more useful filter: Optimized `node2nix` ignores.
A common Nix utility `node2nix` uses `nix-gitignore.gitignoreSourcePure`
to pluck just the `package.json` and `package-lock.json` files from a
source tree.
In this case, `nix-gitignore.gitignoreSourcePure` is somewhat unnecessary,
since the underlying predicate “only keep these two files” is dead simple.
Really, when we are building a single package, you should really use
`builtins.pathExists` with `srcs = [...]`; but because `node2nix` is
designed to possibly build groups of packages in a tree, it needs to
preserve `package[-lock].json` files recursively.
However, we can still improve the existing implementation by writing a
filter directly, which will save us from evaluating an unnecessarily
expensive routine.
In summary, our goal is simply: filter a source tree, preserving only
`package.json` and `package-lock.json` files.
1. Rough Draft
2. A Working Filter
3. Diving Off the Deep End
4. Recommended Solution
---
### Author:  [@Ten](https://discourse.nixos.org/u/Ten)
#### Post date: [May 1, 2024, 7:28pm UTC](https://discourse.nixos.org/t/filtering-source-trees-with-nix-and-nixpkgs/19148/2 "2024-05-01T19:28:54Z")
> [@Growpotkin](#):
>
> and would still get the hash we care about
> for our “real build” - which is what we actually care about
I’m struggling to understand how fullyClean will actually have a hash that does not depend on the empty directories of mostlyClean, since I don’t see anything that makes it not-source-addressed. (I’m trying to prevent recompilations that may be caused by adding and removing directories in an otherwise largely filtered tree)
---
### Author:  [@Infinisil](https://discourse.nixos.org/u/Infinisil)
#### Post date: [May 1, 2024, 7:46pm UTC](https://discourse.nixos.org/t/filtering-source-trees-with-nix-and-nixpkgs/19148/3 "2024-05-01T19:46:28Z")
Oh you might be interested in checking out [`lib.fileset`](https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset), which hasn’t existed when the original post was published 🙂
---
### Author:  [@Ten](https://discourse.nixos.org/u/Ten)
#### Post date: [May 1, 2024, 7:56pm UTC](https://discourse.nixos.org/t/filtering-source-trees-with-nix-and-nixpkgs/19148/4 "2024-05-01T19:56:14Z")
Thanks! I was just looking at it but I still couldn’t find a way in there to do the regular directory traversal that filterSource does, but only creating a directory once we’re about to put an actual (not filtered-out) file inside, in order to not update hashes and cause recompilations based on unrelated directories.
It does look like something that I’m very likely not the first to need and should consequently already exist in Nix…
Is there such a thing?
EDIT: Ah [this post](https://discourse.nixos.org/t/easy-source-filtering-with-file-sets/29117) seems to suggest that maybe lib.fileset by default does filter empty directories? Does this mean then that [#8820](https://github.com/NixOS/nix/issues/8820) (that you opened!) is out of date?
---
### Author:  [@Ten](https://discourse.nixos.org/u/Ten)
#### Post date: [May 1, 2024, 8:45pm UTC](https://discourse.nixos.org/t/filtering-source-trees-with-nix-and-nixpkgs/19148/5 "2024-05-01T20:45:21Z")
So I’m trying to use [`fileset.fileFilter`](https://nixos.org/manual/nixpkgs/stable/#function-library-lib.fileset.fileFilter) and the documentation specifies that `type` may not be `directory`, unlike with `cleanSourceWith`.
Does that mean that `fileset` will always traverse the entire file tree recursively?
(I would like to avoid e.g. `target` and `node_modules` because those tend to be huge so slow to traverse and I know nothing interesting is in there)
EDIT: looks like combing it with [`lib.fileset.fromSource`](https://nixos.org/manual/nixpkgs/stable/#function-library-lib.fileset.fromSource) may be a fix…
EDIT2: Yeah doing `fromSource` of a filtered source then `toSource` again works.
---
### Author:  [@Infinisil](https://discourse.nixos.org/u/Infinisil)
#### Post date: [May 1, 2024, 9:53pm UTC](https://discourse.nixos.org/t/filtering-source-trees-with-nix-and-nixpkgs/19148/6 "2024-05-01T21:53:56Z")
> [@Ten](#):
>
> EDIT: Ah [this post](https://discourse.nixos.org/t/easy-source-filtering-with-file-sets/29117) seems to suggest that maybe lib.fileset by default does filter empty directories? Does this mean then that [#8820](https://github.com/NixOS/nix/issues/8820) (that you opened!) is out of date?
Indeed, `lib.fileset` cannot even at all express the notion of “empty directories” because it (conceptually) has no notion of directories at all (it’s just a set of files)! ~~So that’s also why the issue there is still up-to-date, because there _might_ be use cases where you do need empty directories, so when you convert to a store path, you’d need some non-fileset argument to specify that.~~ The issue there however is just for _efficiency_ reasons. Having that feature would allow `lib.fileset` to be faster 🙂
> [@Ten](#):
>
> Does that mean that `fileset` will always traverse the entire file tree recursively?
`fileset.fileFilter` _does_ recurse all the way if you use it as-is, but because the combinators are lazy, you can prevent that from happening:
```nix
# All rust files in the current directory that are not in ./target
# This prevents recursing into ./target!
difference
(fileFilter (file: file.hasExt "rs") ./.)
./target
```
As you discovered, you can also convert from `lib.sources`-based values, though I’d avoid that since you lose some laziness properties that way.
Note that for the use-case of filtering out directories if you just know their name, but not their path, there’s a recent discussion in [`lib.fileset` should have a way to filter directories · Issue #271307 · NixOS/nixpkgs · GitHub](https://github.com/NixOS/nixpkgs/issues/271307) 🙂