Friday hackday project: nix-filter

I know there are already a lot of nix source filters out there, but bear with me.

  • the various .gitignore source filters are not good enough! In monorepos, I only want to add what I need and not rebuild the world otherwise.
  • lib.cleanSource: the MVP of filters but clearly not powerful enough.
  • lib also has some source filters but I find them confusing to use.

So his is a different attempt. It’s a very small and self-contained project. If successful, I think it should be added to the nixpkgs/lib.

Please have a look, and give me feedback!

16 Likes

Looks nice! Does it rebuild when the root folder of the source is renamed?

Thanks!

It shouldn’t rebuild since the derivation name defaults to “source” in either case and the content would be the same.

2 Likes

This is fantastic!

I’m using it to reduce the size of the monorepo I’m working with from 7GB to 500M for each project.

However now I want to take the next step and use it not only when building from ./. but also when fetching the sources from git.

It appears to me that’s not trivially possible, right?

Git has a shallow clone feature to only fetch one commit in the history. This helps if the repo has a heavy history, but not if the tree itself is large, as you seem to suggest. I think Microsoft is researching that with GVFS but that’s out of scope for nix-filter I’m afraid.

Seems like this would benefit a lot from the changes in source combinators: extend, trace, cutAt, focusAt and more by roberth · Pull Request #112083 · NixOS/nixpkgs · GitHub. It would simplify your implementation and make the functionality composable and interoperable (among other improvements).

Personally I don’t like that include and exclude are parameters to the same function. This means that as a user, I can not decide which one is more important than the other, and perhaps worse is that I can’t see it at a glance when reading it. That’s why I’ve proposed a union-like function sources.extend in #112083. It allows you to express both choices through composition with sources.filter.

It’s interesting that my PR is four months older than your project. Goes to show that there’s a need for [RFC 0082] lib.experimental by fare · Pull Request #82 · NixOS/rfcs · GitHub. I’ve nominated myself as shepherd. Is anyone else interested?

1 Like

Yeah, it looks interesting. I honestly didn’t know about it until last week. Let’s try and merge both works. I think it can be useful outside of nixpkgs, and then get merged later in once the value is demonstrated.

1 Like

Yeah sounds good. Most of it can be external except the WIP commit that improves the interaction with stdenv.mkDerivation { src }.

Hosted by Flying Circus.