Friday hackday project: nix-filter

Development
1

I know there are already a lot of nix source filters out there, but bear with me.

  • the various .gitignore source filters are not good enough! In monorepos, I only want to add what I need and not rebuild the world otherwise.
  • lib.cleanSource: the MVP of filters but clearly not powerful enough.
  • lib also has some source filters but I find them confusing to use.

So his is a different attempt. It’s a very small and self-contained project. If successful, I think it should be added to the nixpkgs/lib.

Please have a look, and give me feedback!

16 Likes
2

Looks nice! Does it rebuild when the root folder of the source is renamed?

3

Thanks!

It shouldn’t rebuild since the derivation name defaults to “source” in either case and the content would be the same.

2 Likes
4

This is fantastic!

I’m using it to reduce the size of the monorepo I’m working with from 7GB to 500M for each project.

However now I want to take the next step and use it not only when building from ./. but also when fetching the sources from git.

It appears to me that’s not trivially possible, right?

5

Git has a shallow clone feature to only fetch one commit in the history. This helps if the repo has a heavy history, but not if the tree itself is large, as you seem to suggest. I think Microsoft is researching that with GVFS but that’s out of scope for nix-filter I’m afraid.

6

Seems like this would benefit a lot from the changes in source combinators: extend, trace, cutAt, focusAt and more by roberth · Pull Request #112083 · NixOS/nixpkgs · GitHub. It would simplify your implementation and make the functionality composable and interoperable (among other improvements).

Personally I don’t like that include and exclude are parameters to the same function. This means that as a user, I can not decide which one is more important than the other, and perhaps worse is that I can’t see it at a glance when reading it. That’s why I’ve proposed a union-like function sources.extend in #112083. It allows you to express both choices through composition with sources.filter.

It’s interesting that my PR is four months older than your project. Goes to show that there’s a need for [RFC 0082] lib.experimental by fare · Pull Request #82 · NixOS/rfcs · GitHub. I’ve nominated myself as shepherd. Is anyone else interested?

1 Like
7

Yeah, it looks interesting. I honestly didn’t know about it until last week. Let’s try and merge both works. I think it can be useful outside of nixpkgs, and then get merged later in once the value is demonstrated.

1 Like
8

Yeah sounds good. Most of it can be external except the WIP commit that improves the interaction with stdenv.mkDerivation { src }.