Back in February I started work on a set of combinator-style functions to work with sources; not just filtering but combining and adding a concept of “focus”: which file inside the source to work on, start with, etc.
Could someone please review it? Or perhaps some of the comparatively simple parts first.
These additions are useful for specifying your sources in “self-packaging” projects like with flakes and for writing language integrations. @davidscherer wants to use it in his C/C++ build framework. haskell.nix uses a similar concept of focus (subdir) which served as an inspiration and works well with the new combinators.
New use cases:
- Construct sources additively. So far we didn’t have a function that takes a source and returns one with more files in it. Addition is more predictable than subtraction, relating to hermeticity.
- Add files to a source from any relative path. Perhaps you have a config file in the parent directory of the package you’re building. That’s just
src = sources.extend (cleanSource ./.) ../linter.yaml
so you don’t have to write a custom filter function and having to change working directory in the right build phase.
And some small improvements:
- provide a conceptually simpler
sources.filter and source.setName as alternative to cleanSourceWith { src, filter, name } to improve discoverability and align api style with the new functions
-
sources.trace for debugging
- some side-catch for path manipulation in
lib.filesystem
Not convinced? It also adds tests and… documentation!
3 Likes
I feel it might be better to provide such features as an external repository / flake. If it gets widely used, it could be added to nixpkgs in the future.
As an aside, a function name like lib.cutAt doesn’t really give me any idea what it does. If they were namespaced better (e.g. lib.sourceCombinators.cutAt), it would convey at least some idea of what it’s related to.
flake
It’s already a flake: source-combinators.url = "github:hercules-ci/nixpkgs/source-combinators";.
I think you’re right, that there’s no need to rush this. Though I do think we should merge it at some point, because some aspects don’t come into their own outside Nixpkgs, such as the stdenv cd change and the docs. There’s value in having a good “standard library”, which is, in a way, what Nixpkgs is all about.
namespaced
They are namespaced. For example, it’s lib.sources.cutAt with no intent to export it as lib.cutAt.
2 Likes
There’s value in having a good “standard library”, which is, in a way, what Nixpkgs is all about.
Right, but it’s very hard to change the standard library. So if we add anything we’d better make sure it’s perfect, since it’s impossible to improve afterwards. Hence it’s better to iterate in an external project (which has the added advantage that it can have its own release schedule).
Ah sorry, I misread the tests (didn’t see the with sources).
1 Like
Could you mention a couple of 10.000 feat level use cases? It sound’s quite interesting, but without that 10.000 feat compass and only from the tests, it’s a bit more difficult to mentally engage.
10.000 feet is a lot of body parts, but alright then. Keep in mind that Nixpkgs itself uses very little source filtering, but rather source fetching.
EDIT: I’ve added this to the top post.
New use cases:
- Construct sources additively. So far we didn’t have a function that takes a source and returns one with more files in it. Addition is more predictable than subtraction, relating to hermeticity.
- Add files to a source from any relative path. Perhaps you have a config file in the parent directory of the package you’re building. That’s just
src = sources.extend (cleanSource ./.) ../linter.yaml
so you don’t have to write a custom filter function and having to change working directory in the right build phase.
And some small improvements:
- provide a conceptually simpler
sources.filter and source.setName as alternative to cleanSourceWith { src, filter, name } to improve discoverability and align api style with the new functions
-
sources.trace for debugging
- some side-catch for path manipulation in
lib.filesystem
4 Likes
I think I like this. I conceptualized some filter that would crawl the Cargo.toml files of a Rust project to end up with only the source files that are needed to test and build, I haven’t actually looked into implementation details but “additive” in this case seems a lot better than “subtractive”.
Having extend you could expose this cargo scanner as a path -> Source rather than a path -> path -> string -> bool that users then have to wire into their cleanSourceWith somehow.
Other use cases I’m trying to read into this:
- It seems to me this also might facilitate (unify) a gitignore filtering.
- with the path prefix comparision code might be able to “detect” if they are contained eithin the same flake. That’s a cool feature to be able to detect for example which modules to reexport because they are actually defined within the same flake and don’t come from inputs. /cc @Pacman99
While the original source combinators haven’t been merged, I recently developed an easier-to-use alternative version. If you’re interested, please take a look at this post, try it out and give feedback about the interface in that thread!
1 Like