This blog post explains my recent work on building Haskell packages incrementally using Nix/Nixpkgs, which we’re already using at work for significant build time savings.
The work is already open source in the form of two pull requests and mainly awaiting review and the post shows you how you can use that work if you don’t mind living on the bleeding edge
For those who want to jump into the code: Add `haskell.lib.incremental` utility by Gabriella439 · Pull Request #204020 · NixOS/nixpkgs · GitHub.
I hope we can generalize it, because nixpkgs#167670 adds similar functionality, but perhaps less adapted to haskell?
It should be pointed out that the most relevant part of the implementation is in a change to Nix: Add optional `date` argument to `builtins.fetchGit` by Gabriella439 · Pull Request #7362 · NixOS/nix · GitHub. Basically, for this feature to be useful we would need to allow a new kind of impurity in builtins.fetchGit. I personally think it may be problematic adding a date argument to fetchGit as it may very well be too enticing.
I believe the date argument does not affect the purity of builtins.fetchGit. In other words, I believe if builtins.fetchGit args were pure then builtins.fetchGit (args // { date = …; }) would also be pure.
Sure because it would presumably ignore date in the latter case. However, I think it has a false air of purity which is a bit concerning. Where plain fetchGit { url = "…"; } would give you clearly different results at different times, with date specified it would be impure only seldomly (i.e. force pushes or other weird situations), so it’s status would not be very intuitive.
Furthermore I’m not sure where it “fits” in Nix so to speak. We should generally encourage using rev as it is actually content-addressed. The date features seems oddly specific to me, i.e. I think it doesn’t have many legitimate uses beyond the one you are proposing which I’m not sure is such a killer feature. Also the same effect could be achieved by a mere shell script wrapper around nix-build.
If it doesn’t “fit”, it should probably just be a Nix plugin (a feature we seem to have after all…).
fetchGit never ignores the date field (for either pure or impure invocations of fetchGit) so I’m not sure what you are referring to.
As a concrete example, if you specify:
fetchGit { url = …; rev = …; date = "1 day before"; }
… then it will be return the last commit at least 1 day before the specified revision. Moreover, that fetchGit invocation would be pure both with or without the date field. Force pushing and/or rebasing wouldn’t change that purity in any way.
I want to encourage formulation of more concrete plans to fold incremental builds: add derivation override functions by messemar · Pull Request #167670 · NixOS/nixpkgs · GitHub and Add `haskell.lib.incremental` utility by Gabriella439 · Pull Request #204020 · NixOS/nixpkgs · GitHub into a joint effort.
Is that likely? Or even already underway?
“snapping” the date
The cool thing about this change is that it doesn’t compromise the purity of builtins.fetchGit. If a given fetchGit specification was pure then adding a date specification preserves that purity.
I think readers who indulge in this motion might benefit from conceptual argumentative proof short of auditing the code (which I still.will do now 
).
We can make it happen, but these days, i have not that much time to work on it further.
Regarding the fetchGit change.
This will work only for a fetchGit use case.
If you wish to override it with local sources or something else, the approach will not work any longer.
Also, my use case for incremental builds: add derivation override functions by messemar · Pull Request #167670 · NixOS/nixpkgs · GitHub are local source overrides, thus it will not beneficial for me, at least.
Just to clarify, the fetchGit patch and the corresponding change to Nixpkgs both work with local source packages (and the demo example from the blog post illustrates this)
Quick question. Are yall trying to write an abstraction to handle Nixpkgs and builtin fetchers “interchangeably”?
If you are let me know. I spent a few weeks making a lib for this and could potentially save you some effort.
As useful this stepped rolling approach is from a dx perspective, I’m afraid that I can’t copy this argument.
An input "one week ago" impurely consumes the current date to produce the output, even if that happens inside git and outside the nix code.
It’s a dilemma.
We may need to create CI actions that can gitops these stepped base build references back into the source repo to achieve this.
Since this is overwhelmingly a CI domain optimization, this seems pretty fair and balanced.
"one week ago" is relative to the specified rev. That hash also “fixes” the commit date, so the relative specification is as good as an absolute time.date + rev overall: Every commit hash also hashes its ancestors, so there is no way to get a different commit even if the upstream branch gets force pushed.Still it doesn’t sit quite right with me that this feature is so specific, but as long as we are not going to add it to pkgs.fetchgit…
I would like to know still if a Nix plugin was considered / would be possible. Maybe the answer would turn out to be the “Nix plugins feature is largely useless”?
No. The “corresponding code in nixpkgs” is an utility that depends on this Nix change.
Thank you for the clarification. I had built up the wrong context here while chewing through the proposal.
In my opinion, nix plugin features are largely underappreciated. But I see their strength in language lockfile support and the avoidance of IFD.
Since it turned out to be pure, I think it’s a good idea in the way it is proposed. The problem (which is a more generic one) with flake support is still outstanding and I wondered if Elco’s already famous branch about Source Accessors (nix#6530) would actually improve this or at least make it easier to fix.