We added support for incremental builds in garnix! The approach (at a very high level) is to override a special input with whatever the latest successful garnix build was; any “intermediates” outputs in one build then become an input in the next build. This can dramatically build times, especially for large projects.
We wrote a blog with more detailed explanation, and a comparison with other approaches.
If you have questions, or feedback, let me know!
Any plans to open source it/contribute it upstream?
Do you have a non-toy example? E.g. for a meson or cmake project? For some reason the example in the blog post is not clicking for me and I have a hard time understanding how to use this
I haven’t figured out a good way to do that, since it relies on infrastructure such as a database with previous build results for the same repo. But I put up this gist with our code for creating the normalized flake (which will be used with --override-input) given the right list of builds to base off of.
Relatedly, it’d be nice to have a standardized protocol here, so using incrementalism across CIs doesn’t result in vendor lock in. I’m happy to organize with anyone who wants to make that happen.
Yeah in retrospect I should have filled that explanation out a bit more.
In general, there are two existing ways in which tooling supports incrementalism: via timestamps and hashes. If with the latter, the idea is to just copy the directory where intermediate files go from the cache and into the build before building, and then out of the build and into $intermediates after the build is done.
GHC > 9.4 is like this, and actually the Nix functions in nixpkgs even already take arguments for incrementalism. Unfortunately they are very brittle (changing the package version breaks the build!). Still, incrementaliziing a build is quite easy:
let haskellPkgs = pkgs.haskell.packages.ghc98;
myOldBuild = haskellPkgs.callCabal2nix ... # or whatever you already had;
in cache: (haskell.lib.overrideCabal myOldBuild (old: {
doInstallIntermediates = true;
enableSeparateIntermediatesOutput = true;
preBuild = ''
${old.preBuild or ""}
mkdir -p dist;
rm -r dist/build
cp -r ${cache}/share/haskell/${old.compiler.version}/${old.pname}-${old.version}/dist/build dist/build || echo "No cache found"
mkdir -p dist/build
find dist/build -exec chmod u+w {} +
find dist/build -exec touch -d '1970-01-01T00:00:00Z' {} +
'';
}));
That should work (it does for our project at least). And when I get around to upstreaming the preBuild changes it’ll be even simpler.
For systems that use a timestamp it’s a bit more work (since the timestamps will be clobbered by Nix and git), but not much. The basic idea is to again copy e.g. object files back and forth, but this time to set the modification time correctly enough. Let’s say you have a build that puts object files in ODIR and has source files in SRCDIR. The changes might look like this:
cache : pkgs.mkDerivation {
# Add "intermediates" to whatever you already had
outputs = [ "out" "intermediates" ];
preBuild = ''
cp -R ${cache}/* $ODIR
# Here we see if the file had the same hash as before. If not, it changed, so we
# update it's timestamp. Note that this might need to change if SRCDIR is not
# modifiable
if [ -f ${cache}/hashes]; then
for FILE in $(find $SRCDIR -type f); do
SHA=$(shasum $FILE)
if [ -z "$(grep $SHA ${cache}/hashes)" ]; then
touch $FILE
fi
fi
''
postInstall = ''
# Here we keep track of the hashes of the source files we saw. In preBuild we use
# this to set the timestamp: if we saw it before, it's old, if not, it's new
find $SRCDIR -not -path '*/\.*' -type f -exec shasum "{}" + > ${cache}/hashes
mv "$ODIR"/* "$intermediates"
'';
# the rest is as usual
}
I haven’t tried this one though, so there probably are bugs. Of course, it too can become simpler if some of the logic is upstreamed or provided as an external lib.
Does that help?
This sounds pretty much like pkgs.checkpointBuildTools!
Yes! I didn’t know about it, but the mkDerivation example may be able to just use checkpointBuildTools!