# Incremental builds in garnix

**URL:** https://discourse.nixos.org/t/incremental-builds-in-garnix/56100
**Category:** Announcements
**Created:** [November 16, 2024, 12:54am UTC](https://discourse.nixos.org/t/incremental-builds-in-garnix/56100 "2024-11-16T00:54:41Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![jkarni](https://discourse.nixos.org/user_avatar/discourse.nixos.org/jkarni/32/6049_2.png) [@jkarni](https://discourse.nixos.org/u/jkarni)
#### Post date: [November 16, 2024, 12:54am UTC](https://discourse.nixos.org/t/incremental-builds-in-garnix/56100/1 "2024-11-16T00:54:41Z")

</div>

We added support for incremental builds in garnix! The approach (at a very high level) is to override a special input with whatever the latest successful garnix build was; any “intermediates” _outputs_ in one build then become an _input_ in the next build. This can dramatically build times, especially for large projects.

We [wrote a blog](https://garnix.io/blog/incremental-builds) with more detailed explanation, and a comparison with other approaches.

If you have questions, or feedback, let me know!

---

<div class="post-metadata">

### Author: ![bbigras](https://discourse.nixos.org/user_avatar/discourse.nixos.org/bbigras/32/56_2.png) [@bbigras](https://discourse.nixos.org/u/bbigras)
#### Post date: [November 16, 2024, 6:39am UTC](https://discourse.nixos.org/t/incremental-builds-in-garnix/56100/2 "2024-11-16T06:39:49Z")

</div>

Any plans to open source it/contribute it upstream?

---

<div class="post-metadata">

### Author: ![arianvp](https://discourse.nixos.org/user_avatar/discourse.nixos.org/arianvp/32/602_2.png) [@arianvp](https://discourse.nixos.org/u/arianvp)
#### Post date: [November 16, 2024, 9:16am UTC](https://discourse.nixos.org/t/incremental-builds-in-garnix/56100/3 "2024-11-16T09:16:00Z")

</div>

Do you have a non-toy example? E.g. for a meson or cmake project? For some reason the example in the blog post is not clicking for me and I have a hard time understanding how to use this

---

<div class="post-metadata">

### Author: ![jkarni](https://discourse.nixos.org/user_avatar/discourse.nixos.org/jkarni/32/6049_2.png) [@jkarni](https://discourse.nixos.org/u/jkarni)
#### Post date: [November 16, 2024, 12:38pm UTC](https://discourse.nixos.org/t/incremental-builds-in-garnix/56100/4 "2024-11-16T12:38:29Z")

</div>

I haven’t figured out a good way to do that, since it relies on infrastructure such as a database with previous build results for the same repo. But I put up [this gist](https://gist.github.com/jkarni/e0a6b6c4c574db25b21e2899508a71a3) with our code for creating the normalized flake (which will be used with `--override-input`) given the right list of builds to base off of.

Relatedly, it’d be nice to have a standardized protocol here, so using incrementalism across CIs doesn’t result in vendor lock in. I’m happy to organize with anyone who wants to make that happen.

---

<div class="post-metadata">

### Author: ![jkarni](https://discourse.nixos.org/user_avatar/discourse.nixos.org/jkarni/32/6049_2.png) [@jkarni](https://discourse.nixos.org/u/jkarni)
#### Post date: [November 16, 2024, 1:18pm UTC](https://discourse.nixos.org/t/incremental-builds-in-garnix/56100/5 "2024-11-16T13:18:58Z")

</div>

Yeah in retrospect I should have filled that explanation out a bit more.

In general, there are two existing ways in which tooling supports incrementalism: via timestamps and hashes. If with the latter, the idea is to just copy the directory where intermediate files go from the cache and into the build before building, and then out of the build and into `$intermediates` after the build is done.

GHC \> 9.4 is like this, and actually the Nix functions in nixpkgs even already take arguments for incrementalism. Unfortunately they are very brittle (changing the package version breaks the build!). Still, incrementaliziing a build is quite easy:

```auto
   let haskellPkgs = pkgs.haskell.packages.ghc98;
        myOldBuild = haskellPkgs.callCabal2nix ... # or whatever you already had;
    in cache: (haskell.lib.overrideCabal myOldBuild (old: {
          doInstallIntermediates = true;
          enableSeparateIntermediatesOutput = true;
          preBuild = ''
            ${old.preBuild or ""}
            mkdir -p dist;
            rm -r dist/build
            cp -r ${cache}/share/haskell/${old.compiler.version}/${old.pname}-${old.version}/dist/build dist/build || echo "No cache found"
            mkdir -p dist/build
            find dist/build -exec chmod u+w {} +
            find dist/build -exec touch -d '1970-01-01T00:00:00Z' {} +
        '';
    }));

```

That should work (it does for our project at least). And when I get around to upstreaming the `preBuild` changes it’ll be even simpler.

For systems that use a timestamp it’s a bit more work (since the timestamps will be clobbered by Nix and git), but not much. The basic idea is to again copy e.g. object files back and forth, but this time to set the modification time correctly enough. Let’s say you have a build that puts object files in ODIR and has source files in SRCDIR. The changes might look like this:

```auto
cache : pkgs.mkDerivation {
   # Add "intermediates" to whatever you already had
   outputs = ["out" "intermediates"];
   preBuild = ''
      cp -R ${cache}/* $ODIR
      # Here we see if the file had the same hash as before. If not, it changed, so we 
      # update it's timestamp. Note that this might need to change if SRCDIR is not
      # modifiable
      if [-f ${cache}/hashes]; then
        for FILE in $(find $SRCDIR -type f); do
          SHA=$(shasum $FILE)
          if [-z "$(grep $SHA ${cache}/hashes)"]; then
              touch $FILE
          fi
      fi
   ''
   postInstall = ''
       # Here we keep track of the hashes of the source files we saw. In preBuild we use 
       # this to set the timestamp: if we saw it before, it's old, if not, it's new 
       find $SRCDIR -not -path '*/\.*' -type f -exec shasum "{}" + > ${cache}/hashes
       mv "$ODIR"/* "$intermediates"
   '';
   # the rest is as usual
}

```

I haven’t tried this one though, so there probably are bugs. Of course, it too can become simpler if some of the logic is upstreamed or provided as an external lib.

Does that help?

---

<div class="post-metadata">

### Author: ![Infinisil](https://discourse.nixos.org/user_avatar/discourse.nixos.org/infinisil/32/26523_2.png) [@Infinisil](https://discourse.nixos.org/u/Infinisil)
#### Post date: [November 16, 2024, 1:32pm UTC](https://discourse.nixos.org/t/incremental-builds-in-garnix/56100/6 "2024-11-16T13:32:35Z")

</div>

This sounds pretty much like [`pkgs.checkpointBuildTools`](https://nixos.org/manual/nixpkgs/unstable/#sec-checkpoint-build)!

---

<div class="post-metadata">

### Author: ![jkarni](https://discourse.nixos.org/user_avatar/discourse.nixos.org/jkarni/32/6049_2.png) [@jkarni](https://discourse.nixos.org/u/jkarni)
#### Post date: [November 16, 2024, 1:42pm UTC](https://discourse.nixos.org/t/incremental-builds-in-garnix/56100/7 "2024-11-16T13:42:18Z")

</div>

Yes! I didn’t know about it, but the `mkDerivation` example may be able to just use `checkpointBuildTools`!

---

<div class="post-metadata">

### Author: ![shivaraj-bh](https://discourse.nixos.org/user_avatar/discourse.nixos.org/shivaraj-bh/32/9607_2.png) [@shivaraj-bh](https://discourse.nixos.org/u/shivaraj-bh)
#### Post date: [December 22, 2024, 3:52pm UTC](https://discourse.nixos.org/t/incremental-builds-in-garnix/56100/8 "2024-12-22T15:52:33Z")

</div>

I have a working example for systems that use timestamps (inspired by the example given by @jkarni) at [test-garnix-incremental-build/nix/modules/flake-parts/haskell.nix at 5a108120c042fbbb5fad16ba4a39e36e0082e9df · shivaraj-bh/test-garnix-incremental-build · GitHub](https://github.com/shivaraj-bh/test-garnix-incremental-build/blob/5a108120c042fbbb5fad16ba4a39e36e0082e9df/nix/modules/flake-parts/haskell.nix#L61-L102) on a Haskell package.
