Hi all. This is my first post here, and it is a long one (so I have sectioned it out).
It will likely only appeal to those who deal with build systems - and have faced similar issues to me when it comes to the fragile interaction between Nix, a system designed to be hermetic, and build systems that try to solve caching etc in their own way.
tl;dr
My hope is to write a build system in Nix, without passing the build steps to other build systems such as make, cmake, bazel, etc. So in this vision, Nix is given all the information about how targets relate to source files, how theyāre built, and can use the store to prevent rebuilds of unchanged targets.
Background
I am a developer working on a large polyrepo, with components written mainly in C++, but with some python and NodeJS components too. All in all, a single configuration builds ~20 tools, and I am usually building ~20 configurations, so a full build will involve ~400 compilations, and this takes about an hour if done from scratch.
Many of my dependencies, external and within the polyrepo, are header-only - which means they do not get ābuiltā, just available as includes to the final binaries. This is by design: much my work involves a lot of C++ template metaprogramming, which inhibits the use of intermediate build steps. So even if I fetch and cache dependencies with Nix, that will save me the (negligible) download time rather than any build time.
My approach, since starting with nix a few months ago (to gain hermeticity of compiler, etc), has been to fetch the top-level project repository, configure it, and invoke another build system (Bazel) within a nix environment to build everything I need. Twaegās approach of using Nix within Bazel (rather than the other way around) doesnāt appeal to me - I want Nix to be my entrypoint, as the build process is only the first step of a larger deployment pipeline written in Nix (outputs get dockerised, k8s files get generated, etc).
Problem
Say I add a single character to a README file and commit. A .tar.gz download of the archive at this new commit thus has a new hash. If I want to perform a rebuild (say, in CI), Nix notices that the new repository hash has not been seen before. Because it isnāt playing the role of build system, it doesnāt understand how files in the repository relate to one another - it is unable to use a previous build cache, at least not without potentially-hermeticity-breaking hacks. It has to start from scratch, rebuilding everything. What could have been an instant āoh, no actual builds have changed, nothing to do hereā becomes a 30 minute slog.
I have trialled other build systems - including cmake - and the fundamental problem remains the same. A benefit of cmake is that I can pull dependencies with Nix and pass them in - but this still doesnāt go far enough. If the project changes in any way, components are still rebuilt from scratch, even if the libraries I depend on arenāt. CMake also requires quite a lot of bootstrap in order for Nix to be able to allow other CMake projects to find eachother, and this seems redundant.
I have now given up trying to use other build systems. Iāve seen a thousand solutions that each generate a thousand new problems, and Iāve gone down too many rabbit holes.
Proposal
What I want to do is write an open-source build system in pure nix, with some inspiration from Bazel.
As for a name: Nix/Bazel = ā¦ Nasal? No. That brings some horrible visions involving RMS.
Nozzle? That might do it. Iāll go with nozzle for now, at least for purposes of demonstration.
Possible Example
In order to provide an example which is simple enough to explain, but not trivial enough to say ājust invoke gccā, hereās an outline of a project which has a multi-step build procedure. It involves running python to generate C++ code, building that C++ code as a library, linking it to another library to produce an executable, and then allowing that executable to be invoked by python.
project/
flake.nix
default.nix
build.nix
project/
build.nix
library.h
library.cpp
codegen-additional-library.py
header-only-library.h
main.cpp
run.py
The flake.nix sets up the external dependencies (which are overwritten in local development if I have local versions undergoing changes), the build.nix files provide detail to the build system, and the default.nix sets up the workspace (e.g. invokes the build).
In this project, the build steps should be:
- Run
codegen-additional-library.pyand write stdout toadditional-library.cpp - Compile
library.cppandadditional-library.cpp, usinglibrary.has the header, to output a compiled static librarylibrary.a. -
header-only-library.hhas no dependencies and no build steps - it just needs to be copied to the nix store. - main.cpp includes on library.h, header-only-library.h, and builds to produce
./main.a. -
main.ais linked with./library.ato produce an executable binary./main. - run.py is then able to invoke the built
./mainthrough thesubprocessmodule. To add complexity, letās say it also depends on thenumpyandpyyamlpackages.
One might then envision project/project/build.nix as being along the lines of:
{pkgs, nozzle}:
self: super: {
additional-library: nozzle.codegen {
invoke = nozzle.python-runnable {
source = [./codegen-addtional-library.py];
python = pkgs.python39;
packages = p: [p.numpy];
};
output = "additional-library.cpp";
};
library: nozzle.cpp-library {
headers = [./library.h];
sources = [./library.cpp self.additional-library];
};
header-only-library: nozzle.cpp-library {
headers = [./header-only-library.h];
};
main: nozzle.cpp-binary {
sources = [./main.cpp];
dependencies = [self.library self.header-only-library];
};
run = nozzle.python-runnable {
source = [./run.py];
python = pkgs.python39;
packages = p: [p.numpy p.pyyaml];
runtime-dependencies = [self.main];
};
}
Explanation
Each build.nix adds an overlay to a general project structure. Iām still a newbie, and overlays still confuse me somewhat, but I think this makes sense. The derivation corresponding run.py can be referred to from another build file as super.project.run, so Nix would āseeā the project as
{
project = {
additional-library = ...;
library = ...;
header-only-library = ...;
main = ...;
run = ...;
};
}
Invoking nix build would then build everything in the project, and perhaps nix build --argstr target "project/run" or something similar could be used to build a single target.
Most importantly, Nix now knows the project. It understands it. Each entity is a different derivation in the nix store. The hashes are based on file contents, not on the hash of the overall project. So:
- If I change
codegen-additional-library.py, Nix knows thatlibrary.ahas to be rebuilt andmain.ahas to be re-linked with it to formmain. - If I change
header-only-library.h, Nix knowsmain.ahas to be rebuilt and linked to the unchangedlibrary.ato producemain. - If I change
run.py, Nix knows that none of the C++ files need to be rebuilt at all -mainremains unchanged. It becomes a simple case of copying the newrun.pyto the nix store, with the existingmainderivation as a dependency.
Even if the source of this project is fetched from github, with a different hash (e.g. because of an additional character in a readme file), Nix is still able to realise that the files it cares about havenāt changed. If some have changed, it still knows what hasnāt changed, and it doesnāt need to rebuild the universe.
Finally, this would be designed to be composable. If another project uses Nozzle (or whatever it ends up being called), it can be brought in as a dependency to another Nozzle project natively. Integration with other build systems might be doable, too - a cmake project might be depended on, for example, if I can figure out how to do that.
Questions
Before I throw too much weight behind this, Iād like to gather some thoughts from the community. Itād save me from wasting time.
- Has this been done already? I havenāt been able to find anything like it yet, but it seems like a fairly nix-like thing to want to do, and maybe Iām just searching for the wrong things.
- Are there any glaring downsides that Iām missing?
- Any changes I should make to this plan?
- Any advice in general?
- Would you use it?
- If this is an open source project, would you assist?
