Advice wanted: writing a build system in Nix

Hi all. This is my first post here, and it is a long one (so I have sectioned it out).

It will likely only appeal to those who deal with build systems - and have faced similar issues to me when it comes to the fragile interaction between Nix, a system designed to be hermetic, and build systems that try to solve caching etc in their own way.

tl;dr

My hope is to write a build system in Nix, without passing the build steps to other build systems such as make, cmake, bazel, etc. So in this vision, Nix is given all the information about how targets relate to source files, how they’re built, and can use the store to prevent rebuilds of unchanged targets.

Background

I am a developer working on a large polyrepo, with components written mainly in C++, but with some python and NodeJS components too. All in all, a single configuration builds ~20 tools, and I am usually building ~20 configurations, so a full build will involve ~400 compilations, and this takes about an hour if done from scratch.

Many of my dependencies, external and within the polyrepo, are header-only - which means they do not get ‘built’, just available as includes to the final binaries. This is by design: much my work involves a lot of C++ template metaprogramming, which inhibits the use of intermediate build steps. So even if I fetch and cache dependencies with Nix, that will save me the (negligible) download time rather than any build time.

My approach, since starting with nix a few months ago (to gain hermeticity of compiler, etc), has been to fetch the top-level project repository, configure it, and invoke another build system (Bazel) within a nix environment to build everything I need. Twaeg’s approach of using Nix within Bazel (rather than the other way around) doesn’t appeal to me - I want Nix to be my entrypoint, as the build process is only the first step of a larger deployment pipeline written in Nix (outputs get dockerised, k8s files get generated, etc).

Problem

Say I add a single character to a README file and commit. A .tar.gz download of the archive at this new commit thus has a new hash. If I want to perform a rebuild (say, in CI), Nix notices that the new repository hash has not been seen before. Because it isn’t playing the role of build system, it doesn’t understand how files in the repository relate to one another - it is unable to use a previous build cache, at least not without potentially-hermeticity-breaking hacks. It has to start from scratch, rebuilding everything. What could have been an instant “oh, no actual builds have changed, nothing to do here” becomes a 30 minute slog.

I have trialled other build systems - including cmake - and the fundamental problem remains the same. A benefit of cmake is that I can pull dependencies with Nix and pass them in - but this still doesn’t go far enough. If the project changes in any way, components are still rebuilt from scratch, even if the libraries I depend on aren’t. CMake also requires quite a lot of bootstrap in order for Nix to be able to allow other CMake projects to find eachother, and this seems redundant.

I have now given up trying to use other build systems. I’ve seen a thousand solutions that each generate a thousand new problems, and I’ve gone down too many rabbit holes.

Proposal

What I want to do is write an open-source build system in pure nix, with some inspiration from Bazel.

As for a name: Nix/Bazel = … Nasal? No. That brings some horrible visions involving RMS.

Nozzle? That might do it. I’ll go with nozzle for now, at least for purposes of demonstration.

Possible Example
In order to provide an example which is simple enough to explain, but not trivial enough to say “just invoke gcc”, here’s an outline of a project which has a multi-step build procedure. It involves running python to generate C++ code, building that C++ code as a library, linking it to another library to produce an executable, and then allowing that executable to be invoked by python.

project/
    flake.nix
    default.nix
    build.nix
    project/
        build.nix
        library.h
        library.cpp
        codegen-additional-library.py
        header-only-library.h
        main.cpp
        run.py

The flake.nix sets up the external dependencies (which are overwritten in local development if I have local versions undergoing changes), the build.nix files provide detail to the build system, and the default.nix sets up the workspace (e.g. invokes the build).

In this project, the build steps should be:

  • Run codegen-additional-library.py and write stdout to additional-library.cpp
  • Compile library.cpp and additional-library.cpp, using library.h as the header, to output a compiled static library library.a.
  • header-only-library.h has no dependencies and no build steps - it just needs to be copied to the nix store.
  • main.cpp includes on library.h, header-only-library.h, and builds to produce ./main.a.
  • main.a is linked with ./library.a to produce an executable binary ./main.
  • run.py is then able to invoke the built ./main through the subprocess module. To add complexity, let’s say it also depends on the numpy and pyyaml packages.

One might then envision project/project/build.nix as being along the lines of:

{pkgs, nozzle}:

self: super: {
    additional-library: nozzle.codegen {
        invoke = nozzle.python-runnable {
            source = [./codegen-addtional-library.py];
            python = pkgs.python39;
            packages = p: [p.numpy];
        };
        output = "additional-library.cpp";
    };
    library: nozzle.cpp-library {
        headers = [./library.h];
        sources = [./library.cpp self.additional-library];
    };
    header-only-library: nozzle.cpp-library {
        headers = [./header-only-library.h];
    };
    main: nozzle.cpp-binary {
        sources = [./main.cpp];
        dependencies = [self.library self.header-only-library];
    };
    run = nozzle.python-runnable {
        source = [./run.py];
        python = pkgs.python39;
        packages = p: [p.numpy p.pyyaml];
        runtime-dependencies = [self.main];
    };
}

Explanation

Each build.nix adds an overlay to a general project structure. I’m still a newbie, and overlays still confuse me somewhat, but I think this makes sense. The derivation corresponding run.py can be referred to from another build file as super.project.run, so Nix would “see” the project as

{
   project = {
       additional-library = ...;
       library = ...;
       header-only-library = ...;
       main = ...;
       run = ...;
   };
}

Invoking nix build would then build everything in the project, and perhaps nix build --argstr target "project/run" or something similar could be used to build a single target.

Most importantly, Nix now knows the project. It understands it. Each entity is a different derivation in the nix store. The hashes are based on file contents, not on the hash of the overall project. So:

  • If I change codegen-additional-library.py, Nix knows that library.a has to be rebuilt and main.a has to be re-linked with it to form main.
  • If I change header-only-library.h, Nix knows main.a has to be rebuilt and linked to the unchanged library.a to produce main.
  • If I change run.py, Nix knows that none of the C++ files need to be rebuilt at all - main remains unchanged. It becomes a simple case of copying the new run.py to the nix store, with the existing main derivation as a dependency.

Even if the source of this project is fetched from github, with a different hash (e.g. because of an additional character in a readme file), Nix is still able to realise that the files it cares about haven’t changed. If some have changed, it still knows what hasn’t changed, and it doesn’t need to rebuild the universe.

Finally, this would be designed to be composable. If another project uses Nozzle (or whatever it ends up being called), it can be brought in as a dependency to another Nozzle project natively. Integration with other build systems might be doable, too - a cmake project might be depended on, for example, if I can figure out how to do that.

Questions

Before I throw too much weight behind this, I’d like to gather some thoughts from the community. It’d save me from wasting time.

  • Has this been done already? I haven’t been able to find anything like it yet, but it seems like a fairly nix-like thing to want to do, and maybe I’m just searching for the wrong things.
  • Are there any glaring downsides that I’m missing?
  • Any changes I should make to this plan?
  • Any advice in general?
  • Would you use it?
  • If this is an open source project, would you assist?
7 Likes

Hey! I’m from Tweag and am currently contracting for a client who created a C build system within Nix, pretty much exactly what you’re looking for, and they are interested in upstreaming it into nixpkgs. The work is in a short pause for now as we’re focusing on some other bits, but I can say that the prototype is working well, isn’t hugely complicated and I expect to be able to PR it to nixpkgs within a couple months!

The main downside is that evaluation is slow, since each C file needs to be its own derivation. This makes it infeasible to be used for packages in nixpkgs itself, but for just a couple packages in development outside of nixpkgs it’s gonna be fine.

Thanks a lot for reaching out on Discourse with this idea, it’s really good to know that other people are interested in this! For now my advice is to wait with spending time on this and stay tuned for the PR we expect to create soonish. I’ll definitely ping you for that, and we might also be able to publish a sneak peak beforehand :slight_smile:

4 Likes

I have an interest in this and making it somewhat automatic. Ask a build system about these fine-grained dependencies and make it automatic to break up such a project into smaller and more specific derivations. This is a driver behind rfcs/0092-plan-dynamism.md at 553b132ca05e0ad19b563b80b08d17330df205cf · NixOS/rfcs · GitHub which is still being implemented.

@tomberek Yes, in the approach we’re working on upstreaming with @infinisil the breakdown is automatic. @YorikSar proposed adding break points to control the granularity, and that would be a possible evolution step following the file-granular mode.

We’re currently picking apart how to sanely deal with file system paths and source directories, and discussing possible venues with @roberth who proposes a source combinators library.

1 Like

Maybe interesting to you: I have a prototype for a pure Nix C build system lying around where every object file is its own derivation. It can also support codegen easily as every source file may be replaced by a derivation. I guess the code (which isn’t too much) can be of interest to you.

Some words of warning:

  • This was a bit of a project just for me, so the code is quite idiosyncratic: It uses a Nix DSL to generate execline scripts which are used as a scripting language for in derivation glue code.
  • It uses unwrapped C compilers and ld which is a bit problematic, because I had to reimplement a lot of logic. Not using nixpkgs’ cc-wrapper may be fine, but using plain ld instead of the C compiler for linking was a mistake, as it requires replicating annoying logic. In any case when that job was the last remaining piece, I lost motivation for doing C stuff and the usecase disappeared.
3 Likes

If it is unfeasable for nixpkgs packages why not release it as a flake in a separate repository? It would also help release it more quickly

1 Like

Thanks everyone for the replies - it’s great to see that I’m not alone in this.

I have had my head down for a couple of days trying to implement something. I’m still fairly new to Nix so there was a lot of tripping up along the way - especially with flakes - but I’ve managed to put together a proof of concept at GitHub - jake-arkinstall/nozzle: A build system for Nix, with example usage at GitHub - jake-arkinstall/nozzle-usage: Example usage for the Nozzle build system.

At the moment, it can handle a header-only library, a static library, and a binary that uses both. It’s simple, but it coming together in one binary is fantastic to see.

I’ll start looking through some of the links sent here and see if I can draw some inspiration from it, particularly @sternenseemann’s prototype. It may be for nothing if the build system being worked on by @Infinisil fits the bill, but it’s a fun project anyway.

I’m looking for some reviews on the nix code (on here, as issues, or even PRs). As I’m a beginner, I don’t know what Nix sins I’m committing in there, and what design flaws I’m introducing. In particular, there’s complete rigidity in the compiler choice and the flags (e.g. --std=c++20 and -O3), which I could do with passing as a configuration option to the user.

I’m also looking for ideas on incorporating existing packages within nixpkgs, such as fmt, spdlog etc. These are CMake projects, but (having just looked) they do update the NIX_CFLAGS_COMPILE and NIX_LDFLAGS environment variables - so it might be easier than I thought to do that. I don’t know whether I should pass them in directly in the dependencies attribute, whether they should first be wrapped so that I can treat them as I treat the cpp-library derivations, or whether I should modify the cpp-library derivations to expose the above environment variables.

Hosted by Flying Circus.