Use derivation as builder

Help
1

Hi!

I am experimenting with Nix as a build tool for C programs. Simple flow (compile object files, then link) works fine. However, I got a problem with code generation.

When I define generator binary as a derivation and then use this derivation as a builder in another derivation, i get an error:

I uploaded MRE to GitHub: GitHub - maxxk/nix-build-with-generator

An here is a complete default.nix:

error: the string 'q2hpfyc1pn7v8c19g7hzzznmqqc2f6yw-generator-source.txt.c.o'
is not allowed to refer to a store path
(such as '!out!/nix/store/10y7a1vv8rxw2z0xb4x29aq1ghfrg9b9-generator-source.txt.c.drv')

{ pkgs ? import <nixpkgs> {}}:
let
    link = { name, inputs, args ? []}: derivation {
        inherit (pkgs) system;
        inherit name;
        builder = "${pkgs.stdenv.cc}/bin/cc";
        args = [
            "-o"
            (placeholder "out")
        ]  ++ inputs ++ args;
    };

    cc = { input, args ? [] }: link {
        name = "${baseNameOf input}.o";
        inputs = [ input ];
        args = [ "-c" ] ++ args;
    };
    
    generator-bin = link ({ name = "generator"; inputs = [ ./generator.c ]; });

    generate = { input }: derivation {
        inherit (pkgs) system;
        name = "${baseNameOf input}.c";
        builder = generator-bin;
        args = [
            input
            (placeholder "out")
        ];
    };

    hello-c = generate { input = ./generator-source.txt; };

    sources = [ hello-c ./program.c ];
    objs = map (source: cc { input = source; }) sources;
    bin = link { name = "program"; inputs = objs; };
    bin2 = link { name = "program2"; inputs = [ (cc { input = ./program.c; }) ]; };
in 
    bin
    # bin2
2

Can you teach your generator to look for $out (the environment variable)? Or maybe use runCommand and pass arguments there (there in a string the placeholder should work, as far as I remember)

1 Like
3

Yeah, it is an option, thank you. It will require a wrapper (or runCommand) in general case to keep generator decoupled from Nix. On the bright side, if the wrapper is required then the generator can be simplified to process stdin to stdout, not the command-line arguments.

However, it’d still be interesting to understand, why
builder = "${pkgs.stdenv.cc}/bin/cc" works fine, while
builder = generator-bin doesn’t.

In REPL both pkgs.stdenv.cc and generator-bin are derivations.

4

Oh right, sorry. I bought your description and I should not have. Indeed, that probably works.

What does not work is your generate passing a string obtained by processing a derivation output path as a name of derivation. Nix is being unhappy with that…

Basically, strings dependent on mangling outpaths are supposed to change when the inputs change; it’s fine for the build process, but derivation names are treated specially… In principle you should be able to do something like builtins.unsafeDiscardStringContext to work around this, or use a different naming scheme

1 Like
5

Thank you, I integrated this solution in the following preprocessing function (not that I’m very proud of the code :slight_smile: )

let
    HASH_LENGTH = 32;
    cleanName = name: 
        let 
            base = baseNameOf name;
            baseLength = builtins.stringLength base; in
        if baseLength > HASH_LENGTH + 1
        then builtins.unsafeDiscardStringContext (
            builtins.substring (HASH_LENGTH + 1) (baseLength - HASH_LENGTH - 1) base
        ) else base;