I have a situation where I’m wanting to persist some information from my derivation that is not strictly an output, but is more than just what I can dump into stdout. This includes:
- Extra log-type info, for example recordings and images generated by test runs.
- The obj files from the build, so that I can have a later derivation “continue on” from where this one left off (eg, build additional targets in the same workspace).
Both of these cases don’t really fit great with the existing nix “output” scheme, for example:
- I’d like the log output to be lean in terms of its run dependencies, but any time a log file references a nix store path (like gcc for example), that obviously ends up in its closure, so that downloading logs becomes potentially huge.
- The obj files end up having to be compressed into a tarball to hide from Nix that there are circular references between these and the main output.
- Ideally these “extra” outputs would be up for garbage collection immediate after being created, but that doesn’t seem to be the case— they stick around as long as the main output does.
Am I asking the impossible here? Or is there a way to get this kind of auxiliary output content out of a Nix derivation build? Are there special log outputs? Or a way to disable/override some of the post-build checks that make doing this difficult?
I’m not sure why your regular output is referring to your “extra” output, but that sounds like the reason for some of your troubles. If your regular output is just supposed to be what you would get out of the derivation if you didn’t include an extra output, then it shouldn’t refer to the extra output. That would avoid the cyclic reference issue, and would allow the extra output to be garbage collected separately (unless you have keep-outputs = true in nix.conf). As for a log file referring to something like GCC, well there’s always pkgs.nukeReferences.
Okay, nukeReferences is interesting, that was new to me—
However, it looks like it’s just replacing the base32 string with a dummy string, so the underlying information is lost about which packages were involved in that build. It would be great if there was a way to suppress the reference-finding altogether for an output, rather than having to modify it like this.
Regarding the circular-references issue, I think it depends a lot on the subtleties of how the thing comes about, for example performing the build in-place in the output location, vs copying it in afterward. But either way, there are lots of cases where generated files (think cmake modules, pip executable passthrus, etc) retain a reference to the cwd that originated them. If that cwd is just /build/build or something, then Nix ignores it, but if you try to perform the build in an output, then it’s trouble.