As a workaround to MongoDB build requires a lot more memory I’ve found that lowering nix.settings.cores allows the build to complete. Is there a way to apply that setting only to MongoDB?
1 Like
https://nix.dev/manual/nix/2.24/command-ref/conf-file.html?highlight=cores#conf-cores
Sets the value of the
NIX_BUILD_CORESenvironment variable in the invocation of thebuilderexecutable of a derivation. Thebuilderexecutable can use this variable to control its own maximum amount of parallelism.
Thanks, but I don’t see what you’re suggesting there. Yes, cores sets NIX_BUILD_CORES—what do I do with that knowledge?
You can override the value of NIX_BUILD_CORES in preBuild.
1 Like
I see. Is there a way to achieve this without affecting the derivation’s hash to keep it machine-independent?
You can’t modify the derivation without modifying the derivation hash, that I know of. I don’t see how it’d be machine-dependent though?
Correct, so implicitly the question is whether there’s a way to affect the value of NIX_BUILD_CORES for specific derivations, which they already respect unmodified.
The value of NIX_BUILD_CORES is machine-dependent, and as I understand it that’s the point; this is designed to accommodate variation in capacity across machines without having to modify the derivation. Accommodating that variation by instead modifying the derivation would make the derivation machine-dependent, i.e. have an unstable identity across machines.
Not if you set it… and are you saying you want to build it on every machine independently? (why?)
I’m sorry, but there is something so fundamentally different about how you’re approaching this—maybe about how you approach problem solving in general—that trying to convey my perspective to you feels like pulling teeth. Thanks for trying.
2 Likes
If you set NIX_BUILD_CORES to 4 on machine A, then distribute that to machines B, C, and D, how is that machine-dependent? I don’t understand what you mean by perspective, it’s objectively not machine dependent. There’s no reason to rebuild the same code on B, C, and D other than licensing restrictions that IMO nixpkgs takes too conservative a stance on.
I suppose you could modify preBuild to:
- Check for total system memory
- If lower than X GB, set NIX_BUILD_CORES to something low
That isn’t too bad IMO, since NIX_BUILD_CORES already varies by system.
The derivation hash would stay the same then.
2 Likes
It appears there is not a way to make this setting vary within a single invocation of nix without changing the derivation like you said. You could do your building in 2 steps but first running something like this (or whichever way makes sense for how your nix files are structured) and then run the command to complete the rest of your build. If the mongodb dependencies are not built yet, they would also be limited to 2 cores. If that’s a problem, they could be built in another pass before the other two. It seems like what you are asking for would be a hypothetical feature like meta.buildCores. I understand not wanting to bother with setting up distributed builds, but I am curious about your objection to changing the hash in this case.
nix-build --expr 'with import <nixpkgs> { config.allowUnfree = true; }; mongodb' --cores 2
1 Like
Thanks, yes, this is what I’ve been doing so far.
Right, I imagined that something like a meta.nix.settings might exist but wasn’t sure how to thoroughly check for it.
I’m foremost interested in learning how to handle this in the general case. The capacity of the build machine isn’t a property of the software being built, so it should not affect the hash. More practically, it would be ironic for this to be the cause of divergence from a communal build cache.
The irony here is in this comment.
If you are using a communal cache, then the capacity of the local machine does not matter, so you don’t even need to set NIX_BUILD_CORES. Just build it on one machine (one that can take the load) and make it available to the others, like I mentioned earlier.
If you are not using a communal cache, then the cache miss does not matter, since you’re building on every single machine anyway. So you can override the package to set NIX_BUILD_CORES per machine suitable to the capacity it can handle (though I personally consider that wasteful of energy and time, but it’s your machines after all).
And in the general case, you may be using a communal cache.
Sure, I am assuming that you have control over the machines you are using, or at least can convince the person who does control the machines to have some intelligent setup here. If not, you’re currently out of luck to do anything other than what you’re already doing. (If there was some better option, then I’d imagine nixpkgs.config.enableParallelBuildingByDefault would have been written to avoid the mass rebuilds that it currently causes.)