how would i go about subdividing nixpkgs.stdenv from a single nix-build into multiple nix-build steps that can then use most/all cached build to produce the final stdenv
specifically for segmented building in a CI where a large build may be split into smaller builds that yield more/less the same result as the large build but can be incrementally built (eg, to reduce job timeout chance)
See splitBuildInstall: split buildPhase and installPhase for large packages and Distributed nix build: split large package into many derivations, though you will be guaranteed to rebuild anything that uses your new abstractions as hydra won’t have built/cached it.
i dont think this will be that simple
the issue is stdenv itself builds many sub-packages in many stages during the bootstrap/building of stdenv itself (substitutions false)
i would like to seperate these into multiple nixpkgs.stdenv-* subpackages that together form nixpkgs.stdenv itself
stdenv- prefix due to stdenv itself using hardcoded package versions that are seperate from the official nixpkgs.* packages eg nixpkgs.stdenv-bash version 4 and nixpkgs.bash version 5
these should be independently buildable as packages without requiring stdenv itself as a dependency (such as nixpkgs.busybox depends on stdenv but stdenv bootstraps a hard-coded busybox and as such we are unable to specify this hardcoded version of busybox to explicitly build and cache in the event that building stdenv with no substitutions in a CI may get timeout-killed and may prevent any binary packages from being cached depending on the CI used)
please note that this is for personal use and i do not intend to upstream it unless nixpkgs desires such
I have been thinking about similar things lately.
I general it would probably be enough for me in my cases to just create stages of derivation, and then build and cache those, and use them in an umbrella package that brings them together. This can be just the phases for me, and so could be as simple as what @waffle8946 pointed to in my case.
But I understand you were thinking of something more complex, that deals with the the derivations that stdenv might build “under the hood” when running a nix build. I believe in this case someone would need to create a nixpkgs lib-like “builder” that is used as an alternative to stdenv, and has the logic you seek.
and as it turns out, building nixpkgs.stdenv with substitutions false DOES indeed cause the CI to time out
The job running on runner GitHub Actions 15 has exceeded the maximum execution time of 360 minutes.
killed at 6H build time
however it DOES upload the cache even on a killed build so we can chrono to continously build provided a package itself does not take >= 6H to build
however, in the event a single package itself takes >= 6H to build, we must figure out how to divide it so portions of it can be built and cached in seperate jobs
for example, if clang took >= 6H we would need to figure out how to correctly split clang into sub-builds that can be cached, eg
and ideally each stage would take less than 6H to build
if a stage takes >= 6H we must further divide, and so on and so forth, say
in which case, for certain packages this would either be easy or difficult depending on the complexity of the package and the build system the underlying package uses
eg if nix <pkg> gets build via cmake then we could subdivide into seperate cmake modules and a corresponding <pkg> sub package
but for a makefile/autotools based package this may be more difficult as makefile/autotools can be very complex when compared to cmake