It is very common for software that has not been updated to be completely rebuilt after updating channels or flake inputs. I have recently been using custom build flags across my whole system as per Build flags - NixOS Wiki and the result is that as far as I can tell, nearly every if not all packages are rebuilt after updates – about 3.5K derivations when I first rebuilt everything and about 3.5K derivations the next update. To be clear this happens whether or not there are build flags applied, but of course it is not so much a problem nor as obvious when most things can be pulled from binary cache.
This issue (if it is an issue) doesn’t break anything, so I’m mostly confused about why things are being rebuilt so often? This can’t be intended behaviour. Is there a way to find out what is changing in all of these thousands of derivations when the packages themselves have not been updated? Am I underestimating the number of updates happening over the course of a week?
This is expected if you are using custom build flags. The cached versions on cache.nixos.org weren’t built with those flags, so Nix has to make your custom build for you locally.
It likely is. Nix will also rebuild any package whose dependencies have changed (recursively), so any change in lower-level libraries will cause a nearly full rebuild. Upstream tries to limit these because they cause a lot of pressure on hydra, but it’s still quite common.
Custom build flags aren’t really intended to be used on general purpose systems. If you do use them, it’s best to set them sparingly for specific packages where you know the performance impact will be worth the build cost.
Likewise, if you modify a low-level dependency in some way on your system (overrides, custom build flags, etc.), that will cause rebuilds of everything dependent on it.
I don’t particularly need custom build flags across my entire system, but using them demonstrated what a high percentage of my system gets rebuilt each time, which I wouldn’t notice when most entries are downloaded from cache. Rebuilding nearly everything after not updating for a couple of weeks was very unexpected
Yes, there are mass-rebuilds when some deep dependency changes. These are typically bundled together in batches and land together with the staging branches. If you’re updating every few weeks, there’s a pretty reasonable chance there has been a staging cycle land in that interval (but probably not every time) and a good chance that other packages have been bumped in the meantime too.
There’s one other thing that may be amplifying the problem for you: garbage collection. Once something needs to be rebuilt, you need the build tools and build-time-only dependencies for it. Those are not normally part of the system closure for the resulting output, and so can get garbage collected quickly. The next time you need them, they are not in your local store anymore and also have to be rebuilt, and of course are not on the public cache with your build flags.
Do you notice that it’s also downloading a lot of source tarballs for what it’s rebuilding, even when those haven’t changed? If so, this will be why.