It’s been several days that I’m trying to update the system, but update gets stuck, this is the line
building '/nix/store/bbc16bd51k2cxmlx5j8lzqbx6b898ilh-user-units.drv'...
It just hangs on that? Further log context? Any idea which line in your config causes that?
i.e. have you tried disabling different parts of your config piecemeal to narrow down a potential source for the hang?
You can also try building with –show-trace and –verbose to see more of what’s going on during build, may also help you narrow down where in the config it’s coming from.
Have you just let it hang before and given it some time? If so, how long did you give it?
I’ve personally had issues in the past with some packages cloning dependencies and nothing showing up in the logs, which made builds appear to be seemingly stuck when in reality it was just executing an incredibly long download or other action.
I’ve been having similar issues for the past few days. It looks like the stable channel is no longer stable. In my case, rolling back to build nixos-26.05.1947.a0374025a863 fixed the problem. Now all that’s left is to wait for a newer build.
There were some issues with the cache. It’s possible that your build is not stuck, but is building some package (which can last hours).
1 Like
You misunderstand the word “stable” as it is used in software versioning.
Stable means that no backwards-incompatible changes are introduced. It does not guarantee bug-freeness, in fact there is no process that could guarantee this.
There have been no recent changes to the stability guarantee policy, so stable is still the same stable it was last week.
That, and specific big builds (e.g. librewolf) just happened to fail recently. That can happen from time to time anyway, the simplest solution is usually to wait a week or so for the next update, though there are alternatives.
I’d recommend running your builds with nh, especially if you don’t really understand nix yet. It does a much better job telling you what’s going on.
2 Likes
So that’s why my terminal spent the past several hours building python and the linux kernel from source.
You are probably right, and I suppose you understand what is going on behind the scene.
This isn’t directed at you, but it’s difficult for me to understand why this happened and how a build that causes so many issues (and missing entries in the cache) ended up being released. If the simplest solution was to “wait a week or so”, why couldn’t the release have waited until then, when the cache would have been ready?
This is discussed every time it happens.
The problem is that nixpkgs is big. Not just big, it outscales the AUR by nearly a factor of 2. Keep in mind that that is a package repo anyone can submit to without any quality or security gates.
Additionally, nix takes the view that if a package changes, all its dependants also have to be rebuilt. So, roughly every two weeks or so, the NixOS foundation rebuilds a package set nearly twice as large as the AUR.
The fact that our build infrastructure can keep up with this pressure is bordering on a miracle.
Anecdotally, I’ve seen billion-dollar company internal build pipelines choke on a tenth of this (in fairness, due to eval overhead of non-nix build tooling, but still).
That in itself doesn’t answer your question though. To check if a specific commit causes any issues, we’d have to - on top of the constant build pressure from staging - have to build every single commit, before merging. nixpkgs sees a few hundred commits a day.
These commit rebuilds aren’t quite as big as the staging rebuilds, but many of them individually would still take hours to days, and everything would need to be redone for all commits about to be merged every time a commit merges. Any random package version bump maps to a commit, to be clear, so not doing this means not even sending out security patches, defying the point of a stable release distro.
Since 100x1h > 24h, it’d become impossible to make any changes to nixpkgs if we had that level of CI gate.
Even if, somehow, we did, this still doesn’t prevent one-off issues like spurious build failures or labeling mistakes causing temporary cache misses.
The big one everyone is currently complaining about is that librewolf was marked as unmaintained - it was re-marked as maintained shortly after. The cause of that was that it indeed was effectively unmaintained (because updates required too much manual work and no committer was assigned, so those changes could not be made in a timely fashion), though the details of organization around that could have been done better.
Regardless, little things like that will always happen. It’s extremely rare for cache misses like this to happen, something big enough to be discussed on discourse happens maybe 2-3 times a year at best, almost always for a desktop package, and it normally only affects a subset of the community.
I appreciate the frustration, but seriously, given the amount of work happening behind the scenes, I’d just be thankful that somehow this boat is kept afloat with, at worst, like a one week update lag for a specific package once or twice a year. Given the amount of funding and general problems in the ecosystem, this is an insanely good track record.
Besides, even if it does happen, it doesn’t take genius-level nix knowledge to still get updates perfectly normally for unaffected packages. You just have to shuffle your channels/npins/flake inputs a lil’. I’ve written tutorials a good dozen times on this discourse, they’re out there.
It’s only necessary if you’re very paranoid or know you’re affected by something important, but you can go out of your way to get your updates if you need them.
All that out of the way, if you’re really upset about this, consider using the -bin variant of your browsers (and electron-dependant packages). Browsers are insanely slow to build and have massive dependency trees. They’re usually the packages affected, and the only ones where you notice anyway, since nothing else takes long enough to build to be noticeable.
Hell, consider switching to using a flatpak for your browser, I’m of the opinion that distros should not ship their own browser packages anyway - we’d get so much more manpower for free if all communities got together and made the browser flatpaks work better instead of reinventing what are the most complex and build-intensive packages for every community.
1 Like
My frustration is not caused by something not working, but that I don’t understand it and can’t find a reasonable explanation for it. So, thank you for the detailed explanation! The scale is indeed enormous. I also found some additional explanations on this topic here: Nix refuses some packages from binary cache - #3 by vcunat Taking everything into account, the overall picture is becoming much clearer. Thanks
So it looks like it is a common problem. To answer some questions, I let it compile for 14 hours, it’s not an issue of not enough time.
I also tried to run nixos-rebuild boot --show-trace |& grep 'while evaluating derivation' but no output was displayed (the compilation was taking place though).
And I also tried nixos-rebuild boot --show-trace, but I can’t tell the difference, the output is not something I understand.
I tried several times during the last week, so if it was a Hydra issue it’s taking more than a week to be sorted.
Yeah, use nh for better logging. Nix will be building something. If you still refuse to use nh, but use flakes, don’t forget the -L flag to actually see build output.
1 Like
If I may express my two cents, everything you said is right and understandable, I would like to give my perspective as a humble user with no particular skills or knowledge.
Indeed many packages in the repo have no maintainer or the maintainers don’t maintain them, they are several versions behind. Lately the NixOS team decided to remove package requests, I understand that, what’s the point of having the biggest repository if the packages are outdated or unmaintained?
Still, the Nix repo is one of the reasons people are attracted to NixOS (yes, Nix can be used on any distro, but using Nix on NixOS makes more sense).
But then any minimal distro would do, just thrown in every AppImage or Flatpak you need and the base system makes no longer a difference. This is the direction the Universal Blue project has taken. Many folks who want a declarative system come to NixOS instead of Guix because of the repository. But admittedly this is also a general trend in Linux in general, Debian users, Mint users, Fedora users, are all being told to just get the Flatpak or the AppImage, at which point the distro doesn’t matter anymore.
But the point of NixOS should be “Declarative, Reproducible, Reliable”. I should declare the system once and for all and have it working indefinitely.
If I have to use Flatpaks and AppImages it’s no longer declarative.
If it only works on an older channel but not a newer one it’s no longer reproducible.
If I must remove portions of the configuration to check issues it’s no longer reliable.
I can’t install nh, nor nix-tree, nor nix-output-monitor because the build fails. I found these in other threads, but I can’t use them.
You can use them via nix-shell or nix shell for the time being. They’re very simple binaries.
Sure, but the repo could be an order of magnitude smaller and be just as attractive. There’s a ton of cruft. Probably more attractive, since the packages people actually use would actually be maintained.
Flatpaks can trivially be managed declaratively. This is true for appimages too. Both are just files to be managed on your filesystem, and no worse than -bin packages, you just need to add appropriate nix integration.
The true loss is not being able to override build sources anymore, which is sad, but I believe with a more concerted effort that kind of stuff could also be done. We’d just have to work on supporting flatpak better.
For browsers specifically, which is what I’m talking about, barely any systems can build them successfully today anyway because of their builds’ tendency to guzzle 10s of GB of memory and days of compute, so using a flatpak (or -bin package) for your browser isn’t in practice a loss in functionality.
NixOS remains useful for detailed declarative system configuration, which other atomic distros don’t do nearly as well, if at all - with the exception of arguably the guix distribution.
Between NixOS and the guix distribution the trade-offs are more detailed, but choice isn’t a bad thing. We’re not in a capitalist war to be the most used distro, if there are several options that do well that’s perfectly fine. We don’t need artificial scarcity.
1 Like
In the end build succeeded. It still compiled something that Hydra failed to build, but this time it went through, it took about 10 days to sort everything out.
For those who had this issue try once again, be sure to run nix-collect-garbage and nix-channel --update before rebuild.
That’s unnecessary and will actively make the situation worse, since it will invalidate anything you have already built locally.
Don’t run random commands just because you feel like it. Nix builds software reproducibly, there is no state to clean up.
The “correct” solution remains to check what is actually building (which you can do with the -L flag if there are no logs because you’re using flakes, or various log processors like nh), and then confirming why you’re getting a cache miss.
It’s even possible this isn’t caused by the recent cache mishaps, and instead by misconfiguration on your end, so I would recommend properly debugging if you run into similar issues again.
2 Likes
I read to do it in another thread, it can be an issue apparently. I don’t mind rebuilding something if it means the build succeeds, if the build fails and I didn’t clean garbage beforehand I will never know if that was the issue.
I didn’t change anything in the configuration during the days it kept failing to build. I simply tried updating the system and retried during the course of several days, eventually it succeeded.
If you have a build failure nothing ends up in the store hence there is no point of garbage collection.
2 Likes
You might have always had some configuration that makes your builds slightly different from the cache from time to time, and just never noticed because you never hit a big rebuild.
Yeah, you either eventually hit a situation in which enough was cached that your system could power through, or finally waited long enough.
Whether the issue is upstream or on your end is impossible to assert without further debugging, or at least a glance at your config.
I know it’s pretty common to solve tech support issues by flinging crap at the wall until something sticks, but NixOS uniquely gives you full control of and insight into the system. This is the one place where you can actually be certain about both the problem and the solution, it’s a waste to accept it potentially recurring.
1 Like
Yeah, I installed those diagnostic tools, next time I will try to dig deeper.