Upgrade to 25.11 fails

I’m on 25.05 and when I try to upgrade a lot of packages begin compiling on my machine. In particular, the one that fails is ungoogled-chromium, though it’s not the only one that compiles, many other packages that should be downloaded precompiled build locally.

On Hydra, ungoogled-chromium seem to fail to build, but chromium builds correctly and ungoogled-chromium is the same package with some flags changed, so I don’t understand if it should be reported as a build failure since the black X indicates the failure is related to another package.

But since several packages are not precompiled I’m wondering if it is still early to upgrade. How long will it take for the whole situation to stabilize and for the upgrade process to be smoothened?

This shouldn’t be happening with 25.11. Are you sure you’ve switched? What do you see if you type

sudo nix-channel --list

?

3 Likes

Alternatively, in case you are using flakes to manage the system inputs, share the output of nix flake metadata, or if public, a link to your configuration flake.

I see no failures on Hydra for these packages.
https://hydra.nixos.org/build/315412381
https://hydra.nixos.org/build/315413996

So it’s definitely an issue with your local configuration. Either you’ve enabled CUDA support, or you’re using the wrong channel.

1 Like

Sorry, I didn’t get any notification.

nixos https://channels.nixos.org/nixos-25.11

nixos-unstable https://nixos.org/channels/nixos-unstable

I don’t use flakes at all.

I didn’t add CUDA support. I simply ran

nix-channel --add https://channels.nixos.org/nixos-25.11 nixos

and then

nixos-rebuild boot --upgrade

changed the packaged that were renamed in the configuration and added a couple of insecure packages that were making the build fail to start. I tried upgrading again, now those other packages that were compiling before are downloaded from the cache, but ungoogled-chromium still builds from source and still fails.

I also tried to switch between cache.nixos.org and the fastly.net beta cache as I thought the second one might receive the freshly built packages with some delay.

I followed the exact procedure as when I upgraded from 24.11 to 25.05, ungoogled-chromium is the main obstacle at the moment, but I need it.

1 Like

Your channels are the right ones. Either way, the package is cached, you’re trying to build something different from upstream for one reason or another.

Your configuration could still do a number of things that would cause this. There are various global nixpkgs config you can enable (like the cuda flag), non-obvious overrides, as well as potentially additional “manual” nixpkgs imports with fetchurl.

You’ll have to share your full configuration if you want help with this one, since it’s not the obvious suspects. I suggest you stop experimenting, it’s an issue with your configuration, but something you added, not something that’s wrong by default.

2 Likes

My configuration is huge and scattered into several files, it’s a bit tricky to post everything.

I use no override at all, but I don’t understand why the rebuild fails only with the 25.11 channel, while it succeeds with the 25.05 channel. If it was a configuration error both should fail or succeed, isn’t that the point of reproducibility?

The point of reproducibility is that, given the exact same configuration, you should be able to reproduce the exact same results. That does not imply that if you change something - including the NixOS version - you get the same result as before (otherwise you would never be able to update anything; that’s obviously not reproducing the same result).

For example, maybe you’ve previously built chromium without noticing and simply been using a locally cached version since. With the change to 25.11 lots of base libraries updated; If this is the case, now you need to build a new chromium, because it’s obviously not the same anymore and therefore you need to produce something different (i.e., this isn’t re-production, you’re not trying to reproduce either your old chromium or the upstream chromium) - and maybe your build fails because your old changes no longer result in a correct build for the new chromium version.

Or maybe they *are* correct

I suspect you’re just running OOM and the build would in fact complete fine after a very long time if you ran it with fewer cores; Not that you should, given the symptoms I suspect it’s likely you are in fact running a release channel, despite the nix-channel output. For now I have to go off your word that that’s not the problem without your code.

You might’ve also just accidentally changed something unrelated, even if you think you made no other changes; I’d recommend using git to guard against that kind of thing.

Reproducibility does mean that if you shared your configuration we could perfectly reproduce your problem and solve it for you; that’s what it buys you (among other things that are usually less exciting to end users). It also means that we can pretty confidently say it’s a configuration mistake and not an upstream problem, because we can reproduce working chromium builds and download it from the cache.

Whatever it is, without the code nobody but you can tell what it is. We can keep guessing random stuff, but we don’t even know if you’re reading your own code correctly, let alone whether you actually understand the suggestions. This thread will be very frustrating if it continues as-is; those two guesses are the last random suggestions you’ll get from me, at any rate. If you create a GitHub or codeberg repo it’s pretty easy to share, but yeah, that’d be a bit more effort.

I guess the alternative is to start writing a new configuration from scratch, adding back bits until you find the mistake? Given you’re using channels, I’d recommend using a VM to ensure there’s no state messing with things.

2 Likes

What I’d do in your shoes:

  1. Run nix-instantiate --eval --raw --expr 'builtins.substring 11 32 (import <nixpkgs/nixos> {}).pkgs.ungoogled-chromium.outPath' and note the result.

  2. Check to make sure that it’s not in the cache.
    a. One way: curl -Ifs "https://cache.nixos.org/$hash.narinfo"
    b. Another way: paste the hash into https://wh0.github.io/nix-cache-view/

  3. Comment out a large chunk of your configuration, run the command in step 1 again, and see if the result is different.

  4. If it is different, check to see if that hash is in the cache. If so, the problem is somewhere in the commented code. Uncomment some of it and repeat. If not, or if the result is unchanged, the problem is in the remaining code.

  5. Repeat steps 3 and 4 until you’ve found the problem. If it’s not obvious what to do about it, post what you’ve found.

2 Likes

Share a git repo on some forge (github, gitlab, codeberg, whatever).You are using git or some version control, right? If not, start now - then you can compare what you changed.

The third option is to iterate on comparing the derivations:

  1. Run nix derivation show on the package in nixpkgs (on the channel you’re using) vs the path that your nixos-rebuild is about to build.
  2. Then use some json diffing tool to compare the two, difftastic seems to work well for me.
  3. If you see some significant differences, zero in on that.
  4. If you don’t, but you see some difference in hashes, diff those paths by repeating the first 2 steps - iterate until you find some difference.

Ultimately this will actually tell you what differs, eventually.
I don’t think the first approach will tell you much other than a yes/no if it’s in the cache or not - but this approach will tell you exactly what nix is seeing and why it’s a cache miss.

I made some other digging and turned out I was wrong, what fails to compile is actually qtwebengine-5.15.19, I was mistaken because I saw “chromium” printed everywhere and ungoogled-chromium wasn’t still compiled by Hydra when I first looked. qtwebengine-5.15.19 is the dependency of another package, I removed that from the configuration and the upgrade went fine.

2 Likes