Why does nixos-rebuild build everything from sources?

Help
1

Hi,

Still trying to build my standard configuration on my newly installed asahi linux.
The RAM problem I had in my previous post is fixed, but now after more than 4h of building, the build has failed due to an error in webkitgtk. That makes me wonder why suddently nix wants to rebuild everything from sources ?

From what I understand of nix, there is a binary cache that can provide most of the usual binaries without rebuilding them from sources at every rebuild. But it seems that in my case, nothing is taken from cache… Does this means there is no such cache for aarch64-linux system ?
I double checked my flake as pointed in some other threads, i’m NOT using the ‘release-23.11’ branch as an input for nixpkgs.

Is there a way to check the cache content to see if the packages i’m building from sources are available ?

Thank’s for help,
JM445

1 Like
2

How are you specifying nixpkgs? This could mean you’re using the master branch, where you can end up on commits that haven’t been built yet or won’t get built at all.

3

Here are my inputs:

inputs = {
    nixpkgs.url = "nixpkgs/nixos-23.11";
    home-manager.url = "github:nix-community/home-manager/release-23.11";
    home-manager.inputs.nixpkgs.follows = "nixpkgs";
    nixpie.url = "git+https://gitlab.cri.epita.fr/cri/infrastructure/nixpie.git";
    nixos-hardware.url = "github:NixOS/nixos-hardware/master";
    myAwesome = {
      url = "path:./pkgs/MyAwesome";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    nixpkgs-unstable.url = "nixpkgs/nixos-unstable";
  };

Full config is publicly available here (quite messy for now): https://github.com/JM445/nixconfig

1 Like
4

Which host is the one building stuff downstream?

There are lots of reasons why this might happen, basically your config needs to be different from what upstream is building in some way. If you’re noticing gtk rebuilding specifically, that might be graphics libraries, for example. I notice you grabbing unstable and stable, mixing those is sometimes a reason for rebuilds to happen, too.

Typically when these kinds of things happen subtle changes to your config can give you nearly the same result while not having to rebuild packages.

1 Like
5

The host building is “asahi-mac” (I forgot to specify I was building on the wayland-sway-migration branch).
As an example, I have added in the “home/asahi-mac/default.nix” file the chromium package. I do not modify anything from this package, just install it in home.packages. However the package is still compiled from sources.

I may be misunderstanding the way binary cache is working. Do I need to build from source each package at least once to be able to use it from cache on the next rebuilds ? I was thinking the binary cache was instead here to provide already built packages for common architectures/ABI

6

No. Someone needs to build each package at least once, of course, but usually this is done upstream by hydra.

Yes, this is the idea.

However, if the binary cache does not have a specific package cached, nix will automatically build it on your computer. This can happen if you make modifications to a package, or somehow depend on a branch that hydra has not built yet or something. Changes in dependency packages will also cause cascading rebuilds all the way to leaf packages you actually use (because there is no way to guarantee ABI/API compatibility), so a small change can cause a lot of churn.

It’s not particularly common to see accidental changes like that, but your flake is quite complex with many different inputs (some of which do not have first-party support and will therefore not be built by hydra), so something might have snuck in.

I think aarch64 also generally builds a bit less than x86_64, so maybe this is indeed a case where aarch64 just doesn’t have this, but usually that’s packages marked as not working on the platform, so you would need to opt-in to attempting to build such packages.

7

I never fully understood the chain of libraries that lead from mesa to gtk, but you’re overriding mesa here: https://github.com/JM445/nixconfig/blob/780e04e4b70e0bc1055edc413e8a785efa2bf5b1/hosts/asahi-mac/apple-silicon-support/modules/mesa/default.nix

Usually when I mess with mesa all graphics-related packages need to be rebuilt, so that will probably be it.

That whole apple-silicon-support directory strikes me as a massive project, did you write that yourself or are you vendoring a third party project? Quite likely they have their own binary cache, or that you can at least import it as a flake input with some trickery.

Edit: Yep, rebuilds are caused by that, as stated in your very own comment (so you probably didn’t write that all ;p): https://github.com/JM445/nixconfig/blob/780e04e4b70e0bc1055edc413e8a785efa2bf5b1/hosts/asahi-mac/apple-silicon-support/modules/mesa/default.nix#L25

1 Like
8

Okay, I got it. That is indeed not my own code, it is provided on the NixOS Asahi installation guide, I actually do not understand very well what is going on inside that. Asahi needs quite a lot of things tweaked from the default nixos to be able to run harware acceleration.
I was thinking I would just need to rebuild the gpu driver related libraries, but now I see the point: chromium is rebuilt because webkit is rebuilt and webkit is rebuilt because mesa is rebuilt etc…