Nixos builds a ton of packages instead of fetching them in the cache

Help
1

I have tried installing a new keyboard layout using the services.xserver.extraLayouts option. It worked, except that since then, each time I run nixos-rebuild, even on some small changes (like changing the layout), it builds tons of packages, and each build takes hours, during which the CPUs fully used. Not only it does not fetch these packages in the binary cache (which is strange), but it also rebuilds them each time.

The rebuilt packages are:

mutter
gnome-shell
gnome-session
gnome-control-center
desktops
xdg
nixos-gsettings-desktop
gnome-tweaks
gtksourceview
tepl
gedit
sushi
gnome-terminal
chrome-gnome-shell
firefox
gnome-calculator
system
pam
useradd
vlock
dbus
unit
user
system
chfn
unit-accounts
userdel
su
unit
sudo
xlock
groupdel
set
etc
passwd
xscreensaver
unit-display
unit-systemd-fsck
unit
system
login
polkit
usermod
sshd
groupmod
groupmems
systemd
i3lock
i3lock
gdm-launch
system
groupadd
chpasswd
runuser
cups
chsh
etc
nixos-system-frisbee

Why? It doesn’t make sense to me… How do I solve this?

2

Obvious question: Does removing the added keyboard layout also stop the rebuilds?

3

This is to be expected, as the option changes xorg libraries, which many packages depend on:

The docs are unfortunately bit of an understatement:

Applying this customization requires rebuilding several packages

If you want to avoid rebuilds, you can rewrite the NixOS module from using overlays to system.replaceRuntimeDependencies .

3 Likes
4

Yeah, quite an understatement.

Unfortunately, I am still a beginner with NixOS, and I don’t know how to do that yet. I don’t really understand where I should put system.replaceRuntimeDependencies to make it work.
Reading the documentation, i have the impression that I should do something like this in my /etc/nixos/configuration.nix

{ pkgs, ... }:
{
  # ...
  system.replaceRuntimeDependencies = [
    {
      original = pkgs.xserver;
      replacement = ???;
    }
  ]
  # ...
}

where ??? should be the overwritten version. Is that right? And, if so, how exactly do I generate that overwritten version? Because currently I don’t overwrite the package, I just change the settings through the interface.
If someone is willing to explain me how to do it, I would be thankful to them. This looks exactly what I am looking for.

However, since I use Gnome, and since Gnome overwrites the xorg layouts (I have to manually add my keyboard layout in the gnome settings for it to take effect):

$ dconf write /org/gnome/desktop/input-sources/sources "[('xkb', 'bsk')]"

Couldn’t I just change the xkb configuration (adding a keyboard layout), without trying to load it in xorg? Is that even possible, or does xorg needs to be rebuilt anyways when changing the configuration of xkb?

Finally, I just have a stupid question: even assuming it has to rebuild everything because a dependency changed, why can’t it load the binaries from the cache? It’s not like the binaries changed because a dependency changed, since the dependency is a runtime one.

1 Like
5

I meant that you could edit the NixOS module itself and submit a pull request.

With overlays, you are taking a specific Nixpkgs attribute and are replacing it with another one (possibly derived from the previous one – in super). That is why it causes so many rebuilds – when an attribute changes, all derivations that (transitively) refer to it are considered different as well.

replaceRuntimeDependencies uses a completely different approach: First your system is built as normal and then, in the resulting system package and its dependencies, the store paths are textually replaced. If the packages you replaced are equivalent in the sense that the packages depending on them would not change if you used overlays (modulo store path changes), you will have saved many builds. But if any package depending on one of the packages you replaced would produce a different output you might get wrong results (for example, if xorgserver parsed the xkeyboardconfig files at build time and compiled the data into the program) or even crashes (if the replaced package was a library and the replacement was not ABI compatible). I do not suspect that to be the case but one should always keep this in mind.

Going the replaceRuntimeDependencies, you would do something like the following untested snippet:

  system.replaceRuntimeDependencies =
    let
      xkeyboardconfig = pkgs.xorg.xkeyboardconfig_custom {
        layouts = config.services.xserver.extraLayouts;
      };

      xorgserver = pkgs.xorg.xorgserver.overrideAttrs (old: {
        configureFlags = old.configureFlags ++ [
          "--with-xkb-bin-directory=${xkbcomp}/bin"
          "--with-xkb-path=${xkeyboardconfig}/share/X11/xkb"
        ];
      });

      setxkbmap = pkgs.xorg.setxkbmap.overrideAttrs (old: {
        postInstall =
          ''
            mkdir -p $out/share
            ln -sfn ${xkeyboardconfig}/etc/X11 $out/share/X11
          '';
      });

      xkbcomp = pkgs.xorg.xkbcomp.overrideAttrs (old: {
        configureFlags = [
          "--with-xkb-config-root=${xkeyboardconfig}/share/X11/xkb"
        ];
      });
    in
    [
      {
        original = pkgs.xorg.xkeyboardconfig;
        replacement = xkeyboardconfig;
      }
      {
        original = pkgs.xorg.xorgserver;
        replacement = xorgserver;
      }
      {
        original = pkgs.xorg.setxkbmap;
        replacement = setxkbmap;
      }
      {
        original = pkgs.xorg.xkbcomp;
        replacement = xkbcomp;
      }
    ];

Actually, looking at the xorg overrides the packages are already passed the flags so only the xkeyboardconfig is really necessary.

Of course, you could also put that code into your configuration.nix and be done with it – it is just another NixOS module after all.

I think, in GNOME Settings, you are only adding an existing layout returned by Xorg. You still need to make Xorg itself aware of it, which is what the extraLayout option does.

Nix cannot know if the same output will be produced when a dependency changes (see above) so it needs to build it. And since Nix bases identity of packages (derivations) on the identity of their dependencies, it would still be considered a different package even if the output were the same (modulo store paths).

There is an experimental feature that will allow early cut-offs when derivation output is the same (modulo store paths) but you will always need to rebuild at least packages directly depending on the package you change.

3 Likes
7

There is still something that I don’t fully understand. Layouts are not part of the source, they are configuration, and adding one should just add a few configuration files (on other distros, to add a new layout you just add a few files in /etc). So why would NixOS rebuild xorg in the first place?

This is all-the-more annoying as I am creating the layout I am trying to install, so I need to “debug” it a lot (trying if typing a way or an other is more practical), and having to wait for the recompilation of half of my xserver each time I change makes it more practical to install a VM… it says everything.

I tried adding it to my configuration.nix before doing a PR (at least to see if it works), but my system is still rebuilding. I couldn’t say if it rebuilds the same derivations, as it’s not really clear from the output (the compilation output is simply overwhelming compared to the few lines nixos-rebuild prints, so finding them is a challenge).

However, rebuilding my system still takes a few hours, which is a lot, so I think it didn’t change the compilations.

1 Like
8

Did you remove the services.xserver.extraLayouts option and pass the value directly to xkeyboardconfig_custom? If you did not, it will still trigger the module in NixOS to add an overlay.

There are three common reasons for loading configuration from the package instead of some location on the system:

  • It is not configuration but data. In such case, it will be loaded from datadir (share directory in package’s store path) and since Nix packages are immutable, you need to rebuild the package to change it.
    • Ideally, the project would load data not just from its own datadir but also all datadirs listed in $XDG_DATA_DIRS environment variable. That way, we could just add the extra layout to environment.systemPackages and it would be installed to /run/current-system/sw/share, where the software would find it. But alas, older software often does not support this.
      • For practical reason, we sometimes point the software to look to /run/current-system/sw/share (example) or even /etc since it is conveniently changeable using environment.etc but that introduces a dependency on NixOS module on NixOS, and is not portable to other Linux distros.
  • We want the package to be reproducible/hermetic/fix incompatibilities.
    • For example, long time ago, fontconfig changed its configuration format but kept using the same directory. This meant that programs linked against the old fontconfig did not work with the new configuration in /etc/fonts. So we decided to patch new fontconfig to use /etc/fonts/2.11. But then distros like Arch switched to unstable fontconfig, which made the config backwards incompatible and the stable fontconfig from Nixpkgs would have had issues with it. To fix this, we made fontconfig hermetic outside of NixOS by loading the configuration from the etc directory inside the package. But of course, this meant fonts were no longer configurable on Arch for packages from Nix without rebuilding fontconfig and everything depending on it. In the end, we resigned on hermeticity for fontconfig and just hold our fingers crossed when loading the configuration from the system, sacrificing programs built against ancient fontconfig.
  • Since Nix packages build with --prefix different from /, most build systems will default to --sysconfdir=$prefix/etc for installing and loading the configuration.
    • This is reasonable for Nix as packages cannot write outside of their own store paths but yeah, it will not be editable without rebuilding the package.
    • If we want the program to read its config from /etc, for Autotools-based projects, we can pass --sysconfdir=/etc to the project’s build system’s configure script and then change the installation path by passing sysconfdir=$out/etc to make install.
      • Other build system may require patching.
      • If the project does not have sane defaults or requires the config file to exist in the location, we will need to make sure the config actually ends up in /etc (e.g. using NixOS module). This also rules out just running it in nix-shell or installing it on non-NixOS distros.
    • Sometimes people creating the package are not aware of this behaviour or do not want to bother.
9

I did that, and now it works, doing

xkeyboardconfig = pkgs.xorg.xkeyboardconfig_custom {
  layouts = { ... }
};

There is however a small catch, is that there are no “default” options, so I have to manually specify option = null; for the options I don’t want to set. That’s not too problematic, but if they add an optional option, it will break my setup.

However, I still don’t understand why rebuilding xorg. It’s not data, it shouldn’t be hermetic, and I think that xorg has sane defaults (it would be surprising that such a big project didn’t have some).

Whatever, thanks for the snippet, it works. Hurray! :slight_smile:

1 Like
10

I do not understand what option do you mean.

The distinction between configuration and data is blurry. In this case, it is in datadir, not sysconfdir. Similarly, whether something should be hermetic needs to be based on case-by-case basis. In this case, since the config format is frozen in time, I agree that it is probably not necessary. I was speaking generally. Though, ancient software like xorg (not sure if xorg specifically) often do not have defaults and require some configuration file in /etc so hermeticity would be required if it were not so prevalent.

11

In layout.bsk, it is defined to have attributes which include, among others, symbolsFile and compatFile. I want to specify a custom symbolsFile, but I don’t have a custom compatFile, therefore I want only to set symbolsFile, which is ok when it has a default value. If I set the layout value to a custom set, it does not have default values, and I thus have to set explicitly compatFile to null. It’s not a problem, it’s just that if, in the future, an optional attribute is added, it will break my configuration file.
Concretely, I dont do

layouts = {
  bsk = {
    symbolsFile = ./my/layout;
    languages = [ "eng" ];
  };
};

but

layouts = {
  bsk = {
    symbolsFile = ./my/layout;
    languages = [ "eng" ];
    compatFile = null;
    geometryFile = null;
    # ...
  };
};

Ok, it makes sense.

12

Oh, I see. Yeah, the default values originate in the extra-layouts submodule so you will need to pass them manually, when you use the xorgconfig_custom directly. There is no nice way to do that but since it will break loudly you can just fix it during update. And I doubt there will be much new options, everyone is working on Wayland these days.

Well, I guess, if you wanted, you could do something like

  bsk = builtins.mapAttrs (name: definition: definition.default or null) (options.services.xserver.extraLayouts.type.getSubOptions ["services" "xserver" "extraLayouts"]) // {
    symbolsFile = ./my/layout;
    languages = [ "eng" ];
  };

But that will add some extra internal attributes like _module (should not matter for this use case) and uses an internal details of the module system so it will probably change more likely than Xorg module.