Overlay for xkeyboard-config causing a lot of other packages to recompile. Why?

Help
1

Hi!
I’m completely new to NixOS, coming from Arch.

TLDR is basically the title. Can I avoid a full rebuild of all packages (or dependencies?) after applying an overlay to xkeyboard-config? Why is Nix doing that in the first place?

I use DreymaR’s Colemak-DH mods, but instead of using the included installer/patcher, I forked the upstream xkeyboard-config package and added the patches to the actual source, so that the package wasn’t stuck at the fixed version that DreymaR distributes it. This makes it a lot easier to update and keep up with the new features of xkeyboard-config (I really needed that caps:escape_shifted_capslock option :P).

I found this thread about installing the same mods, but it was mostly about using the official bash script that overwrites the existing XKB directory, which I’d rather avoid using. I want to use my fork instead.

On Arch, this is trivial to install. Just make an AUR package that provides+conflicts with xkeyboard-config from the official repositories and then install it.

Here on NixOS, the right way to do it seems to be to use an overlay that overrides the source (and build process) of the package. I based myself off of the example on the NixOS wiki and also added some extra steps so that the package would build correctly. This is the overlay I have now:

self: super:
{
  xorg = super.xorg.overrideScope' (
    selfx: superx: {
      xkeyboardconfig = superx.xkeyboardconfig.overrideAttrs (old: {
        src = super.fetchFromGitHub {
          owner = "SeerLite";
          repo = "xkeyboard-config-bbkt";
          rev = "99e90ad6a17bc42a55f3b7965261de8017fbd3ed";
          sha256 = "0lbnq9ba81ppp5wrp18yyz8l68h3gpzbkfva3bxdnicj5qzafg2j";
        };
        preBuild = ''
          for file in $(find . -name '*.py'); do
            patchShebangs $file
          done
        '';
        nativeBuildInputs = [ self.autoreconfHook self.xorg.utilmacros ] ++ old.nativeBuildInputs;
      });
  });
}

After adding this to nixpkgs.overlays in configuration.nix, nixos-rebuild succesfully builds the package. My issue is that, for some reason, it also rebuilds a bunch of packages that I didn’t touch. Not only that, it doesn’t download the binary version but tries to compile them all locally. And looking at the packages it’s compiling, I’m guessing this would take a long time, which is inconvenient for me. So, why is this happening? Is there a way I can avoid it?

I could also use some feedback on the overlay above or if there’s a better way to do it. I’ve read that there’s a way to add custom XKB layouts to configuration.nix directly and I might end up just doing that if this becomes too complicated, but even then I’d like to get the xkeyboard-config fork working too, at least to learn a bit more about Nix.

Thank you!

2

If you change something through an overlay, than anything that has that as a direct or indirect input, it has to get recompiled, as it’s inputs changed, this is how nix works.

Sadly I’m not sure if there is a less intrusive way to change this package.

3

Thank you for the answer. Turns out there are a few workarounds (that might not be the right Nix way).

First I tried using services.xserver.xkbDir: Making a new xkeyboardconfig-bbkt package with my fork as source and leaving the original untouched. Then setting services.xserver.xkbDir to ${pkgs.xkeyboardconfig-bbkt}/etc/X11/xkb. This did work pretty well, except that I was unable to set the xkb options in configuration.nix because “xkbvalidate” was using the default xkb dir (xkeyboardconfig’s) instead. And even if I did override xkbvalidate too, setxkbmap (which I use to switch between Colemak-DH and qwerty) was also giving me trouble as it also uses the default xkb dir instead of the option given to the server. Then I saw xkbcomp was yet another package that uses xkeyboardconfig’s xkb.

Overriding these packages to use my derivation’s xkb dir (or to use the xkbDir option directly) seemed too excessive to me so I tried looking for something else.

I went back to my original override of xkeyboardconfig and looked at some of the dependencies that were being rebuilt. They all seemed to be caused by depending on libxkbcommon, which uses xkeyboardconfig at build time.
Now, I had used my fork on Arch for a lot of time, with a lot of software and never ran into any issues when using the pre-built libxkbcommon from the repositories, and reading the description on the README, it seems to me that it’s used for key configuration/layout detection by GUI programs, to setup keyboard bindings or something similar. Maybe I’m wrong but it I concluded that the changes introduced by the mods don’t really affect what libxkbcommon is most commonly used for.
So I decided to override xkeyboardconfig for every package except libxkbcommon. This would allow me to easily put my layout config in configuration.nix and use programs like setxkbmap and xkbcomp without any problems, while not having to rebuild every single GUI program.

I accomplish this by adding this to my overlay:

libxkbcommon = super.libxkbcommon.override {
  xkeyboard_config = super.xorg.xkeyboardconfig;
};

This uses xkeyboardconfig from super instead of self, so the derivation before my fork override.

There’s a catch though: libxkbcommon is also used by xkbvalidate to, well, validate the layout. So I’m still unable to configure my layout in configuration.nix! But after thinking about it for a while I realized the following would work:

xkbvalidate = super.xkbvalidate.override {
  libxkbcommon = super.libxkbcommon;
};

It’s using libxkbcommon from super now, which means it uses the derivation before my override that tells it not to use my fork. This means I’m using a new build of libxkbcommon that does use my fork, only to use it for xkbvalidate. Nix really blew my mind when this worked.

So yeah, my layout is working now and I’m kinda proud of the solution I came up with haha.

I’ll probably be staying on NixOS for a while, it’s been really fun so far.

2 Likes
4

Excellent thread! Since I have come from a very similar place (ArchLinux+custom layout) it’s very interesting to me that you ended up with a libxkbcommon setup that is very similar to that of the xorg.extra-layouts module module that I now use. I would say theirs is a bit less convoluted, a bit less double negation-heavy, but I think they are equivalent.

In any case welcome aboard. I see there’s some interest in BBKT in the package requests and it seems your fork is more nix-friendly, so maybe we can work with that…

2 Likes
5

Wow, I completely forgot checking the implementation for extra layouts! I thought it did something more complex haha. I see they ended up going for my first idea of manually overriding everything that depends on xkeyboard-config.

IMHO I think my way of doing it is a bit better. Like you said, it has the same result, but also it overrides less packages (manually) and thus less code has to be written. Besides it’s easier to maintain: if a new package that depends on xkeyboard-config pops up, or if the steps and flags that were overriden change for any of the already overriden packages (both very unlikely but still!), it’s gonna be handled seamlessly by my overlay while it would require manual updating of the existing one. Maybe I should open a PR for this?

Also this explains why I had no issues with libxkbcommon on Arch:

environment.sessionVariables = {
  # runtime override supported by multiple libraries e. g. libxkbcommon
  # https://xkbcommon.org/doc/current/group__include-path.html
  XKB_CONFIG_ROOT = "${pkgs.xkb_patched}/etc/X11/xkb";
};

I’ll be adding that to mine too.

Ooh, yes! This is the exact purpose of my fork. Though I’ve been neglecting it for a while. I’ll fix some missing stuff and try to keep it more up to date with the upstream releases from now on.

I’m just a bit confused about how packaging this would work. Would an overlay like mine be good enough? Where would I add it? I could really use an example of existing packages with alternatives or variations of themselves. I can’t think of any myself

6

good point. I would disagree with that however for two reasons:

  1. The current implementation is designed to minimize the number of packages that the user has to rebuild. (while still maintaining functionality) Hence, it decides which packages to override by inclusion. Since your overlay works by exclusion, it could hit new or existing packages that you don’t know about. In fact, one such package is fcitx5, which does not get rebuilt under the current implementation (I tested) but would if I added your overlay (unless I missed something)

  2. Your current solution relies a lot on overlay magic, in that you have to know for a given package which of self or super is vanilla or not at any point, which is a lot of cognitive load. I got confused by this when reading it. Introducing a new name like the current implementation does (xkb_patched) is much clearer. So these are cool tricks indeed, but for maintainability’s sake it’s better to keep it simple.

In the context of NixOS, there are all of the .package options that do pretty much exactly that. For inclusion in Nixpkgs, I would consider two steps:

  1. Make a xkeyboard-config-bbkt package with proper metadata and all. Piggy back on the vanilla one if possible. It goes on hydra and people can even pull it in custom overlays if they want.
  2. Hack that into the extra-layouts NixOS module with a .package option. I think it’s the right approach, but I might be wrong.

Feel free to ping me if I can help!

1 Like