How to add extra build input to linux kernel?

Help
1

I am trying to build a lz4-compressed kernel for faster boot, at very first I did it by

boot.kernelPatches = [ {
    name = "lz4 compressed";
    patch = null;
    extraConfig = ''
         KERNEL_LZ4 y
    '';
} ];

But this won’t work, as lz4 is not a default build input of the kernel. So next I tried adding:

boot.kernelPackages = with pkgs; linuxPackages_5_6 // {
    kernel = linux_5_6.overrideAttrs (old: {
        nativeBuildInputs = old.nativeBuildInputs ++ [ lz4 ];
    });
};

This was inspired by the make menuconfig trick on the wiki. However the kernel still ended up failing to build because of missing lz4 (and I haven’t figured out why yet). Next, I tried replacing the above with:

boot.kernelPackages = with pkgs; recurseIntoAttrs (linuxPackagesFor (
    linux_5_6.overrideAttrs (...)
));

This time, I got the error that some kernel configs essential to NixOS, like CONFIG_DEVTMPFS and CONFIG_CGROUPS , are not set. I tried removing the kernel patch for extra config, and the problem remains. So it is caused by overriding the kernel’s build inputs. My explanation is that overriding build inputs changes the kernel’s derivation and somewhat makes it failed to find the correct config file in /nix/store.

Finally, I tried overriding linux_5_6 directly in nixpkgs, but no luck. The same “essential config not set” error remains.

Of course, manually supply a config file the same as NixOS’s defalut can solve the problem. But kernel config varies from version to version, and I favor Nix’s approach of extraConfig a lot, as I can alter kernel config freely, without the need to scan-through all config changes on kernel update.

So my question is: how can one add build inputs to the linux kernel in NixOS, while keeps using the config generated by NixOS (perhaps with patches later), not having to manually supply a config file?

Thanks for any suggestions in advance ; )

1 Like
2

Hi, I don’t have an answer to your question. Can I ask you a question? why do you believe it would be faster? I can guess faster load into memory but that would be offset by perhpas orders of magnitude more time used in decompression in my knowledge.

3

Generally, disk, NVMe, eMMC, SPI Flash etc. are all so much slower than your CPU that loading something uncompressed is much slower that loading a compressed thing and then decompressing. decompression is usually possible to do in parallel, when the storage may be queried asynchronously and the algorithm may work with partial data or in a streaming fashion. Modern storage is usually accessed asynchronously and LZ4 may be streamed.

For example, loading a LZ4 compressed kernel from NVMe is faster than an uncompressed kernel on my single Cortex-A53 @ 1.4GHz by a factor that is almost the same as the compression factor. That is, in my testing, you can’t easily detect any latency added by compression.

Still, I encourage you to do your own testing, as you may have a setup that does not benefit from compression (off hand, I can’t think of anything like that). Be warned that Linux normally keeps everything in RAM in a cache of your filesystem because it’s so much faster for accessing, so any testing would have to include flushing this cache.

TL;DR: modern decompression algorithms are orders of magnitude faster than non-volatile memory today.

2 Likes
4

Thank you for answering. Typically what are the kernel load times? (I get your point about the benefit seen from lz4 compression). Also, I wonder if something like this exists for elf binaries, seeing as only only the loader will have to change

5

I have seen 2-4 seconds for fast storage systems, like the aforementioned NVMe + Cortex-A53 setup. For booted operating systems, You can enable full-disk compression for similar benefits. Modern filesystems such as btrfs and zfs support these features out of the box. Perhaps that would get the speedup you’re after.

Further, when you’re running an elf for the second time, you’re likely to have it cached in RAM from the previous run, unless it has been paged out.

Also, there is a swap-space-like-thing based on the principal that compression is much much faster than disk called zram or zswap, which can further reduce the chance to do a round trip to the disk.

1 Like
6

I am actually moving from gzip to lz4… When I scanned through make menuconfig, I didn’t manage to find the option of uncompressed (or sometimes referred to as “cat compressed”) kernel, and among those algorithms supported by the kernel, lz4 is the fastest.
When I moved from gzip’ed initrd to lz4, I actually observed a 10% improvement in kernel boot time, according to systemd-analyze.

7

The reason why overrideAttrs doesn’t quite work here is because of this:

Here extendDerivation adds features and commonStructuredConfig to all the derivation outputs, but since overrideAttrs is done in stdenv.mkDerivation, the result won’t include the extra attributes.

One way to get around this is to use overrideDerivation instead, like this:

{ lib, pkgs, ... }:

{
  boot.kernelPackages = pkgs.linuxPackages.extend (lib.const (super: {
    kernel = super.kernel.overrideDerivation (drv: {
      nativeBuildInputs = (drv.nativeBuildInputs or []) ++ [ pkgs.lz4 ];
    });
  }));
}

The reason why this works, is because the latter is applied in callPackage instead.

Another issue in your above approach however is that the config doesn’t actually validate, since by default KERNEL_XZ is enabled, which in turn conflicts with the choice type option. So to cope with this you also need to disable KERNEL_XZ.

So in summary, the full config would look something like this (untested):

{ lib, pkgs, ... }:

{
  boot.kernelPatches = lib.singleton {
    name = "lz4 compressed";
    patch = null;
    extraConfig = ''
      KERNEL_XZ n
      KERNEL_LZ4 y
    '';
  };
  boot.kernelPackages = pkgs.linuxPackages.extend (lib.const (super: {
    kernel = super.kernel.overrideDerivation (drv: {
      nativeBuildInputs = (drv.nativeBuildInputs or []) ++ [ pkgs.lz4 ];
    });
  }));
}

Looking at the Kconfig file the name should be KERNEL_UNCOMPRESSED.

1 Like
8

Yep, it works now. Thanks a lot.

I am not sure if I have understand it correctly. So overrideAttrs actually overrides the attribute set supplied to mkDerivation. But kernel expressions in nixpkgs are formed in a more complicated way, in that they are derivations created by mkDerivation extended with some other stuffs later. So calling overrideAttrs on kernel expressions can only override the part from the mkDerivation call and can not affect those parts extended afterwards. And to alter those extended parts, overrideDerivation, which can directly modify the final derivation, is needed. Am I right?

As for KERNEL_XZ, actually I have turned it off, and my actual kernel patch has a lot more things, not mentioned for simplicity. But good catch anyway ; )

9

Correct.

The issue here is not overrideAttrs itself but at which point it’s applied. To illustrate this in a very simplified way:

let
  myDerivation = rec {
    attrs = { aaa = 456; bbb = 789; };
    overrideAttrs = f: attrs // f attrs;
  };
  extendDerivation = drv: drv // { foo = 111; bar = 222; };
  callPackage = pkg: pkg // {
    overrideDerivation = f: pkg // f pkg;
  };
in callPackage (extendDerivation myDerivation)

Here is the difference between overrideDerivation and overrideAttrs from the above:

> overrideAttrs (_: { ccc = 666; })
{ aaa = 456; bbb = 789; ccc = 666; }

> overrideDerivation (_: { ccc = 666; })
{ attrs = { ... }; bar = 222; ccc = 666; foo = 111; overrideAttrs = «lambda»; }

Of course, the real overrideAttrs/overrideDerivation/callPackage functions have vastly different implementations, but the above is just to make the issue bit more obvious in a simplified example.

Yep - as mentioned - this is because the overrideDerivation attribute is added in callPackage outside of generic.nix:

https://github.com/NixOS/nixpkgs/blob/98c9ae41de70a724517f64c0a39d4c5f68373a7c/pkgs/top-level/all-packages.nix#L17175-17181

1 Like
10

Oh I see. The extensibility is not builtin magic, but something implemented inside utilities like mkDerivation and callPackage (In your example, you showed how to implement support of overrideAttrs manually, if I am correct). Thanks for your explanation!