Nixpkgs.overlays not applied in another overlay

Help
1

Hi! first of all, I’m new to nix (I moved my main machine to nixos around 2 months ago), but already enjoying both nix/nixos :).

I’m trying to build nixos for raspberry pi 2, I know armv7 is not supported anymore and I have to build everything from source, but I thought it will be a nice experiment to learn more about nix and extend the life of the pi :).

On my first try building the sdimage, I found out that libxcrypt won’t pass tests in armv7/qemu userspace arm, there is actually an issue about this in their github.

So, I created a patch that disables the test that fails and applied it as an overlay to nixpkgs in the flake I’m making to build the image.

If I build only libxcrypt like, nix build .#nixosConfigurations.rpi2.pkgs.libxcrypt everything works fine, result/lib/libcrypt.so.2.0.0 ends up as an ARM ELF 32-bit.

The problem arises when building the whole image, ie. nix build .#nixosConfigurations.rpi2.config.system.build.sdImage, the build fails because the test I’m removing with the patch is failing, my theory is that the overlay I’m using for the raspberry image, "${nixpkgs}/nixos/modules/installer/sd-card/sd-image-raspberrypi.nix" is instantiating an unpatched version of nixpkgs, but I have no idea how to debug/fix this.

Any help both debugging or fixing this is welcome!

2

Have a close look which libxcrypt build is failing. There are at least two variants in Nixpkgs; overriding one won’t affect the other.

3

Thanks for the fast response!

I think that you refer to libxcrypt and libxcrypt-legacy, right? from the buid logs, I think that the one failing is libxcrypt (error: builder for '/nix/store/1r5wac68j0rsvpj3h2mmkpflf3b5a3ra-libxcrypt-4.4.33.drv' failed with exit code 2;)

Full logs of the build:

$ nix build .#nixosConfigurations.rpi2.config.system.build.sdImage
error: builder for '/nix/store/1r5wac68j0rsvpj3h2mmkpflf3b5a3ra-libxcrypt-4.4.33.drv' failed with exit code 2;
       last 10 log lines:
       > # ERROR: 0
       > ============================================================================
       > See ./test-suite.log
       > Please report to https://github.com/besser82/libxcrypt/issues
       > ============================================================================
       > make[2]: *** [Makefile:2599: test-suite.log] Error 1
       > make[2]: Leaving directory '/build/libxcrypt-4.4.33'
       > make[1]: *** [Makefile:2707: check-TESTS] Error 2
       > make[1]: Leaving directory '/build/libxcrypt-4.4.33'
       > make: *** [Makefile:3237: check-am] Error 2
       For full logs, run 'nix log /nix/store/1r5wac68j0rsvpj3h2mmkpflf3b5a3ra-libxcrypt-4.4.33.drv'.
error: 1 dependencies of derivation '/nix/store/hi3xgnh0r63binpq8wy98p0djzhlba0b-xgcc-12.2.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/i2vas21mpxy883f6lmbmlklvapizjwr8-bootstrap-stage2-gcc-wrapper-12.2.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/qd226szbbvf8q7hdhavpqsw34w8iw2f3-bootstrap-stage3-gcc-wrapper-12.2.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/6g3cdw3b4w9g15hm290yjsdcs695788v-glibc-2.37-8.drv' failed to build
error: 1 dependencies of derivation '/nix/store/gasm7c2ki91sqjxmjw3kb6zi61mh7qya-stdenv-linux.drv' failed to build
error: 1 dependencies of derivation '/nix/store/sq94an1a74v1bh5724s2imfikwr1zv1b-stdenv-linux.drv' failed to build
error: 1 dependencies of derivation '/nix/store/rg5r2n1iya5h28lsjyis6pk0a1l43rsg-linux-6.1.21-1.20230405.drv' failed to build
error (ignored): error: cannot unlink '/tmp/nix-build-strace-armv7l-unknown-linux-gnueabihf-6.4.drv-0/strace-6.4': Directory not empty
error (ignored): error: cannot unlink '/tmp/nix-build-boost-armv7l-unknown-linux-gnueabihf-1.79.0.drv-0/boost_1_79_0/boost': Directory not empty
error (ignored): error: cannot unlink '/tmp/nix-build-perl-armv7l-unknown-linux-gnueabihf-5.36.0.drv-0/perl-5.36.0': Directory not empty
error: 1 dependencies of derivation '/nix/store/k1fsz7wk0hzlksplcgnac319y02schfy-nixos-system-nixos-23.05.20230711.8163a64.drv' failed to build
error: 1 dependencies of derivation '/nix/store/3gnj86xj4y2k4d1qwbd1y63qmpxffak3-ext4-fs.img.zst-armv7l-unknown-linux-gnueabihf.drv' failed to build
error (ignored): error: cannot unlink '/tmp/nix-build-unbound-1.17.1.drv-0/unbound-1.17.1': Directory not empty
error: 1 dependencies of derivation '/nix/store/q6hfh6ni01im0bd2xsjw21ixvi4fbm0d-nixos-sd-image-23.05.20230711.8163a64-armv7l-linux.img-armv7l-unknown-linux-gnueabihf.drv' failed to build
4

Those are just the attr names and those don’t necessarily correspond with the pnames. In this case, libxcrypt and libxcrypt-legacy have the same pname as it’s the same package but with a different set of hashes enabled.

  libxcrypt-legacy = libxcrypt.override { enableHashes = "all"; };

I don’t fully know how overlays play into this override business but overriding libxcrypt should propagate to -legacy. I’d try overriding libxcrypt-legacy too though to make sure.

5

Tried patching also libxcrypt-legacy but that also fails, so I replaced the input of nixpkgs with a local checkout and added the patch there, instead of using an overlay, that worked just fine, but still I don’t know how to figure out why the overlay is not working when I try to build the image.

On a side note, now I don’t know how feasible is to build this in my machine with 16Gb of ram, the build was killed by the OOM killer when it was building llvm… so, the next try will be doing nix build --max-jobs 1 and hope that I have enough ram to build the big dependencies one by one.

6

max-jobs = 1 is a bit overkill and will take forever. 8 ought to be enough (2GiB/cc). I’d also highly recommend zram swap to buffer temporary spikes a little.

7

Hmm, it failed to build xgcc because it couldn’t allocate memory even using max-jobs = 1, maybe it is because I don’t have a lot of swap, only 8Gb. Thanks to remind me about zram, I totally forgot to setup that, will try to build xgcc again with it enabled.

8

xgcc should never allocate anywhere near 16GB RAM. There’s something else going on here. Monitor RAM usage and which processes are using it.

9

Actually, that’s true, is not allocating anything near 16Gb. I just realized that it is building xgcc using qemu-arm and not cross-compiling, so it might be related to the address space available in the 32 bit emulation.

What is weird is the error, cc1plus: out of memory allocating 67424184 bytes after a total of 152682496 bytes, same numbers in several tries.

10

So you’ve set up arm emulation via binfmt? Try configuring your Pi system to cross-compile from x86_64 instead.

11

I have arm emulation via binfmt, but unrelated to this build (I’m using it to debug arm docker images).

I thought I was already using cross-compile, that’s why it was a surprise seeing qemu-arm in ps, this is how my nixosSystem looks:

rpi2 = nixpkgs.lib.nixosSystem {
  system = "x86_64-linux";
  modules = [
  "${nixpkgs}/nixos/modules/installer/sd-card/sd-image-raspberrypi.nix"
  {
    nixpkgs.crossSystem.system = "armv7l-linux";
    ...
  }];
  ...
}
12

Disabling binfmt I get:

error: a 'armv7l-linux' with features {} is required to build '/nix/store/8c85knhag7wc6l7zirw4pdsrgzxhsdwl-xgcc-12.2.0.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}

But then, if I do nix build .#nixosConfigurations.rpi2.pkgs.hello it builds and the binary is for the correct architecture:

$ file result/bin/hello
result/bin/hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /nix/store/1nydrvf5vv1p2pfnzydxssmm5bahm8mq-glibc-armv7l-unknown-linux-gnueabihf-2.37-8/lib/ld-linux-armhf.so.3, for GNU/Linux 3.10.0, not stripped

So, might be that xgcc needs binfmt because it is actually the bootstrap compiler to build gcc in the end? if I understand correctly, gcc is built in 3 stages, xgcc, which is used to build gcc and with that gcc rebuild itself to get the final artifact.

13

Untested but try:

{
  nixpkgs.hostPlatform = lib.systems.examples.armv7l-hf-multiplatform;
  nixpkgs.buildPlatform = lib.systems.elaborate "x86_64-linux";
}

These are the options you want to work with, I’m not sure the second one can be set like that.

14

Cross never needs to execute the host platform’s code. That’s the entire point of cross.

15

Finally, I figured out what was going on! I was replacing the kernel using the wrong nixpkgs, one that didn’t set the cross config or patches for libxcrypt (also, I don’t need those patches, because is not affected while cross compiling instead of using qemu-arm).

So, the culprit on my config was:

boot.kernelPackages = nixpkgs.lib.mkForce nixpkgs.legacyPackages.armv7l-linux.linuxPackages_rpi2;

Besides that ended up replacing sd-image-raspberrypi.nix with sd-image-armv7l-multiplatform.nix, since the raspberrypi one has uboot configuration for armv6l (the first version of the pi) and nix build was complaining about an unsupported platform.

So, finally, the build worked and after dding into the sdcard, it booted! thanks @Atemu for all the help!

Here is the gist with the flake that works.

1 Like
16

Note that this issue would easily be side-stepped if you just used pkgs from the module’s arguments. This whole flake business introduces some really nasty confusion and IMO configuring everything in the flake.nix file is a huge anti-pattern.

Usually, modules are a function with parameters such as { pkgs, lib, config, ...}: { ... }. Traditionally, they sit in individual files and that’s how NixOS is structured internally.

This is how I’d recommend you do it:

{ pkgs, ... }: {
   system.stateVersion = "23.05";
   boot.kernelPackages = pkgs.linuxPackages_rpi2;
}

Ideally put this into its own file and add that file to the NixOS module imports so that you can’t even think about using flake-specific parts in the wrong places.

2 Likes
17

Yes, that makes sense, I’m actually doing that on my laptop’s config, but started this setup isolated and the idea was to start simple and then split the module into a different file.

I’ll keep in mind to always use modules as a function and passing arguments instead of using stuff from the outer scope. Thanks a lot!

1 Like