populateFirmwareCommands usage in a cross-compilation context

Help
1

Hello,

I am trying to cross-build an SD card image to boot a raspberrypi3.
My experiment is available at https://github.com/gautaz/phoenix/tree/raspberrypi and is flake based.

My current build process follows:

❯ nix-shell

phoenix on  raspberrypi [!] via ❄️  impure (phoenix)
❯ phx-build echidna
warning: Git tree '/home/del/github.com/gautaz/phoenix' is dirty
error: builder for '/nix/store/n8xalnwdi24ay0j5km5rakjjky1grhs7-nixos-sd-image-23.11.20231017.e585645-armv7l-linux.img-armv7l-unknown-linux-gnueabihf.drv' failed with exit code 1;
       last 10 log lines:
       > /nix/store/xn9s6irhj753nckqbwvlldikjapx7xma-nixos-sd-image-23.11.20231017.e585645-armv7l-linux.img-armv7l-unknown-linux-gnueabihf/sd-image/nixos-sd-image-23.11.20231017.e585645-armv7l-linux.img1      16384   77823   61440   30M  b W95
       > /nix/store/xn9s6irhj753nckqbwvlldikjapx7xma-nixos-sd-image-23.11.20231017.e585645-armv7l-linux.img-armv7l-unknown-linux-gnueabihf/sd-image/nixos-sd-image-23.11.20231017.e585645-armv7l-linux.img2 *    77824 4138815 4060992  1.9G 83 Linu
       >
       > The partition table has been altered.
       > Syncing disks.
       > 4060992+0 records in
       > 4060992+0 records out
       > 2079227904 bytes (2.1 GB, 1.9 GiB) copied, 5.12259 s, 406 MB/s
       > mkfs.fat 4.2 (2021-01-31)
       > cp: cannot create regular file 'firmware/config.txt': Permission denied
       For full logs, run 'nix log /nix/store/n8xalnwdi24ay0j5km5rakjjky1grhs7-nixos-sd-image-23.11.20231017.e585645-armv7l-linux.img-armv7l-unknown-linux-gnueabihf.drv'.

So the process currently fails during the populateFirmwareCommands step, in particular on https://github.com/gautaz/phoenix/blob/99c92502a35de61a0e9daa8e3b5937248c280400/sd-image-raspberrypi.nix#L26:

cp ${configTxt} firmware/config.txt

The whole process succeeds without populateFirmwareCommands but this will in fact result in an SD card image with an empty firmware partition.

I initially looked at examples like https://github.com/NixOS/nixpkgs/blob/189d2641716be9b862f9619bb3948a7e25f62f41/nixos/modules/installer/sd-card/sd-image-aarch64.nix to grasp what should be the “standard way” to provision the firmware partition but I am not sure this is applicable in a cross-compilation context.

Looking at https://github.com/n8henrie/nixos-btrfs-pi/blob/82df1c61560ece8f95e33ac375468dac38f24a0d/btrfs-sd-image.nix#L298, it seems @n8henrie needs to mount the firmware partition manually.
This feels strange as I would have expected the populateFirmwareCommands to be executed in a context where the firmware partition would already be mounted and available for provisioning.

In fact, looking at the build logs, the partition has been formatted (mkfs.fat 4.2) but does not seem to be mounted afterwards.
Is this missing intentionally?
If so:

If anyone could shed some light on how populateFirmwareCommands should be used in a cross-compilation context, it would save my day :sweat_smile:.

2

Hi!

populateFirmwareCommands is defined in various places but is used here.

As you can see in the several lines beneath that, it populates the firmware files into a local directory which is then copied onto the partition in subsequent steps.

I’m pretty new to nix and this was one of my first projects, so no promises that I’m doing things correctly :slight_smile:

3

Thanks @n8henrie for the insight :+1:.

I did not realize that the call to populateFirmwareCommands was next to the other files I was looking at in nixpkgs.

So basically firmware is a temporary directory created just before the call to populateFirmwareCommands. Once populated by populateFirmwareCommands, all the files are transferred in the real firmware filesystem on the SD card image.

What I do not understand now is why cp ${configTxt} firmware/config.txt fails in populateFirmwareCommands with a Permission denied.
As the firmware directory has been created just before the call to populateFirmwareCommands with obviously the same user, how can this happen?

I have added the following before the cp in populateFirmwareCommands:

pwd
ls -ld ./firmware
ls -la ./firmware
whoami

Which displays this during the build:

> mkfs.fat 4.2 (2021-01-31)
> /build
> drwxr-xr-x 1 nixbld nixbld 0 Oct 29 09:37 ./firmware
> total 0
> drwxr-xr-x 1 nixbld nixbld  0 Oct 29 09:37 .
> drwx------ 1 nixbld nixbld 66 Oct 29 09:37 ..
> nixbld
> cp: cannot create regular file 'firmware/config.txt': Permission denied

So populateFirmwareCommands is executed with the nixbld user and is trying to copy files to the firmware directory, owned by nixbld and nixbld can write to firmware.
I am most probably missing something but this feels awkward.

Investigating further lead me to a way to work around the issue but, still, I don’t understand why the workaround succeeds.

In order to avoid cp to fail with a Permission denied, I am ensuring that ./firmware/config.txt exists first:

> ./firmware/config.txt
cp ${configTxt} firmware/config.txt

No more Permission denied, the build succeeds, this is even awkwarder…

4

Definitely weird. Can you add a set -x at the top so we can double check the commands being run?

And maybe ensure set -e is set, in case there is a failure somewhere above that isn’t being noted? Although I suspect this is already set (since the whole thing fails when the cp fails).

5

Maybe related – aren’t some of these steps done in a VM?

https://github.com/NixOS/nixpkgs/issues/224889

6

I was running into the same issue importing the installer/sd-card/sd-image-aarch64.nix and setting the populateFirmwareCommands variable.

A workaround that worked for me was importing installer/sd-card/sd-image.nix instead, copying the sdImage settings for aarch64 and then just changing the configTxt to suit my needs.