Hello all,
I am trying to set up NixOS on a raspberry pi 4b with the mainline kernel booting from a USB. However, every single time, I get a message like “Timed out waiting for device…” or that relevant systemd mounting service failed. There’s no /dev/sd* in emergency shell, and I see in journalctl -xb that various USB devices are loaded from the hub. I’ve already ensured my EEPROM firmware is updated.
At the moment, I am trying to use U-Boot with systemd-boot, which is directly outputs an image file that is written to a USB stick. This gets me booting to initrd just fine. I’ve tried using the vendor kernel and systemd-boot successfully mounts the root partition, but the login prompt doesn’t show up. I’m not sure why it doesn’t finish successfully booting, but the point is that systemd-boot can see the USB drive with the vendor kernel unlike the mainline kernel. Thus, I must be missing some kernel module or device tree overlay that vendor kernel has that mainline doesn’t.
I’ve tried a lot of different modules, but general consensus seems to be that the kernel modules defined in nixos-hardware (usbhid, usb-storage, vc4, pcie-brcmstb, reset-raspberrypi) are enough. I don’t need the uas module since the storage device is a USB plugged into the raspberry pi directly and is not an enclosure.
I’ve spent a lot of time on this, but it seems that I shouldn’t be running into such problems (e.g., I had the same config as Eisfunke and was using the same UEFI firmware, but still ran into all the issues listed above, leading me to this config). I would appreciate any assistance!
Here are the relevant parts of my config:
{inputs, pkgs, lib, modulesPath, ...}: {
imports = [
inputs.nixos-hardware.nixosModules.raspberry-pi-4
"${modulesPath}/profiles/minimal.nix"
];
boot = {
loader = {
systemd-boot.enable = true;
generic-extlinux-compatible.enable = false;
efi.canTouchEfiVariables = false;
};
initrd.systemd = {
enable = true;
emergencyAccess = true;
}
kernelPackages = lib.mkForce pkgs.linuxPackages_latest;
blacklistedKernelModules = [
# Disable wifi
"brcmfmac"
"brcmutil"
# Disable bluetooth
"btbmc"
"hci_uart"
];
};
console.enable = false;
environment.systemPackages = with pkgs; [
libraspberrypi
raspberrypi-eeprom
];
hardware.deviceTree = let
# Using pinned firmware or else my HDMI doesn't work
# See comment at https://git.eisfunke.com/config/nixos/-/blob/90b677e694dfccc079ce8d4d40b931082c9639d8/devices/amethyst.nix#L68
# All CPUs come up in `journalctl -xb` without the mentioned overlay for me
raspberrypifw = pkgs.raspberrypifw.overrideAttrs {
version = "pinned-2023.05.12";
src = pkgs.fetchFromGitHub {
owner = "raspberrypi";
repo = "firmware";
rev = "b49983637106e5fb33e2ae60d8c15a53187541e4";
hash = "sha256-Ia+pUTl5MlFoYT4TrdAry0DsoBrs13rfTYx2vaxoKMw=";
};
};
in {
enable = true;
dtbSource = pkgs.device-tree_rpi.override {inherit raspberrypifw;};
name = "broadcom/bcm2711-rpi-4-b.dtb";
overlays = let
upstreamOverlay = name: raspberrypifw + /share/raspberrypi/boot/overlays/${name}.dtbo;
in [
{
name = "upstream-pi4";
dtboFile = upstreamOverlay "upstream-pi4";
}
];
};
facter.reportPath = ./facter.json;
powerManagement.cpuFreqGovernor = "ondemand";
services.jitterentropy-rngd.enable = true;
boot.kernelModules = ["jitterentropy_rng"];
}
and I am generating the sd image used the repart module:
# Reference: https://github.com/MatthewCroughan/raspberrypi-nixos-example/blob/a4d2595b0839a7e7f1bc297dc9f006042d1d958d/repart.nix
{lib, config, pkgs, modulesPath, ...}: let
efiArch = pkgs.stdenv.hostPlatform.efiArch;
configTxt = pkgs.writeText "config.txt" ''
[pi4]
kernel=u-boot.bin
enable_gic=1
armstub=armstub8-gic.bin
disable_overscan=1
arm_boost=1
[all]
arm_64bit=1
enable_uart=1
avoid_warnings=1
'';
in {
imports = [
"${modulesPath}/image/repart.nix"
];
systemd.repart.enable = true;
systemd.repart.partitions."01-root".Type = "root";
boot.initrd.systemd.root = "gpt-auto";
boot.initrd.supportedFilesystems.ext4 = true;
image.repart = {
name = "image";
partitions = {
"01-esp" = {
contents = {
"/EFI/BOOT/BOOT${lib.toUpper efiArch}.EFI".source = "${pkgs.systemd}/lib/systemd/boot/efi/systemd-boot${efiArch}.efi";
"/EFI/Linux/${config.system.boot.loader.ukiFile}".source = "${config.system.build.uki}/${config.system.boot.loader.ukiFile}";
"/u-boot.bin".source = "${pkgs.ubootRaspberryPi4_64bit}/u-boot.bin";
"/armstub8-gic.bin".source = "${pkgs.raspberrypi-armstubs}/armstub8-gic.bin";
"/config.txt".source = configTxt;
"/".source = "${pkgs.raspberrypifw}/share/raspberrypi/boot";
};
repartConfig = {
Type = "esp";
Format = "vfat";
Label = "ESP";
SizeMinBytes = "512M";
};
};
"02-root" = {
storePaths = [config.system.build.toplevel];
repartConfig = {
Type = "root";
Format = "ext4";
Label = "nixos";
Minimize = "guess";
GrowFileSystem = true;
};
};
};
};
}
Additional notes:
- I’ve thrown together the above snippets from my actual configuration, please let me know if there’s any mistakes, as I haven’t had any major problems with evaluating nor building. For example, I’ve excluded my systemd-network configuration
- I also have a wireguard configuration in systemd-network and a systemd service for mounting at
multi-user.target, but I don’t think either of these contribute to my issue - Perhaps the issue lies in whether the device tree is from systemd-boot (installed via
boot.loader.systemd-boot.installDeviceTreesincehardware.deviecTree.nameis not null) or from U-Boot (firmware in ESP partition), but I don’t think that’s the issue since the HDMI doesn’t work without pinningraspberrypifwand overridinghardware.deviceTree.dtboSourcewith the pinned version, so systemd-boot device tree must be the one being used? (I’m not too well-versed in the boot process besides what I’ve learned from this week)
EDIT: I just realized that the image building process is using raspberrypifw from nixpkgs, but as stated above, the HDMI functionality is sensitive to the pinned version of raspberrypifw, so I don’t think this is the source of the problem. Plus, I’ve experienced this with the other UEFI firmware as well.