After a couple of trial and error with zfs and a few other things the above flake successfully builds an image and I can dd it to the sd card. However the raspberry pi zero w does not boot and instead fails with File not found /boot/extlinux/../nixos/[...]/bcm2835-rpi-zero-w.dtb and now is stuck at Starting kernel ....
After reading this issue on GitHub, I tried to just copy the bcm2708-rpi-zero-w.dtb to bcm2835-rpi-zero-w.dtb as I read that they are somehow similar, but that results in a kernel panic.
I then found that there is a correct dts file for the bcm2835 in the uboot repository, but I haven’t found any guide on how to use them and get the dtb file.
Any ideas how I can get a nixos system onto my raspberry pi zero w?
What exactly do you mean with customize? Shouldn’t the overlays be actually defined in the correct dts file?
Filtering for bcm2835* results in the following error, however filtering for bcm2708* works:
error: builder for '/nix/store/xjmixw4i2avv0fabi0vd6yhhihvhvfd9-dtbs-filtered.drv' failed with exit code 123;
last 2 log lines:
> cp: missing file operand
> Try 'cp --help' for more information.
For full logs, run 'nix log /nix/store/xjmixw4i2avv0fabi0vd6yhhihvhvfd9-dtbs-filtered.drv'.
error: 1 dependencies of derivation '/nix/store/bnplzdzkh16pas8mlqi5zy9xh6z7mfb7-nixos-system-nixos-23.11.20240105.c1be43e.drv' failed to build
error: 1 dependencies of derivation '/nix/store/cy0c0w6gz6x34b4pfyscxr08d5znccf6-ext4-fs.img.zst-armv7l-unknown-linux-gnueabihf.drv' failed to build
error: 1 dependencies of derivation '/nix/store/4xwhhyrjmn1xw4bypi5l4j507w58232b-nixos-sd-image-23.11.20240105.c1be43e-armv7l-linux.img-armv7l-unknown-linux-gnueabihf.drv' failed to build
I now also found out, that in the official raspberry pi os there is only a bcm2708[...].dtb and nothings starts with bcm2835, so I suspect that the bcm2708 should work but is not used by uboot with that SoC?
But what is interesting is, that there is no bcm2708 in the uboot repo, but only a bcm2835 and my image only contains a bcm2708. So is nixos using something else?
And then I also found this line which seems to do exactly that copy I had to do manualy. Why is that not performed automatically?
I have the original rpi zero w since the nixos system should later be really minimal and just run one service. And you are right about the armv6l, just checked that with uname -a running Raspberry pi os. But however changing the host platform to armv6l-linux (everything else is just like the flake in the initial post) results in the following error when building which I am not able to solve because I don’t find anything about it on the internet.
error: builder for '/nix/store/jv5h2678lrh4mqj267s85h8zrxc891ng-compiler-rt-libc-armv6l-unknown-linux-gnueabihf-16.0.6.drv' failed with exit code 1;
last 10 log lines:
> inlined from '__sanitizer::uptr* __sanitizer::StackStore::BlockInfo::GetOrUnpack(__sanitizer::StackStore*)' at /tmp/nix-build-compiler-rt-libc-armv6l-unknown-linux-gnueabihf-16.0.6.drv-0/compiler-rt-src-16.0.6/compiler-rt/lib/sanitizer_common/sanitizer_stack_store.cpp:283:35:
> /tmp/nix-build-compiler-rt-libc-armv6l-unknown-linux-gnueabihf-16.0.6.drv-0/compiler-rt-src-16.0.6/compiler-rt/lib/sanitizer_common/sanitizer_stack_store.cpp:192:17: warning: 'diff' may be used uninitialized [-Wmaybe-uninitialized]
> 192 | previous += diff;
> | ^~~~
> /tmp/nix-build-compiler-rt-libc-armv6l-unknown-linux-gnueabihf-16.0.6.drv-0/compiler-rt-src-16.0.6/compiler-rt/lib/sanitizer_common/sanitizer_stack_store.cpp: In member function '__sanitizer::uptr* __sanitizer::StackStore::BlockInfo::GetOrUnpack(__sanitizer::StackStore*)':
> /tmp/nix-build-compiler-rt-libc-armv6l-unknown-linux-gnueabihf-16.0.6.drv-0/compiler-rt-src-16.0.6/compiler-rt/lib/sanitizer_common/sanitizer_stack_store.cpp:190:10: note: 'diff' was declared here
> 190 | sptr diff;
> | ^~~~
> [327/666] Building CXX object lib/sanitizer_common/CMakeFiles/RTSanitizerCommonNoHooks.armhf.dir/sanitizer_thread_registry.cpp.o
> ninja: build stopped: subcommand failed.
For full logs, run 'nix log /nix/store/jv5h2678lrh4mqj267s85h8zrxc891ng-compiler-rt-libc-armv6l-unknown-linux-gnueabihf-16.0.6.drv'
You’re fortunate: I’ve just finished building an installer image of 23.11 for the Raspberry Pi B rev 2 (armv6-linux). Pretty much everything still works out-of-the-box (because of various efforts last year).
As you’ve figured already, there a couple of minor issues. The main one being compiler-rt that has accumulated a few regressions on ARM. Versions up to 14 have been patched in nixpkgs, version 15+ have a partial fix and haven’t been patched but effectively still don’t build. You can simply set llvmPackages = super.llvmPackages_14 to work around it.
Then, I’ve noticed a link problem in cmake, I haven’t figured out the root cause but you can fix it with cmake = super.cmake.overrideAttrs (old: { env.NIX_CFLAGS_COMPILE = "-latomic"; });.
That’s pretty much it if you’re building a headless system.
Oh, instead of patching that function to ignore missing modules, do this:
Hello, sorry for that late response, I had a lot todo the last days so I hadn’t a chance until now to test it. Thank you very much for your help, with those changes added building the image and booting now works. However I cannot use the nix commands, because I get this error for nix-shell -p hello:
Nothing wrong really, you just need to run nix-channel --update to download the channel.
Alternatively, you could import ${modulesPath}/installer/cd-dvd/channel.nix, which will copy the channel into the image, so you don’t need to download it later.
/nix/store/9daif9c2pmwa7hwnzrazkxi6nykrl0v3-extlinux-conf-builder.sh: line 138: cd: /nix/var/nix/profiles: No such file or directory
Preparing store paths for image...
Creating an EXT4 image of 1744146432 bytes (numInodes=107152, numDataBlocks=211513)
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: done
Creating filesystem with 425817 4k blocks and 106496 inodes
Filesystem UUID: 44444444-4444-4444-8888-888888888888
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Copying files into the device: ext2fs_symlink: Could not allocate inode in ext2 filesystem while creating symlink "coreutils"
__populate_fs: Could not allocate inode in ext2 filesystem while writing symlink"coreutils"
mkfs.ext4: Could not allocate inode in ext2 filesystem while populating file system
However running nix-channel --update works, but it takes a while. Then I’am able to run the nix-shell -p hello command, but it builds a lot of things from scratch because there are a lot of gcc processes shown when I run top in a second tty.
So I’m now asking me a few things:
how can I fix the channel error from above by importing it directly into the image to speed up things?
how can I include some packages like hello or python already when I build the image so that they don’t need to be compiled on the rpi itself?
If I include things like users, service configurations, … in the sd-image build flake, is that included later in the sd-image, and where do I find the nix configuration file defining that in the sd image? (I couldn’t find anything like that in /etc/nix or /etc/nixos). And is that even necessary when I’m not planning on rebuilding the system on the device directly?
Sorry if those questions sound easy, I am currently trying to get my first nixos system running. And again, thank you really much for your great help, really appreciate it.
I’m not sure: it may be that this module is slightly incompatible with the sd image builder. It’s normally used for the installer image.
but it builds a lot of things from scratch because there are a lot of gcc processes
That’s unfortunately expected: there is no binary cache at all for armv6, so any package not included in the image will have to be built.
how can I fix the channel error from above by importing it directly into the image to speed up things?
No idea, sorry.
how can I include some packages like hello or python already when I build the image so that they don’t need to be compiled on the rpi itself?
Just add them to environment.systemPackages, like you normally would for NixOS.
If I include things like users, service configurations, … in the sd-image build flake, is that included later in the sd-image
Yes, you’re essentially building a normal system (in the /run/current-system directory sense), but the result and all its dependencies are collected into a compressed archive (the image).
and where do I find the nix configuration file defining that in the sd image? (I couldn’t find anything like that in /etc/nix or /etc/nixos). And is that even necessary when I’m not planning on rebuilding the system on the device directly?
No, it’s not necessary if you don’t plan on editing the configuration later (and running nixos-rebuild).
What I do is: I write a typical configuration.nix file, import the image builder module, build the image, flash unto the SD card and plug it in the raspberry pi. On the first boot the image will even auto-expand to fill the whole card, so it will look like a normal install.
Thank you very much for your help and being patient with me. Installing packages like python through environment.systemPackages already works now. I also think I understood a bit further how the sd image and the configuration.nix works.
I’m now trying to get i2c working, so first I added the i2c-tools package, build and bootet, but the command i2c-detect -y 1 now shows me this error (also tested with sudo):
Error: Could not open file `/dev/i2c-1' or `/dev/i2c/1': No such file or directory"
So I thought that maybe i2c is not enabled.
First I tried adding this device tree overlay to my flake, but I still get the same error:
Thank you very much for your help and being patient with me. Installing packages like python through environment.systemPackages already works now. I also think I understood a bit further how the sd image and the configuration.nix works.
No problem, I’m glad to be of help.
Do you got i2c working on your model B or have any idea what I could try next?
I have never really tried i2c. I used SPI to flash chips but that was some time back, so I don’t remember whether I was using NixOS or alarm.
IIRC the dtparam options won’t work because they are a feature of the raspberry pi bootloader, but we are using uboot. So, the right approach is to use a device tree overlay, as in your first attempt.
I gave a quick look at the dtb (if you build linuxPackages_rpi1.kernel, you can inspect the tree with nix run -f '<nixpkgs>' dtc result/dtbs/bcm2835-rpi-zero-w.dtb) and I think something like this should work:
Thanks for your help. Trying your provided device tree doesn’t work. Reading the pin assignments from the bcm2835 datasheet it seems like i2c1 should be right one, because the pins 2,3 (these are what I’m using, already tested with raspberry pi os) are named SDA1 and SCL1. GPIO0 and 1 have the SDA0 and SCL0 label. But either way, shouldn’t there be a /dev/i2c* file or directory which I never saw with nixos.
What syntax is that dts actually? It looks really weird to me. And where do you have that &i2c1 thing from? Why is that outside of the compatible scope? I can’t find something like that on the internet.
I now played a bit further with the dts by actually extracting the dtb files from the second partition and converting them to dts files using nix run -f '<nixpkgs>' dtc m/boot/nixos/<some-hash>-device-tree-overlays/bcm2835-rpi-zero-w.dtb > current.dts and found out that:
if I use your provided device tree overlay there is no diff in the dts file to just using nothing.
if I use the device tree overlay (I now found the source again, I just used the one from the rpi4 on the nixpgs and adjusted the chip) from my first post about it (post #16), I get the following diff compared to just adding nothing. But this still not works:
I then tried to manually modify the dtb file in the image by mounting the image, converting it to a dts file, applying the following patch and compiling it back with the following command. Then I moved it into the image, unmounted and tried to boot it. I then also verified that uboot showed the loading of the correct .dtb file on boot.
What syntax is that dts actually? It looks really weird to me.
It seems to be a syntactic sugar12 that has been available for a while, but few people have picked it up. Anyway, this is what the kernel docs tell you to write, now.
And where do you have that &i2c1 thing from?
I took the label from the base device tree: you can dump it as I said before.
Why is that outside of the compatible scope? I can’t find something like that on the internet.
I suppose it’s to avoid repeating it for every property being changes. I can
assure you it’s correct: i have several overlays written like this (enabling
audio, gpu, etc.)
I don’t remember where I first learned about overlays, probably from someone in #nixos-on-arm:nixos.org. You should try asking there, because I think this is as far as I can go.
But also that way, there is no /dev/i2c* so the i2cdetect command fails. Do you have any idea how I can get that to work?
I don’t know, AFAIK hardware.i2c.enable = true is all you should need to get the i2c devices set up.
