Nixos-rebuild always fails on ARMv7l (It's not the .xz bug)

I’m trying to setup my Raspberry Pi 3 with NixOS. I’ve read the WiKi page about it and I think I understand the overall process, thankfully it’s rather simple. However, I’m having trouble with the nixos-rebuild switch --upgrade command I’m supposed to run after editing /etc/nixos/configuration.nix. Thanks to the outstanding .xz decompression Nix bug, @dezgeg’s image still doesn’t work but @qolii’s image does - the WiKi is accurate.

I’ve had 2 nixos-rebuild attempts which failed differently. It was very hard to record the errors it spit, since I ran this inside the native tty (can’t use UART, never mind…).

Both failures occurred during the build phase of GCC which nix somehow decided it needs to build by himself. Other packages which required an update were either compiled locally fine or retrieved from a cache. The full command I ran was:

nixos-rebuild switch --fast --option binary-caches 'https://nixos-arm.dezgeg.me/channel' --option binary-cache-public-keys 'nixos-arm.dezgeg.me-1:xBaUKS3n17BZPKeyxL4JfbTqECsT+ysbDJz29kLFRW0=%'

As proposed in this section of the WiKi page.

The 1st failure of nixos-rebuild was probably due to an overheat of the CPU and so I didn’t record it.

The 2nd attempt however, failed with the following error:

rm -f include/stdint.h
if [ wrap = wrap ]; then 
  rm -f include/stdint-gcc.h; 
  cp ../../gcc-7.4.0/gcc/ginclude/stdint-gcc.h include/stdint-gcc.h; 
  chmod a+r include/stdint-gcc.h; 
  cp ../../gcc-7.4.0/gcc/ginclude/stdint-wrap.h include/stdint.h; 
  chmod a+r include/stdint.h; 
elif [ wrap = provide ]; then 
  cp ../../gcc-7.4.0/gcc/ginclude/stdint-gcc.h include/stdint.h; 
  chmod a+r include/stdint.h; 
fi
set -e; for ml in `cat fixinc_list`; do 
sysroot_headers_suffix=`echo ${ml} | sed -e 's/;.*$//'`; 
multi_dir=`echo ${ml} | sed -e 's/^[^;]*;//'`; 
fix_dir=include-fixed${multi_dir}; 
if true ; then 
  cat ../../gcc-7.4.0/gcc/limitx.h ../../gcc-7.4.0/gcc/glimits.h ../../gcc-7.4.0/gcc/limity.h > tmp-xlimits.h; 
else 
  cat ../../gcc-7.4.0/gcc/glimits.h > tmp-xlimits.h; 
fi; 
/nix/store/65sxp86ypi3ca914pq069blazwq9qzr1-bootstrap-tools/bin/bash ../../gcc-7.4.0/gcc/../mkinstalldirs ${fix_dir}; 
chmod a+rx ${fix_dir} || true; 
/nix/store/65sxp86ypi3ca914pq069blazwq9qzr1-bootstrap-tools/bin/bash ../../gcc-7.4.0/gcc/../move-if-change 
tmp-xlimits.h  tmp-limits.h; 
rm -f ${fix_dir}/limits.h; 
cp -p tmp-limits.h ${fix_dir}/limits.h; 
chmod a+r ${fix_dir}/limits.h; 
done
rm -f include-fixed/README
cp ../../gcc-7.4.0/gcc/../fixincludes/README-fixinc include-fixed/README
chmod a+r include-fixed/README
echo timestamp > stmp-int-hdrs
/build/build/./gcc/xgcc -B/build/build/./gcc/ -nostdinc -x c /dev/null -S -o /dev/null -fself-test=../../gcc-7.4.0/gcc/testsuite/selftests
compilation terminated due to -fmax-errors=3070189568.
make[3]: *** [Makefile:1933: s-selftest] Error 1
make[3]: *** Waiting for unfinished jobs....
rm gcc.pod
make[3]: Leaving directory '/build/build/gcc'
make[2]: *** [Makefile:4584: all-stage2-gcc] Error 2
make[2]: Leaving directory '/build/build'
make[1]: *** [Makefile:21366: stage2-bubble] Error 2
make[1]: Leaving directory '/build/build'
make: *** [Makefile:21570: bootstrap] Error 2
builder for '/nix/store/9riiqqd5zwb9nz2brdxd775c0513xl6m-gcc-7.4.0.drv' failed with exit code 2
cannot build derivation '/nix/store/4jmdjhkpbadyjh5h6dpfdwv2qdqj5v5a-bootstrap-stage4-gcc-wrapper-7.4.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/m44xjf4b39ym1ckbqfk0s2bvimnfbpwh-gcc-wrapper-7.4.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/c18dr6xznmlmjr27p5mn49rwvz3lipxv-perl-5.28.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/inkfxkv8qf9in6ww3bxkdjaj3r73a5rf-perl-5.28.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/inmrk391y06kdidn71b6zg65pnx57fm7-perl-5.28.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/123w8ip31552d1kliahk7lzh2029zi8a-stdenv-linux.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/rps3r6mdp6yfy3kflgzirl0r9jbrx2jn-stdenv-linux.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/icy8mgk40nkvl7zydlyfhd9z3vfwr2sq-nixos-system-NIXPI-19.09pre186563.b5f5c97f7d6.drv': 1 dependencies couldn't be built
error: build of '/nix/store/icy8mgk40nkvl7zydlyfhd9z3vfwr2sq-nixos-system-NIXPI-19.09pre186563.b5f5c97f7d6.drv' failed

I’ve found no clue as for this -fmax-errors error… Could it be a general problem with the build that occurs only on an armv7l machine?

Generally speaking, the problem is that there seems no binary cache which contains an updated GCC derivation. I hope I can get @dezgeg’s attention by mentioning him and perhaps learn why did he choose only a subset of packages and not include GCC there…

I can’t describe to you how frustrating it was to find out after 3-4 hours of the GCC build on my pour Raspberry Pi, that it failed and that I have no idea what is this error all about…

If I could only get this derivation precompiled, I will be happy.

Are there any alternative binary cache I can use? I read this announcement and I couldn’t understand exactly what does it mean for me as an armv7l end-user… The WiKi says there’s a binary cache by @clever, and a arm cachix cache (announcement thread).

Is there a way to specify nixos-rebuild to use all of those binary caches together? This way I could verify that none of them helps me…

2 Likes

I had the same issue and tried the arm64 image from hydra. Worked like a charm.

https://hydra.nixos.org/job/nixos/release-19.03/nixos.sd_image.aarch64-linux

Thank you so much @itorres, I’d never imagined the problem is with the image I’m using…

Are uou are using it a Raspberry Pi 3? All of those aarch64/ armv7l code names are very confusing… Only now, after reading several Wikipedia articles, I kind of feel I understand most of it. Anyway, since the official NixOS distribution supports the 64 bit version, it’s very peculiar that @dezgeg and the WiKi page authors are instructing to use the armv7l image (which is 32bit) - the Wiki needs a rewrite!

nixos-rebuild switch finally didn’t totally fail for me :smiley: 1 I’ve also managed to use the unstable channel with it as well.

I’m having one problem though: How exactly am I supposed to set the root password or any other user? I ran passwd after the first reboot and I was hoping it will be set it in the next reboot but again I was automatically logged in as root since it’s password is empty.

Another detail that bothers me, is that during boot, I see a message stating: setting up /etc .... Is it normal? I’m not sure I see it on my main computer’s NixOS installation… Feels like a very familer message from Installation images. If this is indicates that something is wrong, It should explain also why root’s password is reset upon every boot - /etc/passwd and /etc/shadow are given their default contents every boot.

Since I assume you have a working and system with this architecture, I have 1 more question: The Wiki says that I should remove the imports line in configuration.nix:

imports = [ <nixos/modules/installer/cd-dvd/sd-image-armv7l-multiplatform.nix> ];

Seems reasonable but I don’t trust the Wiki anymore… Have you done the same? I get the feeling as if the problem I’m having with root’s password is related somehow…


1: It printed upon finish that the new configuration has a different init and so I need to reboot in order to use the new configuration.

Hi! I think that, yes, the NixOS on ARM wiki page needs a good cleanup.

Though, I am unsure what exactly made you think you needed to use the ARMv7 images. Is it because Installation section of the main wiki page only uses the armv7l images in the instructions? Would it have helped if that section used a placeholder rather than use armv7l? Might it help if we add an introductory bit with “what’s up with the weird names”?

As for knowing which images to use on a particular device, and the peculiarities of a device, it is a good idea to follow the links in the devices list, to the particular devices.

Though, as you’re now using the aarch64 image, it should be smoother.

You shouldn’t put that much distrust in the Wiki. The instructions are generally fine. Though do note that only armv6 and armv7 platforms need alternative channels in the configuration.

And yes, you should remove the <nixos/modules/installer/*> paths from your configuration. What it does is setup your system like an installer media. The base sd_image imports it at build-time. There is a tracking issue to remove that line from the generated image to make it less confusing for end-users.

I’m unsure what mislead me, reading again the pages and knowing what I’m looking for it looks obvious. But at the time of my first visit something in the instructions (maybe the formatting) made me assume that I had to download the armv7l images.

Edit: It could also be this paragraph, which reduces the options to armv6l and armv7l:

The installation images come in two flavors: sd-image-armv6l-linux.img is built for the ARMv6 architecture and it comes with the Raspberry Pi kernel. sd-image-armv7l-linux.img is built for the ARMv7 architecture and comes with the mainline multiplatform ARMv7 kernel (multi_v7_defconfig). Make sure you download the correct image for your board!

I think the issue could be here it reads:

First follow the generic installation steps to get the installer image and install using the installation and configuration steps.

To me it would make more sense to have the image links on the device page, directly linked from the top table, rather than showing the architecture and pointing to the generic instructions.

I’ll only add to this subject of the discussion “Why did I choose to use the arm7l” that I did it because I’ve used the same architecture back when I had Arch Linux on my Raspberry Pi 3 and I didn’t anticipate that it can run both.

Additionally, the fact that one is named with arm at the beginning and the other image’s architecture is named aarch64 makes you think these are totally different things. It’s only until today, after I read many stack-exchange QAs that I’ve finally understood how severe is the lack of solid conventions for naming these architectures.

Also, as for that root’s password issue, the problem was that I ran the 1st nixos-rebuild with that imports = [ ... ] in my configuration.nix. The unfortunate problem with that, was that even after I deleted that line and reran nixos-rebuild, it didn’t help; No idea why :confused:.

Never mind that, my installation is working fine now after I started it all over again.

I have another small question for you guys, hoping you’ll be kind enough to help me a bit more: I’m supposed to use my Raspberry Pi with a certain hardware that specifies that I should write certain lines in my /boot/config.txt.

It seems NixOS handles that via boot.loader.raspberryPi.firmwareConfig? The Wiki page seems to imply that but it refers to /boot/configuration.txt and not /boot/config.txt. Does it matter?

I’ve succesfully managed to operate this hardware with an Arch Linux installation and I noticed that within that installation, /boot/ doesn’t contain only a config.txt file, but also many binary files and an additional directory called overlays/. In that directory there’s a README with the following contents:

Introduction

This directory contains Device Tree overlays. Device Tree makes it possible
to support many hardware configurations with a single kernel and without the
need to explicitly load or blacklist kernel modules. Note that this isn’t a
“pure” Device Tree configuration (c.f. MACH_BCM2835) - some on-board devices
are still configured by the board support code, but the intention is to
eventually reach that goal.

On Raspberry Pi, Device Tree usage is controlled from /boot/config.txt. By
default, the Raspberry Pi kernel boots with device tree enabled. You can
completely disable DT usage (for now) by adding:

device_tree=

to your config.txt, which should cause your Pi to revert to the old way of
doing things after a reboot.

In /boot you will find a .dtb for each base platform. This describes the
hardware that is part of the Raspberry Pi board. The loader (start.elf and its
siblings) selects the .dtb file appropriate for the platform by name, and reads
it into memory. At this point, all of the optional interfaces (i2c, i2s, spi)
are disabled, but they can be enabled using Device Tree parameters:

dtparam=i2c=on,i2s=on,spi=on

However, this shouldn’t be necessary in many use cases because loading an
overlay that requires one of those interfaces will cause it to be enabled
automatically, and it is advisable to only enable interfaces if they are
needed.

Configuring additional, optional hardware is done using Device Tree overlays
(see below).

GPIO numbering uses the hardware pin numbering scheme (aka BCM scheme) and
not the physical pin numbers.

raspi-config

The Advanced Options section of the raspi-config utility can enable and disable
Device Tree use, as well as toggling the I2C and SPI interfaces. Note that it
is possible to both enable an interface and blacklist the driver, if for some
reason you should want to defer the loading.

Modules

As well as describing the hardware, Device Tree also gives enough information
to allow suitable driver modules to be located and loaded, with the corollary
that unneeded modules are not loaded. As a result it should be possible to
remove lines from /etc/modules, and /etc/modprobe.d/raspi-blacklist.conf can
have its contents deleted (or commented out).

Using Overlays

Overlays are loaded using the “dtoverlay” config.txt setting. As an example,
consider I2C Real Time Clock drivers. In the pre-DT world these would be loaded
by writing a magic string comprising a device identifier and an I2C address to
a special file in /sys/class/i2c-adapter, having first loaded the driver for
the I2C interface and the RTC device - something like this:

modprobe i2c-bcm2835
modprobe rtc-ds1307
echo ds1307 0x68 > /sys/class/i2c-adapter/i2c-1/new_device

With DT enabled, this becomes a line in config.txt:

dtoverlay=i2c-rtc,ds1307

This causes the file /boot/overlays/i2c-rtc.dtbo to be loaded and a “node”
describing the DS1307 I2C device to be added to the Device Tree for the Pi. By
default it usees address 0x68, but this can be modified with an additional DT
parameter:

dtoverlay=i2c-rtc,ds1307,addr=0x68

Parameters usually have default values, although certain parameters are
mandatory. See the list of overlays below for a description of the parameters
and their defaults.

The Overlay and Parameter Reference

Should I use strictly the firmwareConfig attribute in my configuration.nix? How come I have no configuration.txt or config.txt file in /boot/ right now? Basically, I need to put 1 line in my config.txt:

dtoverlay=allo-piano-dac-plus-pcm512x-audio,glb_mclk

I’d have jumped right at it by trying to just edit (create) /boot/config.txt with the contents I’ve had on my Arch Linux’s /boot/config.txt, but I doubt that it will work, mainly because of the overlays/ directory missing.

Anyway, I’d be glad to understand what’s NixOS equivalent instruction as for the distro-agnostic and well known “add this line to your /boot/config.txt”. I’ll gladly add this information to the WiKi page.

This generally is not trivial to do using the mainline kernel, which is the default for aarch64. Generally a lot of assumptions are made for Raspberry Pi gizmos and things, that the user will run the raspberry pi foundation’s kernel.

Sadly, I don’t have the personal experience to tell whether you will be forced to drop the use of u-boot, or not, for use with these things.


I have fixed the wiki page, the source code does not mention configuration.txt and configuration.txt would be wrong.

In addition, the newer SD images for aarch64 have different assumptions about the filesystem layout, and the current boot.loader.raspberryPi.* set of options would be incompatible with the assumptions. I would have to think about better ways to handle the situation. The image is not specific to the Raspberry Pi, it is a generic image, making this harder to deal with.

And, there stops the things I know, and I wouldn’t start taking guesses as for your next best step. I guess it would be better if someone with actual experience making dtoverlays work with the raspberry pi foundation’s stuff could answer.

Thanks for editing the WiKi. I guess this will come in a different thread. I will note here though, that I’ve tried using the firmwareConfig option and I noticed a nixos-rebuild didn’t actually update /boot/config.txt.

I’ve also tried disabling u-boot by explicitly setting boot.loader.raspberryPi.uboot.enable = false; but it didn’t help.

Never mind now all the details. I’ll ask about everything in a new thread later. I’ll mention you hoping you’ll be notified.