You’ll need to run nixos-rebuild
again; that will free up the space on /boot
that was associated with the deleted generations.
Well that is precisely the problem. I cannot run nixos-rebuild
since it fails with no space left on device:
[lassi@snadi:~]$ sudo nixos-rebuild boot
[sudo] password for lassi:
building Nix...
building the system configuration...
cat: write error: No space left on device
warning: error(s) occurred while switching to the new configuration
OK. Not sure how to help then, but can give some background (as I’ve had this in the past too).
In my case, compressed kernel and initrd images accumulate under /boot/EFI/nixos/
(I’m on x86_64 and using systemd-boot) due to updated kernels and changes affecting the initrd in successive generations, noting that each image may be shared across a number of generations. I’ve never tried NixOS on an RPi, but I imagine the partition is filling up for similar reasons.
Clearing a sufficient number of old generations (so that the associated kernel and initrd images aren’t needed anymore) than running nixos-rebuild
should work, as it’s what’s suggested for this issue under Chapter 35.1: NixOS Boot Entries in the NixOS Manual. In my case, it always has worked: nixos-rebuild
has cleared the unneeded images off the /boot
partition first, and then applied the new generation… which succeeds as there is now free space under /boot
to write the new images.
Hopefully someone with more knowledge than me (I’m also fairly new to NixOS) can help you out.
…which I’ve just realised is the opposite behaviour to what is described in that open bug report you linked in the original post! But nonetheless, I really have solved it that way on my machine in the past.
Really hoping someone with more knowledge than me stops by this thread to help you out!
Assuming the links to the old generations in /nix/var/nix/profiles
have been deleted and nix-collect-garbage
has been run, you should be able to clear out the extra entries in /boot
with:
/run/current-system/bin/switch-to-configuration switch
I’m harvesting some questions coming from newcomers (and not-so-new as well) and this appears more than I’d think it would, maybe we should state clearly in the manual that not only the store increase in size during normal operation but /boot
does as well if it is a separate partition.
The issue is that nixos-rebuild
removes old kernels if the generations referencing them have been garbage collected but the operation is done after the kernel of the generation you’re trying to switch/boot into is installed.
At the time I didn’t know this and installed NixOS on a small boot partition and haven’t move since then so this is the workflow I’ve developed. It assumes you’re a normal user with sudo privilege, if you’re not do everything as root.
Make sure you understand every step or you could end up with a non bootable system
- Do a
sudo nixos-rebuild build
so that you’re sure that the build of your current configuration can be carried out - Do a garbage collection to remove old system generations with
sudo nix-collect-garbage -d
- Manually make some space in boot. Find your kernels and
rm
them. - Run
sudo nixos-rebuild switch
orsudo nixos-rebuild boot
. This time your bootloader will be installed correctly along with the new kernel and initrd - Make sure point 4 was executed correctly by looking at the output and reboot
- [optional] remove the
result
directory created by point 1
Thanks for the instructions. How do I know which files are safe to delete?
Well, I tried removing the oldest kernel. After it, sudo nixos-rebuild boot
gave me this:
rm: cannot remove '/boot/nixos/zbpbdvcazhh44sfzv9rgy9wllnab66v5-linux-4.18.7-dtbs/broadcom': Read-only file system
warning: error(s) occurred while switching to the new configuration
Edit: If it helps, I have this extra bit in my configuration.nix (as suggested here: NixOS on ARM/Raspberry Pi - NixOS Wiki):
hardware.enableRedistributableFirmware = true;
hardware.firmware = [
(pkgs.stdenv.mkDerivation {
name = "broadcom-rpi3-extra";
src = pkgs.fetchurl {
url = "https://raw.githubusercontent.com/RPi-Distro/firmware-nonfree/54bab3d/brcm80211/brcm/brcmfmac43430-sdio.txt";
sha256 = "19bmdd7w0xzybfassn7x4rb30l70vynnw3c80nlapna2k57xwbw7";
};
phases = [ "installPhase" ];
installPhase = ''
mkdir -p $out/lib/firmware/brcm
cp $src $out/lib/firmware/brcm/brcmfmac43430-sdio.txt
'';
})
];
The boot operation actually rebuild everything it need so if a file is missing it will be added and if it’s part of an old generation it will be removed. I never encountered the read-only filesystem issue, on the ARM board /boot is maybe handled differently? (pinging @dezgeg as the builder/maintainer of some ARM installation images)
Thanks for checking up! I didn’t, I still get the same error of Read-only file system when nixos-rebuild
is removing old files. Don’t really know what to do to it since sudo rm -rf <the file>
also results in Read-only filesystem.
Edit.
I just noticed that ls
cannot access the directory: ls: cannot access '/boot/nixos/zbpbdvcazhh44sfzv9rgy9wllnab66v5-linux-4.18.7-dtbs/broadcom': Input/output error
I had this disk full problem when upgrading a machine to 19.03, I guess the 77th generation was just one too many!
This saved the day, cleared out a bunch of generations, and nixos-rebuild switch --upgrade
was able to complete. Happy days!
post the output for ls -alhR /boot
here so we can see exactly what’s taking up space
nixos-rebuild
calls switch-to-configuration
which calls the system.build.installBootLoader
script on boot | switch
.
That logic varies between bootloaders and I haven’t verified how they work. All of them should have a logic similar to this:
- Clean /boot from all the generations
- Install the /boot entries from the newest to the oldest
- On disk full, complain and let the user know which generations are missing, but don’t fail the script
Make sure that (3) fails if there is not generation and make sure that (2) doesn’t fail.
I’m pretty sure some of them try to install the new generation before cleaning things up. That’s a safer heuristic but also fails hard if /boot
is full. Combined with the fact the nix-store --gc
doesn’t collect old boot entries, it’s a deadly mix.
My /boot is only 30m. I can’t even build a single generation. Any suggestion how to increase that?
This is a BUG. It’s extremely not good that a system can get broken by continuing to do nixos-rebuild switch (because nixos is meant to be a fearless tinkering system - that’s meant to be its strength).
If there’s only space for one more generation then it should fail with an error saying “Please run nixos-garbage-collect
first and then rerun this command with --force
.”
Just to note as this has happened to several people. If you start getting locale problems with perl (and in my case gnome-terminal didn’t want to start). That could be due to the /boot partition running out of space. If you set it in the gui to something then that seems to fix it again (not something that nixos-rebuild would fix unfortunately).
My two cents since I’ve had issues with that on my Raspberry Pi: the root cause of the bug is putting more things than strictly necessary on firmware/EFI partitions. This has been solved for RPi for a while, and it’s where it happens most often. However, an UEFI system with 512MB /boot
partitions as specified in the manual could potentially encounter this issue too, albeit more rarely. Should we consider moving that to /boot/efi
there as well?
I had windows slapped on the box as wasn’t sure if nixos would take (and chillblast doesn’t have a nixos preinstalled option yet) but the /boot partition is 100mb and 25mb seems to be windows boot which is mostly fonts!