No space left on /boot

Help
1

Help!

building the system configuration...
ignoring default profile in the list of boot entries because of the following error:
[Errno 28] No space left on device: '/nix/store/s3pyaxds208immg12b07v2j7wpbgl768-initrd-linux-6.0.12/initrd' -> '/boot/efi/nixos/s3pyaxds208immg12b07v2j7wpbgl768-initrd-linux-6.0.12-initrd.efi'

df -h:

Filesystem       Size  Used Avail Use% Mounted on
devtmpfs         1.6G     0  1.6G   0% /dev
tmpfs             16G  400M   16G   3% /dev/shm
tmpfs            7.8G  8.2M  7.8G   1% /run
tmpfs             16G  432K   16G   1% /run/wrappers
/dev/mapper/enc  899G  673G  221G  76% /
/dev/mapper/enc  899G  673G  221G  76% /home
/dev/nvme0n1p1   511M  511M     0 100% /boot
tmpfs            3.2G   20K  3.2G   1% /run/user/1000

Tried nix-collect-garbage -d, but it didn’t help.

1 Like
2

nix-collect-garbage -d will remove old generations and clean up the nix store

The boot entries for those kernels / generations don’t get updated until the next rebuild, when they all get put in place and a new grub menu (or whatever bootloader) is generated.

If you’re stuck at that step because it can’t start, you can just manually delete a few kernels and initrd’s

3 Likes
3

You need to remove some of the not needed anymore kernels and initrd images from the boot partition. One or 2 are usually enough.

Then rebuild and switch.

The activation script performs the following scripts in that order to remain consistent:

1, copy new kernel and image
2. update menu
3, delete kernels and images that are not in the new menu anymore

There is also an option that will make sure that the number of boot entries does not exceed a certain number, though I do not remember nor can I find it

1 Like
4

How do I do this? I’m using systemd boot. When I delete items from /boot/EFI/nixos and /boot/loader/entries, they are just recreated when I run nixos-rebuild switch.

5

The only ones that should be recreated are the ones for current generations, which should have been reduced by your garbage collect.

How many system-*-link entries do you have in /nix/var/nix/profiles?

Do you have something else taking up space in boot?

6

About 100 dating back to July. Shouldn’t they be getting deleted with garbage collect?

7

Only if your user has permission to delete them; i.e. you need to run nix-collect-garbage -d as root to cause those to be deleted.

7 Likes
8

Oof, yeah. Though a this point something less drastic than -d will be fine, say --delete-older-than 10d. Since these are gc roots, your store will also get a big clean after this.

Once you’re up and going again, I suggest adding something like this to your config, and never thinking about it again (because it will run with the right perms)

  nix.gc = {
    automatic = true;
    randomizedDelaySec = "14m";
    options = "--delete-older-than 10d";
  };
9

Crap. How stupid of me. Sorry to waste your time.

After doing a sudo nix-collect-garbage --delete-older-than 30d, /boot was still 100% full, but sudo nixos-rebuild switch was able to run and when finished, /boot was only 60% full. All good.

I have set nix.gc.automatic = true;, so obviously I don’t quite have it right. I remember thinking I would set this without options and see what it does. It looks like it needs options?

10

Right, without the options it’s just removing stray garbage (from ephemeral nix-shell invocations and the like), no gc roots.

11

Right, nix.gc.automatic = true; will only configure it to automatically run the GC, not to include a --delete-older-than option or anything. So paths in the store with no GC roots will get deleted, but all the existing GC roots will remain. So you want something like nix.gc.options = ["--delete-older-than 30d"]; as well.

12

Honestly, this feels like a case of “there should be a better default”.

3 Likes
13

Awesome. You guys are so helpful. This really is a great community. I am going to make those changes to my configuration.nix.

2 Likes
14

Another option if you’re using systemd-boot is you can set boot.loader.systemd-boot.configurationLimit. The number you set it to depends on your setup, e.g. I have it set to just 2 on my desktop because I’m dual-booting Windows which takes some space, and I’m loading gpu drivers into the initrd which makes my boot images bigger and fill up the partition.

2 Likes
15

That looks interesting. So, does it garbage collect as well, or just remove generations from the systemd boot list? I assume it removes kernels and initramfs from /boot, but without also doing a garbage collect, won’t those /boot items be recreated at next nixos-rebuild? The docs aren’t clear on this.

16

It will not garbage collect. It will make it so that nixos-rebuild will only allow there to be a maximum number of boot entries created in /boot. So the oldest generations are truncated and removed from /boot after the limit during nixos-rebuild.

17

So then the number of boot entries will be out-of-sync with the garbage collection? Why keep old store paths not refered to by any old generations?

18

The old generations still exist, they just aren’t in the boot menu. You can still switch to them from a running system.

19

Ahh… Wow. So much to learn. So, presumably, from the command line if you switch to an old generation that is no longer listed in the boot menu, the appropriate kernel and initramfs gets built on-the-fly?

20

Here’s what happens when you do nixos-rebuild.

  1. The NixOS system derivation is evaluated and built.
  2. The /nix/var/nix/profiles/system profile is set to a new generation, pointing at that system derivation’s store path.
  3. The system generation has a script called switch-to-configuration that gets run to actually do the switch. This has a few modes that are relevant to us; switch, test, and boot.
  4. In the switch and boot modes, the boot loader is reconfigured. Up to a limit of boot.configurationLimit, other system generations as well as the new one have their kernels and initramfs copied to the /boot partition while obsolete generations or generations beyond the configurationLimit cutoff have theirs removed; by now these have already long-since-been built, residing in their /nix/var/nix/profiles/system-*-link/ store path. The new generation is made the default boot entry.
  5. In the switch and test modes, system services are reconfigured to match those of the new generation, and are stopped, started, and restarted to match. Note that the kernel and initramfs don’t come into play here. Those are only relevant when booting a generation.

The case where this isn’t quite accurate is when switching to an entirely different generation. This can be done with nixos-rebuild --rollback or by manually issuing some of the steps above. In this case, the “new” generation is whatever you’ve picked. And even if it’s your oldest generation, it will be included on /boot.

3 Likes