No space left on device error when rebuilding, but plenty of storage available!

Help
1

Hello!
I’ve been presented with an error code while trying to rebuild my system. The full error message is:

❯ sudo nixos-rebuild switch --flake .
[sudo] password for sylvia: 
building the system configuration...
error:
       … while calling the 'head' builtin

         at /nix/store/3kwj19dbdfxnjbcns4hw307ylhz3wgrm-source/lib/attrsets.nix:1541:11:

         1540|         || pred here (elemAt values 1) (head values) then
         1541|           head values
             |           ^
         1542|         else

       … while evaluating the attribute 'value'

         at /nix/store/3kwj19dbdfxnjbcns4hw307ylhz3wgrm-source/lib/modules.nix:809:9:

          808|     in warnDeprecation opt //
          809|       { value = builtins.addErrorContext "while evaluating the option `${showOption loc}':" value;
             |         ^
          810|         inherit (res.defsFinal') highestPrio;

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: writing to file: No space left on device

I originally assumed this meant the device that houses /nix and /nix/store was full, and I expanded it from 100GiB to 150GiB. It currently has around 70GiB available.

After that didn’t fix my issue, I figured it was the /boot partition that was full. I have a 1GiB EFI partition, and it also isn’t full.

After the first attempt to build that resulted in this error, I ran nix store gc, then sudo nix-collect-garbage -d and then removed the two generations that aren’t current with nix-env --delete-generations 101 102. None of this fixed my issue!

For reference, here is the result of lsblk -f:

NAME          FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                                            
└─sda1        LVM2_member LVM2 001       7nDU4K-infE-WkWU-ZUY8-4bwU-Mv0K-b0i680                
  └─HDDs-data ext4        1.0            b45c54e2-19b2-49a5-a72b-311340ad83aa      6.7T     6% /data
sdb                                                                                            
└─sdb1        LVM2_member LVM2 001       xatkEb-L4Vy-pQfP-s9CR-iSE9-drAd-RFcqIC                
  └─HDDs-data ext4        1.0            b45c54e2-19b2-49a5-a72b-311340ad83aa      6.7T     6% /data
nvme1n1                                                                                        
├─nvme1n1p1                                                                                    
└─nvme1n1p2   ntfs                       0E624E23624E0FBF                                      
nvme0n1                                                                                        
├─nvme0n1p1   vfat        FAT32          79F5-7E04                               940.9M     8% /boot
├─nvme0n1p2   swap        1              4e5a17cd-b803-42f5-93e6-0653ea37b1be                  [SWAP]
├─nvme0n1p3   ext4        1.0            8b552fdc-32d2-4b2a-8a7a-0ae0cb3a73a1     84.5G    37% /nix/store
│                                                                                              /
└─nvme0n1p4   ext4        1.0            37e33df7-de68-44cf-993f-2afa7f2c69af    175.6G    35% /home

and df -hl for good measure:

Filesystem             Size  Used Avail Use% Mounted on
devtmpfs               794M     0  794M   0% /dev
tmpfs                  7.8G   58M  7.7G   1% /dev/shm
tmpfs                  3.9G  6.3M  3.9G   1% /run
tmpfs                  7.8G  1.2M  7.8G   1% /run/wrappers
/dev/nvme0n1p3         148G   56G   85G  40% /
efivarfs               192K   48K  140K  26% /sys/firmware/efi/efivars
/dev/nvme0n1p4         294G  103G  176G  37% /home
/dev/nvme0n1p1        1022M   82M  941M   8% /boot
/dev/mapper/HDDs-data  7.3T  444G  6.8T   7% /data
tmpfs                  1.6G  4.9M  1.6G   1% /run/user/1000

Also, I have 16Gb of RAM, a 16GiB SWAP partition, and I’m on an intel CPU, just in case any of this is relevant.
Thank you in advance :3

1 Like
2

That looks like an error during the evaluation of your NixOS configuration! I’m not sure what’s going on with the “no space left on device” part but I think that’s happening after the Nix expression has failed to evaluate. I think if you have enough space left it might be a permissions error.

You should review the changes you made since the last rebuild and maybe run nixos-rebuild again with --show-trace enabled.

1 Like
3

Perhaps the store isn’t currently mounted? What’s the output of the following command?

$ mount | grep "/nix/store"
1 Like
4

Did you run out of inodes? sudo tune2fs -l <device> | grep ^Free\ inodes

1 Like
5

This can also happen if you run out of /tmp space in very big derivations. Not sure if that list of mounts you posted is complete, it seems to lack /nix/store, too.

1 Like
6

/nix/store is mounted on the same partition as /, which is a 150GiB partition on a 500Gb NVMe M.2 SSD. As for the tmp files, I looked into it and realized that the boot.tmp.cleanOnBoot setting is false by default, so I added it to my configuration.nix, manually deleted the contents of /tmp, and then rebooted to try the build again, but this didn’t do it.
I don’t have much of an understanding of what temporary filesystems are, what they’re for, or where they’re located, so I’m not sure how to move on from here on my own. I appreciate your answer, I think you might be onto something.

7

I’m not sure if I’m missing something, but that command outputs this:

tune2fs: Bad magic number in super-block while trying to open /dev/nvme0n1p1

and nothin` else :<

EDIT: I was doing it on the wrong partition, silly me,
this is what it actually looks like:

❯ sudo tune2fs -l /dev/nvme0n1p3 | grep ^Free\ inodes
[sudo] password for sylvia: 
Free inodes:              9124345
8

The nix store is definitely mounted, its just not on it’s own partition or anything.
The output of mount | grep "nix/store" is:

/dev/nvme0n1p3 on /nix/store type ext4 (ro,relatime,stripe=32)

You can also see in the regular lsblk output that both / and /nix/store are mounted on /dev/nvme0n1p3

NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda             8:0    0   3.6T  0 disk 
└─sda1          8:1    0   3.6T  0 part 
  └─HDDs-data 254:0    0   7.3T  0 lvm  /data
sdb             8:16   0   3.6T  0 disk 
└─sdb1          8:17   0   3.6T  0 part 
  └─HDDs-data 254:0    0   7.3T  0 lvm  /data
nvme1n1       259:0    0 476.9G  0 disk 
├─nvme1n1p1   259:1    0    16M  0 part 
└─nvme1n1p2   259:2    0 476.9G  0 part 
nvme0n1       259:3    0 465.8G  0 disk 
├─nvme0n1p1   259:4    0     1G  0 part /boot
├─nvme0n1p2   259:5    0    16G  0 part [SWAP]
├─nvme0n1p3   259:6    0   150G  0 part /nix/store   } Its here I swear!!!
│                                       /
└─nvme0n1p4   259:7    0 298.8G  0 part /home
1 Like
9

Ok this is a little strange, but maybe there’s something obvious here that I’m missing.

I decided to just omit the module that I had added just before getting this issue. It’s a single file called transmission.nix that’s imported into apps.nix which gets imported into my configuration.nix.

After I commented it out in apps.nix, the build went through just fine. I’ll post the contents of transmission.nix below;

{ config, pkgs, userSettings, ... }:
{
  # Enable and Configure Transmission
  environment.systemPackages = [ pkgs.transmission-gtk ];
  services.transmission = {
    enable = true;
    home = /home/${userSettings.username};
    webHome = pkgs.flood-for-transmission;
    settings = {
      download-dir = /data/Torrents;
    };
  };
}

I added the entire services.transmission section in one go, so it really could be any part of that, but I guess I know where to start looking now. If you have any ideas though, I would really appreciate some guidance.

Thanks everybody!

10

That’s probably the culprit. In nix flake evaluation, if unquoted, nix would try to copy the directory into the store. Try quoting it and the home attribute value

8 Likes
11

This solved my issue! Thank you very much, I wasn’t aware that declaring a directory like that would try to clone the whole thing to my store! There’s probably more than a few places I’ve done that, I’ll have to go through and see if I can clean up my store a little bit…