Surprise build error (no space left on device)

I am seeing a surprise build failure on my ARMv7 board. I am trying to finish off a big system update, and am down to the last few packages. I am having trouble with spidermonkey_60, as follows:

# nixos-rebuild build --builders ''
building Nix...
building the system configuration...
these derivations will be built:
  /nix/store/8nfapinw9rpy4gj34mf30lzaaxbmaa44-spidermonkey-60.4.0.drv
  /nix/store/ahy4cp5lfxllzxxx452ajac7bv7yv1dr-polkit-0.116.drv
  /nix/store/gm6v74sa9y829si2f5qrsg86yj99spp4-udisks-2.8.2.drv
  /nix/store/1wv4kv2d2h49rpwnlaskfziwldd7gvnv-unit-systemd-udevd.service.drv
  /nix/store/bcfa4ry4fc3pi2fzg1lwjp366zccslkm-system-path.drv
  /nix/store/d3y7z3ha9m3iqm0ifj6vxykvgz024sb7-unit-polkit.service.drv
  /nix/store/js28a7y31wibjwf8cnr5lv7179mm1zmw-unit-systemd-fsck-.service.drv
  /nix/store/v7kj996l1imw15hbl1m9pid1mhs0j13l-dbus-1.drv
  /nix/store/lx84r2qf66hi6z538sf9zhw9ln6w4fvv-unit-dbus.service.drv
  /nix/store/0akkzndcm5bn1r0mj3x6dcawf3jda480-system-units.drv
  /nix/store/b5zvng1rrnyyw82hqw4yz0wfwrhgjixj-udev-rules.drv
  /nix/store/b48hxyb5s1a8yw9dr6k4rdzsmc50rih9-unit-dbus.service.drv
  /nix/store/p1rnm97k9y06x16p0301npicm7r2yb4s-user-units.drv
  /nix/store/w9p9q53zk3p62imvqg83qfy0dk24gcr9-hwdb.bin.drv
  /nix/store/zwv5hkvxgnx84jhcpwsqfsd5zaiw1wpb-etc.drv
  /nix/store/847brfr6yycazdjvbdzqnhm8nmnmwjlj-nixos-system-node_0-19.09.git.ef8addb.drv
building '/nix/store/8nfapinw9rpy4gj34mf30lzaaxbmaa44-spidermonkey-60.4.0.drv'...
unpacking sources
unpacking source archive /nix/store/swyv5s21k0bbff0byh6wcjlanbwmnzbj-firefox-60.4.0esr.source.tar.xz
tar: firefox-60.4.0/js/src/tests/non262/object/15.2.3.6-dictionary-redefinition-17-of-32.js: Cannot open: No space left on device
...
[millions of the same message, for every file in the archive]

The obvious question is, am I out of space on my root volume? I do not think so!

# df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/disk/by-label/NIXOS_SD   59G   33G   24G  59% /
tmpfs                         16G     0   16G   0% /tmp
...

You can see I am using tmp-on-tmpfs, but the tmpfs is set to 16GB in size, and I have a 16GB swapfile backing it (because the system RAM is only 2GB on this board):

# swapon
NAME      TYPE SIZE USED PRIO
/swapfile file  16G  31M   -2

I can unpack the firefox source file in question on this machine with no problems, and it is about 2.2GB in size.

What could cause this problem? I think 16GB of space in /tmp should be enough, right?

(Also, FWIW, if I watch mem and swap usage over time in htop, the swap use does not get anywhere near 16GB before it fails)

# df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/disk/by-label/NIXOS_SD   59G   33G   24G  59% /
tmpfs                         16G     0   16G   0% /tmp
...

Please run df -ih to check if you are out of inodes.

2 Likes

Please run df -ih to check if you are out of inodes.

Argh, this community is the best. This was exactly it. Thanks so much, @7c6f434c!!!