Remote builder fails unexplicably with "No space left on device"

Help
1

Hi there,

I tried building a patched kernel to get my webcam working on my business notebook. Since it was taking ages, I set up a remote builder on my desktop PC.

It seemed to work, as I could rebuild my system using the remote builder with an older version of the configuration. However, when I tried to rebuild with the patched kernel, it always failed with some error message containing “No space left on device”. It wasn’t the same message every time, but more than once it failed with this:

building the system configuration...
warning: Git tree '/home/$USER/.config/dotfiles' is dirty
error: build of '/nix/store/757db64dkaggr06s59lyzfq0pd6prdnv-linux-6.7.1.drv' on 'ssh://root@purple' failed: builder for '/nix/store/757db64dkaggr06s59lyzfq0pd6prdnv-linux-6.7.1.drv' failed with exit code 2
error: builder for '/nix/store/757db64dkaggr06s59lyzfq0pd6prdnv-linux-6.7.1.drv' failed with exit code 1;
       last 10 log lines:
       > make[2]: *** [../scripts/Makefile.modfinal:56: drivers/gpu/drm/i915/i915.ko] Error 1
       > make[2]: *** Deleting file 'drivers/gpu/drm/i915/i915.ko'
       > objcopy: drivers/gpu/drm/amd/amdgpu/stu4OeMA[.debug_str]: No space left on device
       > objcopy: drivers/gpu/drm/amd/amdgpu/stu4OeMA[.BTF]: No space left on device
       > btf_encoder__write_elf: failed to add .BTF section to 'drivers/gpu/drm/amd/amdgpu/amdgpu.ko': 2!
       > Failed to encode BTF
       > make[2]: *** [../scripts/Makefile.modfinal:56: drivers/gpu/drm/amd/amdgpu/amdgpu.ko] Error 1
       > make[2]: *** Deleting file 'drivers/gpu/drm/amd/amdgpu/amdgpu.ko'
       > make[1]: *** [/build/linux-6.7.1/Makefile:1844: modules] Error 2
       > make: *** [../Makefile:234: __sub-make] Error 2
       For full logs, run 'nix log /nix/store/757db64dkaggr06s59lyzfq0pd6prdnv-linux-6.7.1.drv'.
error: 1 dependencies of derivation '/nix/store/4wmn1j6m5p0y1b79nlpv0slh9b6xbgqp-nixos-system-moooh-24.05.20240121.612f972.drv' failed to build

I watched resources on both systems closely and neither RAM nor disk space seems to be running out on either of them.

Any suggestions how I could debug this?

2

It might be running out of space in the tmpdir where builds occur, if that’s its own volume?

3

When /tmp is mounted as tmpfs, there is limited space.

I am personally using:
systemd.services.nix-daemon.environment.TMPDIR = "/srv/hdd/tmp/";

in configuration because I want to place data on spinning hdd.

1 Like