I find that /tmp is very important for nix build. To ensure fast builds and reduce disk write wear, we usually enable boot.tmp.useTmpfs = true to place temporary build results in memory. However, this can easily lead to failures due to insufficient disk capacity when building large projects. I know that using a large swap space and adjusting boot.tmp.tmpfsSize to a large value can bypass this limitation, but this doesn’t seem to be a solution specifically optimized for nix builds, and I am also skeptical about whether swap can work reasonably in the context of nix builds. I wonder if anyone has developed an optimized solution specifically for nix build scenarios
3 Likes
I used to use tmpfs, but due to memory management differences, tmpfs (and swap) work against the zfs ARC, and I prefer to work with it, so now I just have a regular fs dataset with some optimisation options to save a few IOPS and syncs:
❯ zfs get all -s local rpool/fmrl/tmp
NAME PROPERTY VALUE SOURCE
rpool/fmrl/tmp quota 16G local
rpool/fmrl/tmp mountpoint legacy local
rpool/fmrl/tmp sync disabled local
rpool/fmrl/tmp redundant_metadata none local
I also have relatime=on inherited from the pool root
Like most filesystems, if the file is created and removed again quickly enough the data will never hit disk(ssd). If it lasts longer and is larger, well, then I want the option of reusing that RAM for something else in the meantime.
1 Like
Yea, this is the main reason that we don’t put /tmp on tmpfs by default. According to man file-hierarchy, /var/tmp is realistically where Nix should perform its builds by default, not /tmp. So I’d like to see a change in Nix to use /var/tmp by default, which would let us enable /tmp on tmpfs by default. You can do this yourself by setting the TMPDIR environment variable on the nix-daemon systemd service.
3 Likes
Good advice, but the problem remains unsolved. If we use /var/tmp for building, how can we reduce the consumption of disk life during the build process?
I genuinely don’t think you need to worry about that with modern SSDs, unless you’re running a heavy duty build farm that’s constantly building things 24/7. Any modern SSD that isn’t complete crap is going to be rated for years of literal constant load before they should fail, as long as you TRIM.
If you are still worried about it, the sync=disabled trick for your tmp build directory on ZFS is a good tip, as @uep mentioned. Otherwise, just leaving the tmp build dir on tmpfs and enabling swap is probably about as good as you can get.
1 Like
less jobs and cores can also prevent OOM and relieve memory pressure (though obviously builds will take longer)
1 Like