Best way to run Nix builds in ramdisk?

Help
1

I manage a Hydra instance that runs a 1000+ CMake based builds where the configure stage in particular does a lot of sequential reading and writing of very small files. Based on what I see in netdata, I believe that even my NVMe-based storage is becoming an IO bottleneck for this.

I’m on a pretty bog-standard NixOS install, where I don’t believe anything is (currently) special about /tmp— it’s not its own mount or special filesystem or anything else:

$ mount | grep tmp
devtmpfs on /dev type devtmpfs (rw,nosuid,size=4944188k,nr_inodes=12358439,mode=755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=24720924k,mode=755)
tmpfs on /run/wrappers type tmpfs (rw,nodev,relatime,mode=755)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=9888368k,nr_inodes=2472092,mode=700,uid=1000,gid=100)

I would be interested in hearing some war stories and/or seeing example configurations from others who have been down this road with NixOS, in particular:

  • was it enough to just add an fstab line to fileSystems for /tmp?
  • rather than putting all of /tmp in RAM, could there be a separate tmp mount elsewhere, with Nix steered to use that for builds instead of /tmp
  • is it possible to use a hybrid drive for this, where there’s an allocated pool of RAM storage (say 50-100GB), but also a real disk backing it, for if builds require additional storage? (thinking bcache, zfs, that kind of thing)
1 Like
2

It should be, but try using boot.tmpOnTmpfs instead, given the option is made for it.

You are describing swap, and I think that will be the default behavior if you have any swap space. But worth double checking the docs on that, fairly sure the kernel documents this.

Interesting question! Note that /tmp is not a tmpfs on NixOS specifically because nix builds in it, and the massive parallel builds quickly overflow whatever amount of RAM you may have.

So this idea is only feasible if you have some beefy servers, and non-nix services probably benefit from having a tmpfs in there anyway.

I’d be curious about one of those compressed tmpfses (forgot the name right now) for a special nix build tmpfs, though.

1 Like