If you want to constrain the nix-daemon (i.e. all builds, not individual ones) using cgroups (via systemd) you could use something like this:
systemd.services.nix-daemon.serviceConfig = {
MemoryHigh = "5G";
MemoryMax = "6G";
};
There are a lot of options (see e.g. systemd.resource-control
and systemd.exec
).
If you want to instead constrains individual builds then it’ll be more difficult and probably require patching Nix (but I haven’t looked into this).