I generate custom VM images too. Some are larger and have run out of memory inside the VM run. My expressions use nixos/lib/make-disk-image.nix, which passes a hard-coded value (1024 megs) to Qemu/KVM for guest memory.
I don’t know how or where attributes from release.nix invoke a VM run, so I can’t say whether it is the same problem. Speculating a bit, maybe the typical VM run use cases don’t need more memory, so it hasn’t been a problem for many users or regular NixOS release activities.
One way to fix the make-disk-image function would be to add a memSize ? 1024 argument so that callers may override it as needed. I have worked around it in my private repo by making a custom-disk-image function that does that. If that is useful it could be merged upstream.
If there are other workarounds, I’d love to learn about them too.