For servers, I have found that this combination of settings avoids basically all the usual problems:
boot.kernel.sysctl = {
# It no longer makes sense to have these be large percentages of
# all system RAM.
"vm.admin_reserve_kbytes" = 131072; # 0x20000 = 128M
# These would ideally be tuned based on the speed at which
# the system's persistent storage can sink random-offset writes.
"vm.dirty_background_bytes" = 1048576; # 0x100000 = 1M
"vm.dirty_bytes" = 2097152; # 0x200000 = 2M
# Disable memory overcommit, allow total AS commit no more than
# swap + 90% of phys RAM
"vm.overcommit_memory" = 2;
"vm.overcommit_ratio" = 90;
};
# oomd is not useful when overcommit is disabled
systemd.oomd.enable = false;
This requires a swap partition with at least twice as much space as the system’s physical RAM to avoid fork failures. It will not actually get used unless something goes horribly wrong. Depending on your workload you may want to adjust dirty_background_bytes and dirty_bytes upward, but be cautious. System RAM getting filled with dirty data pages faster than the disk can absorb them is, in my experience, the #1 root cause of OOM conditions on servers.
(If you know a reliable way to measure “the speed at which the system’s persistent storage can sink random-offset writes”, please tell me. All the tools I’ve tried are statistically naive and/or specifically designed for plain old spinning rust with no software RAID or anything on top.)
I think these settings are also appropriate for well-nigh all desktop builds, but I haven’t comprehensively tested them on desktops.