First off I really don’t know where I should be reporting this, but as of Linux 6.11 (on nixos-unstable), my system is unable to run neither nix-index nor nixpkgs-review (on staging specifically), both of which internally call nix-env. Running watch "zramctl && free -m" shows that swap is only filled up approximately 25% when under extreme memory pressure before oom-killer fires and kills nix-env. This behaviour is NOT present in Linux 6.10, and possibly below. Here, swap fills up almost 60% of the way before nix-env finishes up and produces the outputs, freeing up swap and relieving memory pressure almost instantly.
My assumption is due to the memory-related changes that landed in 6.11, swap isn’t pressured enough and long running programs with a high oom_score (like nix-env) are subject to termination. I have no idea how to “tune” them, nor do I have any idea where to actually look for them, to determine what’s going on. All I know from a new features article that memory changes happened.
This is a pretty major nuisance in terms of development. I’d love to begin properly analyzing but I really don’t know where to begin, and I somehow feel even less confident trying to report this directly up to the kernel.
Here’s some valuable logging information from free -m and dmesg:
$ free -m
total used free shared buff/cache available
Mem: 11705 1131 9572 247 1001 9663
Swap: 17557 149 17408
$ sudo dmesg
[ 146.727755] nix-env invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
[ 146.727761] CPU: 7 UID: 1000 PID: 2243 Comm: nix-env Not tainted 6.11.0 #1-NixOS
[ 146.727763] Hardware name: Dell Inc. Inspiron 14 5410 2-in-1/0TPG56, BIOS 2.23.0 11/13/2023
[ 146.727764] Call Trace:
[ 146.727765] <TASK>
[ 146.727767] dump_stack_lvl+0x64/0x90
[ 146.727771] dump_header+0x44/0x1c0
[ 146.727776] oom_kill_process+0x104/0x210
[ 146.727778] out_of_memory+0x256/0x590
[ 146.727779] __alloc_pages_noprof+0xe27/0x1130
[ 146.727783] alloc_pages_mpol_noprof+0xd9/0x1f0
[ 146.727786] folio_alloc_mpol_noprof+0x14/0x40
[ 146.727787] vma_alloc_folio_noprof+0x6b/0xd0
[ 146.727789] do_swap_page+0xb78/0x10d0
[ 146.727792] ? update_load_avg+0x7e/0x7f0
[ 146.727795] ? __pte_offset_map+0x1b/0x190
[ 146.727798] __handle_mm_fault+0x86e/0x1060
[ 146.727800] ? __smp_call_single_queue+0xb8/0x120
[ 146.727803] handle_mm_fault+0x17f/0x2e0
[ 146.727805] do_user_addr_fault+0x177/0x6b0
[ 146.727809] exc_page_fault+0x71/0x160
[ 146.727812] asm_exc_page_fault+0x26/0x30
[ 146.727814] RIP: 0033:0x7f931373676b
[ 146.727832] Code: 39 e8 0f 82 51 fe ff ff 48 8b 38 48 83 ea 10 49 39 fe 77 09 49 39 ff 0f 87 f7 00 00 00 48 83 e8 10 48 39 ea 0f 82 2f fe ff ff <48> 8b 3a 49 89 c5 0f 18 82 00 ff ff ff 49 39 fe 77 c3 49 39 ff 76
[ 146.727833] RSP: 002b:00007fffec6f1220 EFLAGS: 00010206
[ 146.727835] RAX: 00007f923b8f43e8 RBX: 00007fffec6f1250 RCX: 0000000000000000
[ 146.727836] RDX: 00007f923b8f43f0 RSI: 0000000000000000 RDI: 0000000000000000
[ 146.727837] RBP: 00007f923b8f4000 R08: 0000000000000000 R09: 00007f911d6560d0
[ 146.727837] R10: 00000000000003f8 R11: 0000000000000000 R12: 0000000000000a88
[ 146.727838] R13: 0000000000000e80 R14: 00007f8ecc4b76ba R15: 00007f9314ac1380
[ 146.727841] </TASK>
[ 146.727841] Mem-Info:
[ 146.727843] active_anon:1853123 inactive_anon:795366 isolated_anon:0
active_file:8055 inactive_file:4549 isolated_file:0
unevictable:8110 dirty:0 writeback:2
slab_reclaimable:13913 slab_unreclaimable:24445
mapped:8590 shmem:12731 pagetables:8417
sec_pagetables:771 bounce:0
kernel_misc_reclaimable:0
free:16623 free_pcp:1071 free_cma:0
[ 146.727846] Node 0 active_anon:7412492kB inactive_anon:3181464kB active_file:32220kB inactive_file:18196kB unevictable:32440kB isolated(anon):0kB isolated(file):0kB mapped:34360kB dirty:0kB writeback:8kB shmem:50924kB shmem_thp:24576kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:5280kB pagetables:33668kB sec_pagetables:3084kB all_unreclaimable? no
[ 146.727849] Node 0 DMA free:76kB boost:0kB min:84kB low:276kB high:468kB reserved_highatomic:0KB active_anon:13124kB inactive_anon:16kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:36kB local_pcp:24kB free_cma:0kB
[ 146.727852] lowmem_reserve[]: 0 1426 11652 0 0
[ 146.727854] Node 0 DMA32 free:7668kB boost:0kB min:8420kB low:27088kB high:45756kB reserved_highatomic:0KB active_anon:429648kB inactive_anon:1037332kB active_file:808kB inactive_file:624kB unevictable:0kB writepending:8kB present:1559724kB managed:1493600kB mlocked:0kB bounce:0kB free_pcp:1544kB local_pcp:1004kB free_cma:0kB
[ 146.727857] lowmem_reserve[]: 0 0 10226 0 0
[ 146.727859] Node 0 Normal free:58748kB boost:0kB min:59076kB low:190044kB high:321012kB reserved_highatomic:0KB active_anon:6969576kB inactive_anon:2144576kB active_file:31412kB inactive_file:17920kB unevictable:32512kB writepending:0kB present:10739712kB managed:10477724kB mlocked:11540kB bounce:0kB free_pcp:2704kB local_pcp:1028kB free_cma:0kB
[ 146.727861] lowmem_reserve[]: 0 0 0 0 0
[ 146.727863] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 2*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 92kB
[ 146.727869] Node 0 DMA32: 2*4kB (UM) 0*8kB 2*16kB (UM) 2*32kB (UM) 1*64kB (U) 1*128kB (U) 1*256kB (U) 2*512kB (UM) 4*1024kB (UM) 1*2048kB (M) 0*4096kB = 7720kB
[ 146.727877] Node 0 Normal: 1094*4kB (UE) 756*8kB (UME) 399*16kB (UME) 190*32kB (UME) 159*64kB (UE) 98*128kB (UME) 20*256kB (UME) 14*512kB (E) 1*1024kB (E) 0*2048kB 0*4096kB = 58920kB
[ 146.727884] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 146.727886] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 146.727886] 30651 total pagecache pages
[ 146.727887] 2938 pages in swap cache
[ 146.727887] Free swap = 14456572kB
[ 146.727888] Total swap = 17978876kB
[ 146.727888] 3078857 pages RAM
[ 146.727889] 0 pages HighMem/MovableOnly
[ 146.727889] 82186 pages reserved
[ 146.727890] 0 pages cma reserved
[ 146.727890] 0 pages hwpoisoned
[ 146.727891] Tasks state (memory values in pages):
[ 146.727891] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[ 146.727899] [ 522] 0 522 9805 1158 0 1158 0 86016 224 -250 systemd-journal
[ 146.727902] [ 555] 0 555 8483 1205 0 1205 0 90112 736 -1000 systemd-udevd
[ 146.727904] [ 1097] 994 1097 3680 1064 32 1032 0 69632 160 -900 systemd-oomd
[ 146.727906] [ 1153] 154 1153 59210 1030 0 1030 0 114688 224 0 systemd-timesyn
[ 146.727908] [ 1173] 4 1173 3043 1021 0 1021 0 69632 288 -900 dbus-daemon
[ 146.727910] [ 1180] 0 1180 97184 737 0 737 0 139264 224 0 thermald
[ 146.727911] [ 1222] 0 1222 3857 1208 32 1176 0 69632 224 0 systemd-logind
[ 146.727913] [ 1275] 0 1275 83845 2085 160 1925 0 163840 544 0 NetworkManager
[ 146.727914] [ 1341] 996 1341 171668 1102 0 1102 0 184320 320 0 nsncd
[ 146.727915] [ 1351] 0 1351 112047 1018 0 1018 0 139264 864 0 nix-daemon
[ 146.727917] [ 1528] 0 1528 3648 1326 0 1326 0 69632 352 0 wpa_supplicant
[ 146.727918] [ 1546] 0 1546 917 909 128 781 0 49152 0 0 greetd
[ 146.727920] [ 1691] 0 1691 2754 2725 319 2406 0 57344 0 0 greetd
[ 146.727921] [ 1722] 1000 1722 4937 1083 0 1083 0 77824 512 100 systemd
[ 146.727923] [ 1724] 1000 1724 4562 322 1 321 0 69632 352 100 (sd-pam)
[ 146.727924] [ 1742] 1000 1742 496008 2374 580 1580 214 614400 7168 0 sway
[ 146.727926] [ 1747] 28 1747 95449 704 0 704 0 122880 608 0 polkitd
[ 146.727927] [ 1792] 1000 1792 4095 579 0 579 0 69632 224 0 swaybg
[ 146.727929] [ 1807] 1000 1807 235666 3639 576 2999 64 393216 1952 0 .waybar-wrapped
[ 146.727930] [ 1809] 1000 1809 18538 485 0 485 0 69632 64 0 wl-clip-persist
[ 146.727932] [ 1811] 1000 1811 5732 643 0 643 0 90112 2400 0 papersway
[ 146.727933] [ 1813] 1000 1813 1070 588 0 588 0 49152 32 0 swayidle
[ 146.727935] [ 1820] 1000 1820 2529 455 0 455 0 61440 96 0 sh
[ 146.727936] [ 1821] 1000 1821 1416 596 0 596 0 57344 96 200 dbus-daemon
[ 146.727937] [ 1823] 1000 1823 895 463 0 463 0 49152 32 0 swaymsg
[ 146.727939] [ 1824] 1000 1824 5732 160 0 160 0 77824 2400 0 papersway
[ 146.727940] [ 1831] 1000 1831 118994 1037 0 1037 0 176128 704 200 .xdg-desktop-po
[ 146.727941] [ 1836] 1000 1836 133515 585 0 585 0 159744 160 200 .xdg-document-p
[ 146.727943] [ 1840] 1000 1840 77554 571 0 571 0 118784 128 200 .xdg-permission
[ 146.727944] [ 1847] 1000 1847 647 448 0 448 0 40960 0 200 fusermount3
[ 146.727945] [ 1850] 1000 1850 103792 831 0 831 0 184320 1184 200 .xdg-desktop-po
[ 146.727947] [ 1858] 995 1858 5139 661 0 661 0 61440 32 0 rtkit-daemon
[ 146.727948] [ 1861] 1000 1861 28987 740 0 740 0 106496 1088 200 pipewire
[ 146.727949] [ 1862] 1000 1862 120461 895 0 895 0 159744 1312 200 wireplumber
[ 146.727950] [ 1872] 1000 1872 132706 1470 0 1470 0 462848 6688 0 Xwayland
[ 146.727952] [ 1880] 1000 1880 87774 1023 0 1023 0 364544 2176 200 .xdg-desktop-po
[ 146.727953] [ 1904] 1000 1904 357740 722 0 722 0 245760 928 0 foot
[ 146.727954] [ 1905] 1000 1905 3707 509 0 509 0 65536 896 0 zsh
[ 146.727956] [ 1929] 1000 1929 39406 650 32 618 0 86016 32 200 gpg-agent
[ 146.727957] [ 2148] 1000 2148 357143 749 96 649 4 237568 800 0 foot
[ 146.727958] [ 2149] 1000 2149 3705 697 23 674 0 73728 896 0 zsh
[ 146.727960] [ 2208] 1000 2208 2276 666 160 506 0 57344 32 0 watch
[ 146.727961] [ 2221] 1000 2221 278822 3699 2208 1491 0 339968 7328 0 .nix-index-wrap
[ 146.727962] [ 2243] 1000 2243 3467629 2634806 2633185 1621 0 27791360 817216 0 nix-env
[ 146.727964] [ 2260] 0 2260 112092 482 0 482 0 122880 896 0 nix-daemon
[ 146.727965] [ 2450] 1000 2450 4248 916 162 754 0 69632 0 0 swaylock
[ 146.727967] [ 2451] 1000 2451 4364 913 292 589 32 69632 0 0 swaylock
[ 146.727968] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-3.scope,task=nix-env,pid=2243,uid=1000
[ 146.727980] Out of memory: Killed process 2243 (nix-env) total-vm:13870516kB, anon-rss:10532740kB, file-rss:6484kB, shmem-rss:0kB, UID:1000 pgtables:27140kB oom_score_adj:0