"nixos-rebuild switch --upgrade" gets me OOM

I hope that someone can me assist me on this. I’ve tried to get rid off the problem for 2 weeks but didn’t find a solution myself.

...
updating GRUB 2 menu...
lsblk: /dev/mapper/no*[0-9]: kein blockorientiertes Gerät
lsblk: /dev/mapper/raid*[0-9]: kein blockorientiertes Gerät
lsblk: /dev/mapper/disks*[0-9]: kein blockorientiertes Gerät
Found Windows 10 on /dev/sda1
activating the configuration...
/nix/store/vmkijy3aslx3cn7ry0qj355ra1wxc5yb-nixos-system-nixos-20.03.2516.674ab2dffa5/activate: line 63:  6185 Killed                  /nix/store/sg42gk32rbx5b3klmw9ql2sy165rnq8v-perl-5.30.1/bin/perl -w -I/nix/store/8s02cl42y2xjn87bj12ckcf7jg6wzh7n-perl5.30.1-File-Slurp-9999.29/lib/perl5/site_perl -I/nix/store/blknnx27sq31hbqdzh1i3z06af8lkd93-perl5.30.1-JSON-4.02/lib/perl5/site_perl /nix/store/w5s7s1nnzsmzdqh06abysffrj2m0nrwf-update-users-groups.pl /nix/store/m5kaj04ly5x646580l2p0x42jiv106qr-users-groups.json
Activation script snippet 'users' failed (137)
setting up /etc...
reloading user units for root...
reloading user units for sddm...
reloading user units for guido...
setting up tmpfiles
reloading the following units: dbus.service
restarting the following units: polkit.service
warning: error(s) occurred while switching to the new configuration

I do get the same error in line 63 when “starting activation script” in stage #2 of the boot process. Booting needs now between 20 - 60 minutes (before 1 - 2 minutes I guess).

Memory consumption is high from nscd, polkit, file.so … So I’ve stopped some services. Still the rebuild fails. The perl script consumes up to 2 GB.

Does booting into an older generation help to overcome the high boot duration?

When booted into an older generation, are you able to upgrade and switch from there?

That was my 1st idea. But it didn’t help.

Then I removed older generations and did a rebuild. That didn’t help either.

extra info on memory consumption

[root@nixos:~]# ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -n 20
966 10.1 /nix/store/q3q926jschmm9k5gsbbslpk70gqfpyrp-polkit-0.116/lib/polkit-1/polkitd
585 9.9 /nix/store/685dw13pxvjyysmdshbhbwvv03prl7jh-systemd-243.7/lib/systemd/systemd-udevd
1354 9.8 /nix/store/dz5kmyfd52hs6fdva2fn2d63f5p47hq0-cups-2.3.3/sbin/cupsd
1260 3.0 /nix/store/vdazblchhz6fvsl7jw29pcijdw0fqbph-plasma-workspace-5.17.5/bin/plasmashell
1245 1.1 /run/current-system/sw/bin/kwin_x11
1365 1.0 /run/current-system/sw/bin/ksysguard
922 1.0 /nix/store/9ihmw0ylna0rajgfsy9frkddk0y5qqa2-xorg-server-1.20.8/bin/X
1387 0.8 /run/current-system/sw/bin/konsole
1285 0.5 /nix/store/z4digpwrvym1x5wiqz0bfhg9z1qp2nrs-powerdevil-5.17.5/libexec/org_kde_powerdevil
1211 0.4 /run/current-system/sw/bin/kded5
1419 0.4 file.so
1477 0.4 file.so
1459 0.3 /nix/store/685dw13pxvjyysmdshbhbwvv03prl7jh-systemd-243.7/lib/systemd/systemd-journald
1207 0.2 /nix/store/aafm8n4p41nmd61j7352ylhf9lwh918x-kinit-5.66.0/libexec/kf5/klauncher
1228 0.2 /nix/store/vdazblchhz6fvsl7jw29pcijdw0fqbph-plasma-workspace-5.17.5/bin/plasma_session
785 0.2 /nix/store/rd8sabd4rvgllw44lx8dpjzsabkcglmn-network-manager-1.22.10/sbin/NetworkManager
1 0.1 systemd
1020 0.1 /nix/store/vqp6kf692aaqz2kjdn1pvg9g0dbcsdkh-kwallet-5.66.0-bin/bin/kwalletd5
1038 0.1 /nix/store/a8zqwbgmdcxc0zmzcnvs5g9znbzqh67d-pulseaudio-13.0/bin/pulseaudio
1226 0.1 /run/current-system/sw/bin/kaccess

[root@nixos:~]# systemctl stop nscd

[root@nixos:~]# systemctl stop polkit

[root@nixos:~]# systemctl stop cups  

[root@nixos:~]# ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -n 20
585 9.9 /nix/store/685dw13pxvjyysmdshbhbwvv03prl7jh-systemd-243.7/lib/systemd/systemd-udevd
1260 3.0 /nix/store/vdazblchhz6fvsl7jw29pcijdw0fqbph-plasma-workspace-5.17.5/bin/plasmashell
1245 1.1 /run/current-system/sw/bin/kwin_x11
1365 1.0 /run/current-system/sw/bin/ksysguard
922 1.0 /nix/store/9ihmw0ylna0rajgfsy9frkddk0y5qqa2-xorg-server-1.20.8/bin/X
1387 0.8 /run/current-system/sw/bin/konsole
1285 0.5 /nix/store/z4digpwrvym1x5wiqz0bfhg9z1qp2nrs-powerdevil-5.17.5/libexec/org_kde_powerdevil
1211 0.4 /run/current-system/sw/bin/kded5
1419 0.4 file.so
1477 0.4 file.so
1459 0.3 /nix/store/685dw13pxvjyysmdshbhbwvv03prl7jh-systemd-243.7/lib/systemd/systemd-journald
1207 0.2 /nix/store/aafm8n4p41nmd61j7352ylhf9lwh918x-kinit-5.66.0/libexec/kf5/klauncher
1228 0.2 /nix/store/vdazblchhz6fvsl7jw29pcijdw0fqbph-plasma-workspace-5.17.5/bin/plasma_session
785 0.2 /nix/store/rd8sabd4rvgllw44lx8dpjzsabkcglmn-network-manager-1.22.10/sbin/NetworkManager
1 0.1 systemd
1020 0.1 /nix/store/vqp6kf692aaqz2kjdn1pvg9g0dbcsdkh-kwallet-5.66.0-bin/bin/kwalletd5
1038 0.1 /nix/store/a8zqwbgmdcxc0zmzcnvs5g9znbzqh67d-pulseaudio-13.0/bin/pulseaudio
1226 0.1 /run/current-system/sw/bin/kaccess
1233 0.1 /run/current-system/sw/bin/ksmserver
1237 0.1 /nix/store/kag6a4h9mxzs6v2l8izmkayfyzwl0i27-kglobalaccel-5.66.0-bin/bin/kglobalaccel5

[root@nixos:~]# ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -n 20
1772 10.4 /nix/store/q3q926jschmm9k5gsbbslpk70gqfpyrp-polkit-0.116/lib/polkit-1/polkitd
585 9.9 /nix/store/685dw13pxvjyysmdshbhbwvv03prl7jh-systemd-243.7/lib/systemd/systemd-udevd
1705 9.5 /nix/store/l6dp0q42rdkshwyv1jr5x4v7fbs922f5-firefox-unwrapped-77.0.1/bin/.firefox-wrapped
1758 4.1 /nix/store/l6dp0q42rdkshwyv1jr5x4v7fbs922f5-firefox-unwrapped-77.0.1/lib/firefox/firefox
1984 3.6 /nix/store/l6dp0q42rdkshwyv1jr5x4v7fbs922f5-firefox-unwrapped-77.0.1/lib/firefox/firefox
1260 3.3 /nix/store/vdazblchhz6fvsl7jw29pcijdw0fqbph-plasma-workspace-5.17.5/bin/plasmashell
1853 3.1 /nix/store/l6dp0q42rdkshwyv1jr5x4v7fbs922f5-firefox-unwrapped-77.0.1/lib/firefox/firefox
1873 2.9 /nix/store/l6dp0q42rdkshwyv1jr5x4v7fbs922f5-firefox-unwrapped-77.0.1/lib/firefox/firefox
1965 1.7 /nix/store/l6dp0q42rdkshwyv1jr5x4v7fbs922f5-firefox-unwrapped-77.0.1/lib/firefox/firefox
1245 1.2 /run/current-system/sw/bin/kwin_x11
922 1.1 /nix/store/9ihmw0ylna0rajgfsy9frkddk0y5qqa2-xorg-server-1.20.8/bin/X
1365 1.0 /run/current-system/sw/bin/ksysguard
1387 0.8 /run/current-system/sw/bin/konsole
1211 0.6 /run/current-system/sw/bin/kded5
1285 0.5 /nix/store/z4digpwrvym1x5wiqz0bfhg9z1qp2nrs-powerdevil-5.17.5/libexec/org_kde_powerdevil
1419 0.4 file.so
1477 0.4 file.so
1459 0.3 /nix/store/685dw13pxvjyysmdshbhbwvv03prl7jh-systemd-243.7/lib/systemd/systemd-journald
1207 0.2 /nix/store/aafm8n4p41nmd61j7352ylhf9lwh918x-kinit-5.66.0/libexec/kf5/klauncher
1228 0.2 /nix/store/vdazblchhz6fvsl7jw29pcijdw0fqbph-plasma-workspace-5.17.5/bin/plasma_session

[root@nixos:~]# systemctl stop polkit

[root@nixos:~]# 

nscd used 1.7 GB of RAM (shown in KSysGuard).

next try: I removed some of the users that I had setup in configuration.nix and did a rebuild.

no solution …

Do I have to install from scratch?

Try to remove some packages from environment.systemPackages and update. If it works, add the packages you removed.

At the risk of sounding like a simpleton, is adding more swap temporarily an option? I understand that if there’s a memory leak in the perl script this won’t help, but if the script just needs a bit more space it might help. Even plugging in an old usb stick might do it.

removing packages didn’t help either

I run this laptop without swap and it worked like a charm before this happened.

I’ve traced the memory allocation of the script and it allocated more than 2 GB (!!!). Even this is a time consuming task. Adding swap will only prolong this.

Sorry but I won’t follow up on this.

Would it be possible to do a

nixos-install

in the existing installation?

Yes, like in the first installation process, you can mount the drives and do the nixos-install again.
(That is very handy if you system freezes in booting process [after you did a garbage collection before reboot])

tried … but got this:


[root@nixos:~]# nixos-generate-config --root /sda5
Failed to execute: btrfs subvol show /sda5/

[root@nixos:~]# echo $?
2

Is BTRFS not supported with this? I only wanted to mount the existing partition of the installation guide (section 2.3.1.) and proceed from there.

The existing installation had been a Linux MINT installation migrated to Nixos.

what if you do

mount /dev/disk/by-label/"${nix}" /mnt 
nixos-generate-config --root /mnt

mount had been:

mount /dev/sda5 /sda5

so your proposal wouldn’t have made a difference; nevertheless: Many thanks for looking into this.

solution: I followed the installation guide (section 2.5.4. step #12 onwards)

I just should have done

users.users.root.initialHashedPassword = "";

from step #9 beforehand. So I had to tinker a little bit on /etc/shadow with a LIVE USB stick.