Getting segmentation fault on 'nixos-rebuild switch' on latest 25.11 channel

Help
1

I haven’t updated in a while (2026-01-04 is my last good channel gen) and was experiencing a graphics issue and decided to do so today

My journey

  • As always, I start with sudo nix-channel --update. This did show a sign of problems to come as I got a non-zero exit code but there were no errors in the output so I thought it was just a weird fluke and just kept going
  • nixos-rebuild switch --upgrade –-sudo seemed to build without issue
  • Get password prompt after build, then seconds later segmentation fault (core dumped) nixos-rebuild switch --sudo
  • I run nixos-rebuild switch --upgrade –-sudo again and the only output in the console is immediately segmentation fault (core dumped) nixos-rebuild switch --sudo
  • Check journalctl
Mar 15 17:09:59 BigNix systemd-coredump[21466]: [🡕] Process 21448 (.nixos-rebuild-) of user 1000 dumped core.
                                                
                                                Module libgcc_s.so.1 without build-id.
                                                Stack trace of thread 21448:
                                                #0  0x00007f7d18515144 PyObject_Hash (libpython3.13.so.1.0 + 0x115144)
                                                #1  0x00007f7d1852bb99 PyDict_GetItemRef (libpython3.13.so.1.0 + 0x12bb99)
                                                #2  0x00007f7d17fad508 _datetime_exec (_datetime.cpython-313-x86_64-linux-gnu.so + 0x16508)
                                                #3  0x00007f7d1861d34e PyModule_ExecDef (libpython3.13.so.1.0 + 0x21d34e)
                                                #4  0x00007f7d1861d6f0 _imp_exec_dynamic (libpython3.13.so.1.0 + 0x21d6f0)
                                                #5  0x00007f7d18533341 cfunction_vectorcall_O (libpython3.13.so.1.0 + 0x133341)
                                                #6  0x00007f7d187c37a9 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x3c37a9)
                                                #7  0x00007f7d1864b66f object_vacall (libpython3.13.so.1.0 + 0x24b66f)
                                                #8  0x00007f7d1864b8db PyObject_CallMethodObjArgs (libpython3.13.so.1.0 + 0x24b8db)
                                                #9  0x00007f7d187576ea PyImport_ImportModuleLevelObject (libpython3.13.so.1.0 + 0x3576ea)
                                                #10 0x00007f7d187cfd90 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x3cfd90)
                                                #11 0x00007f7d187d5ae7 PyEval_EvalCode (libpython3.13.so.1.0 + 0x3d5ae7)
                                                #12 0x00007f7d187eb22f builtin_exec (libpython3.13.so.1.0 + 0x3eb22f)
                                                #13 0x00007f7d1857bca0 cfunction_vectorcall_FASTCALL_KEYWORDS (libpython3.13.so.1.0 + 0x17bca0)
                                                #14 0x00007f7d187c37a9 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x3c37a9)
                                                #15 0x00007f7d1864b66f object_vacall (libpython3.13.so.1.0 + 0x24b66f)
                                                #16 0x00007f7d1864b8db PyObject_CallMethodObjArgs (libpython3.13.so.1.0 + 0x24b8db)
                                                #17 0x00007f7d187576ea PyImport_ImportModuleLevelObject (libpython3.13.so.1.0 + 0x3576ea)
                                                #18 0x00007f7d187cfd90 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x3cfd90)
                                                #19 0x00007f7d187d5ae7 PyEval_EvalCode (libpython3.13.so.1.0 + 0x3d5ae7)
                                                #20 0x00007f7d187eb22f builtin_exec (libpython3.13.so.1.0 + 0x3eb22f)
                                                #21 0x00007f7d1857bca0 cfunction_vectorcall_FASTCALL_KEYWORDS (libpython3.13.so.1.0 + 0x17bca0)
                                                #22 0x00007f7d187c37a9 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x3c37a9)
                                                #23 0x00007f7d1864b66f object_vacall (libpython3.13.so.1.0 + 0x24b66f)
                                                #24 0x00007f7d1864b8db PyObject_CallMethodObjArgs (libpython3.13.so.1.0 + 0x24b8db)
                                                #25 0x00007f7d187576ea PyImport_ImportModuleLevelObject (libpython3.13.so.1.0 + 0x3576ea)
                                                #26 0x00007f7d1857bca0 cfunction_vectorcall_FASTCALL_KEYWORDS (libpython3.13.so.1.0 + 0x17bca0)
                                                #27 0x00007f7d187c37a9 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x3c37a9)
                                                #28 0x00007f7d1864b66f object_vacall (libpython3.13.so.1.0 + 0x24b66f)
                                                #29 0x00007f7d1864b8db PyObject_CallMethodObjArgs (libpython3.13.so.1.0 + 0x24b8db)
                                                #30 0x00007f7d18756dad PyImport_ImportModuleLevelObject (libpython3.13.so.1.0 + 0x356dad)
                                                #31 0x00007f7d187cfd90 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x3cfd90)
                                                #32 0x00007f7d187d5ae7 PyEval_EvalCode (libpython3.13.so.1.0 + 0x3d5ae7)
                                                #33 0x00007f7d187eb22f builtin_exec (libpython3.13.so.1.0 + 0x3eb22f)
                                                #34 0x00007f7d1857bca0 cfunction_vectorcall_FASTCALL_KEYWORDS (libpython3.13.so.1.0 + 0x17bca0)
                                                #35 0x00007f7d187c37a9 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x3c37a9)
                                                #36 0x00007f7d1864b66f object_vacall (libpython3.13.so.1.0 + 0x24b66f)
                                                #37 0x00007f7d1864b8db PyObject_CallMethodObjArgs (libpython3.13.so.1.0 + 0x24b8db)
                                                #38 0x00007f7d187576ea PyImport_ImportModuleLevelObject (libpython3.13.so.1.0 + 0x3576ea)
                                                #39 0x00007f7d187cfd90 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x3cfd90)
                                                #40 0x00007f7d187d5ae7 PyEval_EvalCode (libpython3.13.so.1.0 + 0x3d5ae7)
                                                #41 0x00007f7d187dcc6a run_eval_code_obj (libpython3.13.so.1.0 + 0x3dcc6a)
                                                #42 0x00007f7d187dce97 run_mod (libpython3.13.so.1.0 + 0x3dce97)
                                                #43 0x00007f7d187df3d5 _PyRun_SimpleFileObject (libpython3.13.so.1.0 + 0x3df3d5)
                                                #44 0x00007f7d187e1b08 Py_RunMain (libpython3.13.so.1.0 + 0x3e1b08)
                                                #45 0x00007f7d1802a4d8 __libc_start_call_main (libc.so.6 + 0x2a4d8)
                                                #46 0x00007f7d1802a59b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a59b)
                                                #47 0x0000558508e49085 _start (/nix/store/44rn0p64x92bnnh7cwn6x6ybvflybmvz-python3-3.13.12/bin/python3.13 + 0x1085)
                                                ELF object binary architecture: AMD x86-64
░░ Subject: Process 21448 (.nixos-rebuild-) dumped core
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ Documentation: man:core(5)
░░ 
░░ Process 21448 (.nixos-rebuild-) crashed and dumped core.
░░ 
░░ This usually indicates a programming error in the crashing program and
░░ should be reported to its vendor as a bug.
Mar 15 17:09:59 BigNix systemd[1]: systemd-coredump@11-8197-21465_21765-0.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit systemd-coredump@11-8197-21465_21765-0.service has successfully entered the 'dead' state.
Mar 15 17:09:59 BigNix systemd[1]: systemd-coredump@11-8197-21465_21765-0.service: Consumed 136ms CPU time, 16.1M memory peak, 1.5M written to disk.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit systemd-coredump@11-8197-21465_21765-0.service completed and consumed the indicated resources.
Mar 15 17:09:59 BigNix drkonqi-coredump-processor[21467]: "/nix/store/44rn0p64x92bnnh7cwn6x6ybvflybmvz-python3-3.13.12/bin/python3.13" 21448 "/var/lib/systemd/coredump/core.\\x2enixos-rebuild-.1000.e4284e927e364939a1dcf56dfa6df40d.21448.1773608999000000.zst"
Mar 15 17:09:59 BigNix systemd[1838]: Started Launch DrKonqi for a systemd-coredump crash (PID 21467/UID 0).
░░ Subject: A start job for unit UNIT has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit UNIT has finished successfully.
░░ 
░░ The job identifier is 1135.
Mar 15 17:09:59 BigNix drkonqi-coredump-launcher[21478]: Unable to find file for pid 21448 expected at "kcrash-metadata/python3.13.e4284e927e364939a1dcf56dfa6df40d.21448.ini"
Mar 15 17:09:59 BigNix drkonqi-coredump-launcher[21478]: Nothing handled the dump :O
Mar 15 17:09:59 BigNix systemd[1]: drkonqi-coredump-processor@11-8197-21465_21765-0.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit drkonqi-coredump-processor@11-8197-21465_21765-0.service has successfully entered the 'dead' state.
Mar 15 17:09:59 BigNix systemd[1]: drkonqi-coredump-processor@11-8197-21465_21765-0.service: Consumed 95ms CPU time, 10.4M memory peak, 12.4M read from disk.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit drkonqi-coredump-processor@11-8197-21465_21765-0.service completed and consumed the indicated resources.
  • nixos-rebuild switch --rollback --sudo immediately outputs segmentation fault (core dumped) nixos-rebuild switch --rollback --sudo
  • I did try rebooting and rolling back on startup, but this is a VM with GPU passthrough and sometimes that’s not possible, so of course this was one of those time
  • sudo nix-channel --rollback <prevGen> succeeded, finally giving me hope
  • nixos-rebuild switch --sudo also succeeded
  • Made small changes to my config just to get a real build and running the switch succeeded again
  • Am able to consistently reproduce it on this system with multiple reboots and garbage collections in between. The only difference is I only got that non-zero exit from sudo nix-channel --update that first time, but everything else results in the same exact issues

Given that the latest threads on here when searching for segmentation fault or even seg fault are from before my last update, except for 1 which OP said was a bad SSD and I’m confident that’s not my issue, I assume it must be a one off issue. Only thing I can think of is something got corrupted in that first channel update and it’s still lingering somehow.

Any ideas on how I can proceed without having to rebuild the whole system would be appreciated

2

More “progress”, figured out how to delete specific generations and deleted the bad build and the channel update I rolled back from. Garbage collection did purge out more data after each, but after trying to update again the same thing happens.

3

If you’re still on 25.11, you can try setting system.rebuild.enableNg to false, since the segfault is coming from Python.

4

That did it, thanks. This was the first I had seen that it was implemented in python, didn’t even consider it was something new

5

After finishing update (after a dozen or so switches cause I had package issues) I was able to re-enable system.rebuild.enableNg, and even added a new package to make sure it had to build, and it still works.

Wondering if when I updated back in Jan I picked up a bugged build of the rebuild python script and just needed this update to succeed to get it fixed.