Custom kernel module ending up in the wrong place

That’s wrong, it must be in the kernel-ABI specific lib dir for it to work with NixOS options.

What happens when you insmod that file?

Are you sure you put it into the correct directory? It must be

$out/lib/modules/${kernel.modDirVersion}/updates/

i.e.

$out/lib/modules/6.6.34/updates/modulenamehere.ko.xz

Check the source. I.e. nixpkgs/pkgs/os-specific/linux/nct6687d/default.nix at 213b7c561bdd292f11ff99373222c2545c53ce7a · NixOS/nixpkgs · GitHub

How and where the module file is installed however is defined by the package’s build system however. That’s what you need to consult. You may need to set a specific installTarget like I had to in the module above.