I don’t use home manager, but maybe this will help you.
nixpkgs.lib.nixosSystem takes a pkgs argument. You can use the nixpkgs input to construct a pkgs that has your custom package in it with an overlay. Here is an example.
defining pkgs
calling nixosSystem