I made a derivation for a program, which is not in nixpkgs. It has a myapp/default.nix, and I usually just use callPackage and can then install the program in home-manager.
I don’t use home manager, but maybe this will help you. nixpkgs.lib.nixosSystem takes a pkgs argument. You can use the nixpkgs input to construct a pkgs that has your custom package in it with an overlay. Here is an example. defining pkgs calling nixosSystem
Move that into your flake as a package output, then make sure that you add _module.args.self = self; to one of your modules, and also pass self to your ./home.nix import, there you should also set _module.args.selg = self;.
You will get self as an argument to all modules then and make it accessible by adding it to the argset.
That can be used then to add self.packages.${system].hello to your home.packages.