How to build a nixos system or just a home directory from a single home-manager module?

To slowly migrate my dotfiles into home-manager, I would like to have the option to inspect and copy legacy dotfiles into a non-nix/os system.
I would like to have a single home.nix in my flake source tree that gets used in a output.nixosSystem and also in a output.homeManagerConfigurations.

Is there a way to define a single home-manager module for both output targets?
Or, alternatively, can I somehow home-manager build just the home directory from a nixosSystem target?

To reproduce the attempt, a minimal flake.nix and home.nix:
flake.nix:

{
  description = "multi target, single home-manager config";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";

    home-manager = {
      url = "github:nix-community/home-manager";
      inputs.nixpkgs.follows = "nixpkgs";
   };
 };

 outputs = { self, nixpkgs, home-manager }:
 {
   nixosConfigurations.OS = nixpkgs.lib.nixosSystem {
     system = "x86_64-linux";
     specialArgs = { inherit home-manager; };
     modules = [
       {
         boot.isContainer = true;
         users.users.USER = { isNormalUser = true; };
       }
       ./home.nix
     ];
   };

   homeConfigurations.HOME = home-manager.lib.homeManagerConfiguration {
     pkgs = nixpkgs.outputs.legacyPackages.x86_64-linux;
     extraSpecialArgs = { inherit home-manager; };
     modules = [ ./home.nix ];
   };
 };
}

home.nix:

{ home-manager, ... }:
{
  imports = [ home-manager.nixosModules.home-manager ];

  home-manager.useGlobalPkgs = true;
  home-manager.useUserPackages = true;

  home-manager.users.USER.home = {
    stateVersion = "24.11";
  };
}

Building the nixos system with nixos-rebuild --flake .#OS works fine but home-manager build --flake .#HOME fails, in this example case with error: The option 'home.stateVersion' is used but not defined.. Other tries failed with infinite recursion or even cache errors.

Just put somewhere in your NixOS config:

  imports = [ home-manager.nixosModules.home-manager ];

  home-manager.useGlobalPkgs = true;
  home-manager.useUserPackages = true;

  home-manager.users.USER.home = import ./home-modules;

And in your flake.nix:

homeConfigurations.HOME = home-manager.lib.homeManagerConfiguration {
   pkgs = nixpkgs.outputs.legacyPackages.x86_64-linux;
   extraSpecialArgs = { inherit home-manager; };
   modules = [ ./home-modules ];
 };

Then in home-modules/default.nix put an actual home-manager module, rather than a NixOS module that configures home-manager.

Here’s an example: