Unfortunately this won’t work.
The module system separate between special arguments (e.g., lib, config, options) and “regular” arguments (the rest). In this case isVM is a regular argument.
When a module is loaded its arguments are introspected and regular arguments mapped to config._module.args.${name} where ${name} is the name of the argument. In other words,
lib.optional (!isVM) ./hardware_configuration.nix
is effectively the same as
lib.optional (!config._module.args.isVM) ./hardware_configuration.nix
which would explain the infinite recursion. Namely, to build the configuration we have to import the modules and to import the modules we have have to evaluate config._module.args.isVM, but to evaluate config._module.args.isVM we have to build the configuration…