I went through similar problems, and can confirm that trying to override Python packages can make you desperate…
For me, it turned out that overriding using pythonPackagesExtensions (see https://github.com/NixOS/nixpkgs/blob/e52b59f96b04a7c87a68596ea36577460574c654/doc/languages-frameworks/python.section.md#how-to-override-a-python-package-for-all-python-versions-using-extensions-how-to-override-a-python-package-for-all-python-versions-using-extensions or Overriding Python Modules - #7 by Ozone6418 for examples) is much more robust for Python packages.