Installing python with system package but with pip working too

I’m trying create nixpkgs environment that includes python but so that I could install packages with pip install --user.

I’m using ~/.nixpkgs/config.nix, which currently has something like:

{
  packageOverrides = pkgs: with pkgs; {
    myPackages = pkgs.buildEnv {
      name = "my-packages";
      ignoreCollisions = true;
      paths = [
        ...
        python3
        ...
      ];
    };
  };
}

which I can install with nix-env -i my-packages. The problem is that installing binary packages with pip install in this mode is impossible - the shared libraries used by binary modules are in Nix cache and not visible to pip-installed packages.

I tried different approach:

{
  packageOverrides = pkgs: with pkgs; {
    myPackages = pkgs.buildEnv {
      name = "my-packages";
      ignoreCollisions = true;
      paths = [
        ...
        (pkgs.python3.withPackages (
          ps: with ps; [
            numpy
            matplotlib
          ]
        ))
        ...
      ];
    };
  };
}

This way installed python already has needed packages (e.g. numpy and matploblib) but it’s a virtualenv environment so I can’t use pip with it.

What I want to have python with some packages specified in Nix config file but installed as non-virtualenv so that I could add packages I need with pip install. Is it possible?