Openai-whisper failing on larger models (NVIDIA / CUDA issues)

Help
21

I’m curious, does this shell work or does it fail to build as well?

# shell.nix
{
  pkgs ? import <nixpkgs> {
    config = {
      allowUnfree = true;
      cudaSupport = false;
    };
    overlays = [
      (final: prev: {
        ctranslate2 = prev.ctranslate2.override {
          withCUDA = true;
          withCuDNN = true;
        };
      })
    ];
  },
}:
pkgs.mkShell {
  packages = with pkgs; [
    whisper-ctranslate2
  ];
}

You can activate it with:

$ nix-shell
22

For some reason that works and I’m able to run large v3 model with seemingly no issue.

1 Like
23

One possible explanation I have for this is that config.cudaSupport = true; enables CUDA for way too many packages than is needed and that might be a bit overkill. When I enabled it in the shell above, everything was compiling okay until it began building onnxruntime and that’s where my system freezes.

Having uncached dependencies indicates to me that they may not being built with this option enabled, but I don’t really know how to check.

On my system, I tried to enable this option once and faced a similar issue, that’s why I’ve opted to enabling CUDA on a per-package basis instead, using overrides/overlays like the one for ctranslate2 and I haven’t had any issues with that approach.

24

It finally works… Thank you for sticking with me throughout this issue

Just for conclusiveness, the final module that works is:

{
  config,
  lib,
  pkgs,
  ...
}:

{

  nixpkgs =  {
    config = {
      cudaSupport = false;
      allowUnfree = true;
    };

    # Set ctranslate2 cuda support
    overlays = [
      (final: prev: {
        ctranslate2 = prev.ctranslate2.override {
          withCUDA = true;
          withCuDNN = true;
        };
      })
    ];
  };


nix.settings = {
  substituters = [ "https://nix-community.cachix.org"];
  trusted-public-keys = [
    # Compare to the key published at https://nix-community.org/cache
    "nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs="
  ];
};



  environment.systemPackages = with pkgs; [
    python3Packages.pytorch-bin
    whisper-ctranslate2

  ];
}

With this I’m able to run the large-v3 whisper model just fine and its much faster than the original whisper

1 Like