Cannot get python312Packages.faster-whisper working with CUDA

Help
1

Hello, I wish to run python312Packages.faster-whisper with CUDA on my GPU. However, this does not seem to work as when I try and do that from the Python REPL, I am simply given the error that ctranslate2 was not compiled with CUDA support. What do I need to do to fix this issue? I cannot run this on the CPU since it’s too slow.

This is my shell.nix:

{ pkgs ? import <nixpkgs> {} }:

pkgs.mkShell {
    buildInputs = with pkgs; [
        python312
        (with python312Packages; [
            faster-whisper
            openai
        ])
        cudaPackages.cudatoolkit

        gcc
        cmake
        ninja
        clang-tools

        onnxruntime
        zlib
        openssl
        libopus
    ];
}

I am on nixpkgs unstable.

REPL log:

>>> from faster_whisper import WhisperModel
>>> model_size = "large-v3"
>>> model = WhisperModel(model_size, device="cuda", compute_type="float16")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/vf1dxjfw04p7sy9k2m64q64b9qf2p1a9-python3.12-faster-whisper-unstable-2024-07-26/lib/python3.12/site-packages/faster_whisper/transcribe.py", line 145, in __init__
    self.model = ctranslate2.models.Whisper(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: This CTranslate2 package was not compiled with CUDA support
2

As the error suggests, the CTranslate2 package, a dependency of faster-whisper, wasn’t compiled with CUDA support. You can fix that issue by overriding the withCUDA attribute in the ctranslate2 package of the faster-whisper package like this:

(with python312Packages; [
    (faster-whisper.override { ctranslate2 = pkgs.ctranslate2.override { withCUDA = true; }; }) 
    openai
])

Unlike some other packages, the withCUDA argument does not default to config.cudaSupport. I hope this fixes your issue!

2 Likes
3

ctranslate2 actually uses config.cudaSupport, but just not in the derivation itself:

So to enable it in the shell, you can do:

{
  pkgs ? import <nixpkgs> {
    config.cudaSupport = true;
    config.allowUnfree = true;
  },
}:

If you need CUDA support for just faster-whisper, then the override by @nanoyaki will work nicely (you may also have to enable withCuDNN as well, though, I’m not sure here).

Also, you can refactor this as:

    (python312.withPackages (
      ps: with ps; [
        faster-whisper
        openai
      ]
    ))
2 Likes
4

This seems to work, but now I get this issue where it’s building pytorch from source. Is there a way to force it to use torch-bin? or torchWithCuda?

5

You can enable the CUDA Cache for Nix Community so a pre-compiled package is downloaded, if it exists. I don’t know if the cache can be enabled per development shell, but I think it would be better if you do it system-wide as it’s quite useful in general:

# configuration.nix
nix.settings = {
  substituters = [ "https://nix-community.cachix.org" ];
  # Compare to the key published at https://nix-community.org/cache
  trusted-public-keys = [ "nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=" ];
};

Note that you’ll have to switch to your configuration with the cache enabled first, before it can take any effects.

1 Like