Skip to content
Whishper

GPU Support

Since v2.0.0, running Whishper with GPU is possible. This means that you can use the GPU in whishper to accelerate the transcriptions.

Some notes:

  • This feature has only been tested in GNU/Linux amd64 with an NVIDIA RTX.
  • This feature needs cuBLAS for CUDA 11 and cuDNN 8 for CUDA 11 as per faster-whisper requirements. I’m not sure it can run on non-NVIDIA GPUs, but this is dependent on the faster-whisper library.
  • If you are running on a different architecture or OS, please test it and report any issues you find. This will help improving GPU support in whishper.

Requirements

You must have a NVIDIA GPU with CUDA support. You can check if your GPU is supported by CUDA here. Then you should configure your OS properly to use the GPU. Here are some guides:

After that, you should install the NVIDIA Container Toolkit so that Docker can use the GPU. You can find the installation instructions here. Make sure you perform the installation and configuration steps.

Enabling GPU support in whishper

The get-whishper.sh script will prompt you to enable GPU support and automatically fetch the necessary files for you and set the necessary environment variables.

If you want to enable it later, you can do so by replacing the default docker-compose.yml file with the contents in the docker-compose.gpu.yml file. Make sure all requirements are satisfied before running docker-compose up -d.

This will also enable GPU support for LibreTranslate.

Upgrading from v2.x to v3.x

Upgrading should be simple:

  • Replace your docker-compose.yml file with the contents of the latest GPU compose file.
  • Make sure your .env file has the necessary environment variables set. You can check the example.env
    • COMPOSE_PROFILES is no longer needed.
  • Run docker-compose up -d to restart the containers with GPU support.

First pull will be slow as the new images are downloaded. The GPU image contains all the CUDA and Torch dependencies, so it’s a heavy image compared to the CPU one (7GB vs 450MB).

Upgrading from v1.x to v2.x

Upgrading should be simple:

  • Replace your docker-compose.yml file with the contents of the latest compose file.
  • Set the COMPOSE_PROFILES environment variable to gpu in the .env file.
  • Run docker-compose up -d to restart the containers with GPU support.