> For the complete documentation index, see [llms.txt](https://deepgram.gitbook.io/help-center/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://deepgram.gitbook.io/help-center/self-hosted/why-is-my-self-hosted-engine-instance-running-on-the-cpu.md).

# Why is my self-hosted Engine instance running on the CPU?

When self-hosting Deepgram services, the Engine container is typically run on an instance with access to a GPU.  The GPU speeds the inference process by orders of magnitude faster than what can be processed by a CPU.  In order for the Engine to make use of the GPU, there needs to be the necessary drivers, configuration settings, and other considerations.  Below are some of the most common reasons why an Engine will not recognize or make use of the GPU.

## Reasons why the Engine does not recognize a GPU

### Compose File does not specify NVIDIA runtime

If using Docker or Podman, ensure that in your Docker Compose or Podman Compose file, the Engine node uses the NVIDIA runtime. If not explicitly specified, Deepgram will not attempt to use a GPU.

Docker:

```
  # The speech engine service.
  engine:
    image: quay.io/deepgram/onprem-engine:release-<version>

    # Utilize a GPU, if available.
    runtime: nvidia
```

Podman:

```
  # The speech engine service.
  engine:
    image: quay.io/deepgram/onprem-engine:release-<version>

    # Utilize a GPU, if available.
    devices:
      - nvidia.com/gpu=all
```

If using Kubernetes, make sure the Engine Deployment manifest requests a GPU. This is automatically applied in the `deepgram-self-hosted` Helm chart as well.

```
apiVersion: apps/v1
kind: Deployment
...
spec:
  template:
    spec:
      containers:
      - name: <engine-pod-name>
        resources:
          requests:
            nvidia.com/gpu: 1
```

### CUDA version incompatabilities

The underlying system needs to have the necessary CUDA version installed.  Since the [October 2023 self-hosted release](https://deepgram.com/changelog/deepgram-on-premises-october-release-231026) (`quay.io/deepgram/onprem-engine:release-231026` or `quay.io/deepgram/onprem-engine:3.59.2`) the minimum CUDA version is `12.1.1`.

### No NVIDIA Container Runtime available

The NVIDIA Container Runtime is a container runtime which allows your containers to access the GPUs on the machine.  It is typically installed with the [NVIDIA Container Toolkit](https://github.com/NVIDIA/nvidia-container-toolkit). &#x20;

With Kubernetes, please consider the [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html) tooling.

### Missing drivers

If the necessary drivers are not installed and the NVIDIA runtime  is enabled, typically this will result in an error.  Please reference our self-hosted [Drivers and Containerization Platforms](https://developers.deepgram.com/docs/drivers-and-containerization-platforms) documentation for information on installing the necessary drivers when using Docker/Podman. For Kubernetes, the `deepgram-self-hosted` [Helm chart](https://github.com/deepgram/self-hosted-resources) will automatically install the proper drivers for you.

### Detecting Common NVIDIA Issues with Docker/Podman

Deepgram's public self-hosted resources has a [diagnostic script](https://github.com/deepgram/self-hosted-resources/blob/main/diagnostics/dg_validate_nvidia_setup.sh) to validate NVIDIA issues when deploying with Docker/Podman. If you're encountering issues, you can download this script to your specific machine and run it to validate your setup.

```sh
wget https://raw.githubusercontent.com/deepgram/self-hosted-resources/refs/heads/main/diagnostics/dg_validate_nvidia_setup.sh
chmod +x ./dg_validate_nvidia_setup.sh
sudo ./dg_validate_nvidia_setup.sh
```

### Google Kubernetes Engine Bug from Oct 2023 - Jan 2024

There is a known library path issue with the following Deepgram self-hosted releases when run on Google Kubernetes Engine:

* `release-231026`
* `release-231114`
* `release-231207`
* `release-240104`

This prevents NVIDIA libraries from being detected, and the Engine containers will fall back to CPU mode even when a GPU is available.

The recommended workaround for these images is to upgrade to `release-240228` or later.

## How can I verify my self-hosted deployment is using the GPU?

On startup, the Engine container will output logs indicating whether it is using the CPU or GPU(s).  Those logs will look like:

```
INFO impeller::config: Using devices: Gpu(0)
```