Why is my self-hosted deployment not as performant as expected?

When planning your self-hosted deployment, your Deepgram Account Representative will work with you to estimate the hardware requirements to meet your latency and throughput needs. However, as your needs evolve, there may be cases where your deployment is not as performant as expected.

Deepgram will work closely with you to optimize your environment. The GPU hardware is the most expensive part of your deployment, and therefore should always be the bottleneck in an efficient setup. That means if a non-GPU factor is limiting your performance, your deployment is being under-utilized. This article describes some of those non-GPU factors.

Networking Throttles

CPU cores and system memory are the most commonly cited factors when allocating compute. However, networking is important to consider as well.

Bandwidth

Aggregate bandwidth should be sufficient to download audio files provided in requests, and return results to clients.

If bandwidth is an issue, you may consider compressed audio formats for pre-recorded audio, or upgrading the available bandwidth for your environment.

Packet-per-Second (PPS) performance

Many cloud providers and ISPs have limits on how many network packets per second will be accepted. Excess packets are often silently dropped. This is particularly disruptive to streaming requests, and can cause performance degradation that is difficult to track down.

Many cloud providers have guides on measuring networking performance to determine if networking throttling is the root cause of your deployment's performance concerns.

• AWS EC2: Monitor network performance for ENA settings on your EC2 instance

• AWS EKS: Capturing the metrics to monitor workloads for network performance issues

• GCP: Performance Dashboard Overview

• See cloud provider documentation for guides on your specific platform

PPS performance is typically an issue in streaming environments, where one requests may send frequent packets over a websocket connection. You may consider sending larger, less frequent audio payloads for streaming connections before upgrading the PPS threshold in your environment. For example, if on one livestreaming websocket connection, you send an audio payload every 20ms, you may consider buffering audio in your code and sending an audio payload every 40ms instead.

However, there