How can I pre-load models to reduce cold start latency?

By default, a model is loaded into GPU memory for the first time once an API request is received that resolves to use the model. This means that the first time a model is served after start-up, a request may experience several seconds of latency.

Deepgram offers an option to pre-load individual models, so that the model-loading time does not impact API requests. This is currently available only for speech-to-text (STT) models.

This functionality is offered as-is, and is not guaranteed to be supported indefinitely. Deepgram reserves the right to modify model loading processes for the overall optimal functioning of the self-hosted offering.

Pre-loading models

Add the following type of configuration block to your engine.toml file:

[preload_models]
models = [
  { model = "2-general-nova", version = "2024-01-11.36317", language = "en", format = false },
  { model = "2-general-nova", version = "2023-11-21.6308",  language = "fr", format = false },
  { model = "2-general-nova", version = "1999-05-16.19331", language = "multi", format = false }
]

It is beneficial to be as specific as possible, to ensure that the desired model(s) are pre-loaded. You can do so by additionally specifying the model version, language, and format (whether the model is smart-formatted).

If no language is specified for a model, the default language is US English (en-US). To specify a language, use the format language = "fr". To specify a multilingual model, use the format language = "multi" .

To determine a model's version, you can use the /GET models endpoint, either on models already loaded in a self-hosted environment (doc), or to list Deepgram's publicly available models (doc).

Verifying that a model was pre-loaded

To verify that a model was pre-loaded, consult your Engine container logs. For instance, with Docker containers:

sudo docker ps
sudo docker logs <engine container ID>

You'll find on startup that several patterns of log lines are generated. When a model pre-load is initiated, a Preloading model line is logged. Once the pre-load has been attempted, you'll see that it either succeeded with two lines (await_model and load_model), or failed with one line (Failed to find model to preload).

A successful model pre-load

2025-05-29T14:36:51.196624617Z  INFO impeller: Preloading model query=ModelQuery { keywords: [], speakers_of_interest: [], soi_threshold: None, model: "2-general-nova", tier: Base, version: "2024-01-11.36317", language: Language("en"), diarize: false, diarize_version: None, identify: false, unify_speaker_id: false, max_speakers: None, punctuate: false, alternatives: 1, context: false, full_vec: false, channels: None, tag: None, profanity_filter: false, detect_language: {}, sentiment: false, format: false, entity_prompt: None }
...
2025-05-29T14:36:51.197408817Z  INFO await_model{model_uuid=1dbdfb4d-85b2-4659-9831-16b3c76229aa}: impeller::model_suppliers: new
2025-05-29T14:36:51.197606267Z  INFO load_model{model_uuid=1dbdfb4d-85b2-4659-9831-16b3c76229aa}: impeller::model_suppliers: new

Note that once a model successfully pre-loads, it logs the model's UUID (in this case, 1dbdfb4d-85b2-4659-9831-16b3c76229aa).

A failed model pre-load

2025-05-29T14:36:58.753059563Z  INFO impeller: Preloading model query=ModelQuery { keywords: [], speakers_of_interest: [], soi_threshold: None, model: "2-general-nova", tier: Base, version: "1999-05-16.19331", language: Language("multi"), diarize: false, diarize_version: None, identify: false, unify_speaker_id: false, max_speakers: None, punctuate: false, alternatives: 1, context: false, full_vec: false, channels: None, tag: None, profanity_filter: false, detect_language: {}, sentiment: false, format: false, entity_prompt: None }
2025-05-29T14:36:58.753085015Z  WARN impeller: Failed to find model to preload

If a model fails to pre-load, no model UUID is logged.

Reasons why models fail to pre-load

• The model exists, but it is not present in the models directory. Download the desired model into the models directory.

• The combination of requested model metadata (model, version, language, format) is not valid (the model does not exist). Correct the metadata.

• It is possible that a set of model metadata is too vague, such that it matches multiple models. Only one is selected and loaded. To select the desired model, be specific with all four types of metadata (model, version, language, format).

• The model is not a STT model. If you would like to load other types of models, please reach out to your Deepgram representative to express interest.