Skip to content

OpenAI Compatible Client

Allows you to use Talemate with a service that exposes an openai-like API.

Use the officual client if it is available

Only use this if the service you are trying access doen't already have official support from Talemate. All the officially supported clients can be found here.

DeepInfra

If you want to add a DeepInfra client, change the Client Type to DeepInfra.

Client DeepInfra

Settings

Client Name

A unique name for the client that makes sense to you.

API Url

The URL of DeepInfra's openai api - currently https://api.deepinfra.com/v1/openai

API Key

Your DeepInfra api key. You can manage your DeepInfra API keys at https://deepinfra.com/dash/api_keys

API handles prompt template

This will cause requests to go to the chat/completions API instead and DeepInfra will be in control of the prompt template. This means if you enable this DeepInfra needs to be configured correctly and Talemate will also lose control of the prompt template, causing a likely loss in quality.

This setting is recommended to keep disabled.

Model name

Currently you need to manually specify which model to use by typing its name as it exists on deepinfra.com.

Decent choices:

microsoft/WizardLM-2-8x22B

nvidia/Nemotron-4-340B-Instruct

meta-llama/Meta-Llama-3-70B-Instruct

mistralai/Mixtral-8x22B-Instruct-v0.1
Context Length

The number of tokens to use as context when generating text. Defaults to 8192.

Parameters

The Parameters tab in the client settings lets you control which sampler parameters are sent with each request. Some OpenAI-compatible services reject requests that include parameters they don't support for the selected model, returning an error instead of a generation. If you run into this, you can turn the offending parameter off so it is left out of the request entirely.

Each parameter has its own toggle:

  • Send temperature
  • Send top_p
  • Send presence_penalty

All three are enabled by default. Turning a toggle off omits that parameter from the request completely (it is not sent as a zero value). Leave them enabled unless the service you are connecting to errors when it receives one of them.

Ready to use

Click Save to add the client.

Once it is added, the client should appear in the clients list and should display the currently loaded model.

Client DeepInfra Ready

Common issues

Generations are weird / bad

Make sure the correct prompt template is assigned.

Response Length Enforcement

Controls whether token limits and/or human-readable response length instructions are sent with prompts. Defaults to Limit tokens and send instructions.

See Response Length Enforcement for details.