AI SDK - OpenAI Compatible Provider

This package provides a foundation for implementing providers that expose an OpenAI-compatible API.

The primary OpenAI provider is more feature-rich, including OpenAI-specific experimental and legacy features. This package offers a lighter-weight alternative focused on core OpenAI-compatible functionality.

Setup

The provider is available in the @ai-sdk/openai-compatible module. You can install it with

1npm i @ai-sdk/openai-compatible

Provider Instance

You can import the provider creation method createOpenAICompatible from @ai-sdk/openai-compatible:

1import { createOpenAICompatible } from '@ai-sdk/openai-compatible';

Example

1import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
2import { generateText } from 'ai';
3
4const { text } = await generateText({
5  model: createOpenAICompatible({
6    baseURL: 'https://api.example.com/v1',
7    name: 'example',
8    apiKey: process.env.MY_API_KEY,
9  }).chatModel('meta-llama/Llama-3-70b-chat-hf'),
10  prompt: 'Write a vegetarian lasagna recipe for 4 people.',
11});

Customizing headers

You can further customize headers if desired. For example, here is an alternate implementation to pass along api key authentication:

1import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
2import { generateText } from 'ai';
3
4const { text } = await generateText({
5  model: createOpenAICompatible({
6    baseURL: 'https://api.example.com/v1',
7    name: 'example',
8    headers: {
9      Authorization: `Bearer ${process.env.MY_API_KEY}`,
10    },
11  }).chatModel('meta-llama/Llama-3-70b-chat-hf'),
12  prompt: 'Write a vegetarian lasagna recipe for 4 people.',
13});

Including model ids for auto-completion

1import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
2import { generateText } from 'ai';
3
4type ExampleChatModelIds =
5  | 'meta-llama/Llama-3-70b-chat-hf'
6  | 'meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo'
7  | (string & {});
8
9type ExampleCompletionModelIds =
10  | 'codellama/CodeLlama-34b-Instruct-hf'
11  | 'Qwen/Qwen2.5-Coder-32B-Instruct'
12  | (string & {});
13
14type ExampleEmbeddingModelIds =
15  | 'BAAI/bge-large-en-v1.5'
16  | 'bert-base-uncased'
17  | (string & {});
18
19const model = createOpenAICompatible<
20  ExampleChatModelIds,
21  ExampleCompletionModelIds,
22  ExampleEmbeddingModelIds
23>({
24  baseURL: 'https://api.example.com/v1',
25  name: 'example',
26  apiKey: process.env.MY_API_KEY,
27});
28
29// Subsequent calls to e.g. `model.chatModel` will auto-complete the model id
30// from the list of `ExampleChatModelIds` while still allowing free-form
31// strings as well.
32
33const { text } = await generateText({
34  model: model.chatModel('meta-llama/Llama-3-70b-chat-hf'),
35  prompt: 'Write a vegetarian lasagna recipe for 4 people.',
36});

For more examples, see the OpenAI Compatible Providers documentation.