Endpoints

Recipes

Recipes

Below are simple code snippets that show how to use your endpoints from a local machine.

Quickstart

All supported frameworks expose an OpenAI-compatible API. See the framework sections below for details and other ways to interact with your endpoints.

Start by installing the OpenAI Python client:

pip install openai

Copy your endpoint URL and API key from the Endpoints page, and specify the correct model ID:

from openai import OpenAI

client = OpenAI(
    base_url="<YOUR-ENDPOINT-URL>/v1/",
    api_key="<YOUR-API-KEY>",
)

chat_completion = client.chat.completions.create(
    model="<MODEL-ID>",
    messages=[
        {
            "role": "user",
            "content": "Say this is a test",
        }
    ],
)

If you are using the Ollama framework, you must pull the model before using it.

Ollama framework

Ollama is an easy-to-use LLM inference framework, ideal when peak throughput is not a priority. You do not need to create a new endpoint for each model because models can be pulled and managed via the Ollama API.

The Ollama Python client simplifies interaction with its API. Install it via pip:

pip install ollama

Pulling a model

Before using a model, pull it from Ollama's model hub. Depending on the model size, pulling may take several minutes. In the following example, we download the gpt-oss:120b model, which is 65 GB.

Copy your endpoint URL and API key from the Endpoints page:

import ollama

client = ollama.Client(
    host="<YOUR-ENDPOINT-URL>",
    headers={"Authorization": f"Bearer <YOUR-API-KEY>"},
)

client.pull("gpt-oss:120b")

# List downloaded models
client.list()

Generating chat messages

After pulling the model, you can run inference. The following example generates the next message in a chat.

Copy your endpoint URL and API key from the Endpoints page:

import ollama

client = ollama.Client(
    host="<YOUR-ENDPOINT-URL>",
    headers={"Authorization": f"Bearer <YOUR-API-KEY>"},
)

response = client.chat(model="gpt-oss:120b", messages=[
    {
        "role": "user",
        "content": "Why is the sky blue?",
    },
])

print(response["message"]["content"])

Check out the official Ollama API reference for a complete overview of the available endpoints and features, as well as the compatibility with OpenAI's API.

vLLM framework

vLLM is a high-performance and easy-to-use inference engine for LLMs. It is optimized for serving multiple requests in parallel with high throughput and low latency.

The vLLM framework focuses on providing an OpenAI-compatible API. Therefore, the recommended way to interact with its HTTP server is using the official OpenAI Python client that you can install via pip:

pip install openai

Choosing the model

When using this framework, you must specify the model in the Framework CLI arguments during endpoint creation. You can choose any vLLM-compatible model from the HuggingFace Model Hub.

Depending on the size of the model and whether it's cached, the endpoint building phase may take several minutes. You can monitor the logs to track model loading progress.

In the following example, we choose the openai/gpt-oss-120b model.

--model=openai/gpt-oss-120b

Configuring the server

vLLM allows you to simply configure the server via the Framework CLI arguments during endpoint creation. You can find the full list of available options in the vLLM documentation.

When using multiple GPUs, you must configure the parallelization strategy. By default we assume tensor parallelism, which shards the model weights across multiple GPUs.

In the following example, we use the smaller openai/gpt-oss-20b model with data parallelism across two GPUs (replicating the model) rather than tensor parallelism.

--model=openai/gpt-oss-20b --data-parallel-size=2

Generating chat messages

Once the endpoint is running, you can start using OpenAI's Completions API, Chat API, and more. For a complete list of supported APIs, see the vLLM documentation.

In the following example, we generate the next message in a chat.

Copy your endpoint URL and API key from the Endpoints page:

from openai import OpenAI

client = OpenAI(
    base_url="<YOUR-ENDPOINT-URL>/v1/",
    api_key="<YOUR-API-KEY>",
)

chat_completion = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "Say this is a test",
        }
    ],
)

vLLM supports parameters that are not supported by OpenAI. Check out the official API reference for a complete overview of available parameters, as well as the compatibility with OpenAI's API.