Skip to content

OpenAI-compatible endpoint

Use a Kojen deployment with LlamaIndex

Call a Kojen-trained model from LlamaIndex with OpenAILike in Python or the OpenAI LLM in LlamaIndex.TS, including chat, streaming and function-calling agents.

Before you start

  • Deploy a model in Kojen (Deployments → New deployment). Its id goes in the base URL: https://kojen.ai/api/v1/endpoints/<deployment-id>.
  • Create a deployment key (kojen_dk_…) under Deployments → Learning, revisions & keys. It can only call that one deployment, so it's the right key to ship in an app. An account key (kojen_…, Settings → API Keys & MCP) of the deployment's owner with the read or write scope also works.
  • Keep the key server-side, e.g. in KOJEN_DEPLOYMENT_KEY. Anyone holding it can call the deployment and spend its owner's token allowance.

Python: OpenAILike

Use OpenAILike, not OpenAI: it doesn't validate the model name against OpenAI's list. Set is_chat_model=True, since the endpoint serves chat completions only, and set context_window to your deployed model's, because LlamaIndex can't look it up.

Python (llama-index-llms-openai-like)
pip install llama-index-llms-openai-like

import os
from llama_index.llms.openai_like import OpenAILike
from llama_index.core.llms import ChatMessage

llm = OpenAILike(
    model="deployed",  # ignored by Kojen
    api_base="https://kojen.ai/api/v1/endpoints/<deployment-id>",
    api_key=os.environ["KOJEN_DEPLOYMENT_KEY"],  # kojen_dk_…
    is_chat_model=True,           # Kojen serves chat completions only
    is_function_calling_model=True,  # tools are forwarded to the model
    context_window=128000,        # set to your deployed model's window
)

print(llm.chat([ChatMessage(role="user", content="I was charged twice")]))

for chunk in llm.stream_complete("App crashes on login"):
    print(chunk.delta, end="", flush=True)

Function-calling agents

With is_function_calling_model=True, LlamaIndex agents send tools and read the model's tool_calls, which the endpoint passes through.

Python
from llama_index.core.agent.workflow import FunctionAgent

def get_order(order_number: str) -> dict:
    """Look up an order by number."""
    return {"order_number": order_number, "status": "delivered", "days_ago": 12}

agent = FunctionAgent(tools=[get_order], llm=llm)
print(await agent.run("Is order 1042 refundable?"))

TypeScript: LlamaIndex.TS

LlamaIndex.TS has no OpenAILike; its OpenAI class takes any model name and a baseURL (or the OPENAI_BASE_URL environment variable), and supports tool calls for any model name.

TypeScript (@llamaindex/openai)
npm install llamaindex @llamaindex/openai

import { OpenAI } from "@llamaindex/openai";

const llm = new OpenAI({
  // Kojen ignores the model name, but LlamaIndex.TS looks it up for the context window
  // (unknown names fall back to 1,024 tokens), so pass an OpenAI model name.
  model: "gpt-4o-mini",
  baseURL: "https://kojen.ai/api/v1/endpoints/<deployment-id>",
  apiKey: process.env.KOJEN_DEPLOYMENT_KEY, // kojen_dk_…
});

const reply = await llm.chat({ messages: [{ role: "user", content: "I was charged twice" }] });
console.log(reply.message.content);

What the endpoint supports

  • model is ignored: the deployment decides the model and system prompt. Any string works; SDKs that require one just need something.
  • stream: true streams real SSE chunks ending in [DONE]; stream_options: { include_usage: true } adds a final usage chunk.
  • tools, tool_choice, parallel_tool_calls and response_format (text, json_object, json_schema) are forwarded to the model, and its tool_calls come back unchanged, so tool calling and structured output work through the SDK as usual.
  • Also forwarded: temperature, top_p, n (up to 4), stop, seed, the penalties and max_tokens / max_completion_tokens (up to 4,096).
  • Only POST /chat/completions and POST /feedback exist under the base URL. There is no /models, /embeddings or Responses API, so use the SDK's chat-completions client.
  • Each reply carries kojen.prediction_id (also the X-Kojen-Prediction-Id header). Post it to /feedback to rate a prediction and feed continuous learning.
  • Errors use OpenAI's { error: { message, type, param, code } } shape; 429s send Retry-After.

Get an API key

Kojen is free to start, no card needed. Train and evaluate an agent, deploy it behind an OpenAI-compatible endpoint, and connect your tools over MCP.

Create a free account

Also see: Vercel AI SDK · LangChain · OpenAI SDK · Claude Code · Cursor · Claude Desktop