Deploy as an OpenAI-compatible endpoint
A deployment serves a model version, or a base model with a system prompt, at its own chat-completions URL. Point any OpenAI SDK at it. Every response carries a prediction id, so feedback from your users flows back into the next training run.
What it does
Drop-in for the OpenAI SDK
Real SSE streaming, tools and tool_calls, response_format, stop, seed and the usual sampling parameters are forwarded to the model; errors use OpenAI's error shape.
Checked before it serves
Deploying makes a real test call first. If it fails, the deployment is marked failed with the error instead of serving a broken endpoint. Start and Stop control serving.
Revisions, rollback and canaries
Every change to what's served is a revision. Roll back to any earlier one, or send a percentage of traffic to another revision and compare their error rates and latency.
Keys and privacy per deployment
Deployment keys (kojen_dk_…) can call only their own endpoint and are metered separately. Choose what each request logs (nothing, metadata or redacted excerpts) and how long excerpts are kept.
Retraining from feedback
Rate a prediction with an optional correction. With auto-retraining on, a rising error rate or share of negative feedback starts a prompt-tuning run on the corrections; the result waits for review and is promoted only if it beats what's serving.
How it works
- 1Pick a model version (or a base model and system prompt) and deploy it.
- 2Create a deployment key and call the endpoint with any OpenAI client.
- 3Send feedback with each response's prediction id.
- 4Watch per-revision metrics, roll back or canary, and promote retrained versions.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://kojen.ai/api/v1/endpoints/<deployment-id>",
api_key=os.environ["KOJEN_DEPLOYMENT_KEY"], # kojen_dk_… for this deployment (or an owner key)
)
reply = client.chat.completions.create(
model="ignored", # the deployment decides the model and system prompt
messages=[{"role": "user", "content": "I was charged twice"}],
temperature=0.2,
max_tokens=256,
)
print(reply.choices[0].message.content)
prediction_id = reply.model_extra["kojen"]["prediction_id"]Limits, plainly
- 120 requests a minute per key and 600 per deployment; max 4,096 output tokens per request.
Questions
- Is the endpoint really OpenAI-compatible?
- It accepts the chat-completions request shape and returns chat.completion responses (or chat.completion.chunk events when streaming), plus a kojen object with the prediction id and revision. The deployment decides the model and system prompt, so the model field is ignored.
- Can I roll back a bad release?
- Yes. Every change creates a revision, and rolling back serves an earlier revision again. A canary split lets you try a new one on part of the traffic first.
- Can LoRA adapters be deployed?
- Not yet: deployments serve hosted models with their tuned prompts. Adapters from Text LoRA are saved as draft model versions.