Agents in Production
Lesson 4 of 6Advanced 8 min

Deploy, monitor and learn from feedback

Shipping is the start of training, not the end. Serve behind a standard API, watch the golden signals, and feed real outcomes back in.

In this lesson you'll

  • Serve a model behind an OpenAI-compatible API
  • Monitor the four golden signals
  • Close the loop with feedback without chasing a proxy

The Chat Completions request shape has become a common interface for serving models [1], so serving yours the same way lets existing SDKs and tools call it unchanged.

Once live, monitor latency, traffic, errors and saturation, the SRE book's four golden signals [2], plus a quality signal: user ratings, corrections or downstream outcomes. Promote changes through staging, and keep the previous version ready to roll back.

Feedback without Goodhart

Thumbs-up rates are a proxy, and optimising hard against any proxy eventually hurts the real goal [3]. Feed negative examples and corrections into your test set, retrain against it, and check that the regression suite still passes before promoting [4].

curl https://kojen.ai/api/v1/endpoints/<deployment id>/chat/completions \
  -H "Authorization: Bearer $KOJEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"I was charged twice."}]}'

In Kojen

Deployments are live OpenAI-compatible endpoints. Every response includes a kojen.prediction_id; rating it at /feedback feeds auto-retraining, which starts new prompt-tuning runs when error rates or negative feedback pass a threshold [5].

Key takeaways

  • Use a standard API shape so anything can call your model.
  • Monitor the golden signals plus a quality signal.
  • Turn feedback into test cases, not just a score to maximise.

Practise in the dojo

Deploy your best model version to staging, call it with curl, and rate a few predictions.

Deploy to staging

Check your understanding

Score 70 or more to complete the lesson. Progress is saved in this browser.

1. Which is not one of the SRE book's four golden signals?

Further reading

Sources

Checked against the original sources in November 2026.

  1. [1]OpenAI (2025). Chat Completions API reference. platform.openai.com.
  2. [2]Beyer, B. et al. (eds.) (2016). Site Reliability Engineering, ch. 6: Monitoring Distributed Systems. O'Reilly / Google.
  3. [3]Gao, L., Schulman, J. & Hilton, J. (2022). Scaling Laws for Reward Model Overoptimization. arXiv:2210.10760.
  4. [4]Anthropic (2026). Demystifying evals for AI agents. Anthropic Engineering, January 2026.
  5. [5]Kojen (2026). Kojen developer docs: REST API, MCP server and endpoints. kojen.ai.