AI Gateway

View Markdown

IDE & client integrations.mdDownload .md

# Runku AI Gateway — IDE / client setup

Point any **OpenAI-compatible** client at `runku-ai`. The gateway holds the
real provider key; clients only see a consumer key (`rk_ai_...`).

## Bootstrap (once)

```bash
export RUNKU_AI_MASTER_KEY=$(openssl rand -hex 32)
runku-ai serve --config /etc/runku/config.ai.yaml

# other terminal
runku-ai admin key create --config /etc/runku/config.ai.yaml
runku-ai provider add --config /etc/runku/config.ai.yaml \
  --name openai --kind openai --api-key "$OPENAI_API_KEY"
runku-ai model enable --config /etc/runku/config.ai.yaml \
  --id gpt-4o-mini --provider openai --upstream gpt-4o-mini
runku-ai consumer create --config /etc/runku/config.ai.yaml --id ide --name "IDE"
runku-ai consumer quota set --config /etc/runku/config.ai.yaml --id ide --max-tokens 5000000
runku-ai consumer key create --config /etc/runku/config.ai.yaml --id ide
# → save the rk_ai_... value; shown once
```

**Base URL:** `http://127.0.0.1:8088/v1`  
**API key:** `rk_ai_...`  
**Model id:** clients always use `runku/<id>` (e.g. `runku/gpt-5-mini`).
The gateway accepts both `runku/<id>` and the bare catalog id.

Streaming (`"stream": true`) is supported and is what Cursor / Continue /
OpenCode use by default. L1 response cache applies only to non-streaming
`temperature: 0` (and embeddings).

Cursor Agent BYOK often sends Responses-shaped `tools` (flat `name` /
`type: "custom"`). The gateway rewrites those to Chat Completions function
tools and drops unsupported custom tools before calling upstream.

Image attachments from Cursor Agent use `{type:"image", data, mimeType}`
(or `image_url` objects with extra keys). The gateway rewrites them to
Chat Completions `{type:"image_url", image_url:{url, detail?}}` so OpenAI
does not reject with "Unexpected keys in a message content image dict."

---

## Cursor

Cursor often **does not** populate the model picker from `GET /v1/models`.
You must add the gateway model id by hand:

1. **Cursor Settings → Models**
2. Enable **OpenAI API Key** → paste `rk_ai_...`
3. Enable **Override OpenAI Base URL** = `http://127.0.0.1:8088/v1`
4. **Add Model** / custom model → type the exact id (e.g. `runku/gpt-5-mini`)
5. Toggle that model **ON**. Fully quit Cursor (**Cmd+Q**) and reopen.
6. In chat, select that model (not Auto / Cursor-only).

`runku-ai plugin install cursor` writes settings + `~/.cursor/runku-models.txt`
with these steps. Provider label: **Runku AI Gateway**.

---

## Claude Code

Claude Code uses the **Anthropic Messages** API (`POST /v1/messages`). Runku
translates that to OpenAI chat completions upstream.

```bash
set -a; source ~/.config/runku/claude-code.env; set +a
claude --model runku/gpt-5-mini -p "ping"
```

Important: `ANTHROPIC_BASE_URL` must be the gateway **origin**
(`http://127.0.0.1:8088`), **not** `.../v1` — the SDK appends `/v1/messages`.
Model ids always carry the `runku/` prefix.

`plugin install claude-code` also sets `model` + `env` in `~/.claude/settings.json`.
To revert Claude to Anthropic defaults (no gateway):

```bash
runku-ai plugin uninstall claude-code
```

---

## OpenCode

Uses OpenAI-compatible chat. Provider display name: **Runku AI Gateway**.
Model selection: `runku/<id>` (e.g. `runku/gpt-5-mini`). Newer OpenAI models
(`gpt-5*`) reject `max_tokens`; the gateway rewrites to `max_completion_tokens`.

```bash
opencode run -m runku/gpt-5-mini "ping"
```

---

## VS Code (Continue, Cline or similar)

Use an extension that accepts an OpenAI-compatible base URL:

### Continue (`continue.dev`)

In `~/.continue/config.json` (or Continue UI):

```json
{
  "models": [
    {
      "title": "Runku",
      "provider": "openai",
      "model": "gpt-4o-mini",
      "apiBase": "http://127.0.0.1:8088/v1",
      "apiKey": "rk_ai_..."
    }
  ]
}
```

### Cline / Roo / similar

Provider: **OpenAI Compatible**  
Base URL: `http://127.0.0.1:8088/v1`  
API Key: `rk_ai_...`  
Model: your registered id

---

## Cursor (custom model registration)

**Do not quit Cursor from an agent chat** — that kills the session.

### Hard limit (not a Runku bug)

Cursor BYOK does **not** call your machine directly:

```text
Cursor app → Cursor cloud → Override OpenAI Base URL
```

So `http://127.0.0.1:8088/v1` / `localhost` / LAN IPs fail with:

> Access to private networks is forbidden

While Override + OpenAI key point at that broken URL, **other OpenAI-family
models in Cursor also fail**. Restore them: Settings → Models → turn **OFF**
Override OpenAI Base URL and the OpenAI API Key toggle (`Cmd+Shift+0`).

VS Code Copilot, OpenCode, Continue, etc. call localhost **directly** — they
keep working with `127.0.0.1`.

### Working Cursor setup

1. Expose the gateway with a public HTTPS tunnel:

```bash
cloudflared tunnel --url http://127.0.0.1:8088
# or: ngrok http 8088
```

2. Install with the **public** `/v1` URL (Terminal.app, Cursor quit):

```bash
runku-ai plugin install cursor \
  --base https://<your-tunnel>/v1 \
  --key rk_ai_… \
  --model gpt-5-mini
# or: ~/.cursor/runku-fix-cursor.sh  (after updating ~/.cursor/runku.json baseUrl)
```

3. Pick `runku/gpt-5-mini` in chat.

The installer writes `openAIBaseUrl`, `useOpenAIKey`, `cursorAuth/openAIKey`,
and `userAddedModels` into `state.vscdb` (not only `settings.json`). For a
private `--base` it will **not** leave Override pointing at localhost.

---

## OpenCode / other OpenAI-compatible CLIs

| Setting | Value |
|---------|--------|
| Base URL | `http://127.0.0.1:8088/v1` |
| API Key | `rk_ai_...` |
| Model | `runku/<id>` in OpenCode, or bare `<id>` elsewhere |

If the client appends `/v1` itself, set base URL to `http://127.0.0.1:8088`
instead (the gateway also serves `/chat/completions` and `/models`).

Smoke:

```bash
opencode run -m runku/gpt-5-mini "ping"
```

---

## OpenAI SDK (smoke)

```bash
OPENAI_BASE_URL=http://127.0.0.1:8088/v1 \
OPENAI_API_KEY=rk_ai_... \
python -c "
from openai import OpenAI
c = OpenAI()
print(c.models.list())
r = c.chat.completions.create(model='gpt-4o-mini', messages=[{'role':'user','content':'hi'}], stream=True)
for chunk in r:
    print(chunk.choices[0].delta.content or '', end='')
"
```

---

## Auth headers accepted

- `Authorization: Bearer rk_ai_...` (preferred)
- `Authorization: rk_ai_...`
- `api-key: rk_ai_...`
- `x-api-key: rk_ai_...`

---

## Portal auth (IdP / OIDC)

Homologous to the MCP self-hosted portal:

| Setting | Behavior |
|---------|----------|
| `auth.enabled: false` | Portal + `/admin/*` open (local/dev) |
| `auth.enabled: true` | Gate: **Log in with SSO** (OIDC PKCE) and/or paste bearer |
| Admin API key | `runku-ai admin key create` → `rk_admin_…` |
| Discovery | `GET /auth/config` (never gated) — same shape as MCP |

```yaml
# Optional OIDC authentication for AI Gateway administrative access.
auth:
  # Require valid authentication on protected routes.
  enabled: true
  oidc:
    # Trusted issuer used for discovery and token verification.
    issuer: http://localhost:8081/realms/runku
    # Audience required in incoming AI Gateway tokens.
    audience: runku-ai
    # Public Authorization Code + PKCE client used by the portal.
    portal_client_id: runku-ai-portal   # public PKCE client
```

Register the portal redirect URI on the IdP, for example
`https://ai.example.com/portal/`.

## Portal

Set `server.public_base_url` in the AI config (e.g.
`https://ai.example.com`). The portal **Plugins**
tab walks operators through installing the `runku-ai` CLI, then **Get script**
for each IDE/agent. Scripts never embed consumer keys — they prompt at runtime.

```bash
runku-ai plugin list
runku-ai plugin script cursor --base http://127.0.0.1:8088/v1 --model gpt-4o-mini
runku-ai plugin install cursor --base http://127.0.0.1:8088/v1 --model gpt-4o-mini
# --key optional; omit to be prompted
```

Targets include Cursor, Windsurf, Zed, Antigravity, Trae, VS Code, Continue,
Cline, Roo, Kilo, JetBrains (via Continue), OpenCode, Claude Code, Aider,
Codex CLI, and Goose.

Open `https://ai.example.com/portal/` and authenticate with SSO or an
`rk_admin_...` key.

- Streaming has **no write timeout**; upstream header timeout is 120s.
- Quotas for streams use `stream_options.include_usage` (injected by the
  gateway when missing). Providers that ignore it still stream; token
  accounting may be zero for that call.
- Never put your real OpenAI/Anthropic key into Cursor/Continue — only
  the `rk_ai_` consumer key.

Runku AI Gateway — IDE / client setup

Point any OpenAI-compatible client at runku-ai. The gateway holds the real provider key; clients only see a consumer key (rk_ai_...).

Bootstrap (once)

export RUNKU_AI_MASTER_KEY=$(openssl rand -hex 32)
runku-ai serve --config /etc/runku/config.ai.yaml

# other terminal
runku-ai admin key create --config /etc/runku/config.ai.yaml
runku-ai provider add --config /etc/runku/config.ai.yaml \
  --name openai --kind openai --api-key "$OPENAI_API_KEY"
runku-ai model enable --config /etc/runku/config.ai.yaml \
  --id gpt-4o-mini --provider openai --upstream gpt-4o-mini
runku-ai consumer create --config /etc/runku/config.ai.yaml --id ide --name "IDE"
runku-ai consumer quota set --config /etc/runku/config.ai.yaml --id ide --max-tokens 5000000
runku-ai consumer key create --config /etc/runku/config.ai.yaml --id ide
# → save the rk_ai_... value; shown once

Base URL: http://127.0.0.1:8088/v1
API key: rk_ai_...
Model id: clients always use runku/<id> (e.g. runku/gpt-5-mini). The gateway accepts both runku/<id> and the bare catalog id.

Streaming ("stream": true) is supported and is what Cursor / Continue / OpenCode use by default. L1 response cache applies only to non-streaming temperature: 0 (and embeddings).

Cursor Agent BYOK often sends Responses-shaped tools (flat name / type: "custom"). The gateway rewrites those to Chat Completions function tools and drops unsupported custom tools before calling upstream.

Image attachments from Cursor Agent use {type:"image", data, mimeType} (or image_url objects with extra keys). The gateway rewrites them to Chat Completions {type:"image_url", image_url:{url, detail?}} so OpenAI does not reject with "Unexpected keys in a message content image dict."

Cursor

Cursor often does not populate the model picker from GET /v1/models. You must add the gateway model id by hand:

Cursor Settings → Models
Enable OpenAI API Key → paste rk_ai_...
Enable Override OpenAI Base URL = http://127.0.0.1:8088/v1
Add Model / custom model → type the exact id (e.g. runku/gpt-5-mini)
Toggle that model ON. Fully quit Cursor (Cmd+Q) and reopen.
In chat, select that model (not Auto / Cursor-only).

runku-ai plugin install cursor writes settings + ~/.cursor/runku-models.txt with these steps. Provider label: Runku AI Gateway.

Claude Code

Claude Code uses the Anthropic Messages API (POST /v1/messages). Runku translates that to OpenAI chat completions upstream.

set -a; source ~/.config/runku/claude-code.env; set +a
claude --model runku/gpt-5-mini -p "ping"

Important: ANTHROPIC_BASE_URL must be the gateway origin (http://127.0.0.1:8088), not .../v1 — the SDK appends /v1/messages. Model ids always carry the runku/ prefix.

plugin install claude-code also sets model + env in ~/.claude/settings.json. To revert Claude to Anthropic defaults (no gateway):

runku-ai plugin uninstall claude-code

OpenCode

Uses OpenAI-compatible chat. Provider display name: Runku AI Gateway. Model selection: runku/<id> (e.g. runku/gpt-5-mini). Newer OpenAI models (gpt-5*) reject max_tokens; the gateway rewrites to max_completion_tokens.

opencode run -m runku/gpt-5-mini "ping"

VS Code (Continue, Cline or similar)

Use an extension that accepts an OpenAI-compatible base URL:

Continue (`continue.dev`)

In ~/.continue/config.json (or Continue UI):

{
  "models": [
    {
      "title": "Runku",
      "provider": "openai",
      "model": "gpt-4o-mini",
      "apiBase": "http://127.0.0.1:8088/v1",
      "apiKey": "rk_ai_..."
    }
  ]
}

Cline / Roo / similar

Provider: OpenAI Compatible
Base URL: http://127.0.0.1:8088/v1
API Key: rk_ai_...
Model: your registered id

Cursor (custom model registration)

Do not quit Cursor from an agent chat — that kills the session.

Hard limit (not a Runku bug)

Cursor BYOK does not call your machine directly:

Cursor app → Cursor cloud → Override OpenAI Base URL

So http://127.0.0.1:8088/v1 / localhost / LAN IPs fail with:

Access to private networks is forbidden

While Override + OpenAI key point at that broken URL, other OpenAI-family models in Cursor also fail. Restore them: Settings → Models → turn OFF Override OpenAI Base URL and the OpenAI API Key toggle (Cmd+Shift+0).

VS Code Copilot, OpenCode, Continue, etc. call localhost directly — they keep working with 127.0.0.1.

Working Cursor setup

Expose the gateway with a public HTTPS tunnel:

cloudflared tunnel --url http://127.0.0.1:8088
# or: ngrok http 8088

Install with the public /v1 URL (Terminal.app, Cursor quit):

runku-ai plugin install cursor \
  --base https://<your-tunnel>/v1 \
  --key rk_ai_… \
  --model gpt-5-mini
# or: ~/.cursor/runku-fix-cursor.sh  (after updating ~/.cursor/runku.json baseUrl)

Pick runku/gpt-5-mini in chat.

The installer writes openAIBaseUrl, useOpenAIKey, cursorAuth/openAIKey, and userAddedModels into state.vscdb (not only settings.json). For a private --base it will not leave Override pointing at localhost.

OpenCode / other OpenAI-compatible CLIs

Setting	Value
Base URL	`http://127.0.0.1:8088/v1`
API Key	`rk_ai_...`
Model	`runku/<id>` in OpenCode, or bare `<id>` elsewhere

If the client appends /v1 itself, set base URL to http://127.0.0.1:8088 instead (the gateway also serves /chat/completions and /models).

Smoke:

opencode run -m runku/gpt-5-mini "ping"

OpenAI SDK (smoke)

OPENAI_BASE_URL=http://127.0.0.1:8088/v1 \
OPENAI_API_KEY=rk_ai_... \
python -c "
from openai import OpenAI
c = OpenAI()
print(c.models.list())
r = c.chat.completions.create(model='gpt-4o-mini', messages=[{'role':'user','content':'hi'}], stream=True)
for chunk in r:
    print(chunk.choices[0].delta.content or '', end='')
"

Auth headers accepted

Authorization: Bearer rk_ai_... (preferred)
Authorization: rk_ai_...
api-key: rk_ai_...
x-api-key: rk_ai_...

Portal auth (IdP / OIDC)

Homologous to the MCP self-hosted portal:

Setting	Behavior
`auth.enabled: false`	Portal + `/admin/*` open (local/dev)
`auth.enabled: true`	Gate: Log in with SSO (OIDC PKCE) and/or paste bearer
Admin API key	`runku-ai admin key create` → `rk_admin_…`
Discovery	`GET /auth/config` (never gated) — same shape as MCP

# Optional OIDC authentication for AI Gateway administrative access.
auth:
  # Require valid authentication on protected routes.
  enabled: true
  oidc:
    # Trusted issuer used for discovery and token verification.
    issuer: http://localhost:8081/realms/runku
    # Audience required in incoming AI Gateway tokens.
    audience: runku-ai
    # Public Authorization Code + PKCE client used by the portal.
    portal_client_id: runku-ai-portal   # public PKCE client

Portal

Set server.public_base_url in the AI config (e.g. https://ai.example.com). The portal Plugins tab walks operators through installing the runku-ai CLI, then Get script for each IDE/agent. Scripts never embed consumer keys — they prompt at runtime.

runku-ai plugin list
runku-ai plugin script cursor --base http://127.0.0.1:8088/v1 --model gpt-4o-mini
runku-ai plugin install cursor --base http://127.0.0.1:8088/v1 --model gpt-4o-mini
# --key optional; omit to be prompted

Targets include Cursor, Windsurf, Zed, Antigravity, Trae, VS Code, Continue, Cline, Roo, Kilo, JetBrains (via Continue), OpenCode, Claude Code, Aider, Codex CLI, and Goose.

Open https://ai.example.com/portal/ and authenticate with SSO or an rk_admin_... key.

Streaming has no write timeout; upstream header timeout is 120s.
Quotas for streams use stream_options.include_usage (injected by the gateway when missing). Providers that ignore it still stream; token accounting may be zero for that call.
Never put your real OpenAI/Anthropic key into Cursor/Continue — only the rk_ai_ consumer key.