🦀 kani

kani is an OpenAI-compatible local proxy that automatically routes LLM requests to the most suitable model. It classifies each request by prompt complexity, required capabilities, and your cost/quality profile.

Use kani when you want to:

• use one OpenAI-compatible endpoint across OpenAI, OpenRouter, local proxies, and other providers
• reduce cost by routing simple prompts to cheaper models
• keep stronger models for complex, agentic, or reasoning-heavy work
• inspect routing decisions through headers, logs, and debug endpoints

• Python 3.13+
• uv
• At least one OpenAI-compatible provider API key, for example OPENROUTER_API_KEY

This classifies a prompt without running the proxy server.

uvx --from git+https://github.com/tumf/kani kani route "hello world"

git clone https://github.com/tumf/kani.git
cd kani
uv sync
cp config.example.yaml config.yaml
cp .env.example .env

Edit .env:

OPENROUTER_API_KEY=your-openrouter-api-key

Start kani:

uv run kani serve

By default, kani listens on http://localhost:18420/v1.

Send an OpenAI-compatible request:

curl http://localhost:18420/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kani/auto",
    "messages": [
      {"role": "user", "content": "explain quicksort"}
    ]
  }'

Debug routing without proxying upstream:

uv run kani route "explain quicksort"

kani speaks the OpenAI API. Change base_url and model; keep the rest of your client code the same.

from openai import OpenAI

client = OpenAI(api_key="sk-...")

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "explain quicksort"}],
)

from openai import OpenAI

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key="sk-or-...",
)

response = client.chat.completions.create(
    model="anthropic/claude-sonnet-4",
    messages=[{"role": "user", "content": "explain quicksort"}],
)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:18420/v1",
    api_key="kani-local-dev",  # ignored unless kani API key auth is enabled
)

response = client.chat.completions.create(
    model="kani/auto",
    messages=[{"role": "user", "content": "explain quicksort"}],
)

response = client.chat.completions.create(
    model="kani/premium",
    messages=[{"role": "user", "content": "prove P != NP"}],
)

Any tool or library that supports the OpenAI API works with kani: LangChain, LlamaIndex, Cursor, Continue, and similar clients.

kani uses two different kinds of keys:

1. Upstream provider keys
   • Used by kani to call OpenAI, OpenRouter, local proxies, and other providers.
   • Configured in config.yaml or environment variables such as OPENROUTER_API_KEY.
2. kani proxy API keys
   • Used by clients to authenticate to kani itself.
   • Optional. If no kani keys are configured, requests are accepted without authentication.

Enable kani proxy authentication:

uv run kani keys add my-client

Use the generated key as the OpenAI client API key:

client = OpenAI(
    base_url="http://localhost:18420/v1",
    api_key="kani-aBcDeFgH...",
)

When at least one kani proxy key exists, every API request must include Authorization: Bearer <key>. /health and /docs are exempt.

Profiles are examples. Tune names, strategies, and model mappings for your workload and cost/quality goals.

config.yaml:

host: "0.0.0.0"
port: 18420

default_provider: openrouter
default_profile: auto

providers:
  openrouter:
    name: openrouter
    base_url: "https://openrouter.ai/api/v1"
    api_key: "${OPENROUTER_API_KEY}"

profiles:
  auto:
    tiers:
      SIMPLE:
        primary: "gpt-4o-mini"
      MEDIUM:
        primary: "gpt-4o-mini"
      COMPLEX:
        primary: "gpt-4o"
      REASONING:
        primary: "gpt-4o"

For a full example with embeddings, model metadata, fallback backoff, tool detection, and context compaction, see config.example.yaml.

Config path resolution order:

1. --config flag
2. $KANI_CONFIG
3. ./config.yaml
4. $XDG_CONFIG_HOME/kani/config.yaml
5. /etc/kani/config.yaml

kani does not rewrite model IDs. Configured model IDs are sent literally to the selected provider.

For OpenRouter, use OpenRouter model IDs:

primary: "anthropic/claude-sonnet-4"

For OpenAI, use OpenAI model IDs:

primary: "gpt-4o"

To route the same profile to different providers, set provider on the model entry or tier.

kani detects required capabilities from the request and routes to a model that supports them. If no model in the scored tier has the required capabilities, kani escalates to higher tiers.

Declare model metadata with prefix matching:

model_rules:
  - prefix: "anthropic/claude-"
    capabilities: [vision, tools, json_mode]
  - prefix: "google/gemini-"
    capabilities: [vision, tools, json_mode]
  - prefix: "gpt-4"
    capabilities: [vision, tools, json_mode]

model_rules is the primary metadata key. The legacy model_capabilities key is accepted only when model_rules is unset.

Use /v1/route to see which tier and model kani would choose without sending the request upstream.

curl http://localhost:18420/v1/route \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kani/auto",
    "messages": [
      {"role": "user", "content": "write a detailed migration plan"}
    ]
  }'

For proxied requests, inspect response headers:

curl -i http://localhost:18420/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"kani/auto","messages":[{"role":"user","content":"hello"}]}'

Look for:

• X-Kani-Tier
• X-Kani-Model
• X-Kani-Score
• X-Kani-Signals

Request → Distilled Feature Classifier → Tier + Agentic Score → Capability Filter → Model Selection → Upstream Provider
                                                   │
                                                   └─ model unavailable → conservative default

Classification pipeline:

1. Deterministic tokenCount plus learned semantic dimensions.
2. Separate complexity and reasoning scores drive tier selection.
3. agenticTask is exposed as agentic_score without affecting tier.
4. Missing feature model, embedding config, embedding request, or prediction path falls back to MEDIUM.
5. Capability filtering detects vision, tools, and JSON mode requirements.

Runtime routing does not call an LLM. LLM usage is limited to optional offline dataset annotation.

kani is distilled-feature-first:

• compute tokenCount deterministically
• infer 14 semantic dimensions with a learned multi-output classifier
• compute separate complexity and reasoning axis scores
• determine SIMPLE, MEDIUM, COMPLEX, or REASONING
• expose agentic_score independently
• return conservative default routing when feature scoring is unavailable

Tier thresholds:

• REASONING: score > 0.72
• COMPLEX: score > 0.58
• MEDIUM: score > 0.20
• SIMPLE: score <= 0.20

simpleIndicators is inverse-scored: high (trivial prompt) maps to 0.0 and lowers the composite score, matching the annotator calibration.

A score landing within a configurable band of a tier boundary is ambiguous between the two adjacent tiers. Each boundary names which tier wins via prefer: LOWER or prefer: UPPER, so the fail direction is a per-boundary cost/quality decision rather than a blanket default:

ambiguous_bands:
  SIMPLE_MEDIUM:     {band: 0.02, prefer: UPPER}   # fail up to MEDIUM
  MEDIUM_COMPLEX:    {band: 0.02, prefer: UPPER}   # fail up to COMPLEX
  COMPLEX_REASONING: {band: 0.02, prefer: LOWER}   # fail down to COMPLEX

When unset, behaviour is plain threshold mapping. See config.example.yaml for full details.

Routing improves through retraining and calibration rather than runtime prompt engineering.

Context compaction is opt-in and disabled by default. It can compact oversized message histories inline before proxying upstream and pre-compute summaries in the background for later reuse.

Minimal shape:

smart_proxy:
  context_compaction:
    enabled: true
    sync_compaction:
      enabled: true
      threshold_percent: 80.0
    background_precompaction:
      enabled: true
      trigger_percent: 70.0
    session:
      header_name: X-Kani-Session-Id
    context_window_tokens: 128000

Compaction headers:

Compaction state is persisted in SQLite under $XDG_DATA_HOME/kani/compaction.db by default. Override with KANI_DATA_DIR.

Set KANI_ADMIN_TOKEN to enable admin-only config reload:

export KANI_ADMIN_TOKEN="your-admin-token"
curl -X POST http://localhost:18420/admin/reload-config \
  -H "Authorization: Bearer ${KANI_ADMIN_TOKEN}"

Reload validates with strict config validation. Changes to host or port are rejected with 409 and require restart.

All routing decisions are logged to $XDG_STATE_HOME/kani/log/routing-YYYY-MM-DD.jsonl, defaulting to ~/.local/state/kani/log/.

Token usage per request is logged to execution-YYYY-MM-DD.jsonl in the same directory. Each completed request produces exactly one execution record (streaming requests included) containing the final cumulative prompt_tokens / completion_tokens / total_tokens and total wall-time elapsed_ms. The dashboard ingests these records into SQLite for analytics.

Use logs to build training data:

uv run python scripts/build_agentic_dataset.py \
  --output data/distilled_feature_dataset.json

Annotate missing semantic labels offline:

uv run python scripts/build_agentic_dataset.py \
  --annotate-missing \
  --output data/distilled_feature_dataset.json

Train the feature classifier bundle:

uv run python scripts/train_classifier.py \
  --data data/distilled_feature_dataset.json \
  --output models

This writes models/feature_classifier.pkl with the classifier, label encoders, weights, thresholds, and embedding metadata.

The classifier head is a scaled multi-layer perceptron (StandardScaler + MLPClassifier((128, 32))) chosen over logistic regression by stratified 5-fold cross-validation (scripts/compare_embeddings.py): mean held-out accuracy 0.741 -> 0.799, macro-F1 0.699 -> 0.742, improving all 14 dimensions. Embedding and classifier-head variants can be re-evaluated on cached embeddings without re-embedding:

uv run python scripts/compare_embeddings.py \
  --models voyage-4 \
  --heads linear,mlp,histgb \
  --folds 5

Optional annotator configuration:

feature_annotator:
  model: "gemini-2.5-flash-lite"
  provider: "openrouter"

Priority is CLI flags, environment variables, config.yaml feature_annotator, then built-in defaults.

Annotation requests ask for a JSON object (response_format.type = "json_object") and use temperature 0; the selected model must support json_mode. Models configured for the mistral provider and deepseek-v4-flash:0731-cloud/ollamacloud support this.

kani serve [--config path] [--host 0.0.0.0] [--port 18420]
kani route "your prompt here" [--config path]
kani config [--config path]
kani keys add <name>
kani keys list
kani keys remove <name|prefix>

src/kani/
├── scorer.py    # distilled feature scoring
├── router.py    # tier to model/provider mapping
├── proxy.py     # FastAPI OpenAI-compatible server
├── config.py    # YAML config loading and env var resolution
├── dirs.py      # XDG-compliant directory paths
├── logger.py    # JSONL routing log
└── cli.py       # Click CLI

uv sync --dev
uv run ruff check src/
uv run ruff format --check src/ tests/
uv run pyright src/
uv run pytest tests/ -q
uv build

Scoring logic ported from ClawRouter under the MIT license.

MIT