Tags: weka/wekai
Tags
build: bump dagger to v0.21.7 and pin the dockerfile frontend
Releases began failing at Directory.dockerBuild with "grpc: failed to unmarshal
the received message: proto: wrong wireType = 0 for field LinuxResources", with
no commit in between and a retry failing identically.
Nothing in this repo changed. Every Dockerfile here begins
`# syntax=docker/dockerfile:1`, a MOVING tag, and Docker rolled it to 1.26
(sha256:ecfaec9e). That frontend speaks a buildkit protocol the engine inside
dagger v0.19.10 cannot unmarshal. A machine that had already cached the previous
frontend kept working until its cache was invalidated, which is why this looked
like a CI-only problem for a few hours and then reproduced locally.
Both halves of the fix are verified locally against this module, by running
dagger call build-router at each version:
v0.18.6 fails instantly — the engine cannot serve the introspection query
the Python SDK sends ("Unknown argument includeDeprecated"), which
is why the workflow pinned past it in the first place
v0.19.10 fails — the LinuxResources unmarshal above
v0.21.7 builds
So the CLI the workflow installs moves to 0.21.7, and dagger.json's engineVersion
moves with it rather than leaning on compat mode, so the two agree.
The frontend is now pinned by version AND digest in all four Dockerfiles. That
is the durable half: the version bump fixes today's break, the pin stops a third
party moving a tag from breaking a release again. It costs a manual bump to pick
up frontend updates, which is the right trade for a release path.
Coverage limit worth stating: build-router exercises Dockerfile.router. The
release also builds the root Dockerfile through push-helm, which needs
credentials and pulls a multi-GB replay image, so it was not run here. The
failure was in the dagger/buildkit protocol rather than in any Dockerfile's
content, and that protocol is now exercised and passing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KE8qPNPCKgPycxZgeXrYKz
fix(router): make the predicted cache fraction a token share, like th… …e observed one router_cache_predicted_fraction emitted matched BLOCKS over total blocks while router_cache_observed_fraction emitted cached TOKENS over prompt tokens, and the Grafana dashboard plotted the two against each other on one panel. Blocks are variable-sized here — kvcache chunks at 1024 bytes, so a 180-byte conversational turn and a full system chunk are both one block — so on agentic traffic the two quantities differ severalfold. The only closed loop the router has on whether prefix prediction is worth anything was comparing two different things. The predicted side is now scored through kvcache.Cover, the same helper the offline replay analyzer uses, so the router, the analyzer and the dashboard cannot drift apart again. The local block-ratio helper it replaces had no other caller and is gone. Two dashboard expressions were also wrong independently of the units: both panels selected a histogram's raw series rather than its mean, which renders one line per bucket. They now read sum/count over the rate interval, with the denominator clamped so an idle interval reads zero instead of NaN. Tests cover the gap rather than the fix. kvcache.Coverage shipped with none at all, so it gets a table covering both fractions, the missing counts, clamping of an out-of-range match and the empty request. At the router level two tests pin the metric itself: one asserts the observed value against a request whose blocks differ wildly in size (1000/10/10/10 tokens, one block matched — 0.97 as tokens, 0.25 as blocks) and names the block answer explicitly in its failure message; the other asserts the emitted value equals kvcache.Cover(...).TokenFraction() so the tie to the shared scorer is checked and not merely intended. Both fail against the previous behaviour, reporting 0.25 and 0.50 respectively. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KE8qPNPCKgPycxZgeXrYKz
fix(router): an API key confines the router to the dialect's own routes
A production config had to say two things to get one outcome:
FORWARD_PATH_ALLOWLIST=/v1/chat/completions,/v1/completions,/v1/embeddings,/v1/models
INBOUND_API_KEY=...
The second now implies the first. Setting a key says this listener faces users,
and a user-facing listener proxying arbitrary paths through to a backend is a
surface nobody asked for — so the allowlist defaults to the dialect's own route
table, all of it requiring the key, and everything else is refused. The
passthrough tier is closed, and so are the admin endpoints; an operator who
wants those asks for them.
The set comes from the dialect rather than a list written out beside it,
because a hand-maintained copy of a route table drifts from the table.
--path-allowlist still overrides it, and without a key nothing changes: empty
still means every path, which is what lets one router front a hosted API on
paths this dialect never claims.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013pb9d6n6D2CKnd8DbjFqHp
fix(deps): upgrade google.golang.org/grpc to v1.82.1 Closes a Dependabot alert. An indirect dependency, pulled in through the Kubernetes client used for pod discovery, so nothing in this repository calls it directly — but it ships in both images. genproto's api and rpc modules moved with it, as go mod tidy requires. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013pb9d6n6D2CKnd8DbjFqHp
test(gateway): say which half failed when the route label is missing
TestMetricsCarryTheMatchedRouteClass reports a delta on a process-global
counter, so its failure message describes a symptom it cannot actually
distinguish. A CI run failed with
router_requests_total{route=chat,dialect=openai} went 210 -> 210, want +1:
the matched route class did not reach the access-log middleware
which sends the reader into the middleware looking for a routing bug. But the
counter is bucketed by status too, so an unchanged count means EITHER the label
was wrong OR the request simply did not return 2xx — and the message asserts
the first while the second is at least as likely.
It now checks the response before the counter, and fails with the status and
body when the request did not succeed, since nothing can be concluded about a
label from a request that failed. The counter assertion also reports the
`unmatched` bucket alongside, which is the other place the count would have
gone.
Not a fix for the flake itself, which does not reproduce here across the
package under -race, the failing pair in sequence, or constrained CPU. It is
what makes the next occurrence say something.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013pb9d6n6D2CKnd8DbjFqHp
test(discovery): pin the pod port precedence rule `pods:<selector>` with no port suffix relies entirely on a pod's sole declared port being used, and that path had no test. The only pod-port case in the file always named its port, so the rule the tidiest configuration depends on was documented and implemented but unverified. Six cases now: a sole declared port beating the configured floor, with and without a name; no declared port falling back to the floor; several unnamed ports falling back rather than guessing; a named port picked out of several; and a named port that is absent falling back — deliberate, since picking a different one would send traffic where nothing is listening. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013pb9d6n6D2CKnd8DbjFqHp
refactor(chart)!: one endpoint grammar, and the last discovery flag i…
…s gone
Two ways to say one thing is bad. Two ways that LOOK alike and are subtly
different is worse, and that is what I had just built: a pod selector was a
list ITEM in `backends` but a SIBLING of `endpoints` inside a route. Same
concept, two nesting levels, and nothing to tell you which one a given context
wanted. `{url: ...}` was a third shape — a second way to write a plain string.
There is now one endpoint grammar, used everywhere an endpoint appears. A URL
is a string; a pod selector is a map. `backends` and a route's `endpoints` are
the same list of the same things:
backends:
- http://legacy-vllm:8000
- pods: {app: vllm}
port: http
routes:
- patterns: [fast]
endpoints:
- pods: {app: vllm, size: 7b}
port: http
Both render through one helper, so the grammar is defined once instead of
twice and left to drift. An endpoint that is neither form now fails the render
with a message saying so, rather than rendering to nothing and starting a
router with no backends.
--discover-kubeconfig is removed too, and it is the same complaint. It offered
to run discovery against a cluster from outside, but the namespace became "the
namespace this pod is in" unconditionally — and outside a pod there is no
service account file to read it from, so discovery failed before it started. A
flag that cannot do what it says is worse than no flag. Discovery is in-cluster
only, and a test now asserts no discover-* flag comes back.
BREAKING CHANGE: a route-level `pods:`/`port:` no longer works; put the
selector in that route's `endpoints` list. The `{url: ...}` endpoint form is
removed; write the URL as a plain string. --discover-kubeconfig is removed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013pb9d6n6D2CKnd8DbjFqHp
fix(gateway): stop publishing why the router ran out of room A caller was getting the router's prefix bookkeeping back in an error body: "every backend holding this prefix is saturated and no other is far enough below it to take a copy" That is a sentence about the router's internal state, on what may well be a public endpoint, and a caller can act on exactly one thing regardless of which mechanism declined — back off and retry. Every capacity rejection now answers the same way: 429, Retry-After, "server is busy; retry shortly". The reason did not disappear, it moved to where it is useful. Each rejection logs its specific cause with the request id and pool, and the counters already distinguish them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013pb9d6n6D2CKnd8DbjFqHp
fix(chart): drop default resource limits, raise requests to 4 CPU / 8Gi Shipping default limits next to default requests set a trap. Setting only resources.requests is the natural override — it is what an operator sizing a pod actually does — but Helm MERGES the maps, so the chart's old limits survived underneath the new requests and the pod was rejected for requesting more than it was allowed. The API error names the limit, so nothing in it points back at the chart. Limits are now unset, which lets a cluster's own LimitRange govern — the usual convention for a proxy. Requests are sized for a router doing real work: the prefix-affinity tree lives in memory and the routing decision is on every request's hot path. Both halves are independently overridable and either can be emptied, in which case the block is omitted rather than rendered empty. Requests are a scheduling floor, and the k8s e2e proved it immediately: four routers of that size do not fit on one k3d node, so the fourth sat Pending and the run failed with "outer router never became ready" — which looks nothing like a resource problem. The e2e now pins small requests, as values.yaml advises for a test cluster, and in doing so exercises the requests-only merge that the old default limits would have broken. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013pb9d6n6D2CKnd8DbjFqHp
PreviousNext