epic(streaming): SSE delivery-path throughput — project/serialize once per role, not per subscriber

[EricAndrechek]

Area: streaming · perf — throughput-ceiling epic · found via WaveHouse-Stats dogfooding (load tests in docs/load-testing/)

The live SSE fan-out re-does per-subscriber what could be done once per (role, table). Measured ceiling ≈ 2 270 deliveries/sec, flat across event size — caps the public dashboard at ~2270 / event_rate clean concurrent viewers (~110 during a ~20 ev/s CI burst). Fine at today's traffic; the #1 lever before the streaming embed (LiveDemo) scales. Grouping the delivery-path optimizations so they're tracked without backlog spam (the #194 / #228 pattern); split into discrete issues when promoted to Ready.

Root cause (shared): projection + serialization live in the per-client read loop (internal/api/stream.go:116-145), not in the shared broadcast (internal/api/hub.go:62-92, which already marshals the envelope once and non-blocking-sends to N channels).

• Project + serialize once per (role, table, column-set), not per subscriber — the big lever. applyStreamPolicy (stream.go:136,162-193) runs json.Unmarshal + policy.Evaluate + filterEventColumns (fresh map alloc) + json.Marshal per subscriber, though the result is byte-identical for a given (role, table). For a single-role audience (the public dashboard, every viewer public) this collapses N re-projections to 1, moving the ceiling from deliveries/s toward events/s. Memoize inside Broadcast, or pre-project in the hub-bridge handler keyed by role.
• Don't start an OTel span per event per subscriber on the delivery path (done in fix(stream): emit SSE heartbeats so idle streams survive proxy timeouts #346 — per-delivery SSE.PushEvent span removed from the fan-out path) — stream.go:129-143 runs propagator.Extract(...) + tracer.Start("SSE.PushEvent") … pushSpan.End() per fan-out leg (N×R spans/s), and the Extract/Start calls execute even with WH_OTEL_ENABLED=false. Trace once at ingest/broadcast, or gate the per-delivery span behind a sampling/debug flag.
• Surface (and optionally shed) slow-consumer drops instead of dropping silently — hub.go:86-91 for ch := range subs { select { case ch <- data: default: } } drops over-capacity messages with no counter/log/client signal; saturation looks identical to "quiet." Count drops (per-topic metric/log); optionally disconnect a persistently-full subscriber so EventSource reconnects + gap-fills via Last-Event-ID. Cross-link feat(observability): latency histograms, error-rate counters, saturation gauges, query-path traces #94.
  • Update (from perf(stream): project SSE frames once per role, not per subscriber #353 review, taitelee): the drop metric shipped in perf(stream): project SSE frames once per role, not per subscriber #353 (wavehouse_sse_dropped_frames_total); the eviction half is what's left, and it has two coupled parts, not one:
    1. Trigger — a consecutive-drop threshold closes Subscriber.Evicted().
    2. Interruptible write — closing Evicted() is inert while the handler is wedged in w.Write (no WriteTimeout on the API server, no SetWriteDeadline on the path), so neither Evicted() nor r.Context().Done() can be observed on a connected-but-stalled client. Bound each write with http.NewResponseController(w).SetWriteDeadline(now + writeTimeout) so a wedged write returns an error and unwinds into the deferred Hub.Remove. Both write sites need it: the live loop (internal/api/stream.go frame pump) and the replay path (sendReplay → w.Write). Pick writeTimeout against the keepalive cadence. Not a perf(stream): project SSE frames once per role, not per subscriber #353 regression — pre-existing since the cap-1 keepalive loop.
• Eliminate redundant JSON (re)parsing on delivery — ~4 JSON ops/delivery: envelope unmarshal (stream.go:121-127), EventMessage unmarshal (:165), output marshal (:188), then extractEventTimestamp (:150-157) re-unmarshals the just-produced output only to read received_timestamp — which applyStreamPolicy already had as evt.ReceivedTimestamp (:185). Return it from applyStreamPolicy; carry trace context out-of-band (or skip when tracing off) to drop the per-client envelope unmarshal. Folds into the project-once fix.
• perf(api): right-size SSE per-subscriber channel buffer (currently hardcoded 64, untested) #152 — right-size the hardcoded 64-slot subscriber buffer (stream.go:73, self-flagged TODO) and reduce broadcast lock/range cost at high N (hub.go:83-91 holds the hub RLock and ranges the whole subscriber set per message). Idle holding ceiling ≈ 5 000 streams.

Related: #152 (buffer + lock), #94 (saturation metrics), #226 (SSE heartbeat).

---

From WaveHouse-Stats WAVEHOUSE-FEEDBACK.md dogfooding (load tests vs dev 70c15b0); validated by code-read against b7c69ef (main) on 2026-06-08 via /pm-triage.

[coderabbitai]

🔗 Related PRs

#124 - chore(api)!: drop hub wildcard fan-out; SSE/WS use ?table= (closes #100) [merged]
#174 - refactor(table names): handle unsafe table names [closed]
#182 - refactor: full api --> ingest --> clickhouse --> dlq refactor [closed]
#346 - fix(stream): emit SSE heartbeats so idle streams survive proxy timeouts [merged]

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!