Quelle: ddx-documentation/adr/0020-voice-audio-transport.md
- **Status:** Accepted (plan `ai-assistant-platform`, task 004, 2026-10-29). No invariant exception: the browser leg stays browser -> ddx-web BFF -> ddx-api -> engine (operator order 2026-10-29). The plan's NEEDS_REVIEW item 1 (browser hop) is resolved without a waiver. - **Refines:** [ADR 0016](/de/docs/adr/0016-sse-bus) (one realtime UI path), [ADR 0017](/de/docs/adr/0017-bff-layered) (route classes), [realtime.md](/de/docs/realtime) (Centrifugo, Postgres authority). - **Plan:** `plans/ai-assistant-platform/` (invariants 2, 7, 13; failure mode FM1). - **Structure reference:** `findings/assistant-job/falcon-voice.md` (HMS Falcon = shape only; no code or values copied). - **Ports:** [../ports.md](/de/docs/ports). Port 9631 is claimed by the change that binds it (task 045). ## Context - **Voice needs a binary, low-latency, two-way audio path.** Microphone audio goes up as PCM16 at 16 kHz; synthesized speech comes down as provider PCM chunks. The voice stretch target is first audio < 800 ms p50 (task 048 measures it). - **Centrifugo is the only UI realtime path** and Postgres `osprey.event` is the authority (`context/_invariants.md`, [realtime.md](/de/docs/realtime)). Centrifugo delivers JSON publications after an outbox relay hop: fine for transcripts and UI events, wrong for 50 audio frames per second. - **The engine has no WebSocket server.** `ddx-osprey-engine/src/server.ts` is `node:http` on 127.0.0.1:9630. ddx-api and ddx-web carry no `ws` dependency. - **The browser may only call the ddx-web BFF** (`/api/v1/**`, ADR 0017). The browser-held Centrifugo socket is the one existing realtime socket; it is NOT a precedent for a direct browser -> ddx-api or browser -> engine socket. - **Web check (bounded, 2026-10-29 01:19 UTC).** Installed `next@16.4.3` has no route-handler WebSocket upgrade: `rg 'upgrade\('` over `node_modules/next/dist/server/web/spec-extension/response.js` and `rg webSockets` over `config-shared.js` both return no match. Upstream status: RFC "WebSocket Upgrades in Route Handlers" <https://github.com/vercel/next.js/discussions/95514> (Draft, target experimental, 2026-07-06) and PRs #89320 / #97027 (open). `next start` closes upgrade requests that resolve to app routes. ddx-web runs plain `next start --port 9600`. ## Decision 1. **Binary audio never rides Centrifugo.** Audio frames use a dedicated voice path. Transcripts (final), speak segments, tool activity, UI-control actions and voice state changes are appended to `osprey.event` by the engine and reach the browser on Centrifugo, as today. Interim captions and playback control ride the voice path (they are not durable state). 2. **Engine leg = WebSocket.** The engine runs a voice gateway on **127.0.0.1:9631** (loopback only, `ws` dependency). It accepts only ddx-api: the upgrade carries identity headers (tenant, user, session, stream id) and the internal engine credential the existing ddx-api -> engine calls use. 3. **ddx-api leg = relay at `/osprey/voice`.** ddx-api owns auth and ownership: session JWT, `OrgContextGuard` / ownership check on the Osprey session, then a single-use **voice ticket** minted by `POST /osprey/sessions/:id/voice-ticket` (TTL 120 s, one stream per ticket, bound to tenant + user + session). Opening a stream consumes the ticket; ddx-api opens the engine WebSocket and relays frames both ways. Closing either side closes the other. 4. **Browser leg = through the ddx-web BFF, no socket.** Until Next.js ships route-handler upgrades, the browser hop is two BFF routes in ONE new route class **`stream-voice`** (ADR 0017 table is data; the row lives in `packages/ddx-contracts/src/bff-routes.ts`, task 047): - **Down:** `GET /api/v1/osprey/voice/:streamId` — one long-lived streamed response (`application/octet-stream`, `no-store`, `no-transform`, header-phase deadline only, sticky node like `stream-sse`). Body = length-prefixed frames (1 byte kind + 4 byte length + payload): PCM chunks and JSON control frames. - **Up:** `POST /api/v1/osprey/voice/:streamId/frames` — `application/octet-stream` body with a monotonic sequence number header; 100 ms of PCM16 16 kHz (5 × 20 ms frames, 3200 B) per POST, and JSON control frames (start, stop, barge-in, activity) as their own POSTs. ddx-api drops duplicates and out-of-order frames by sequence. - **Class policy:** `no-store`, bearer credential, rate limit sized for audio (≥ 20 POSTs/s per session); it is the only class with that budget, so no other route gains a looser limit. The same-origin gate, identity layer and bridge stay as they are. 5. **Frame format.** Up: 20 ms PCM16 mono 16 kHz frames (batched 5 per POST on the HTTP leg, one per message on WebSocket legs). Down: provider-native PCM chunks (sample rate declared in the `start` ack) plus a JSON control channel: `start`, `stop`, `barge-in`, `state` (listening | thinking | speaking | error), `caption` (interim), `audio_end`, `fallback` (provider switched). The schemas live in `@ddx/contracts/osprey-voice` (task 005), Zod 4, versioned. 6. **Upgrade path.** When `next` ships `NextResponse.upgrade()` (RFC #95514), the browser leg becomes a WebSocket at the same BFF path under a `voice-ws` route class; the engine and ddx-api legs and the frame schemas do not change. That switch needs a one-row amendment of this ADR. Considered (rejected): a direct browser WebSocket to ddx-api or the engine with the Centrifugo ticket pattern — it breaks invariant 7 and the operator ruled it out on 2026-10-29. ## Rejected | Option | Why not | |---|---| | Audio over Centrifugo | JSON publications after an outbox hop; base64 PCM at 50 frames/s through the event log would flood `osprey.event` and the relay (FM1). | | Direct browser socket (ddx-api or engine) | Breaks "browser -> BFF only" (invariant 7). Operator order 2026-10-29: no waiver. | | Custom Next.js server with an `upgrade` listener | Replaces `next start`, re-implements routing and auth outside `route.ts`, and conflicts with dev HMR sockets; the RFC lists exactly these costs. | | Streaming request body (`fetch` with `duplex: 'half'`) | Chromium allows upload streaming only over HTTP/2+; dev and the BFF run HTTP/1.1; Safari and Firefox lack it. | | LiveKit / WebRTC rooms | Out of scope (WS7 NOT); a second media server and port set. | ## Approved dependencies Resolved live on **2026-10-29** (UTC 01:19) on dudoxxmac, per ADR 0005 Decision 4. | Package | App | Kind | Approved version | Version source (command, 2026-10-29) | |---|---|---|---|---| | `ws` | ddx-osprey-engine, ddx-api | dep | `^8.22.0` | `pnpm view ws version` → `8.22.0` (dist-tag `latest`) | No other package is approved here. ddx-web needs no new dependency (fetch streams). ## Consequences - Good: one realtime UI path stays (Centrifugo); audio never enters the event log. - Good: the BFF stays the only browser entry; auth, ownership and rate limits stay where they are. - Good: engine and ddx-api legs are already WebSocket, so the later browser switch is local. - Cost: the HTTP up leg adds ~100 ms batching plus one BFF hop per POST to the up path. The down path (first audio) streams, so the < 800 ms first-audio stretch stays reachable; task 048 measures it. - Cost: one more loopback port (9631), claimed with its listener (task 045). - Risk: a lost or reordered POST. Mitigation: sequence numbers, drop-late, and a `stop` on gap > 1 s. - Follow-up for the lead: task 047 adds the `stream-voice` class (contracts + route-table rows); task 046's relay needs the two HTTP endpoints next to the ticket mint. ## Validation ```sh rg -l -i 'voice.*transport' ddx-documentation/adr | rg -q . rg -q '9631' ddx-documentation/adr/0020-voice-audio-transport.md rg -q 'Centrifugo' ddx-documentation/adr/0020-voice-audio-transport.md rg -q 'ADR 0020|0020-voice' ddx-documentation/realtime.md ``` --- Last reconciled: 2026-10-29 · task 004 (ai-assistant-platform)
ai-assistant-platform, task 004, 2026-10-29). No invariant
exception: the browser leg stays browser -> ddx-web BFF -> ddx-api -> engine (operator
order 2026-10-29). The plan's NEEDS_REVIEW item 1 (browser hop) is resolved without a waiver.plans/ai-assistant-platform/ (invariants 2, 7, 13; failure mode FM1).findings/assistant-job/falcon-voice.md (HMS Falcon = shape only; no
code or values copied).osprey.event is the authority
(context/_invariants.md, ). Centrifugo delivers JSON publications
after an outbox relay hop: fine for transcripts and UI events, wrong for 50 audio frames per second.ddx-osprey-engine/src/server.ts is node:http on
127.0.0.1:9630. ddx-api and ddx-web carry no ws dependency./api/v1/**, ADR 0017). The browser-held
Centrifugo socket is the one existing realtime socket; it is NOT a precedent for a direct
browser -> ddx-api or browser -> engine socket.next@16.4.3 has no route-handler
WebSocket upgrade: rg 'upgrade\(' over node_modules/next/dist/server/web/spec-extension/response.js
and rg webSockets over config-shared.js both return no match. Upstream status:
RFC "WebSocket Upgrades in Route Handlers"
(Draft, target experimental, 2026-07-06) and PRs #89320 / #97027 (open). next start
closes upgrade requests that resolve to app routes. ddx-web runs plain next start --port 9600.osprey.event by the engine and reach the browser on Centrifugo, as today. Interim captions
and playback control ride the voice path (they are not durable state).ws dependency). It accepts only ddx-api: the upgrade carries identity headers
(tenant, user, session, stream id) and the internal engine credential the existing
ddx-api -> engine calls use./osprey/voice. ddx-api owns auth and ownership: session JWT,
OrgContextGuard / ownership check on the Osprey session, then a single-use voice ticket
minted by POST /osprey/sessions/:id/voice-ticket (TTL 120 s, one stream per ticket, bound to
tenant + user + session). Opening a stream consumes the ticket; ddx-api opens the engine
WebSocket and relays frames both ways. Closing either side closes the other.stream-voice
(ADR 0017 table is data; the row lives in packages/ddx-contracts/src/bff-routes.ts, task 047):
GET /api/v1/osprey/voice/:streamId — one long-lived streamed response
(application/octet-stream, no-store, no-transform, header-phase deadline only,
sticky node like stream-sse). Body = length-prefixed frames (1 byte kind + 4 byte length +
payload): PCM chunks and JSON control frames.POST /api/v1/osprey/voice/:streamId/frames — application/octet-stream body with a
monotonic sequence number header; 100 ms of PCM16 16 kHz (5 × 20 ms frames, 3200 B) per POST,
and JSON control frames (start, stop, barge-in, activity) as their own POSTs. ddx-api drops
duplicates and out-of-order frames by sequence.no-store, bearer credential, rate limit sized for audio
(≥ 20 POSTs/s per session); it is the only class with that budget, so no other route gains
a looser limit. The same-origin gate, identity layer and bridge stay as they are.start ack) plus a JSON control channel: start, stop, barge-in, state
(listening | thinking | speaking | error), caption (interim), audio_end, fallback
(provider switched). The schemas live in @ddx/contracts/osprey-voice (task 005), Zod 4,
versioned.next ships NextResponse.upgrade() (RFC #95514), the browser leg
becomes a WebSocket at the same BFF path under a voice-ws route class; the engine and ddx-api
legs and the frame schemas do not change. That switch needs a one-row amendment of this ADR.Considered (rejected): a direct browser WebSocket to ddx-api or the engine with the Centrifugo ticket pattern — it breaks invariant 7 and the operator ruled it out on 2026-10-29.
Resolved live on 2026-10-29 (UTC 01:19) on dudoxxmac, per ADR 0005 Decision 4.
No other package is approved here. ddx-web needs no new dependency (fetch streams).
stop on gap > 1 s.stream-voice class (contracts + route-table rows);
task 046's relay needs the two HTTP endpoints next to the ticket mint.Last reconciled: 2026-10-29 · task 004 (ai-assistant-platform)
adr/
deploy/
research/
rg -l -i 'voice.*transport' ddx-documentation/adr | rg -q .rg -q '9631' ddx-documentation/adr/0020-voice-audio-transport.mdrg -q 'Centrifugo' ddx-documentation/adr/0020-voice-audio-transport.mdrg -q 'ADR 0020|0020-voice' ddx-documentation/realtime.md