Quelle: ddx-documentation/deploy/FLOW.md
Read this first in a new session that touches production or the public site. It is the story and the state. The tick-list is [CHECKLIST.md](/de/docs/deploy/CHECKLIST); the full runbook is [README.md](/de/docs/deploy/README). State written 2026-11-16, release `77a16fd` (deploy 11). ## 1. State right now | | | |---|---| | Public site | https://osprey.dudoxx.com (live, `/api/health` 200) | | Host | `dudoxx-forge`, app root `~/projects/ddx-osprey` (`repo`, `releases`, `current`, `shared/{env,logs,backups,ibis-packs}`, `.deploy-state`) | | Release | `77a16fd` on `main` (deploy 11). A release is a `git archive` of the PUSHED `main` | | PM2 (namespace `osprey`) | `ddx-osprey-engine` 9630/9631, `ddx-osprey-api` 9601, `ddx-osprey-web` 9600; canaries `-canary` 9632/9633, 9603, 9620 | | Containers (`osprey-prod`) | postgres, redis, minio, qdrant, centrifugo, litellm; subnet `172.30.196.0/24` pinned in the compose file | | Databases | Prisma DBs + engine DB `osprey_engine` (inside container `osprey-prod-postgres`) | | Neighbours (never touch) | booter (`ddx-web-boot`, ports 16300/16310, `booter.dudoxx.com`), Sentinel (`sentinel-*`, 115xx). Both healthy at last check | | Captcha | Cloudflare Turnstile (widget made via Playwriter). ALTCHA swap = phase 2, plan tasks 021-033, not blocking | | Voice | TTS + STT through DDX Omni (`tts:ddx-omni`, `stt:ddx-omni` ready in `osprey doctor`) | | Tools in the Assistant | `current_datetime`, `get_weather`, `search_news`, `web_search` (Exa then Brave) all verified live | Diagnosis knowledge on production: 197 of 198 cards (cardiology 86, dermatology 99, gastroenterology 12), 300 drugs, index 24,261 points. `DDC-CARD-0076` (Takotsubo) is only in the dev DB and has **0 findings**: it cannot score, so it was not shipped. Author it for real with `osprey dx cards author` before it goes anywhere. Demo logins (public list, no admins): `staff`, `member`, `client` of `acme-health` and `globex-care`. Admins use a generated `DDX_API_SEED_ADMIN_PASSWORD` kept in the forge env, never listed. Roles: `staff` creates diagnosis cases; `member` sees "New case" but cannot create (RBAC by design, vague error); `client` is read-mostly. ## 2. The path of one request ``` browser ─ https://osprey.dudoxx.com ─► dudoxx-home Apache :443 (certbot cert, exp 2027-02-14, 503 page when down) ─► forge Apache :80 (tier2 vhost, edge IP only) ─► ddx-web 127.0.0.1:9600 (Next.js, BFF /api/v1) ─► ddx-api 127.0.0.1:9601 (needs a NODE token) ─► engine 127.0.0.1:9630 Centrifugo client paths /connection/ and /emulation go to 127.0.0.1:9650. ``` DNS: `osprey` CNAME → the ddns chain, managed by bind9 / ddx-dns-manager, never a hard IP. ## 3. How a deploy works (every update) Entry point: `./ddx-manage.sh remote <status|deploy|route|rollback|logs|seed-demo>`, which calls `deploy/scripts/*.sh` (`deploy.sh`, `smoke.sh`, `preflight.sh`, `check-env.sh`, `seed-demo.sh`, `gen-secrets.sh`, `backup-db.sh`, `rollback.sh`, `boot-setup.sh`, `lib.sh`, `remote.sh`). Target config: `deploy/targets/forge.conf`. Compose: `deploy/docker/docker-compose.prod.yml`. PM2: `deploy/ecosystem.config.cjs`. Env templates: `deploy/env/*.env.example`. Steps, in order: preflight · fetch · shared · secrets · env-check · install · build-api · build-web · data-tier · ensure-db · backup · migrate · services · engine-migrate · engine-doctor · cluster-mint · canary · canary-smoke · swap · identity · admit-live · pm2-save · live-smoke · finish. A failure before the swap leaves the live release untouched. A failed live smoke rolls back by itself. The routine, from the Mac, in this order: 1. Commit and **push to `main`** (the release is the pushed ref). Main-only, no branches. 2. `deploy/scripts/deploy.sh --target forge --ref main --dry-run` → exit 0. 3. `DDX_DEPLOY_CONFIRM="OPERATOR_CONFIRMED: forge deploy" ./ddx-manage.sh remote deploy --target forge --ref main` with `run_in_background`, read `/tmp/<log>` at the end → `result: OK <from>..<to> live on forge`. 4. `deploy/scripts/smoke.sh --target forge --public` → all layers OK. 5. `https://osprey.dudoxx.com/api/health` 200 **and** `https://booter.dudoxx.com/api/health` 200. 6. Headed browser check as a demo account (see §5). Other confirm lines: route publish `OPERATOR_CONFIRMED="forge publish-route"` with `remote route --publish` (default is `--check`); demo seed `DDX_DEPLOY_CONFIRM="OPERATOR_CONFIRMED: forge seed-demo"`. Production steps run from the main session only, never from a subagent. ## 4. What the first go-live taught (read before changing deploy code) 1. PM2 names are global per daemon. Our names are `ddx-osprey-*`. Osprey once took over booter's `ddx-api` / `ddx-web`; preflight (`pm2-names`) and the post-swap `identity` step guard it. 2. Production `ddx-api` answers 401 `NODE_TOKEN_INVALID` to anything but ddx-web. Scripts use the BFF `127.0.0.1:9600/api/v1` with a session cookie and `origin: https://osprey.dudoxx.com` (`DDX_SEEDER_VIA_BFF=1`). 3. A rendered env file is never overwritten; `gen-secrets.sh` only appends missing keys. 4. `NEXT_PUBLIC_APP_URL` and the Turnstile site key are baked at build time. 5. A migration that only ran on dev broke run 5 (`029`). Test SQL on a fresh PG first. 6. One 502 right after a swap = the deleted canary still in the pool for 15 s. Retry. 7. The forge vLLM answers 401 to an anonymous probe: alive, not down. 8. Sentinel shares the host and broke once on a Node upgrade (`argon2` native build). After any forge OS or Node change, check `https://sentinel.dudoxx.com/`. 9. Never `pm2 restart all`, `pm2 delete` or `pm2 save` blindly: neighbours live in the same daemon. ## 5. How to verify after a deploy (the proof we use) Headed `jev-browser`, owner watching. Sign in from the login page with a demo account. - `member`: Assistant turn that calls tools (time, weather, news, web search). - `staff`: `/en/portal/acme-health/staff/diagnosis/new`, save a case, **Start analysis**, answer or skip questions, then walk the tabs: Overview, Differential, Reasoning, Treatment, Prognosis, Map, Evidence, Dossier. - Page `data-revision` equals the database revision. - Treatment tab shows no "Unnamed item"; explain, what-if and compare answer 200. - Query forge DB read-only: `ssh dudoxx-forge "docker exec -i osprey-prod-postgres sh -c 'psql -U \$POSTGRES_USER -d osprey_engine -At'" < file.sql`. - Select dropdowns in the case form are easy to mis-click: check the specialty chip before you press Save (a cardiology case was created by mistake on 2026-11-16). ## 6. Fixed on 2026-11-16 (release `77a16fd`) - Explain, what-if, compare returned 409 "does not reproduce its resultHash": the rescore now reads the stored round evidence from the workflow snapshot (`insight-service.ts`, `eval-jobs.ts`). - Treatment tab "Unnamed item" ×1,639: the evidence endpoint now resolves every citation id the output bodies carry (`diagnosis/gateway.ts`), not only the 50-id top list. - Case dossier had an empty title: `dossier-brief.ts` stores a short title before the run starts. - Boot mail probe read TLS-only ports as down (`STACK_MAIL_DOWN`): `boot-stack-probe.service.ts` now reads the greeting over TLS. A real mail send is still untested. ## 7. Open items (candidates for the next sessions) Product (the owner's next focus: missing features, UI, public site): - Intake form lacks medications, allergies and conditions fields. - `member` sees "New case" but cannot create; the error is vague. Hide the button or explain. - Cardiology overview once showed 2 red flags as "Unnamed item" (label resolution of rule ids). - The public site (marketing pages, login page) has had no design pass. - Author `DDC-CARD-0076` for real; card count is then 198 of 198. Platform: - Phase 2: ALTCHA instead of Turnstile (plan `osprey-deploy` tasks 021-033). - Plan close-out tasks 034/035, deployment report (`deployment-specialist`, mode `report`) and `/unify plans/osprey-deploy/_index.md` are not done. - Real mail send test; `hms-1.conf` still says docker subnet `172.30.163.0/24` (owner item). - Voice round trip by speaking has not been tested; other voice-provider keys and the OpenRouter key are not copied to forge. - Sentinel sign-in untested (no credentials); engine typecheck has 3 old errors in `src/plan/runtime.ts` and `src/suites/plan-build-events.ts`. - Two `plan-overview*.html` files were committed by `/quick-git --all`; remove if unwanted. ## 8. Rules that stay true Never print or commit a secret (key names and lengths only). Never `down -v` or remove a prod volume. Never touch another app's PM2 process. Commit on `main` only. Validate in the headed browser. Ask before deleting data or touching another live system.
Read this first in a new session that touches production or the public site. It is the story
and the state. The tick-list is ; the full runbook is
. State written 2026-11-16, release 77a16fd (deploy 11).
Diagnosis knowledge on production: 197 of 198 cards (cardiology 86, dermatology 99,
gastroenterology 12), 300 drugs, index 24,261 points. DDC-CARD-0076 (Takotsubo) is only in the
dev DB and has 0 findings: it cannot score, so it was not shipped. Author it for real with
osprey dx cards author before it goes anywhere.
Demo logins (public list, no admins): staff, member, client of acme-health and
globex-care. Admins use a generated DDX_API_SEED_ADMIN_PASSWORD kept in the forge env, never
listed. Roles: staff creates diagnosis cases; member sees "New case" but cannot create (RBAC
by design, vague error); client is read-mostly.
DNS: osprey CNAME → the ddns chain, managed by bind9 / ddx-dns-manager, never a hard IP.
Entry point: ./ddx-manage.sh remote <status|deploy|route|rollback|logs|seed-demo>, which calls
deploy/scripts/*.sh (deploy.sh, smoke.sh, preflight.sh, check-env.sh, seed-demo.sh,
gen-secrets.sh, backup-db.sh, rollback.sh, boot-setup.sh, lib.sh, remote.sh).
Target config: deploy/targets/forge.conf. Compose: deploy/docker/docker-compose.prod.yml.
PM2: deploy/ecosystem.config.cjs. Env templates: deploy/env/*.env.example.
Steps, in order: preflight · fetch · shared · secrets · env-check · install · build-api · build-web · data-tier · ensure-db · backup · migrate · services · engine-migrate · engine-doctor · cluster-mint · canary · canary-smoke · swap · identity · admit-live · pm2-save · live-smoke · finish. A failure before the swap leaves the live release untouched. A failed live smoke rolls back by itself.
The routine, from the Mac, in this order:
main (the release is the pushed ref). Main-only, no branches.deploy/scripts/deploy.sh --target forge --ref main --dry-run → exit 0.DDX_DEPLOY_CONFIRM="OPERATOR_CONFIRMED: forge deploy" ./ddx-manage.sh remote deploy --target forge --ref main
with run_in_background, read /tmp/<log> at the end → result: OK <from>..<to> live on forge.deploy/scripts/smoke.sh --target forge --public → all layers OK.https://osprey.dudoxx.com/api/health 200 and https://booter.dudoxx.com/api/health 200.Other confirm lines: route publish OPERATOR_CONFIRMED="forge publish-route" with
remote route --publish (default is --check); demo seed
DDX_DEPLOY_CONFIRM="OPERATOR_CONFIRMED: forge seed-demo". Production steps run from the main
session only, never from a subagent.
ddx-osprey-*. Osprey once took over booter's
ddx-api / ddx-web; preflight (pm2-names) and the post-swap identity step guard it.ddx-api answers 401 NODE_TOKEN_INVALID to anything but ddx-web. Scripts use
the BFF 127.0.0.1:9600/api/v1 with a session cookie and origin: https://osprey.dudoxx.com
(DDX_SEEDER_VIA_BFF=1).gen-secrets.sh only appends missing keys.NEXT_PUBLIC_APP_URL and the Turnstile site key are baked at build time.029). Test SQL on a fresh PG first.argon2 native build). After any
forge OS or Node change, check https://sentinel.dudoxx.com/.pm2 restart all, pm2 delete or pm2 save blindly: neighbours live in the same daemon.Headed jev-browser, owner watching. Sign in from the login page with a demo account.
member: Assistant turn that calls tools (time, weather, news, web search).staff: /en/portal/acme-health/staff/diagnosis/new, save a case, Start analysis, answer or
skip questions, then walk the tabs: Overview, Differential, Reasoning, Treatment, Prognosis,
Map, Evidence, Dossier.data-revision equals the database revision.ssh dudoxx-forge "docker exec -i osprey-prod-postgres sh -c 'psql -U \$POSTGRES_USER -d osprey_engine -At'" < file.sql.77a16fd)insight-service.ts, eval-jobs.ts).diagnosis/gateway.ts), not only the 50-id top list.dossier-brief.ts stores a short title before the run starts.STACK_MAIL_DOWN): boot-stack-probe.service.ts
now reads the greeting over TLS. A real mail send is still untested.Product (the owner's next focus: missing features, UI, public site):
member sees "New case" but cannot create; the error is vague. Hide the button or explain.DDC-CARD-0076 for real; card count is then 198 of 198.Platform:
osprey-deploy tasks 021-033).deployment-specialist, mode report) and
/unify plans/osprey-deploy/_index.md are not done.hms-1.conf still says docker subnet 172.30.163.0/24 (owner item).src/plan/runtime.ts and src/suites/plan-build-events.ts.plan-overview*.html files were committed by /quick-git --all; remove if unwanted.Never print or commit a secret (key names and lengths only). Never down -v or remove a prod
volume. Never touch another app's PM2 process. Commit on main only. Validate in the headed
browser. Ask before deleting data or touching another live system.
adr/
deploy/
research/
browser ─ https://osprey.dudoxx.com ─► dudoxx-home Apache :443 (certbot cert, exp 2027-02-14, 503 page when down) ─► forge Apache :80 (tier2 vhost, edge IP only) ─► ddx-web 127.0.0.1:9600 (Next.js, BFF /api/v1) ─► ddx-api 127.0.0.1:9601 (needs a NODE token) ─► engine 127.0.0.1:9630Centrifugo client paths /connection/ and /emulation go to 127.0.0.1:9650.