Quelle: ddx-documentation/deploy/README.md
> Current state and the full flow in one page: [FLOW.md](/de/docs/deploy/FLOW). Tick-list for an update: [CHECKLIST.md](/de/docs/deploy/CHECKLIST). **Quick path for the next update: [CHECKLIST.md](/de/docs/deploy/CHECKLIST).** This page is the full reference. **Rule first: deploys happen only on operator request.** A deploy, rollback, route change or boot setup runs from the main session only, after the operator writes an `OPERATOR_CONFIRMED` line for that target and action. Never from a worker, a push, a workflow, a runner, a git hook, a webhook, a timer or cron (`context/_invariants.md` rows 15 and 42, plan invariants 1-2). One chained ssh call at a time, never two in parallel. The engine is the scripts under `deploy/`; this page says when and in which order to call them. The detail of each step lives in the script header (`deploy/scripts/<name>.sh`) and in `./ddx-manage.sh remote help`. Ports are never written here: the one table is [ports.md](/de/docs/ports). Hosts, IPs and users come only from `~/.claude/DDX_LAN_INVENTORY.md`. ## 1. Targets | What | Where | |---|---| | Target registry (id, host, mode, rail) | `context/DEPLOY_TARGETS.md` | | One conf per target | `deploy/targets/<id>.conf` (schema in `context/DEPLOY_TARGETS.md`) | | Default target (no `--target`) | `deploy/targets/default` (today `forge`) | | App root layout on the target | `<app_root>/{repo,releases/<id>,current,shared/{env,cluster,logs,backups},.deploy-state}` | `forge` is a shared host: the scripts touch only the `pm2_namespace` and the `compose_project` of the conf, never another app. PM2 app names are global per daemon and the namespace does not isolate them (`pm2 startOrReload` matches by name), so every Osprey PM2 app carries the `ddx-osprey-` prefix (one definition: `DDX_PM2_APPS` in `deploy/scripts/lib.sh`, same names in `deploy/ecosystem.config.cjs`). ## 2. Confirm forms | Action | Confirm line (env, set for the one command) | |---|---| | deploy | `DDX_DEPLOY_CONFIRM="OPERATOR_CONFIRMED: <id> deploy"` | | rollback | `DDX_DEPLOY_CONFIRM="OPERATOR_CONFIRMED: <id> rollback"` | | boot setup | `DDX_DEPLOY_CONFIRM="<id> boot-setup ..."` (the operator's confirm line) | | publish route | `OPERATOR_CONFIRMED="<id> publish-route"` and the flag `--publish` | | remove route | `OPERATOR_CONFIRMED="<id> remove-route"` | | seed demo data | `DDX_DEPLOY_CONFIRM="<id> seed-demo ..."` (section 13); never part of a deploy | Without the line a real run refuses and connects nowhere. `--dry-run` never needs it. `./ddx-manage.sh remote route` defaults to `--check`: a real publish needs `--publish` plus the line. ## 3. First deploy of a target (order) 1. Target prerequisites (one time): GitHub deploy key on the target for `<app_root>/repo`; BatchMode ssh from the target to its `backup_remote`; `deploy/` committed and pushed (the release archive must contain `deploy/scripts/deploy.sh`). 2. Preflight: `./ddx-manage.sh remote status --target <id>` and `deploy/scripts/deploy.sh --target <id> --ref <ref> --dry-run`. 3. Route: `./ddx-manage.sh remote route --target <id> --check`; publish only on request (section 6). A change of the vhost templates needs a re-publish. 4. Operator-filled keys, BEFORE the deploy. `gen-secrets.sh` writes them as commented `# KEY=` lines in `<app_root>/shared/env/`; replace the line with `KEY=value` (mode 600, key names only in any report). Names only: | File | Key | Rule | |---|---|---| | `docker.env` | `DUDOXX_LOCAL_API_KEY`, `DUDOXX_EMBED_API_KEY` | REQUIRED (LiteLLM upstreams) | | `docker.env` or engine env | `DDX_OPENROUTER_API_KEY` | optional | | `ddx-web.env` + `ddx-api.env` | Turnstile site key and secret key (go-live captcha) | both or neither; ALTCHA replaces it in phase 2 | | `ddx-api.env` | `DDX_API_SYSTEM_ADMIN_EMAIL`, `DDX_API_SMTP_*`, `DDX_API_IMAP_*` | operator values | | `ddx-osprey-engine.env` | `<from-shared-infra>` keys (portal values) | WARN while empty | The engine env template keeps the dev hostnames for `DUDOXX_TTS_URL` / `DUDOXX_STT_URL`: the operator confirms the target host reaches them. 5. First deploy: `./ddx-manage.sh remote deploy --target <id> --ref <ref>` with the confirm line. `gen-secrets.sh` builds four files in `shared/env/` (`docker`, `ddx-api`, `ddx-web`, `ddx-osprey-engine`) and `shared/cluster/*` on the target, then `check-env.sh` (step env-check) stops the deploy BEFORE install and build while a REQUIRED key is empty, a placeholder remains, or exactly one key of a both-or-neither pair is set. Later steps (ensure the engine database, start the Centrifugo and LiteLLM containers, engine migrate and `osprey doctor`): see the header of `deploy/scripts/deploy.sh`. Exit 3 of the doctor means a dead dependency; the deploy fails (no fallback). 5a. Fill any remaining `<from-shared-infra>` keys that `gen-secrets.sh` listed as TODO (values from the shared-infra portal, never from the repo), then run the deploy again. 6. First boot (cluster admission) is automatic. Production keeps manual admission with enforced node auth (invariant 39); on an empty registry nobody can sign in, so the deploy runs the host-only ddx-api CLI `cluster:bootstrap` of the release (`ddx-api/src/cluster/CLAUDE.md`), never over HTTP: 1. `prepare`, after migrate: `mint` a join token (`shared/cluster/<id>.join-token`, 0600, TTL 3600 s) for the canary pair `api-prod-901` / `web-prod-901` and the live pair (`DDX_API_NODE_ID` / `DDX_WEB_NODE_ID`, paths from `DDX_*_NODE_JOIN_TOKEN_PATH`). A node with a registry record prints `known <id> <status>` and gets nothing. 2. After the canary pm2 start: `admit api-prod-901`, then `admit web-prod-901` (wait ≤ 60 s for `joining`). A failure stops `prepare`; the live release is untouched. 3. `swap`, after the live reload: `admit` the live api, then the live web node. A failure prints `STEP admit-live FAIL`; the live smoke then fails and rolls back. By hand on the target (cwd `<release>/ddx-api`, the ddx-api env loaded): `node dist/src/cluster/cluster-bootstrap-cli.js status`. The canary keys are made by `gen-secrets.sh`. Audit actor of every bootstrap write: `host-bootstrap`. 7. Boot persistence: `deploy/scripts/boot-setup.sh --target <id>` with the confirm line, then `--check` (section 8). 8. Smoke the public route: `deploy/scripts/smoke.sh --target <id> --public`. It also checks the realtime path (Centrifugo client endpoint on the vhost); flags: see the script header. 9. Demo data is a separate, confirmed step after the smoke (section 13). A deploy never seeds. 10. Write the deployment report (section 10). ## 4. Deploy ``` DDX_DEPLOY_CONFIRM="OPERATOR_CONFIRMED: <id> deploy" ./ddx-manage.sh remote deploy --target <id> --ref <ref> ``` Also driven by `/ddx-deploy <target> <ref>` (main session). Flow (`deploy/scripts/deploy.sh`): preflight, phase A (fetch, `git archive` into a new `releases/<UTC ts>Z_<sha7>/`, prepare: gen-secrets, install, build, compose up, backup, migrate), canary smoke, phase C (atomic `current` swap, PM2 reload, `pm2 save`), live smoke, phase E (ok marker, prune, journal). The ref must be listed by `git ls-remote origin`. `--no-migrate` skips the migrate step. Seed policy: a deploy never seeds; demo data is `seed-demo.sh` (section 13). Apps and containers of a release: three PM2 apps in the `pm2_namespace` (`ddx-osprey-engine`, `ddx-osprey-api`, `ddx-osprey-web`; canary set `-canary`; engine starts first, the api is its only caller). The folders `ddx-api/` and `ddx-web/`, the env files `ddx-api.env` and `ddx-web.env` and the PM2 log files `shared/logs/<PM2 name>.log` keep their own names. Guards: preflight row `pm2-names` (a name of ours held by another app: FAIL), deploy `STEP pm2-names` before the canary start and before the swap, `STEP identity` after each start / reload and as the first smoke row (namespace, cwd under the release, exec path under `<app_root>`). Besides postgres, redis, minio and qdrant, the compose project holds two more containers: `centrifugo` and `litellm`. All bind `127.0.0.1`. Data tier on forge: the compose network is `172.30.196.0/24`, pinned by the compose file itself. **In-flight marker.** `<app_root>/.deploy-state/inflight` holds `<release id> <user>@<host>:<pid> <UTC time>` from phase A to `finish` or `fail`. While it exists, a new deploy and a manual rollback refuse (exit 75). No auto-expiry. A crashed operator run leaves it: the refusal prints the path and the holder. Clear it by hand ONLY when no deploy runs (check `remote status` and the journal first), in one confirmed ssh call: `rm <app_root>/.deploy-state/inflight`. ## 5. Rollback ``` DDX_DEPLOY_CONFIRM="OPERATOR_CONFIRMED: <id> rollback" ./ddx-manage.sh remote rollback --target <id> [--to <release>] ``` `deploy/scripts/rollback.sh` walks the release stack: the target is `--to <release>`, else the newest release older than the live one that carries `.deploy-state/ok`. Run twice, it steps back two releases. Nothing is rebuilt. The database is not touched: migrations are forward-only, the older release stays compatible with the schema. Rollback limits: the database (Prisma and `osprey_engine`), the Centrifugo and LiteLLM containers are NOT rolled back. A release that predates the engine app has no engine in its ecosystem file: the rollback does not stop the engine, it prints `WARN engine not in <release>: left running from the newer release`. The journal line carries `engine=<reloaded|left-running|?>`. Details: header of `rollback.sh`. A failed live smoke during a deploy rolls back by itself (exit 2; exit 3 when the rollback failed too). ## 6. Route `deploy/route/publish-route.sh --target <id>` (or `./ddx-manage.sh remote route`, which defaults to `--check`): | Mode | Effect | |---|---| | `--dry-run` | print every step, ssh line and rendered vhost; no ssh, no DNS call | | `--check` | read-only: CNAME, redirect, https health, configtest, vhosts, 503 page, UFW (own rule or an existing `allow <port>/tcp`) | | `--publish` | publish (same form as `sentinel.dudoxx.com`): CNAME, target vhost, UFW rule (skipped as COVERED when the port is already open), edge HTTP vhost + 503 page, cert, edge HTTPS vhost | | `--remove` | reverse order; restores a pre-existing vhost; the cert stays | Every vhost write keeps a backup and runs `apachectl configtest` before a reload. A failed configtest restores the backup (exit 3). A half-made route is undone with `--remove`. ## 7. Remote logs and status | Command | What it shows | |---|---| | `./ddx-manage.sh remote logs [api\|web\|engine\|all] --target <id>` | PM2 logs of `ddx-osprey-api` / `ddx-osprey-web` / `ddx-osprey-engine` in this namespace, same filters as local logs (`--level`, `--scope`, `--req`, `--err`, `--lines`) | | `./ddx-manage.sh remote logs req <id> --target <id>` | one request across `ddx-osprey-api` and `ddx-osprey-web`, time order | | `./ddx-manage.sh remote status --target <id>` | PM2 rows, docker ps, disk, live release + sha, boot-setup check | | `./ddx-manage.sh remote pull-logs --target <id>` | rsync `shared/logs/` to `logs/remote/<target>/<UTC ts>/` | `remote logs --follow` streams until killed: run it in the background only. ## 8. Boot setup `deploy/scripts/boot-setup.sh --target <id>` makes PM2 and the prod containers come back after a host restart: PM2's own boot unit, `pm2 save`, docker enabled, restart policy on the data containers. `--with-logrotate` adds pm2-logrotate (global to the PM2 daemon: NEEDS_REVIEW on forge). `--check` is read-only and prints one `CHECK` row per item. **Never reboot a host to prove it**: `boot-setup.sh --check` is the proof. ### Node on a shared host (`node_dir`) forge is shared: other PM2 apps and the PM2 daemon run on the system Node 20. This app runs on its own Node 24 at `~/.local/opt/node-24` (conf key `node_dir=~/.local/opt/node-24/bin`). The system `node` stays as it is. - Install once per target: the official tarball `node-v24.x-linux-x64.tar.xz` (nodejs.org/dist/latest-v24.x/), `mkdir -p ~/.local/opt/node-24 && tar -xJf <tarball> --strip-components=1 -C ~/.local/opt/node-24`. `preflight.sh` FAILs the `node` row while `<node_dir>/node` is missing, is not 24.x or is not first on the ssh PATH. - Every ssh line puts `<node_dir>` before `~/.local/bin`, so pnpm, builds, gen-secrets and every remote script run on Node 24. - PM2: deploy.sh and rollback.sh export `DDX_DEPLOY_NODE_BIN=<node_dir>/node`; `deploy/ecosystem.config.cjs` sets it as the `ddx-osprey-web` interpreter and as the node that `wait-deps.sh` execs for `ddx-osprey-api` and `ddx-osprey-engine`. `pm2 save` stores it, so a boot resurrect keeps Node 24. - The pm2 CLI itself never runs on Node 24. It talks to the shared PM2 daemon, can spawn it, restarts modules (`pm2 set`) and writes the boot unit (`pm2 startup`). Every pm2 call in the scripts goes through `ddx_pm2` (`deploy/scripts/lib.sh`): the ssh line exports `DDX_DEPLOY_NODE_DIR`, and `ddx_pm2` removes that dir from PATH for the one call. The host's default node then runs the CLI. `DDX_DEPLOY_NODE_BIN` stays in the env, so our apps still get Node 24. `boot-setup.sh --with-logrotate` runs `pm2 set` only for a key whose value differs from 50M / 14. - `boot-setup.sh --check` row `pm2-node` proves the three apps run on `<node_dir>/node`. - A rollback to a release whose ecosystem file predates `DDX_DEPLOY_NODE_BIN` gives PM2 plain `node`; the `pm2-node` row then FAILs and names the app until the next deploy. ## 9. Secrets `deploy/scripts/gen-secrets.sh` runs ON the target (deploy.sh calls it). It renders `deploy/env/*.env.example` into `<app_root>/shared/env/{ddx-api,ddx-web,ddx-osprey-engine,docker}.env` (files mode 600, folder 700) and the node keys into `shared/cluster/`. It never overwrites a value; it only appends missing keys. Linked groups hold one value across files: the Postgres password (docker, ddx-api, engine, LiteLLM database URL), the MinIO root pair (docker, ddx-api, engine), the three Centrifugo keys (docker, ddx-api, and the API key in the engine) and `LITELLM_MASTER_KEY` (the engine API key). Two values that disagree, or an empty linked key, stop the run. `check-env.sh --target <id>` re-checks the four files on the target (key names only). `--rotate KEY` renews one standalone key. Output holds key names only. No env file, key or token is ever committed. ## 10. Deployment report After each deploy or rollback, `deployment-specialist` (mode `report`) writes `context/deploy-reports/<target>/<YYYYMMDD-HHMM>_<from7>..<to7>.md` and appends one line to `context/deploy_journal_<host>.md`. A pull with no report is an unverified deploy. ## 11. Open NEEDS_REVIEW (owner) | Item | Decision needed | |---|---| | hms-1 `docker_subnet` | `hms-1.conf` still says `172.30.163.0/24`; the compose file pins `172.30.196.0/24` and ignores the conf key. Align the conf and confirm the subnet is free on hms-1 | | hms-1 `backup_remote` | confirm `dudoxx-forge:backups/ddx-osprey-hms-1` as the off-host copy | | pm2-logrotate on forge | approve `boot-setup.sh --with-logrotate` (rotates every app on the daemon) or accept a FAIL row in `--check` | | seed policy | decided 2026-11-16 (D1): full demo, logins live on the public site; run by `seed-demo.sh` (section 13) | ## 12. Troubleshooting | Symptom | Check | Fix | |---|---|---| | preflight `node ... need 24.x` | `node -v` on the target (forge `/usr/bin/node` is 20) | install Node 24 user-scope (nvm/fnm in `$HOME`), its bin before `/usr/bin` | | preflight `pnpm missing or broken` | `pnpm -v`; corepack cannot run the pinned major | `npm install -g --prefix ~/.local pnpm@<major>` (major from root `package.json`) | | `pnpm` / `pm2` not found over ssh | `echo $PATH` in a non-login ssh shell | `lib.sh` puts `~/.local/bin` first; install tools with `npm -g --prefix ~/.local` | | backup step fails, no `pg_dump` | `command -v pg_dump` on the host is empty by design | nothing to install: `backup-db.sh` runs `pg_dump` inside the `<compose_project>-postgres` container | | route `--check` https fails, edge times out | `sudo ufw status` on the target (forge default DROP) | publish adds the rule for `edge_ip`; re-run `publish-route.sh` with its confirm line | | PM2 name owned by another app: preflight `pm2-names` FAIL, `STEP pm2-names FAIL` or `STEP identity FAIL <app>: namespace=<x> cwd=<y>` | `pm2 jlist` on the target: `name`, `pm2_env.namespace`, `pm2_env.pm_cwd` of that app | never `startOrReload`, `delete` or `restart` it: the other app owns the name. Stop, find the owner; a real leftover of ours is removed by the operator by hand, then redeploy. An app taken over by mistake is restored from ITS own ecosystem file | | rollback `ROLLBACK_FAIL <release> names the PM2 apps ddx-api / ddx-web` | the release is from before the `ddx-osprey-` PM2 names | no rollback across that change: deploy a new release (or a fixed ref) instead | | deploy or rollback exit 75 | `<app_root>/.deploy-state/inflight` and the lock | wait for the running deploy; clear a stale marker by hand (section 4) | | `API_NODE_NOT_ADMITTED` in ddx-api log | `nodes` state of the node id | run the bootstrap (section 3 step 6) | | browser shows only "Loading the page…" | console: `Executing inline script violates ... script-src` | production CSP must keep `script-src 'self' 'unsafe-inline'` (`ddx-web/next.config.ts`): App Router flight data is inline; a nonce does not fit `cacheComponents`, SRI skips inline scripts | | sign-in / any POST answers 403 `CSRF_ORIGIN` | ddx-web log `Cross-origin state change refused origin=foreign` | behind a proxy `nextUrl.origin` is the bind address; `NEXT_PUBLIC_APP_URL` in `ddx-web.env` must be the public `https://<fqdn>` (the BFF origin gate accepts its origin; it is baked at build, so redeploy) | | one 502 right after a deploy | ddx-web log `Upstream not delivered` with `nodeId` `api-prod-901` | the deleted canary api is still in the pool for its lease (15 s); retry. Open item: drain the canary before delete | | a case-only file rename builds on the Mac, fails on the target | `git ls-files <dir>` vs the disk name | `git mv -f <Old> <new>` (macOS ignores case, Linux does not) | | realtime stuck "connecting" | `smoke.sh --public` realtime row; Centrifugo container health on `realtime_port` | the vhost must forward `/connection/` and `/emulation` to `realtime_port`; re-publish the route after a template change. Never expose `realtime_api_port` | | LiteLLM cannot reach vLLM | `docker logs osprey-prod-litellm --tail 50`; `sudo ufw status` on the target | UFW must admit the docker subnet to the vLLM port on the host (`llm_chat_base` is `host.docker.internal`); do not add a fallback model | | `osprey doctor` exits 3 | the doctor output names the dead dependency (vLLM, embedder, Postgres, Qdrant, Centrifugo) | fix that dependency; the deploy does not continue and no mock stands in | | deploy stops at env-check | `check-env.sh --target <id>` lists key names | fill the named key in `<app_root>/shared/env/<file>.env`, deploy again; nothing was built | | `compose up` fails, subnet overlap | `docker network ls` and `docker network inspect` on the target | the compose file pins `172.30.196.0/24`; pick a free subnet in the compose file (the conf key is not read) | ## 13. Demo data `deploy/scripts/seed-demo.sh --target <id> [--dry-run] [--force]`, from the main session. Owner decision D1 (2026-11-16): full demo (orgs, demo logins, Osprey content), logins live on the public site. A deploy or redeploy NEVER seeds. ``` DDX_DEPLOY_CONFIRM="<id> seed-demo ..." ./ddx-manage.sh remote seed-demo --target <id> ``` - Runs ON the target through the web BFF on loopback (`127.0.0.1:<web_port>/api/v1`, session cookie + the public origin, `DDX_SEEDER_VIA_BFF=1`): production ddx-api enforces node authentication (invariant 39), so a direct call answers 401 `NODE_TOKEN_INVALID` and only ddx-web may call it. Needs a live release and `DDX_API_SEED_USER_PASSWORD` set in `ddx-api.env`. - Public demo list (`DDX_API_DEMO_ACCOUNTS_PUBLIC=true`): the login page lists staff, member and client with the demo password. `super-admin` and `org-admin` take `DDX_API_SEED_ADMIN_PASSWORD` (generated by `gen-secrets.sh`), are never listed, and the auth seed refuses a public list without it. The seeder signs in as org-admin and uses the admin password when it is set. - Order: `backup-db.sh` first (a failed backup stops the run), Prisma seed, then `ddx-seeder`. - Marker `<app_root>/.deploy-state/seed-demo.done`: a second run prints `SEED_SKIP` and creates nothing. `--force` re-runs (upserts, no duplicates) after a new backup. - Refuses while a deploy is in flight (exit 75) or when the process env says production (the seeder refuses it; lifting that is an owner decision). - All seed texts stay NEEDS_REVIEW (operator Q5): confirm the wording is demo-safe first. ## 14. Diagnosis knowledge, tool keys and voice The deploy installs code and schema, not knowledge. Packs, card folders and the search index are an operator step (recipe and expected counts: [CHECKLIST.md](/de/docs/deploy/CHECKLIST) section 6); the IBIS packs live outside the repo and are copied to `<app_root>/shared/ibis-packs/` with `rsync --checksum`. Engine env keys that switch capabilities on (file `<app_root>/shared/env/ddx-osprey-engine.env`, mode 600; a restart of `ddx-osprey-engine` with `--update-env` applies them): | Capability | Keys | Proof | |---|---|---| | Search tools (web, news) | `OSPREY_SEARCH_PROVIDERS` (`exa,brave`), `OSPREY_EXA_API_KEY`, `OSPREY_BRAVE_API_KEY` | an Assistant turn calls `web_search` / `search_news` and the tool result is not an error | | Weather, time | none (keyless) | `get_weather`, `current_datetime` | | Images | `OSPREY_IMAGE_PROVIDER=router`, `OSPREY_IMAGE_MODEL` | router alias must exist | | Voice (Omni) | `DUDOXX_TTS_URL`, `DUDOXX_TTS_API_KEY`, `DUDOXX_STT_URL`, `DUDOXX_STT_API_KEY` | `osprey doctor`: `tts:ddx-omni` and `stt:ddx-omni` ready; engine log `voice <id>: started stt=ddx-omni tts=ddx-omni` | Keys are copied from a sibling production app or the shared-infra portal on the target itself (key names in chat, never values); back up `shared/env` before any edit. ## 15. Neighbours on the shared host forge also runs `booter.dudoxx.com` (PM2 namespace `ddx-web-boot`, ports 16300/16310, compose project `ddx-web-boot-prod`) and Sentinel (`sentinel-*`, ports 115xx). PM2 app names are global per daemon, so a deploy that reuses another app's name takes its processes over (2026-11-16: booter was down 6 minutes). Rules: our six PM2 names carry the `ddx-osprey-` prefix; preflight row `pm2-names` and the post-swap `identity` step check owner, namespace and folder; never run `pm2 restart all`, `delete` or `save` from a script that does not know every owner. After any forge change, check the neighbours answer (CHECKLIST.md section 4).
Current state and the full flow in one page: . Tick-list for an update: .
Quick path for the next update: . This page is the full reference.
Rule first: deploys happen only on operator request. A deploy, rollback, route change or
boot setup runs from the main session only, after the operator writes an
OPERATOR_CONFIRMED line for that target and action. Never from a worker, a push, a
workflow, a runner, a git hook, a webhook, a timer or cron (context/_invariants.md rows 15
and 42, plan invariants 1-2). One chained ssh call at a time, never two in parallel.
The engine is the scripts under deploy/; this page says when and in which order to call
them. The detail of each step lives in the script header (deploy/scripts/<name>.sh) and in
./ddx-manage.sh remote help. Ports are never written here: the one table is
. Hosts, IPs and users come only from ~/.claude/DDX_LAN_INVENTORY.md.
forge is a shared host: the scripts touch only the pm2_namespace and the
compose_project of the conf, never another app. PM2 app names are global per daemon and
the namespace does not isolate them (pm2 startOrReload matches by name), so every Osprey
PM2 app carries the ddx-osprey- prefix (one definition: DDX_PM2_APPS in
deploy/scripts/lib.sh, same names in deploy/ecosystem.config.cjs).
Without the line a real run refuses and connects nowhere. --dry-run never needs it.
./ddx-manage.sh remote route defaults to --check: a real publish needs --publish plus the line.
<app_root>/repo;
BatchMode ssh from the target to its backup_remote; deploy/ committed and pushed (the
release archive must contain deploy/scripts/deploy.sh)../ddx-manage.sh remote status --target <id> and
deploy/scripts/deploy.sh --target <id> --ref <ref> --dry-run../ddx-manage.sh remote route --target <id> --check; publish only on request
(section 6). A change of the vhost templates needs a re-publish.Operator-filled keys, BEFORE the deploy. gen-secrets.sh writes them as commented
# KEY= lines in <app_root>/shared/env/; replace the line with KEY=value (mode 600,
key names only in any report). Names only:
The engine env template keeps the dev hostnames for DUDOXX_TTS_URL / DUDOXX_STT_URL:
the operator confirms the target host reaches them.
./ddx-manage.sh remote deploy --target <id> --ref <ref> with the confirm
line. gen-secrets.sh builds four files in shared/env/ (docker, ddx-api, ddx-web,
ddx-osprey-engine) and shared/cluster/* on the target, then check-env.sh (step
env-check) stops the deploy BEFORE install and build while a REQUIRED key is empty, a
placeholder remains, or exactly one key of a both-or-neither pair is set. Later steps (ensure
the engine database, start the Centrifugo and LiteLLM containers, engine migrate and
osprey doctor): see the header of deploy/scripts/deploy.sh. Exit 3 of the doctor
means a dead dependency; the deploy fails (no fallback).
5a. Fill any remaining <from-shared-infra> keys that gen-secrets.sh listed as TODO
(values from the shared-infra portal, never from the repo), then run the deploy again.First boot (cluster admission) is automatic. Production keeps manual admission with
enforced node auth (invariant 39); on an empty registry nobody can sign in, so the
deploy runs the host-only ddx-api CLI cluster:bootstrap of the release
(ddx-api/src/cluster/CLAUDE.md), never over HTTP:
prepare, after migrate: mint a join token (shared/cluster/<id>.join-token, 0600,
TTL 3600 s) for the canary pair api-prod-901 / web-prod-901 and the live pair
(DDX_API_NODE_ID / DDX_WEB_NODE_ID, paths from DDX_*_NODE_JOIN_TOKEN_PATH). A node
with a registry record prints known <id> <status> and gets nothing.admit api-prod-901, then admit web-prod-901 (wait ≤ 60 s
for joining). A failure stops prepare; the live release is untouched.swap, after the live reload: admit the live api, then the live web node. A failure
prints STEP admit-live FAIL; the live smoke then fails and rolls back.
By hand on the target (cwd <release>/ddx-api, the ddx-api env loaded):
node dist/src/cluster/cluster-bootstrap-cli.js status. The canary keys are made by
gen-secrets.sh. Audit actor of every bootstrap write: host-bootstrap.deploy/scripts/boot-setup.sh --target <id> with the confirm line, then
--check (section 8).deploy/scripts/smoke.sh --target <id> --public. It also checks the
realtime path (Centrifugo client endpoint on the vhost); flags: see the script header.Also driven by /ddx-deploy <target> <ref> (main session). Flow (deploy/scripts/deploy.sh):
preflight, phase A (fetch, git archive into a new releases/<UTC ts>Z_<sha7>/, prepare:
gen-secrets, install, build, compose up, backup, migrate), canary smoke, phase C (atomic
current swap, PM2 reload, pm2 save), live smoke, phase E (ok marker, prune, journal).
The ref must be listed by git ls-remote origin. --no-migrate skips the migrate step.
Seed policy: a deploy never seeds; demo data is seed-demo.sh (section 13).
Apps and containers of a release: three PM2 apps in the pm2_namespace (ddx-osprey-engine,
ddx-osprey-api, ddx-osprey-web; canary set -canary; engine starts first, the api is its
only caller). The folders ddx-api/ and ddx-web/, the env files ddx-api.env and
ddx-web.env and the PM2 log files shared/logs/<PM2 name>.log keep their own names. Guards:
preflight row pm2-names (a name of ours held by another app: FAIL), deploy STEP pm2-names
before the canary start and before the swap, STEP identity after each start / reload and
as the first smoke row (namespace, cwd under the release, exec path under <app_root>).
Besides postgres, redis, minio and qdrant, the compose project holds two more containers:
centrifugo and litellm. All bind 127.0.0.1. Data tier on forge: the compose network is
172.30.196.0/24, pinned by the compose file itself.
In-flight marker. <app_root>/.deploy-state/inflight holds
<release id> <user>@<host>:<pid> <UTC time> from phase A to finish or fail. While it
exists, a new deploy and a manual rollback refuse (exit 75). No auto-expiry. A crashed
operator run leaves it: the refusal prints the path and the holder. Clear it by hand ONLY
when no deploy runs (check remote status and the journal first), in one confirmed ssh
call: rm <app_root>/.deploy-state/inflight.
deploy/scripts/rollback.sh walks the release stack: the target is --to <release>, else
the newest release older than the live one that carries .deploy-state/ok. Run twice, it
steps back two releases. Nothing is rebuilt. The database is not touched: migrations are
forward-only, the older release stays compatible with the schema.
Rollback limits: the database (Prisma and osprey_engine), the Centrifugo and LiteLLM
containers are NOT rolled back. A release that predates the engine app has no engine in its
ecosystem file: the rollback does not stop the engine, it prints WARN engine not in <release>: left running from the newer release. The journal line carries
engine=<reloaded|left-running|?>. Details: header of rollback.sh. A failed live smoke
during a deploy rolls back by itself (exit 2; exit 3 when the rollback failed too).
deploy/route/publish-route.sh --target <id> (or ./ddx-manage.sh remote route, which defaults to --check):
Every vhost write keeps a backup and runs apachectl configtest before a reload. A failed
configtest restores the backup (exit 3). A half-made route is undone with --remove.
remote logs --follow streams until killed: run it in the background only.
deploy/scripts/boot-setup.sh --target <id> makes PM2 and the prod containers come back
after a host restart: PM2's own boot unit, pm2 save, docker enabled, restart policy on the
data containers. --with-logrotate adds pm2-logrotate (global to the PM2 daemon:
NEEDS_REVIEW on forge). --check is read-only and prints one CHECK row per item.
Never reboot a host to prove it: boot-setup.sh --check is the proof.
node_dir)forge is shared: other PM2 apps and the PM2 daemon run on the system Node 20. This app runs
on its own Node 24 at ~/.local/opt/node-24 (conf key node_dir=~/.local/opt/node-24/bin).
The system node stays as it is.
node-v24.x-linux-x64.tar.xz
(nodejs.org/dist/latest-v24.x/), mkdir -p ~/.local/opt/node-24 && tar -xJf <tarball> --strip-components=1 -C ~/.local/opt/node-24. preflight.sh FAILs the node row while
<node_dir>/node is missing, is not 24.x or is not first on the ssh PATH.<node_dir> before ~/.local/bin, so pnpm, builds, gen-secrets and
every remote script run on Node 24.DDX_DEPLOY_NODE_BIN=<node_dir>/node;
deploy/ecosystem.config.cjs sets it as the ddx-osprey-web interpreter and as the node that
wait-deps.sh execs for ddx-osprey-api and ddx-osprey-engine. pm2 save stores it, so a boot resurrect keeps Node 24.pm2 set) and writes the boot unit (pm2 startup). Every pm2 call in
the scripts goes through ddx_pm2 (deploy/scripts/lib.sh): the ssh line exports
DDX_DEPLOY_NODE_DIR, and ddx_pm2 removes that dir from PATH for the one call. The host's
default node then runs the CLI. DDX_DEPLOY_NODE_BIN stays in the env, so our apps still
get Node 24. boot-setup.sh --with-logrotate runs pm2 set only for a key whose value
differs from 50M / 14.boot-setup.sh --check row pm2-node proves the three apps run on <node_dir>/node.DDX_DEPLOY_NODE_BIN gives PM2
plain node; the pm2-node row then FAILs and names the app until the next deploy.deploy/scripts/gen-secrets.sh runs ON the target (deploy.sh calls it). It renders
deploy/env/*.env.example into <app_root>/shared/env/{ddx-api,ddx-web,ddx-osprey-engine,docker}.env
(files mode 600, folder 700) and the node keys into shared/cluster/. It never overwrites a
value; it only appends missing keys. Linked groups hold one value across files: the Postgres
password (docker, ddx-api, engine, LiteLLM database URL), the MinIO root pair (docker, ddx-api,
engine), the three Centrifugo keys (docker, ddx-api, and the API key in the engine) and
LITELLM_MASTER_KEY (the engine API key). Two values that disagree, or an empty linked key,
stop the run. check-env.sh --target <id> re-checks the four files on the target (key names only). --rotate KEY renews one standalone key. Output holds
key names only. No env file, key or token is ever committed.
After each deploy or rollback, deployment-specialist (mode report) writes
context/deploy-reports/<target>/<YYYYMMDD-HHMM>_<from7>..<to7>.md and appends one line to
context/deploy_journal_<host>.md. A pull with no report is an unverified deploy.
deploy/scripts/seed-demo.sh --target <id> [--dry-run] [--force], from the main session.
Owner decision D1 (2026-11-16): full demo (orgs, demo logins, Osprey content), logins live on
the public site. A deploy or redeploy NEVER seeds.
127.0.0.1:<web_port>/api/v1, session cookie +
the public origin, DDX_SEEDER_VIA_BFF=1): production ddx-api enforces node authentication
(invariant 39), so a direct call answers 401 NODE_TOKEN_INVALID and only ddx-web may call it.
Needs a live release and DDX_API_SEED_USER_PASSWORD set in ddx-api.env.DDX_API_DEMO_ACCOUNTS_PUBLIC=true): the login page lists staff, member and
client with the demo password. super-admin and org-admin take DDX_API_SEED_ADMIN_PASSWORD
(generated by gen-secrets.sh), are never listed, and the auth seed refuses a public list
without it. The seeder signs in as org-admin and uses the admin password when it is set.backup-db.sh first (a failed backup stops the run), Prisma seed, then ddx-seeder.<app_root>/.deploy-state/seed-demo.done: a second run prints SEED_SKIP and creates
nothing. --force re-runs (upserts, no duplicates) after a new backup.The deploy installs code and schema, not knowledge. Packs, card folders and the search index are
an operator step (recipe and expected counts: section 6); the IBIS
packs live outside the repo and are copied to <app_root>/shared/ibis-packs/ with rsync --checksum.
Engine env keys that switch capabilities on (file <app_root>/shared/env/ddx-osprey-engine.env,
mode 600; a restart of ddx-osprey-engine with --update-env applies them):
Keys are copied from a sibling production app or the shared-infra portal on the target itself (key
names in chat, never values); back up shared/env before any edit.
forge also runs booter.dudoxx.com (PM2 namespace ddx-web-boot, ports 16300/16310, compose
project ddx-web-boot-prod) and Sentinel (sentinel-*, ports 115xx). PM2 app names are global
per daemon, so a deploy that reuses another app's name takes its processes over (2026-11-16:
booter was down 6 minutes). Rules: our six PM2 names carry the ddx-osprey- prefix; preflight row
pm2-names and the post-swap identity step check owner, namespace and folder; never run
pm2 restart all, delete or save from a script that does not know every owner. After any
forge change, check the neighbours answer (CHECKLIST.md section 4).
adr/
deploy/
research/
DDX_DEPLOY_CONFIRM="OPERATOR_CONFIRMED: <id> deploy" ./ddx-manage.sh remote deploy --target <id> --ref <ref>DDX_DEPLOY_CONFIRM="OPERATOR_CONFIRMED: <id> rollback" ./ddx-manage.sh remote rollback --target <id> [--to <release>]DDX_DEPLOY_CONFIRM="<id> seed-demo ..." ./ddx-manage.sh remote seed-demo --target <id>