Fuente: ddx-documentation/deploy/CHECKLIST.md
State and story: [FLOW.md](/es/docs/deploy/FLOW). The short path for an ordinary update of **https://osprey.dudoxx.com** on `dudoxx-forge`. The full runbook is [README.md](/es/docs/deploy/README); this page is the order to follow and the traps that cost time. Run everything from the main session, one chained ssh call at a time, long steps in the background. Every production step needs its confirm line (README section 2). ## 0. Facts to hold | | | |---|---| | Target | `forge` (`deploy/targets/forge.conf`), host `dudoxx-forge`, app root `~/projects/ddx-osprey` | | Public path | browser → `dudoxx-home` Apache :443 (cert) → forge Apache :80 → `127.0.0.1:9600` | | PM2 apps (namespace `osprey`) | `ddx-osprey-engine`, `ddx-osprey-api`, `ddx-osprey-web` (+ `-canary`) | | Containers (`osprey-prod`) | postgres, redis, minio, qdrant, centrifugo, litellm | | Neighbours on the same host | `booter.dudoxx.com` (namespace `ddx-web-boot`, ports 16300/16310), Sentinel (`sentinel-*`). Never touch them | | Release | a `git archive` of the **pushed** `main`; uncommitted work is not deployed | ## 1. Before the deploy (offline, on the Mac) - [ ] Work is committed and **pushed to `main`** (`git status` clean, `git log origin/main..` empty). - [ ] If you changed deploy scripts: all self-tests exit 0 with 0 FAIL (`deploy`, `rollback`, `boot-setup`, `smoke`, `seed-demo`, `gen-secrets`, `check-env`, `backup-db`, `preflight` — each `bash deploy/scripts/<name>.sh --self-test`). - [ ] `./ddx-manage.sh lint` exit 0 (lead-owned, once). - [ ] A **SQL migration** (`ddx-osprey-engine/sql/NNN_*.sql`, Prisma): test it on a fresh database first. A migration that only ever ran on dev broke the first production run (029). - [ ] New env key? Add it to `deploy/env/*.env.example` AND check `check-env.sh` knows it (a required key that is missing stops the deploy before the build, by design). - [ ] New port or service? `ddx-documentation/ports.md` and root `CLAUDE.md` §5 in the same commit. ## 2. Preflight (read-only on forge) ``` ./ddx-manage.sh remote status --target forge deploy/scripts/deploy.sh --target forge --ref main --dry-run ``` - [ ] `remote status`: 3 PM2 apps online, 6 containers up, disk above the floor. - [ ] Dry-run exit 0 (a `tree` FAIL means `deploy/` is uncommitted: commit first). - [ ] Preflight rows OK. `pm2-names` FAIL = another app owns one of our PM2 names: **stop**, never reload over it. `boot-setup` is a WARN on the live release's own check, not a blocker. ## 3. Deploy ``` DDX_DEPLOY_CONFIRM="OPERATOR_CONFIRMED: forge deploy" ./ddx-manage.sh remote deploy --target forge --ref main ``` Run it with `run_in_background`, read the log when it ends. Steps, in order: preflight · fetch · shared · secrets · env-check · install · build-api · build-web · data-tier · ensure-db · backup · migrate · services · engine-migrate · engine-doctor · cluster-mint · canary · canary-smoke · swap · identity · admit-live · pm2-save · live-smoke · finish. - [ ] `result: OK <from>..<to> live on forge`. - [ ] A failure before the swap leaves the live release untouched: read the failing step, fix, commit, push, deploy again (an updated ref needs a new push; the release is the pushed ref). - [ ] A failed live smoke rolls back by itself (exit 2). Rollback to a release from before the PM2 rename is refused: fix forward. ## 4. Verify (all of these, not only the deploy log) - [ ] `deploy/scripts/smoke.sh --target forge --public` exit 0 (layers 1-6, realtime paths). - [ ] `https://osprey.dudoxx.com/api/health` → 200. - [ ] `deploy/scripts/boot-setup.sh --target forge --check` → 12 OK, 0 FAIL. - [ ] **Neighbours untouched:** `https://booter.dudoxx.com/api/health` → 200 and `https://sentinel.dudoxx.com/` answers. `pm2 jlist` on forge: their apps keep their own namespace and cwd. - [ ] Headed browser (jev-browser, owner watching): sign in with a demo account from the login page, open Assistant, send one turn that calls a tool; open Diagnosis as `staff`, save a case, start the analysis. Page `data-revision` equals the database revision. - [ ] One 502 in the first seconds after a deploy is the deleted canary still in the pool for its 15 s lease: retry before you debug. ## 5. Only when needed | Change | Extra step | |---|---| | New or changed **env key / secret** | Edit `~/projects/ddx-osprey/shared/env/<file>.env` on forge (mode 600, key names only in chat), then deploy or `pm2 restart <exact app name> --update-env`. Never `restart all`. Back up the env folder first. | | **Vhost / route** template | `OPERATOR_CONFIRMED="forge publish-route" ./ddx-manage.sh remote route --publish --target forge`, then `remote route --check`. | | **Demo data** (new demo content) | `DDX_DEPLOY_CONFIRM="OPERATOR_CONFIRMED: forge seed-demo" ./ddx-manage.sh remote seed-demo --target forge [--force]`. Runs through the web BFF on loopback (production API only accepts ddx-web). A deploy never seeds. | | **Diagnosis cards / packs** | Section 6 below. | | **Voice** | `DUDOXX_TTS_*` / `DUDOXX_STT_*` in the engine env (Omni), `pm2 restart ddx-osprey-engine --update-env`, then `osprey doctor` shows `tts:ddx-omni` and `stt:ddx-omni` ready. | | **Search tools** | `OSPREY_SEARCH_PROVIDERS` + `OSPREY_EXA_API_KEY` / `OSPREY_BRAVE_API_KEY` in the engine env. Weather and time need no key. | ## 6. Diagnosis knowledge (packs and card folders) Production holds knowledge the deploy never installs. After a database reset, or when the repo gained card folders, run on forge (engine env loaded, from the live release): ``` osprey dx pack install-all osprey dx pack import-ibis <packs>/CORE/1.3.0 osprey dx pack import-ibis <packs>/DERM/1.3.0 --specialty dermatology --supersedes DERM osprey dx pack import-ibis <packs>/CARD/1.3.0 --specialty cardiology --supersedes CARD osprey dx cards install # the 109 card folders in ddx-osprey-engine/datasets/diagnosis/cards osprey dx index rebuild osprey dx index reconcile # expect upserted=0 deleted=0 ``` - The IBIS packs are NOT in the repo (`OSPREY_IBIS_PACKS_DIR`, pinned by `datasets/diagnosis/ibis-packs.lock.json`). Copy `CORE`, `DERM`, `CARD` at `1.3.0` from the HMS sandbox to `~/projects/ddx-osprey/shared/ibis-packs/` with `rsync --checksum`, then verify `sha256sum -c SHA256SUMS` in each folder before the import. - Counts to expect on production: cardiology 86, dermatology 99, gastroenterology 12 cards. `DDC-CARD-0076` exists only in the dev database (never saved as a card folder). - Run it as ONE script with `bash -s` over ssh and a log file under `shared/logs/`; stop at the first non-zero step. ## 7. Traps that already cost hours 1. **PM2 names are global per daemon.** The namespace is only a label. Osprey once took over booter's `ddx-api` / `ddx-web` (same names, same template): booter went down and Osprey ran booter's code while every smoke stayed green. Our names are `ddx-osprey-*`; preflight and the post-swap identity check guard it. 2. **Production API needs a node token.** A direct call to `127.0.0.1:9601` answers 401 `NODE_TOKEN_INVALID`. Only ddx-web (the BFF) may call it; scripts go through `127.0.0.1:9600/api/v1` with the session cookie and the public origin. 3. **`NEXT_PUBLIC_APP_URL` is baked at build.** A wrong value gives 403 `CSRF_ORIGIN` on every sign-in; fix the env, redeploy. 4. **A rendered env file is never overwritten.** `gen-secrets.sh` only appends missing keys; a key that already exists keeps its value (that is how operator keys survive redeploys). 5. **Docker subnets.** The prod compose pins `172.30.196.0/24`; the conf key `docker_subnet` is not read by it. A clash with another app stops `compose up` (preflight `docker-net`). 6. **Model server key.** The forge vLLM answers 401 to an anonymous probe: alive, not down. 7. **Mail.** The API boot probe logs `STACK_MAIL_DOWN` for a TLS-only port (465): it waits for a plain greeting. Not an outage; test a real send before you trust mail. 8. **Public demo list.** `DDX_API_DEMO_ACCOUNTS_PUBLIC=true` lists staff, member and client with the demo password. Admins use `DDX_API_SEED_ADMIN_PASSWORD` and are never listed. 9. **Sentinel is not ours**, but it shares the host: after any forge OS or Node change, check `https://sentinel.dudoxx.com/` (its `argon2` native build broke on the Node 22 upgrade). 10. **Verify the new behaviour, not only the smoke.** A deploy can be green while a tab shows "Unnamed item" or explain answers 409: open the case in the headed browser (FLOW.md §5). 11. **Case form selects.** Check the specialty chip before Save; a wrong click makes a case of the wrong specialty. 12. **Forge DB read-only:** `ssh dudoxx-forge "docker exec -i osprey-prod-postgres sh -c 'psql -U \$POSTGRES_USER -d osprey_engine -At'" < file.sql` (database `osprey_engine`). ## 8. Close - [ ] `deployment-specialist` (mode `report`) writes `context/deploy-reports/forge/<ts>_<from>..<to>.md` and one journal line. A pull with no report is an unverified deploy. - [ ] Docs touched by the change updated in the same commit (this page too, when a step changed).
State and story: . The short path for an ordinary update of on dudoxx-forge. The
full runbook is ; this page is the order to follow and the traps that
cost time. Run everything from the main session, one chained ssh call at a time, long steps in
the background. Every production step needs its confirm line (README section 2).
main (git status clean, git log origin/main.. empty).deploy, rollback, boot-setup, smoke, seed-demo, gen-secrets, check-env,
backup-db, preflight — each bash deploy/scripts/<name>.sh --self-test)../ddx-manage.sh lint exit 0 (lead-owned, once).ddx-osprey-engine/sql/NNN_*.sql, Prisma): test it on a fresh database
first. A migration that only ever ran on dev broke the first production run (029).deploy/env/*.env.example AND check check-env.sh knows it
(a required key that is missing stops the deploy before the build, by design).ddx-documentation/ports.md and root CLAUDE.md §5 in the same commit.remote status: 3 PM2 apps online, 6 containers up, disk above the floor.tree FAIL means deploy/ is uncommitted: commit first).pm2-names FAIL = another app owns one of our PM2 names: stop, never
reload over it. boot-setup is a WARN on the live release's own check, not a blocker.Run it with run_in_background, read the log when it ends. Steps, in order: preflight · fetch ·
shared · secrets · env-check · install · build-api · build-web · data-tier · ensure-db · backup ·
migrate · services · engine-migrate · engine-doctor · cluster-mint · canary · canary-smoke · swap ·
identity · admit-live · pm2-save · live-smoke · finish.
result: OK <from>..<to> live on forge.deploy/scripts/smoke.sh --target forge --public exit 0 (layers 1-6, realtime paths).https://osprey.dudoxx.com/api/health → 200.deploy/scripts/boot-setup.sh --target forge --check → 12 OK, 0 FAIL.https://booter.dudoxx.com/api/health → 200 and
https://sentinel.dudoxx.com/ answers. pm2 jlist on forge: their apps keep their own
namespace and cwd.staff, save a
case, start the analysis. Page data-revision equals the database revision.Production holds knowledge the deploy never installs. After a database reset, or when the repo gained card folders, run on forge (engine env loaded, from the live release):
OSPREY_IBIS_PACKS_DIR, pinned by
datasets/diagnosis/ibis-packs.lock.json). Copy CORE, DERM, CARD at 1.3.0 from the
HMS sandbox to ~/projects/ddx-osprey/shared/ibis-packs/ with rsync --checksum, then verify
sha256sum -c SHA256SUMS in each folder before the import.DDC-CARD-0076 exists only in the dev database (never saved as a card folder).bash -s over ssh and a log file under shared/logs/; stop at the
first non-zero step.ddx-api / ddx-web (same names, same template): booter went down and Osprey ran
booter's code while every smoke stayed green. Our names are ddx-osprey-*; preflight and the
post-swap identity check guard it.127.0.0.1:9601 answers 401
NODE_TOKEN_INVALID. Only ddx-web (the BFF) may call it; scripts go through
127.0.0.1:9600/api/v1 with the session cookie and the public origin.NEXT_PUBLIC_APP_URL is baked at build. A wrong value gives 403 CSRF_ORIGIN on every
sign-in; fix the env, redeploy.gen-secrets.sh only appends missing keys; a key
that already exists keeps its value (that is how operator keys survive redeploys).172.30.196.0/24; the conf key docker_subnet is not
read by it. A clash with another app stops compose up (preflight docker-net).STACK_MAIL_DOWN for a TLS-only port (465): it waits for a
plain greeting. Not an outage; test a real send before you trust mail.DDX_API_DEMO_ACCOUNTS_PUBLIC=true lists staff, member and client with the
demo password. Admins use DDX_API_SEED_ADMIN_PASSWORD and are never listed.https://sentinel.dudoxx.com/ (its argon2 native build broke on the Node 22 upgrade).ssh dudoxx-forge "docker exec -i osprey-prod-postgres sh -c 'psql -U \$POSTGRES_USER -d osprey_engine -At'" < file.sql (database osprey_engine).deployment-specialist (mode report) writes context/deploy-reports/forge/<ts>_<from>..<to>.md
and one journal line. A pull with no report is an unverified deploy.adr/
deploy/
research/
./ddx-manage.sh remote status --target forgedeploy/scripts/deploy.sh --target forge --ref main --dry-runDDX_DEPLOY_CONFIRM="OPERATOR_CONFIRMED: forge deploy" ./ddx-manage.sh remote deploy --target forge --ref mainosprey dx pack install-allosprey dx pack import-ibis <packs>/CORE/1.3.0osprey dx pack import-ibis <packs>/DERM/1.3.0 --specialty dermatology --supersedes DERMosprey dx pack import-ibis <packs>/CARD/1.3.0 --specialty cardiology --supersedes CARDosprey dx cards install # the 109 card folders in ddx-osprey-engine/datasets/diagnosis/cardsosprey dx index rebuildosprey dx index reconcile # expect upserted=0 deleted=0