# Production VM Runbook Status: **current production operations**. Last reconciled with repository configuration on 2026-08-25. This runbook covers the authoritative OVHcloud single-VM stack. Historical Phase-B provisioning, Neon restore, DNS cutover, and Vercel/Fly coexistence instructions remain in the migration records; they are not normal runbook steps. ## Runtime Inventory | Service | Role | Public exposure | | ---------------------------------------- | ------------------------------------------------- | ---------------------------------------------------- | | `caddy` | TLS termination and reverse proxy | ports 80 and 443 | | `app` | Next.js control plane | `stratofusion.io`, `www.stratofusion.io` | | `rclone` | transfer data plane | `rclone.stratofusion.io` | | `postgres` | authoritative application and GlitchTip databases | Compose network only | | `weaviate` | semantic vector index when enabled | Compose network only | | `cron` | production schedule replay | Compose network only; one replica | | `backup` | PostgreSQL dump and offsite-copy schedule | Compose network only | | `glitchtip-*`, `redis` | error tracking | GlitchTip UI through Caddy; backing services private | | `prometheus`, `grafana`, `node-exporter` | metrics and dashboards | Grafana through Caddy; collectors private | | `pgadmin`, `redisinsight` | optional database administration | disabled by default; host loopback only | The canonical Compose definition is `deploy/docker-compose.prod.yml`; do not operate from an undocumented copy. ## Normal Deployment 1. Merge the reviewed release to `main`. 2. Confirm the `VM Deploy` workflow builds both images for the merge SHA. 3. Confirm the deploy job is enabled and succeeds. 4. Verify `/opt/stratofusion` and both OCI revision labels match the intended SHA. 5. Verify app, rclone, PostgreSQL, Caddy, cron, backup, and observability health. The workflow invokes `deploy/deploy.sh`, which requires a 40-character Git SHA, verifies both pulled first-party OCI revision labels before replacement, drains active rclone operations, rolls the digest-pinned Compose stack, recreates and validates Caddy, and checks `/api/health`. Do not bypass the revision or drain gate for routine work. ## Routine Verification From `/opt/stratofusion/deploy` on the VM: ```bash docker compose -f docker-compose.prod.yml ps docker compose -f docker-compose.prod.yml logs --since 15m app rclone cron docker compose -f docker-compose.prod.yml exec -T app wget -qO- http://127.0.0.1:3000/api/health docker compose -f docker-compose.prod.yml exec -T rclone wget -qO- http://127.0.0.1:3001/health ``` From an operator workstation, verify the canonical public endpoints: ```powershell curl.exe -fsS https://stratofusion.io/api/health curl.exe -fsS https://rclone.stratofusion.io/health curl.exe -fsSI https://grafana.stratofusion.io curl.exe -fsSI https://glitchtip.stratofusion.io ``` Do not use the stopped Fly endpoints as routine health probes. Admins may use `/admin/infrastructure` for a read-only summary of current node resources, application/database readiness, rclone worker health, Prometheus scrape state, queue capacity, release metadata, sanitized Compose replica state, the cron replica invariant, and backup freshness. Optional pgAdmin and RedisInsight rows appear only while their management profile containers exist. Partial or unavailable dashboard data is not proof of a VM or service outage, and the dashboard cannot restart, deploy, roll back, reboot, restore, or open a host shell. Continue to use the commands above and the relevant runbook section for maintenance and incident decisions. Private database-tool setup is covered by [Private VM Administration](./VM_ADMIN_ACCESS.md). ## Scheduler Invariant Production uses exactly one `cron` replica. A second scheduler can duplicate backup, sync, reconciliation, trial-expiry, and monitoring work. ```bash docker compose -f docker-compose.prod.yml ps cron docker compose -f docker-compose.prod.yml logs --since 15m cron ``` If scheduled work is unsafe, set `CRON_REPLICAS=0` in the protected VM environment and recreate only the cron service. Restore one replica only after the cause is understood and direct authenticated route checks pass. ## Rclone Queue And Deployment Safety - Confirm active, running, and queued operation counts before maintenance. - Let `deploy/deploy.sh` enforce the normal drain gate. - Keep the worker's five-minute shutdown grace period aligned with its shutdown timeouts. - Treat partial success, provider throttling, expired credentials, watchdog timeouts, and stale heartbeats as distinct terminal or recovery states. - Never run manual destructive rclone commands without explicit source, destination, direction, flags, dry-run consideration, and approval. ### Approved operation cancellation Do not bypass the deployment drain gate. If an identified operation is known to be disposable or stuck, obtain explicit approval for that exact operation ID, then manually dispatch `Cancel Production Rclone Operation` from the default branch. Enter the lowercase operation UUID in both confirmation fields. The workflow validates the UUID, refuses the local-VM Compose override, sends one authenticated cancellation request from inside the production rclone container, and waits for the operation to leave the active set. The workflow runs the immutable script extracted from the release commit with `STRATOFUSION_DEPLOY_DIR=/opt/stratofusion/deploy`, so its Compose context does not depend on the temporary extraction path. The workflow is deliberately not a general remote-shell facility. It cannot select a source, destination, rclone command, or destructive transfer flags. After cancellation succeeds, rerun the failed `VM Deploy` job for the intended exact release SHA and complete the normal health and scheduler verification. ### First dedicated token-key bootstrap Before deploying a release that first requires `TOKEN_ENCRYPTION_KEY` and `TOKEN_ENCRYPTION_KEY_ID`, dispatch `Bootstrap Production Token Encryption` from `main` with the exact confirmation `bootstrap-production-token-encryption`. It shares the `vm-deploy` lock, runs an immutable script from the selected release, generates the key on the VM, and emits only the non-secret key ID and backup path. The script refuses local VM configuration, partial key configuration, and a key that equals the Clerk secret. After the application deploys, run the documented token-rotation utility in dry-run mode before authorizing any database rewrite. ### Read-only token migration audit After the release containing the audit entrypoint is healthy, dispatch `Audit Production Token Encryption` from `main` with the exact confirmation `audit-production-token-encryption`. The workflow shares the deployment lock, requires the running app OCI revision to equal the selected `main` SHA, and runs only `/app/operations/audit-token-encryption.cjs`. The scanner uses a repeatable-read, read-only transaction and reports aggregate counts without logging tokens, account IDs, user IDs, or row IDs. A result with `rewrittenRows: 0` confirms the audit remained read-only. A nonzero `candidateRows` value means legacy/plaintext envelopes remain and need a separately reviewed migration window. Do not remove `TOKEN_ENCRYPTION_PREVIOUS_KEYS` and do not run the write-enabled rotation utility based solely on this audit. ## Database And Backup Safety - Compose PostgreSQL is the production source of truth. - Never treat the retained Neon database as current without an explicit, approved reconciliation. - Confirm the backup container runs and produces restorable dumps. - Keep an offsite copy outside the VM failure domain. - Validate archives with `pg_restore --list` and periodically prove restore into an isolated database; listing alone is not a complete recovery test. - Before schema repair or restoration, capture a source-preserving pre-change archive and record the exact revision and database target. ## Rollback ### Application Or Worker Image Redeploy a previously verified Git SHA through the same drain-gated deployment path. Verify both image revision labels afterward. ### Database Or Whole-Stack Recovery Stop schedulers and user writes, drain rclone work, identify the authoritative database point, restore into an isolated target first, and reconcile all writes before changing production routing. The legacy Vercel, Neon, and Fly resources do not form a safe DNS-only rollback stack after the first VM production write. Starting a legacy Fly Machine or changing canonical DNS requires an explicitly approved recovery plan. ## Administration Boundary Use VS Code Remote SSH over the key-only `deploy` account for normal operator work. Tailscale is the private management plane. Cockpit, when enabled, remains loopback-bound and is published only through Tailscale Serve. Do not add a public Cockpit port or hostname. See [Private VM Administration](./VM_ADMIN_ACCESS.md) for access setup, least-privilege roles, verification, and rollback. ## Historical And Transitional References - [Architecture Evolution](../ARCHITECTURE_EVOLUTION.md) explains the current, legacy, transitional, and planned models. - [Deployment](../DEPLOYMENT.md) contains the completed Phase-B/Phase-C model, current environment ownership, and legacy Fly recovery rules. - `docs/phase-b-checklist.md`, `docs/local-vm-rehearsal.md`, and `docs/cutover-phase-c-worklog.md` are dated migration evidence. - `docs/runbook.md` is the pre-consolidation runbook and remains a cleanup candidate until deletion is separately approved.