Status: current production operations. Last reconciled with repository
configuration on 2026-08-25.
This runbook covers the authoritative OVHcloud single-VM stack. Historical
Phase-B provisioning, Neon restore, DNS cutover, and Vercel/Fly coexistence
instructions remain in the migration records; they are not normal runbook
steps.
Runtime Inventory
Service
Role
Public exposure
caddy
TLS termination and reverse proxy
ports 80 and 443
app
Next.js control plane
stratofusion.io, www.stratofusion.io
rclone
transfer data plane
rclone.stratofusion.io
postgres
authoritative application and GlitchTip databases
Compose network only
weaviate
semantic vector index when enabled
Compose network only
cron
production schedule replay
Compose network only; one replica
backup
PostgreSQL dump and offsite-copy schedule
Compose network only
glitchtip-*, redis
error tracking
GlitchTip UI through Caddy; backing services private
prometheus, grafana, node-exporter
metrics and dashboards
Grafana through Caddy; collectors private
pgadmin, redisinsight
optional database administration
disabled by default; host loopback only
The canonical Compose definition is deploy/docker-compose.prod.yml; do not
operate from an undocumented copy.
Normal Deployment
Merge the reviewed release to main.
Confirm the VM Deploy workflow builds both images for the merge SHA.
Confirm the deploy job is enabled and succeeds.
Verify /opt/stratofusion and both OCI revision labels match the intended
SHA.
Verify app, rclone, PostgreSQL, Caddy, cron, backup, and observability health.
The workflow invokes deploy/deploy.sh, which requires a 40-character Git SHA,
verifies both pulled first-party OCI revision labels before replacement, drains
active rclone operations, rolls the digest-pinned Compose stack, recreates and
validates Caddy, and checks /api/health. Do not bypass the revision or drain
gate for routine work.
Do not use the stopped Fly endpoints as routine health probes.
Admins may use /admin/infrastructure for a read-only summary of current node
resources, application/database readiness, rclone worker health, Prometheus
scrape state, queue capacity, release metadata, sanitized Compose replica
state, the cron replica invariant, and backup freshness. Optional pgAdmin and
RedisInsight rows appear only while their management profile containers exist.
Partial or unavailable dashboard data is not proof of a VM or service outage,
and the dashboard cannot restart, deploy, roll back, reboot, restore, or open a
host shell. Continue to use the commands above and the relevant runbook section
for maintenance and incident decisions. Private database-tool setup is covered
by Private VM Administration.
Scheduler Invariant
Production uses exactly one cron replica. A second scheduler can duplicate
backup, sync, reconciliation, trial-expiry, and monitoring work.
If scheduled work is unsafe, set CRON_REPLICAS=0 in the protected VM
environment and recreate only the cron service. Restore one replica only after
the cause is understood and direct authenticated route checks pass.
Rclone Queue And Deployment Safety
Confirm active, running, and queued operation counts before maintenance.
Let deploy/deploy.sh enforce the normal drain gate.
Keep the worker's five-minute shutdown grace period aligned with its shutdown
timeouts.
Treat partial success, provider throttling, expired credentials, watchdog
timeouts, and stale heartbeats as distinct terminal or recovery states.
Never run manual destructive rclone commands without explicit source,
destination, direction, flags, dry-run consideration, and approval.
Approved operation cancellation
Do not bypass the deployment drain gate. If an identified operation is known to
be disposable or stuck, obtain explicit approval for that exact operation ID,
then manually dispatch Cancel Production Rclone Operation from the default
branch. Enter the lowercase operation UUID in both confirmation fields. The
workflow validates the UUID, refuses the local-VM Compose override, sends one
authenticated cancellation request from inside the production rclone
container, and waits for the operation to leave the active set. The workflow
runs the immutable script extracted from the release commit with
STRATOFUSION_DEPLOY_DIR=/opt/stratofusion/deploy, so its Compose context does
not depend on the temporary extraction path.
The workflow is deliberately not a general remote-shell facility. It cannot
select a source, destination, rclone command, or destructive transfer flags.
After cancellation succeeds, rerun the failed VM Deploy job for the intended
exact release SHA and complete the normal health and scheduler verification.
First dedicated token-key bootstrap
Before deploying a release that first requires TOKEN_ENCRYPTION_KEY and
TOKEN_ENCRYPTION_KEY_ID, dispatch Bootstrap Production Token Encryption
from main with the exact confirmation
bootstrap-production-token-encryption. It shares the vm-deploy lock, runs
an immutable script from the selected release, generates the key on the VM,
and emits only the non-secret key ID and backup path. The script refuses local
VM configuration, partial key configuration, and a key that equals the Clerk
secret. After the application deploys, run the documented token-rotation
utility in dry-run mode before authorizing any database rewrite.
Read-only token migration audit
After the release containing the audit entrypoint is healthy, dispatch Audit Production Token Encryption from main with the exact confirmation
audit-production-token-encryption. The workflow shares the deployment lock,
requires the running app OCI revision to equal the selected main SHA, and
runs only /app/operations/audit-token-encryption.cjs. The scanner uses a
repeatable-read, read-only transaction and reports aggregate counts without
logging tokens, account IDs, user IDs, or row IDs.
A result with rewrittenRows: 0 confirms the audit remained read-only. A
nonzero candidateRows value means legacy/plaintext envelopes remain and need
a separately reviewed migration window. Do not remove
TOKEN_ENCRYPTION_PREVIOUS_KEYS and do not run the write-enabled rotation
utility based solely on this audit.
Database And Backup Safety
Compose PostgreSQL is the production source of truth.
Never treat the retained Neon database as current without an explicit,
approved reconciliation.
Confirm the backup container runs and produces restorable dumps.
Keep an offsite copy outside the VM failure domain.
Validate archives with pg_restore --list and periodically prove restore into
an isolated database; listing alone is not a complete recovery test.
Before schema repair or restoration, capture a source-preserving pre-change
archive and record the exact revision and database target.
Rollback
Application Or Worker Image
Redeploy a previously verified Git SHA through the same drain-gated deployment
path. Verify both image revision labels afterward.
Database Or Whole-Stack Recovery
Stop schedulers and user writes, drain rclone work, identify the authoritative
database point, restore into an isolated target first, and reconcile all writes
before changing production routing.
The legacy Vercel, Neon, and Fly resources do not form a safe DNS-only rollback
stack after the first VM production write. Starting a legacy Fly Machine or
changing canonical DNS requires an explicitly approved recovery plan.
Administration Boundary
Use VS Code Remote SSH over the key-only deploy account for normal operator
work. Tailscale is the private management plane. Cockpit, when enabled, remains
loopback-bound and is published only through Tailscale Serve. Do not add a
public Cockpit port or hostname.