# Cut Over VM Staging To Production You are working on StratoFusion. Execute the controlled Phase C production cutover from the proven OVH rehearsal stack at `https://vm-staging.stratofusion.io` to the canonical production origin `https://stratofusion.io`. This prompt is explicit authorization to prepare, commit, push, and perform the production cutover, including the final database restore and DNS changes, but only after every hard gate below passes. It is not authorization to destroy or decommission Vercel, Fly.io, Neon, the `vm-*` rehearsal aliases, or any other rollback resource. Decommissioning is Phase D and requires separate approval after a stability window. Do not stop after writing a plan. Work through preparation, verification, cutover, and reporting in this thread. Pause only for a hard gate, an account login/MFA handoff, a material conflict with the repository, or a failed check that makes proceeding unsafe. ## Required Context Before changing anything, read: - `AGENTS.md` - `README.md` - `public/docs/developer/AI_OPERATING_PROTOCOL.md` - `docs/ai-skills/INDEX.md` - `docs/ai-skills/01-architecture.md` - `docs/ai-skills/03-rclone-data-plane.md` - `docs/ai-skills/08-testing.md` - `docs/ai-skills/09-authentication-oauth.md` - `docs/ai-skills/11-deployment.md` - `docs/phase-b-checklist.md` - `docs/runbook.md` - `deploy/restore-from-neon.sh` - `deploy/deploy.sh` - `deploy/Caddyfile` - `deploy/docker-compose.prod.yml` - `deploy/.env.production.example` - `.github/workflows/deploy.yml` - `.github/workflows/fly-rclone-deploy.yml` - `vercel.json` - `public/docs/DEPLOYMENT.md` - `public/docs/RCLONE_SERVICE.md` - `public/docs/operations/CLERK_OAUTH_MULTI_ENVIRONMENT_SETUP.md` Use PowerShell for Windows-side repository work, Git, PNPM, DNS checks, and SSH orchestration. Never use WSL or bare `bash`. Invoke Git Bash explicitly only for repository `.sh` scripts that require POSIX behavior. On the Linux VM, normal POSIX shell commands are appropriate. Use the in-app Browser for authenticated provider dashboards and end-to-end browser verification when required. Use Computer control only when a task cannot be completed reliably through the browser or terminal. Reuse signed-in sessions, ask me to take over for login or MFA, and never ask me to paste a password, secret, OAuth token, private key, or database URL into chat. ## Current Known State Treat these facts as starting context and verify all mutable facts before use: - Repository: `C:\code\stratofusion`. - Migration branch: `migrate/vm-consolidation`. - Current known branch head: `ad1613515b6bfa546c0de1a3e2224b1d5eb1f1b0`. - OVH VM public IPv4: `15.204.243.136`. - Expected SSH identity: `C:\Users\rhoun\.ssh\stratofusion-vm-deploy`. - The rehearsal app and rclone health endpoints have previously returned healthy responses. - Rehearsal names currently point to the OVH VM: - `vm-staging.stratofusion.io` - `vm-rclone.stratofusion.io` - `vm-grafana.stratofusion.io` - `vm-glitchtip.stratofusion.io` - The current production app remains on Vercel, its production database remains Neon, and its data plane remains Fly.io until cutover. - The VM uses compose Postgres restored from Neon and must have `CRON_REPLICAS=0` until the scheduler handoff is proven safe. - `AUTH_ACCESS_PAUSED` only pauses sign-in/sign-up. It does not freeze writes from already authenticated users and is not an acceptable write-freeze by itself. - The current Caddyfile serves only the `vm-*` names and needs production hostnames before cutover. - The migration-branch GitHub workflow deliberately bakes `vm-staging` and `vm-rclone` public URLs. The final image must be built from `main` or another explicitly corrected production build path with apex URLs. - `vercel.json` still defines production cron schedules. DNS movement alone does not prove that Vercel Cron has stopped. - Google and Dropbox rehearsal callbacks were previously added to the production provider apps. Treat Microsoft Entra, Stripe, Clerk, and all current dashboard state as unverified until directly confirmed. - Preserve unrelated working-tree changes. In particular, do not modify or delete an unrelated untracked `vercel-stratofusion-dns-table.png` if present. ## Target Production Topology | Service | Production hostname | Target | | --- | --- | --- | | App | `stratofusion.io` | OVH VM `15.204.243.136` | | App alias | `www.stratofusion.io` | canonical redirect or proxy consistent with existing production behavior | | Rclone | `rclone.stratofusion.io` | OVH VM `15.204.243.136` | | Grafana | `grafana.stratofusion.io` | OVH VM `15.204.243.136` | | GlitchTip | `glitchtip.stratofusion.io` | OVH VM `15.204.243.136` | | Database | compose Postgres `stratofusion-prod` | fresh final restore from Neon | | Scheduler | compose cron container | exactly one replica only after legacy schedules are disabled | Keep the `vm-*` hostnames available through the initial rollback window unless their presence creates a verified security or routing problem. ## Non-Negotiable Safety Invariants - Never print complete `.env` files or reveal secrets in commands, logs, chat, screenshots, commits, or documentation. Compare secret length or SHA-256 hashes when identity must be verified. - Never point the VM at Neon as its writable `DATABASE_URL`. Neon is read-only input to `restore-from-neon.sh`; the VM app must use compose Postgres. - Keep VM cron at `CRON_REPLICAS=0` until the old Vercel scheduler is disabled and the new production stack passes core validation. - Drain and verify all active rclone/Fly operations before the write freeze and final database dump. - Do not run `pnpm db:migrate` after restoring the Neon dump. The dump already contains the migration history. - Do not perform destructive provider operations. Manual rclone commands use `--dry-run` unless this prompt specifically requires a live non-destructive smoke operation. Do not run delete, purge, move, sync-delete, bisync recovery, or overwrite tests. - Capture existing DNS records, TTLs, deployment IDs, image tags, Git SHAs, and rollback commands before changing production state. - Never enable writes on both the legacy and VM databases at the same time. - Do not claim that local and production images are byte-identical. Different baked `NEXT_PUBLIC_*` URLs normally produce different image digests. Prove release equivalence with the Git commit/tree identity, reviewed source diff, build inputs, and image labels. - If users can write to the VM after the final restore, a DNS-only rollback to Neon can lose those writes. Keep production access paused until the no-write rollback window has passed, and explicitly report when rollback changes from DNS-only to a database reconciliation exercise. ## Hard Stop Conditions Do not change production DNS if any of these is true: - Phase B evidence is incomplete for app health, rclone health, Clerk, Google Drive, OneDrive, Dropbox, upload, download, Stripe webhook, Grafana, GlitchTip, backup, or drain-gate behavior. - Microsoft Entra does not contain both the apex production callback and the temporary `vm-staging` callback. - The selected cutover source tree differs from the rehearsed tree in unreviewed application logic. - Active transfer operations cannot be drained or their state is ambiguous. - There is no tested mechanism that blocks authenticated writes on legacy production during the final dump. - Vercel Cron cannot be disabled or proven inactive before VM cron activation. - The final Neon restore reports missing tables, migration inconsistencies, or unexplained row-count differences. - The VM image was built with `vm-staging`/`vm-rclone` public URLs instead of production URLs. - The production Caddy routes or runtime environment are incomplete. - A rollback path has not been recorded and checked. - DNS TTL is materially higher than expected and the previous TTL has not elapsed; report the risk and obtain explicit confirmation before proceeding. When blocked, continue all safe diagnostics and preparation, then report the exact failed gate and the smallest required operator action. ## Phase 1: Inventory And Evidence 1. Inspect the working tree, branch, recent commits, branch relationship to `main`, and open PR/check state. Do not overwrite unrelated changes. 2. Record current public DNS answers and TTLs for: `stratofusion.io`, `www.stratofusion.io`, `rclone.stratofusion.io`, `grafana.stratofusion.io`, `glitchtip.stratofusion.io`, and all four `vm-*` names. 3. Capture the current Vercel production deployment ID/commit and the current Fly production worker release without changing them. 4. Connect to the OVH VM and capture, without secrets: - host identity and disk/memory health; - repository commit and tree hash; - compose service status; - app and rclone container image names, tags, and image digest labels; - current non-secret hostname-related environment values; - `CRON_REPLICAS`, `AUTH_ACCESS_PAUSED`, and whether the cron container is actually absent/stopped; - recent backup status and available restore points. 5. Run real TLS health checks against `vm-staging` and `vm-rclone` and inspect Caddy logs for certificate or routing errors. 6. Reconcile the Phase B checklist. Produce a concise pass/fail table and do not infer a pass from a generic `/api/health` response. ## Phase 2: Prepare The Cutover Release Make the smallest coherent repository changes required for final production. At minimum, inspect and address all of the following: 1. Caddy must serve the five production hostnames. Keep `vm-*` aliases during the rollback window. Preserve unlimited/unbuffered rclone streaming. 2. Decide and implement the established `www` canonical behavior; do not create a redirect loop or change callback hosts. 3. Remove staging-only hardcoding from production compose configuration. GlitchTip domain, Grafana root URL, app URL, rclone URL, DSNs, and related host-sensitive settings must resolve to production values at cutover while rehearsal values remain usable before cutover. Prefer existing environment configuration patterns over duplicate compose files. 4. The final VM runtime values must be: ```env DEPLOY_ENV=prod NEXT_PUBLIC_SITE_URL=https://stratofusion.io NEXT_PUBLIC_APP_URL=https://stratofusion.io FLYIO_RCLONE_SERVICE_URL=https://rclone.stratofusion.io NEXT_PUBLIC_FLYIO_RCLONE_SERVICE_URL=https://rclone.stratofusion.io PROD_RCLONE_HOST=rclone.stratofusion.io GOOGLE_REDIRECT_URI=https://stratofusion.io/api/google ONEDRIVE_REDIRECT_URI=https://stratofusion.io/api/onedrive DROPBOX_REDIRECT_URI=https://stratofusion.io/api/dropbox CRON_REPLICAS=0 AUTH_ACCESS_PAUSED=true ``` Remove or leave empty `OAUTH_REHEARSAL_ORIGIN` for final production. Remove `vm-staging.stratofusion.io` from `CRON_EXTRA_ALLOWED_HOSTS` when it is no longer needed. Set the production GlitchTip/Grafana host variables and the apex Stripe webhook signing secret as required by the existing config. 5. The GitHub Actions production build variables must bake: - `NEXT_PUBLIC_SITE_URL=https://stratofusion.io` - `NEXT_PUBLIC_APP_URL=https://stratofusion.io` - `NEXT_PUBLIC_FLYIO_RCLONE_SERVICE_URL=https://rclone.stratofusion.io` - the existing live Clerk and Stripe publishable keys. 6. Remove the temporary migration-branch deploy trigger or hardcoded rehearsal build branch only when the same source tree is safely available on `main`. 7. Disable production Vercel cron schedules as part of the scheduler handoff. Preserve development schedules if the `dev` deployment still needs them. Do not enable VM cron merely because `vercel.json` was edited; verify the production Vercel deployment containing the change is active and that its scheduled jobs are absent/inactive. 8. Establish a real, tested legacy-production maintenance/write-freeze mechanism. `AUTH_ACCESS_PAUSED` is insufficient. Prefer a reversible, environment-controlled gate that can be enabled on Vercel while remaining disabled on the VM. It must block authenticated UI/API mutations, OAuth callbacks, cron execution, and write-producing webhooks on legacy production while allowing a minimal health/maintenance response. Drain active operations before enabling it. Ensure external webhook providers receive a retryable failure rather than a false success if events cannot be processed during the freeze. 9. Add focused tests for any maintenance gate, host validation, Caddy/compose configuration helper, or scheduler behavior changed. Update canonical deployment, OAuth, rclone, testing, and runbook documentation where behavior changes. 10. Run `pnpm env:guard`, focused tests, type checking, and the broader checks warranted by the diff. Validate compose and Caddy configuration without exposing interpolated secrets. Commit and push the cutover preparation intentionally. Ensure CI passes. Deploy the candidate source tree to `vm-staging` with rehearsal build arguments first where practical, then rerun all affected Phase B checks. Record the rehearsed Git tree hash. The final `main` release may have a different commit SHA due to merge mechanics, but its source tree and application logic must match the reviewed candidate except for explicitly documented production build/runtime configuration. ## Phase 3: Confirm External Production Configuration Using authenticated dashboards where necessary, verify before DNS movement: - Clerk live instance recognizes `stratofusion.io` as its production domain and allows the apex origin/callback behavior required by the app. - Google production OAuth client contains `https://stratofusion.io/api/google`. - Microsoft Entra production app contains `https://stratofusion.io/api/onedrive`, supports the intended personal and organizational account types, and has no unresolved tenant/login blocker. - Dropbox production app contains `https://stratofusion.io/api/dropbox`. - Stripe live webhook endpoint is `https://stratofusion.io/api/webhooks/stripe`, and the VM uses that exact endpoint's signing secret rather than the temporary `vm-staging` secret. - Production DNS records can be edited and their previous values/TTLs have been recorded. Keep both apex and `vm-staging` callbacks during the rollback window. Do not remove the apex callbacks at any point. Hand browser control to me for login or MFA and resume immediately after I confirm completion. ## Phase 4: Build And Stage The Final Production Image 1. Select and record `CUTOVER_SHA`, its Git tree hash, and the previous known good VM tag. 2. Merge or fast-forward the reviewed migration tree to `main` using the repository's normal workflow. Do not force-push or rewrite shared history. 3. Confirm CI and the VM build/deploy workflow pass for `CUTOVER_SHA`. 4. Confirm the final app image labels contain `CUTOVER_SHA` and that its baked public URLs are apex production values. Do not rely on the tag `latest`. 5. Before deployment, update `/opt/stratofusion/deploy/.env` atomically and securely with production hostnames, apex OAuth callbacks, the apex Stripe webhook secret, production observability URLs, `CRON_REPLICAS=0`, and `AUTH_ACCESS_PAUSED=true`. Do not display the file. 6. Deploy `TAG=` through the normal drain-gated workflow. Verify internal app/rclone health, compose status, image labels, and Caddy config. 7. Keep user access and cron paused. The apex certificate may not issue until DNS points to the VM; this is expected, but all other pre-DNS checks must pass. ## Phase 5: Freeze Legacy Writes And Take The Final Restore This is the start of the maintenance window. 1. Announce the maintenance start in the work log with an exact UTC timestamp. 2. Verify active Fly and VM rclone operations are zero and no backup/sync job is launching. Do not terminate a live transfer merely to shorten the window. 3. Enable the tested write-freeze on legacy Vercel production and verify from both signed-out and authenticated paths that mutations cannot reach Neon. 4. Verify the production Vercel cron schedules are inactive. Keep VM cron at zero. 5. Confirm webhook/OAuth behavior during the freeze is retryable or safely blocked and does not acknowledge unprocessed writes. 6. Run the final `restore-from-neon.sh` on the OVH VM using the Neon production URL supplied securely outside chat. Let the script stop app/rclone, recreate compose Postgres, restore, analyze, and compare counts. 7. Capture the dump timestamp, restore completion time, migration count, sequence checks, critical-table checks, and row-count comparison. Explain every discrepancy before proceeding. 8. Verify the VM app and rclone restart healthy against compose Postgres and that the VM has not written to Neon. 9. Take or confirm a fresh VM/provider snapshot and offsite backup where the existing runbook supports it. If this phase fails, leave production DNS unchanged, keep VM cron at zero, disable public VM access as appropriate, restore legacy Vercel service, and report the blocker. ## Phase 6: DNS Cutover Only after all prior gates pass, update the production DNS zone so these names route to the OVH VM: | Name | Required destination | | --- | --- | | `stratofusion.io` | `15.204.243.136` | | `www.stratofusion.io` | canonical apex or `15.204.243.136`, consistent with the chosen DNS design | | `rclone.stratofusion.io` | `15.204.243.136` | | `grafana.stratofusion.io` | `15.204.243.136` | | `glitchtip.stratofusion.io` | `15.204.243.136` | Remove conflicting Vercel records only as required to make these answers unambiguous. Do not delete the recorded previous values. Keep TTL at about 300 seconds through the rollback window. Monitor authoritative and public resolvers until the expected answers appear. Then monitor Caddy until valid public certificates are issued for every final hostname. Do not bypass TLS errors in the production pass criteria. ## Phase 7: Production Validation Keep VM cron at zero and keep general access paused during the first smoke checks. Record evidence for each item: 1. `https://stratofusion.io/api/health` returns `200` with a valid public TLS certificate and reports healthy database connectivity. 2. `https://www.stratofusion.io` follows the intended canonical behavior. 3. `https://rclone.stratofusion.io/health` returns `200` with valid TLS. 4. Grafana and GlitchTip load at their production hostnames without redirecting to `vm-*` names. 5. Container image labels and `/api/health` or build metadata, if available, identify `CUTOVER_SHA`. 6. Application and rclone logs contain no startup validation, CORS, database, OAuth-origin, certificate, or secret errors. 7. Send a Stripe live test event and confirm signature verification and a `2xx` response on the apex endpoint. 8. Open the production sign-in flow and verify Clerk loads from the apex. Use a tightly controlled operator window if access must be unpaused. 9. Connect or reconnect Google Drive, OneDrive, and Dropbox one at a time. Each redirect and return must remain on `stratofusion.io`, each account must be stored in compose Postgres, and no callback may land on Vercel/Neon. 10. Perform a small, non-destructive upload and download using an operator-owned test account and test folder. Browser traffic must use `https://rclone.stratofusion.io`; verify exact-origin CORS and byte success. Clean up test data through the normal UI only if the operation and target are explicit and approved; otherwise leave it labelled for later cleanup. 11. Confirm Grafana receives current metrics and GlitchTip receives separate app and worker smoke events. 12. Verify backup configuration and run a non-destructive backup smoke if it was part of the rehearsed checklist. If any core check fails before general user access is opened, roll back using the no-write procedure below. Do not keep debugging with production partially open. ## Phase 8: Scheduler Handoff And Reopen 1. Reconfirm Vercel Cron is disabled and no legacy scheduler has fired during the maintenance window. 2. Reconfirm restored jobs were not unintentionally parked with rehearsal-only timestamps. Compare against the final Neon source and the documented Phase B parking procedure before changing schedules. 3. Set `CRON_REPLICAS=1`, apply compose, and prove exactly one cron container is running. 4. Observe at least two expected execution cycles for sync reconciliation and backup scheduling. Confirm no duplicate launches, correct compose Postgres writes, and correct VM rclone operation IDs. 5. Restore normal production sign-in/sign-up by setting `AUTH_ACCESS_PAUSED=false` only after the operator smoke tests pass. 6. Disable the legacy Vercel write-freeze only if Vercel must remain an active rollback target and doing so cannot reopen writes to Neon. Normally leave the legacy deployment in maintenance during the rollback window. 7. Record the exact UTC time general production access reopened. 8. Monitor health, error rate, OAuth failures, Stripe events, cron logs, queue depth, database connections, disk, CPU, memory, and backups closely for at least the first hour. At the moment general users can write to compose Postgres, state clearly in the work log: `DNS-only rollback is no longer data-safe; rollback now requires VM write freeze and database reconciliation back to Neon.` ## Rollback Prepare and record both rollback modes before cutover. ### No-Write Rollback Use this only before general writes have been accepted on the VM: 1. Keep or restore `CRON_REPLICAS=0` on the VM. 2. Keep VM user access paused. 3. Restore the captured DNS records and TTLs to Vercel/Fly. 4. Promote or restore the recorded previous-good Vercel production deployment. 5. Restore Vercel cron schedules only after verifying the VM scheduler is off. 6. Disable the legacy write-freeze and verify Vercel/Neon health. 7. Confirm Stripe, Clerk, provider callbacks, and Fly rclone are again serving the legacy production path. ### Post-Write Rollback Do not perform a DNS-only rollback after VM writes begin. First: 1. Freeze VM writes and disable VM cron. 2. Drain VM rclone operations. 3. Quantify database and external side-effect divergence since the final dump. 4. Design and verify a forward data migration/reconciliation from compose Postgres back to Neon, including Stripe/webhook and provider-operation state. 5. Obtain explicit operator approval for that reconciliation plan. 6. Only then restore legacy DNS and schedulers. If the app image alone is faulty while VM data remains authoritative, prefer `TAG= ./deploy.sh` on the VM over moving the database back to Neon. Verify compatibility before rolling back across schema changes. ## Deferred Cleanup Do not perform these in the immediate cutover without separate approval: - deleting Vercel projects or deployments; - deleting Fly apps, machines, volumes, secrets, or workflows; - deleting Neon branches/projects or Weaviate Cloud data; - removing `vm-*` DNS aliases or callback entries needed for rollback; - rotating Clerk or token-encryption secrets; - deleting old Docker images, backups, snapshots, or database dumps; - removing monitoring history; - beginning Phase D repository cleanup. After the agreed stability window, present a separate Phase D cleanup plan that includes retention, export, rollback expiry, cost, and secret-rotation choices. ## Final Report Provide a concise but complete report containing: - cutover start, DNS change, general reopen, and completion times in UTC; - `CUTOVER_SHA`, Git tree hash, app/rclone image tags and digest labels; - source/rehearsal equivalence evidence and expected build-argument differences; - files changed, commits created, branch/PR/merge state, and CI results; - old and new DNS records plus observed propagation and TLS status; - final Neon dump/restore verification and any count differences; - legacy write-freeze and scheduler-disable evidence; - production runtime hostname values without secrets; - pass/fail evidence for health, Clerk, Google, OneDrive, Dropbox, Stripe, upload/download, Grafana, GlitchTip, backup, and cron; - whether general writes have started and which rollback mode remains safe; - deferred cleanup items, remaining risks, and the recommended stability window. Do not report success unless every core production check passes. If the cutover is blocked or rolled back, state the exact phase reached, current source of truth for user data, active scheduler, active DNS destination, and next safe action.