# Deployment Environments

## Overview

StratoFusion uses local Windows development, optional local-VM rehearsal, CI
validation, and authoritative production. The Phase-C cutover is complete:
OVHcloud hosts the production application, PostgreSQL database, rclone data
plane, scheduler, observability, and backup source.

Vercel production, Neon production, and both Fly rclone apps are legacy
rollback resources. They are not safe standalone production rollback targets
after the first VM write; any rollback requires explicit database
reconciliation. The former shared-development deployment is retired from
repository runtime policy and must not be replaced by another permanent shared
development target on the production VM. Its remaining external resources are
shutdown candidates tracked in the
[shared-development retirement runbook](./operations/SHARED_DEVELOPMENT_RETIREMENT.md).
All Fly Machines are intentionally stopped.

## Environment Matrix

| Environment        | Git source                   | App runtime                              | Primary origin                                                    | Database                                  | Rclone runtime                            | Scheduler                           |
| ------------------ | ---------------------------- | ---------------------------------------- | ----------------------------------------------------------------- | ----------------------------------------- | ----------------------------------------- | ----------------------------------- |
| Local Windows      | working tree                 | local Node.js                            | `http://localhost:3000`                                           | `stratofusion-local`                      | `http://127.0.0.1:3001`                   | explicit local replay only          |
| Local VM rehearsal | exact reviewed revision      | local Compose stack                      | `https://local-vm.stratofusion.io` through the Windows hosts file | local Compose PostgreSQL                  | `https://local-vm-rclone.stratofusion.io` | disabled by default                 |
| CI validation      | pushed revision              | ephemeral GitHub runner                  | none                                                              | deterministic fixtures/mocks              | deterministic worker tests                | none                                |
| Temporary staging  | explicitly approved revision | approved migration/rehearsal target only | exact approved origin                                             | isolated or restored snapshot per runbook | exact approved worker                     | disabled unless separately approved |
| Production         | `main`                       | OVHcloud VM Compose (`app`)              | `https://stratofusion.io`                                         | VM Compose PostgreSQL                     | `https://rclone.stratofusion.io`          | exactly one VM cron replica         |

The Git `dev` branch is not a deployment environment. CI validates pull
requests plus long-lived `main` and `dev` pushes, but only successful
push-triggered `main` CI drives the automatic VM deploy workflow.

For API request-integrity checks, preserve Caddy's public `Host` header and
its sanitized `X-Forwarded-Proto` scheme (consumed by Next.js). The app compares
browser origins using that Host and request scheme, not the standalone
container's listening address. Do not rewrite Host to `app:3000` or enable
arbitrary forwarded-host trust to work around origin failures.

## System Audit Logs

System Logs are available at `/admin/system-logs` with admin and recent MFA
verification. The exact legacy `/dev/system-logs` URL redirects there; all
other development surfaces retain their production restrictions. These are
application audit logs; use private VM administration for Caddy logs.

## Private VM Administration

The September 2026 security baseline uses Next.js and `eslint-config-next`
15.5.24, Sharp 0.35.4, and worker Morgan 1.12.0. Both production dependency
audits remain required release gates; install from the checked-in lockfiles.

The VM remains a headless Ubuntu server. Operators use VS Code Remote SSH over
the existing key-only `deploy` account, with Tailscale as the private management
network and optional Cockpit access behind Tailscale Serve. Cockpit is a host
package bound to loopback; it is not part of the application Compose stack and
must not receive a public Caddy hostname or firewall rule.

Public SSH remains available until the GitHub Actions deploy job is migrated to
an independently verified private path. Do not restrict port `22` merely because
interactive operator access works over Tailscale.

Use [Private VM Administration](./operations/VM_ADMIN_ACCESS.md) for setup,
least-privilege authentication, verification, and rollback. The host helper is
`deploy/vm-admin-access.sh`.

When production SSH credentials are available only to GitHub Actions, use the
manual [CI admin bootstrap](./operations/VM_ADMIN_BOOTSTRAP.md). It inspects
the production host before installing Tailscale and registering a separate
operator public key, while preserving existing deployment access.
When inspection finds an authentication mismatch, its separately confirmed
`harden-ssh` action applies key-only OpenSSH settings with validation,
rollback on apply failure, and a fresh CI key-authenticated connection check.

## Read-Only Infrastructure Dashboard Configuration

The admin infrastructure overview queries Prometheus from the Next.js server
over the private Compose network. Production uses
`PROMETHEUS_BASE_URL=http://prometheus:9090`; this variable is server-only and
must never use a `NEXT_PUBLIC_*` name. Local Windows development may leave it
unset, in which case the dashboard renders an explicit metrics-unavailable
state while application and local rclone health remain independently visible.

The optional `INFRASTRUCTURE_NODE_NAME`,
`INFRASTRUCTURE_NODE_ENVIRONMENT`, `INFRASTRUCTURE_NODE_PROVIDER`, and
`INFRASTRUCTURE_NODE_REGION` values are non-secret display labels. Leave an
unknown value blank rather than inserting a hostname, public IP, provider
service ID, or guessed region. `GIT_COMMIT_SHA` remains the release source in
the production image. Grafana and GlitchTip links are emitted only when their
configured HTTPS origins are within the `stratofusion.io` domain family; the
private Cockpit tailnet URL is never part of the response.

Compose passes `GRAFANA_ROOT_URL` and `GLITCHTIP_DOMAIN` explicitly to the app,
sharing the same defaults as Grafana and GlitchTip. Missing or blank `.env`
values use the configured production ingress origins
(`https://grafana.stratofusion.io` and `https://glitchtip.stratofusion.io`).
The local-VM Compose override sets matching local origins on the app as well
as the tools. Outside Compose, the resolver still requires explicit values.
Environment changes require recreating the app container; restarting it does
not reload `.env`. Apply tracked Compose changes through the normal exact-SHA
deployment workflow.

After production deployment, GitHub Actions verifies the app container's release
SHA, allowlisted Grafana/GlitchTip runtime destinations, application/database
readiness, and tool HTTP reachability using `deploy/verify-management-links.mjs`.
The read-only probes have bounded timeouts and retries; they report no credentials.
Private database-tool configuration is reported separately without probing the
operator's tunnel. A final runner-side probe checks public application readiness.

The dashboard associates these destinations with their Compose service rows.
Application, rclone, Caddy, Weaviate, and backup rows use fixed same-origin
operations pages. Prometheus and node-exporter use the allowlisted Grafana
origin, while GlitchTip web and worker use the allowlisted GlitchTip origin.
Rejected or absent external URLs render as not configured. Link configuration
does not grant the application any credentials or mutation capability.

### Optional Private Database Tools

`pgadmin` and `redisinsight` live behind the optional Compose `management`
profile. Production leaves `COMPOSE_PROFILES` blank by default. A local VM may
set `COMPOSE_PROFILES=management` after replacing the pgAdmin password and
RedisInsight encryption-key placeholders in its protected `.env` file. The
services use immutable image digests, dedicated persistent volumes, and host
bindings `127.0.0.1:${PGADMIN_HOST_PORT:-5050}` and
`127.0.0.1:${REDISINSIGHT_HOST_PORT:-5540}`. They are not Caddy upstreams and
must not receive public DNS records or firewall rules.

`PGADMIN_MANAGEMENT_URL` and `REDISINSIGHT_MANAGEMENT_URL` control whether the
corresponding dashboard row has an action. The server accepts only loopback
HTTP (for SSH forwarding) or private `*.ts.net` HTTPS origins; leave either
blank when its private access path is not configured. These variables are
server-only and must never use a `NEXT_PUBLIC_*` prefix. See
[Private VM Administration](./operations/VM_ADMIN_ACCESS.md) for operator
setup and verification.

This configuration adds no Docker socket, SSH key, OVHcloud credential,
Tailscale credential, or host-command capability to the app container.

The Compose stack adds two isolated operational telemetry services. The
no-network `operations-collector` performs a fixed read of Docker container
state and writes only allowlisted aggregate counts to `operations_telemetry`.
The unprivileged `operations-exporter` has no Docker socket, reads that volume
plus sanitized numeric backup state, and is reachable only by Prometheus on the
internal `telemetry_scrape` network. The app remains on the existing
server-only Prometheus boundary.

`INFRASTRUCTURE_BACKUP_FRESHNESS_HOURS` defaults to `36` and may be set from 1
through 168 hours. The nightly backup writes atomic numeric state files for its
latest attempt, latest success, and current/final result; it never exports dump
names, paths, destinations, credentials, or rclone output. A missing first-run
state is unavailable, a failed run or an old success is degraded, and a current
run remains healthy only when a prior successful backup is still fresh.

Production `deploy.sh` now waits for an exporter health response and a non-empty
sanitized Compose snapshot before succeeding. It then requires exactly one
running cron replica. The local-VM override remains exempt from that deploy
gate because its scheduler is deliberately scaled to zero; its dashboard will
show the scheduler invariant as degraded, making the safety difference
explicit.

## Historical VM Naming And Cutover Procedure

The procedure below records the completed rehearsal and cutover model. Do not
repeat its Phase-B or Phase-C DNS steps during normal operation. Current
production state and rollback constraints are recorded in
`docs/cutover-phase-c-worklog.md`.

Use three distinct hostname sets so local rehearsal, public VM rehearsal, and
production cutover cannot be confused:

| Stage                          | App                                      | Rclone                            | Grafana                            | GlitchTip                            |
| ------------------------------ | ---------------------------------------- | --------------------------------- | ---------------------------------- | ------------------------------------ |
| Local-only VM rehearsal        | `local-vm.stratofusion.io`               | `local-vm-rclone.stratofusion.io` | `local-vm-grafana.stratofusion.io` | `local-vm-glitchtip.stratofusion.io` |
| OVH public rehearsal           | `vm-staging.stratofusion.io`             | `vm-rclone.stratofusion.io`       | `vm-grafana.stratofusion.io`       | `vm-glitchtip.stratofusion.io`       |
| Final production after cutover | `stratofusion.io`, `www.stratofusion.io` | `rclone.stratofusion.io`          | `grafana.stratofusion.io`          | `glitchtip.stratofusion.io`          |

Required OVH rehearsal DNS records in the Vercel-hosted zone:

| Record         | Type | Value            |
| -------------- | ---- | ---------------- |
| `vm-staging`   | A    | `15.204.243.136` |
| `vm-rclone`    | A    | `15.204.243.136` |
| `vm-grafana`   | A    | `15.204.243.136` |
| `vm-glitchtip` | A    | `15.204.243.136` |

Do not change apex, `www`, or the final production service records during
Phase B. If temporary legacy records such as `staging.stratofusion.io` still
point at the VM, remove them only after the `vm-*` records are verified and no
current rehearsal depends on them.

The OVH rehearsal runtime must keep:

```env
NEXT_PUBLIC_SITE_URL=https://vm-staging.stratofusion.io
NEXT_PUBLIC_APP_URL=https://vm-staging.stratofusion.io
OAUTH_REHEARSAL_ORIGIN=https://vm-staging.stratofusion.io
FLYIO_RCLONE_SERVICE_URL=https://vm-rclone.stratofusion.io
NEXT_PUBLIC_FLYIO_RCLONE_SERVICE_URL=https://vm-rclone.stratofusion.io
PROD_RCLONE_HOST=vm-rclone.stratofusion.io
CRON_REPLICAS=0
AUTH_ACCESS_PAUSED=true
```

The local VM rehearsal uses local-only `.test` names and the `local-vm`
deployment identity:

```env
DEPLOY_ENV=local-vm
NEXT_PUBLIC_SITE_URL=https://local-vm.stratofusion.io
NEXT_PUBLIC_APP_URL=https://local-vm.stratofusion.io
OAUTH_REHEARSAL_ORIGIN=https://local-vm.stratofusion.io
FLYIO_RCLONE_SERVICE_URL=https://local-vm-rclone.stratofusion.io
NEXT_PUBLIC_FLYIO_RCLONE_SERVICE_URL=https://local-vm-rclone.stratofusion.io
CRON_REPLICAS=0
AUTH_ACCESS_PAUSED=true
```

Local-VM releases use the manually dispatched
`.github/workflows/deploy-local-vm.yml` workflow. It builds both images with
the local public URLs, publishes only commit-SHA image coordinates with
provenance and SBOM attestations, and never runs the production SSH deploy.
Its build uses only the dedicated repository variables
`LOCAL_VM_NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` and
`LOCAL_VM_NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY`; both must be test-mode
`pk_test_...` values. The workflow fails before building instead of falling
back to production public variables. One immutable SHA image can therefore be
deployed to multiple local-VM guests that share the same local/test public
origin contract.

After the build succeeds, an operator checks out the exact SHA on the private
guest, runs the guarded `bootstrap-local-vm-token-encryption.sh` command, and
then runs `TAG=<sha> ./deploy.sh`. The bootstrap requires the local Compose
override and `DEPLOY_ENV=local-vm`, generates the active key only on the guest,
preserves the legacy local key as decrypt-only material, and refuses production
configuration. See `docs/local-vm-rehearsal.md` for the complete setup and
verification sequence.

At runtime, the passed `TAG` is authoritative for both the selected images and
the `GIT_COMMIT_SHA` exposed by the app and worker. An older value left in an
untracked guest `.env` cannot replace the release SHA shown by the
infrastructure dashboard.

External callback and webhook dashboards must be updated for the OVH rehearsal
before OAuth or billing tests:

- Clerk production instance: add `https://vm-staging.stratofusion.io`.
- Google OAuth production client: add `https://vm-staging.stratofusion.io/api/google`.
- Microsoft Entra production app: add `https://vm-staging.stratofusion.io/api/onedrive`.
- Dropbox production app: add `https://vm-staging.stratofusion.io/api/dropbox`.
- Stripe live dashboard: add a temporary webhook endpoint at `https://vm-staging.stratofusion.io/api/webhooks/stripe` if webhook validation is part of the rehearsal.

Rollback during Phase B is DNS-safe: leave production DNS untouched, restore
the previous VM image tag with `TAG=<previous-good-sha> ./deploy.sh`, or remove
the `vm-*` DNS records if the rehearsal must be abandoned. Keep cron disabled
and access paused while investigating.

Phase C cutover checklist:

1. Explicit operator approval for cutover.
2. Prove every Phase-B checklist item, backup, drain gate, and rollback path.
3. Deploy a SHA-tagged release built with apex production URLs and keep VM cron at zero.
4. Disable Vercel schedules and enable the Vercel-only
   `LEGACY_PRODUCTION_WRITE_FREEZE` gate. Verify it returns retryable `503`
   responses for UI/API writes, OAuth callbacks, cron, and webhooks while
   leaving `GET`/`HEAD /api/health` available.
5. Drain Fly and VM rclone operations, then run the final restore with
   `RESTORE_MODE=cutover`. Do not run migrations after the restore.
6. Repoint apex, `www`, `rclone`, `grafana`, and `glitchtip` DNS records to the VM.
7. Smoke test app/rclone health, OAuth callbacks, Stripe webhooks, upload and
   download, observability, backups, image labels, and compose database use.
8. Enable exactly one compose cron replica only after legacy schedules are
   proven inactive and the production smoke checks pass.
9. Reopen access only after the no-write validation window. Keep temporary
   `vm-*` aliases and provider callbacks through the initial rollback window.

### Offsite database backup

The production backup container creates timestamped custom-format dumps of the
application and GlitchTip databases under `/backups`, then uses source-preserving
`rclone copy` to `BACKUP_RCLONE_REMOTE`. An `offsite` OneDrive remote uses a
dedicated single-tenant Entra client and is supplied through the protected VM
`RCLONE_CONFIG_OFFSITE_*` variables: `TYPE`, `CLIENT_ID`, `CLIENT_SECRET`,
`TOKEN`, `DRIVE_ID`, `DRIVE_TYPE`, tenant-specific `AUTH_URL`/`TOKEN_URL`, and
optional `REGION`. Complete OAuth interactively and set `ACCESS_SCOPES` to
rclone's read/write profile without SharePoint-site browsing. Never commit or
print those values. For a controlled cutover proof, set
`BACKUP_PRUNE_LOCAL=false` so the run cannot remove older local dumps. Do not use
`sync`, remote cleanup, or source-removal flags.

The final VM runtime uses apex URLs, an empty `OAUTH_REHEARSAL_ORIGIN`,
`CRON_EXTRA_ALLOWED_HOSTS`, and `CRON_REPLICAS=0` until scheduler handoff.
`deploy/deploy.sh` requires an immutable git-SHA tag; `latest` is never a
deployable rollback reference. Both production images carry the SHA in the
`org.opencontainers.image.revision` OCI label, and the deploy script refuses
to roll the stack unless both pulled labels match the requested SHA. Production
Compose references pin third-party images by manifest digest. The rclone image
also validates the official archive SHA-256 before installation.

Routine releases must not bypass the rclone drain gate. When an exact active
operation has explicit cancellation approval, use the manually dispatched
`Cancel Production Rclone Operation` workflow from `main`, enter the lowercase
UUID twice, and retain its sanitized pre/post evidence. The immutable script is
executed with the production deploy directory explicitly selected, even though
the script itself is extracted to a temporary file. After the operation
leaves the active set, rerun the failed deploy job for the same immutable SHA;
do not rebuild under a different release coordinate merely to retry cutover.

All external GitHub Actions are pinned to reviewed 40-character commits and
CI dependency installs use frozen lockfiles. Dependabot proposes weekly npm,
GitHub Actions, and Docker updates; those proposals still require the normal
review and complete test/build gate before merge. A digest or action pin must
never be refreshed without reviewing the corresponding upstream release.

## Source Of Truth

Use these in order of authority:

- local development: `.env.local`.
- local-VM rehearsal: the untracked local VM `deploy/.env` assembled from the
  production example plus `deploy/local-vm.env.example` overrides.
- production deployment: `/opt/stratofusion/deploy/.env` and the immutable
  `main` image built after successful push-triggered `main` CI.
- approved temporary staging: the explicit rehearsal/migration runbook and its
  isolated configuration for that one window.

Repository env templates are reference material only. Retired Vercel or Neon
snapshots must not be treated as an active development source of truth.

`AUTH_ACCESS_PAUSED` controls the public `/sign-in` and `/sign-up` migration
guard at runtime. Leave it unset or set it to `true` to show the pause notice;
set it to `false` to render the normal Clerk sign-in and sign-up entry points.
Restart the local Next.js server or redeploy the authoritative VM application
after changing this value for the intended environment.

`LEGACY_PRODUCTION_WRITE_FREEZE` is a separate Phase-C safety gate. It can
activate only when its value is exactly `true` on a Vercel production runtime
whose configured site is the apex or `www`. The VM cannot activate it. Keep it
off during ordinary operation and use it only after transfers are drained.

## Environment Ownership Inventory

Each environment has separate application, billing, database, provider OAuth,
and rclone ownership. Keep secrets, full OAuth client IDs, and full account
keys out of public documentation; use this table to verify that a deployment is
pointing at the intended environment family.

| Environment        | Application target                           | Database                               | Clerk                        | Stripe                                  | Provider OAuth callbacks                                | Rclone                                         |
| ------------------ | -------------------------------------------- | -------------------------------------- | ---------------------------- | --------------------------------------- | ------------------------------------------------------- | ---------------------------------------------- |
| Local              | `http://localhost:3000` from `.env.local`    | `stratofusion-local`                   | local test instance          | Stripe sandbox through the CLI listener | localhost callbacks                                     | local service at `http://127.0.0.1:3001`       |
| Local VM           | local hosts-file origin                      | local Compose PostgreSQL               | local/test instance          | sandbox through a local forwarder       | exact local-VM callbacks in separate test provider apps | local VM gateway                               |
| Production         | `https://stratofusion.io` on the OVHcloud VM | VM Compose PostgreSQL                  | live instance                | Stripe live                             | apex callbacks                                          | VM gateway at `https://rclone.stratofusion.io` |
| Approved rehearsal | exact, time-bounded origin only              | isolated database or approved snapshot | explicitly selected instance | explicitly selected mode                | exact approved callbacks                                | exact approved worker                          |

The root and Fly rclone environment templates should use the same section order
for OAuth, Clerk, Stripe, Neon, and Fly settings. Template drift is a deployment
risk because it makes cross-environment comparison harder during OAuth,
billing, database, and rclone incidents.

## Multi-Environment Clerk OAuth

Clerk social sign-in and direct provider OAuth are environment-sensitive.
Local and local-VM use test resources; production uses live resources. Phase B
is a controlled exception that temporarily reuses production resources from
the exact staging origin.

Required rules:

- Local uses its own Clerk test instance and only localhost redirect URIs.
- Local VM uses the local/test Clerk instance and callbacks registered in
  separate test provider apps for the exact local-VM origin.
- Production uses its own Clerk live instance and only `stratofusion.io` redirect URIs.
- Phase B requires `OAUTH_REHEARSAL_ORIGIN=https://vm-staging.stratofusion.io`,
  matching runtime/build-time app URLs, and exact Google, OneDrive, and Dropbox
  redirect URIs on that origin.
- Arbitrary previews, wildcard origins, and other `*.stratofusion.io` hosts are
  not valid OAuth rehearsal environments.
- The environment-specific Google and Microsoft apps may be reused by both direct provider OAuth and the matching Clerk social connection, but they must contain both redirect URIs for that environment.
  - the StratoFusion callback route such as `/api/google` or `/api/onedrive`.
  - the Clerk callback URI copied from the matching Clerk instance.

The application now validates the following at startup:

- `NEXT_PUBLIC_SITE_URL`.
- `NEXT_PUBLIC_APP_URL`.
- `OAUTH_REHEARSAL_ORIGIN` when Phase B is enabled.
- `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY`.
- `CLERK_SECRET_KEY`.
- `GOOGLE_CLIENT_ID` and `GOOGLE_REDIRECT_URI`.
- `ONEDRIVE_CLIENT_ID` and `ONEDRIVE_REDIRECT_URI`.
- `DROPBOX_CLIENT_ID` and `DROPBOX_REDIRECT_URI`.

Direct-provider callback state is HMAC signed. Configure a unique 32+ byte
`OAUTH_STATE_SECRET` per environment when practical. If it is blank, the
server falls back to `TOKEN_ENCRYPTION_KEY` and then `CLERK_SECRET_KEY`, so no
production configuration change is required for the code to fail closed.
Rotating the active signing secret invalidates only in-flight OAuth state,
which expires after ten minutes.

Provider tokens require a dedicated `TOKEN_ENCRYPTION_KEY` of at least 32
bytes and a stable `TOKEN_ENCRYPTION_KEY_ID`. The application does not derive
this key from Clerk or session secrets. New records use an authenticated
`enc:v2:<key-id>` envelope. During rotation, put earlier key IDs and secrets in
the server-only `TOKEN_ENCRYPTION_PREVIOUS_KEYS` JSON object, set a new active
key and ID, deploy, then run `pnpm exec tsx
scripts/casa/rotate-token-encryption.ts` first in its default dry-run mode.
Database writes require both `--apply` and
`TOKEN_ROTATION_APPLY_CONFIRM=reencrypt-user-service-tokens`. Retain prior keys
until a second dry run reports zero candidate rows; never put key values in
logs or captured evidence.

Before the first release that enforces the dedicated key, dispatch
`Bootstrap Production Token Encryption` from `main` and enter
`bootstrap-production-token-encryption` exactly. The workflow runs under the
production deployment lock, generates the active key on the VM, retains the
existing Clerk-derived key only as `legacy-v1` decrypt-only material, updates
`/opt/stratofusion/deploy/.env` atomically, and never sends key values through
GitHub Actions output. It is idempotent for an already-valid dedicated key and
fails closed for partial configuration or a local-VM Compose override.

After the matching release deploys, dispatch `Audit Production Token
Encryption` from `main` and enter `audit-production-token-encryption` exactly.
The workflow verifies that the running app image has the selected release SHA,
then scans token envelopes inside a repeatable-read, read-only database
transaction. Its JSON result contains only the active key ID and aggregate
`scanned`, `candidateRows`, and `rewrittenRows` counts; `rewrittenRows` is
always zero. It cannot invoke the write-enabled rotation path. Treat a nonzero
candidate count as migration planning input, not authorization to rewrite
production data.

Expected runtime mapping:

| Environment       | Site/app URL                         | Rehearsal opt-in      | Clerk key mode | Google redirect                                 | OneDrive redirect                                 | Dropbox redirect                                 |
| ----------------- | ------------------------------------ | --------------------- | -------------- | ----------------------------------------------- | ------------------------------------------------- | ------------------------------------------------ |
| Local             | `http://localhost:3000`              | unset                 | test           | `http://localhost:3000/api/google`              | `http://localhost:3000/api/onedrive`              | `http://localhost:3000/api/dropbox`              |
| Local VM          | `https://local-vm.stratofusion.io`   | exact local-VM origin | test           | `https://local-vm.stratofusion.io/api/google`   | `https://local-vm.stratofusion.io/api/onedrive`   | `https://local-vm.stratofusion.io/api/dropbox`   |
| Phase-B rehearsal | `https://vm-staging.stratofusion.io` | exact staging origin  | live           | `https://vm-staging.stratofusion.io/api/google` | `https://vm-staging.stratofusion.io/api/onedrive` | `https://vm-staging.stratofusion.io/api/dropbox` |
| Production        | `https://stratofusion.io`            | unset                 | live           | `https://stratofusion.io/api/google`            | `https://stratofusion.io/api/onedrive`            | `https://stratofusion.io/api/dropbox`            |

The generic runtime environment classifier may continue to report Phase B as
`unknown`. The OAuth-specific validator reports `staging` only when the exact
rehearsal opt-in and all matching URLs are present, while resource-safety
validation still requires the production database name, live Stripe mode, the
configured production rclone host, and a strong cron secret.

For the full operator sequence, use [CLERK_OAUTH_MULTI_ENVIRONMENT_SETUP.md](./operations/CLERK_OAUTH_MULTI_ENVIRONMENT_SETUP.md).

## Active Cron Jobs

The production cutover release intentionally contains no `crons` entry in
[vercel.json](../../vercel.json). A production Vercel deployment of that
release must be active and its dashboard schedules must be absent before the
compose cron container can scale above zero. Editing the file alone is not
proof of scheduler handoff.

The runtime cron guard allows these jobs only on the stable custom domains:

- `https://stratofusion.io`.
- `https://www.stratofusion.io`.

Preview deployments and unknown hosts still return a skipped `200` response even if the cron route is called manually.
Local development uses `scripts/local-cron.ps1` for replay and debugging instead of Vercel cron. Local replay requires `ALLOW_LOCAL_CRON=true` in `.env.local`, a local request host such as `localhost` or `127.0.0.1`, and the usual `CRON_SECRET` bearer token. Restart `pnpm dev` after changing `.env.local` so the Next.js server sees the flag.

The AI reconciliation schedule may remain installed while AI indexing is
disabled. After cron authentication and production-host validation, the
`reconcile-ai-indexing` route returns a successful skipped response when
`AI_INDEXING_ENABLED` is not true and does not inspect provider accounts.

## Legacy Fly.io rclone Service

Use Git Bash on Windows. Do not use WSL.

The OVHcloud VM is the authoritative production rclone runtime. Both legacy Fly
apps are stopped. The production app remains an approval-gated recovery input;
the former development app is a retirement candidate. Do not probe either
public endpoint while stopped because older deployed configuration may still
allow a request to wake a Machine.

The active recovery script supports only `fly-rclone/fly.prod.toml` from
`main`. `fly-rclone/fly.toml` is retained as external-deletion evidence and is
not a deploy target.

Retained rollback policy:

- Retained configurations record `auto_start=false`, `auto_stop="stop"`, and
  `min_machines_running=0`.
- The Fly deployment workflow is manual-only and requires an explicit paid
  legacy-capacity confirmation plus a rollback/recovery reason.
- Active operation polling and remote cancellation requests now dual-write to PostgreSQL inside `fly-rclone`.
- ZIP download endpoints have been removed from both Next.js and `fly-rclone`.
- Retained stopped Machines cannot be started by Fly Proxy; an operator must
  intentionally start or redeploy them for an approved recovery.
- Queue metrics are now exposed on an internal Fly metrics port so queue depth can be scraped without publishing `/metrics` on the public app service.
- A separate `fly-autoscaler` app can be added later if backlog-based scaling is needed.

### Manually recover the Prod Fly App

```bash
cd fly-rclone
./deploy.sh --env prod
```

Do not create a burst pool during normal VM operation. For an explicitly
approved Fly recovery only:

```bash
cd fly-rclone
./deploy.sh --env prod --machine-pool-size 5
```

### Fly Verification During an Approved Recovery

```bash
curl https://stratofusion-rclone-prod.fly.dev/health
```

Do not run those public checks while the legacy apps are intentionally stopped.

Local metrics verification:

```bash
cd fly-rclone
curl http://localhost:9091/metrics
```

`/health` now returns `503` with `status: "unhealthy"` when Fly service-to-service authentication is missing. A successful health check therefore verifies that `FLYIO_RCLONE_API_KEY` is configured on the Fly app.

Required per-app secrets:

- `NODE_ENV`.
- `LOG_LEVEL`.
- `FLYIO_RCLONE_API_KEY` so the Next.js app can authenticate to protected Fly routes.
- `DATABASE_URL` for shared Fly active operation state.

Configured per-app provider secrets:

- `GOOGLE_CLIENT_ID`.
- `GOOGLE_CLIENT_SECRET`.
- `ONEDRIVE_CLIENT_ID`.
- `ONEDRIVE_CLIENT_SECRET`.
- `DROPBOX_CLIENT_ID`.
- `DROPBOX_CLIENT_SECRET`.

Optional per-app secrets:

- `SENTRY_DSN`.
- `SENTRY_ENVIRONMENT`.
- `DATABASE_URL`.

Current note:

- The VM rclone container is authoritative; separate Fly apps are stopped
  legacy resources.
- The Next.js app generates the rclone config and sends it to the VM rclone
  service per request, so provider secrets are not a separate data-plane
  credential source today.
- Local/test resources and production/live resources remain separated.
- The Fly rclone image pins `pnpm@9.15.0` and installs a pinned upstream `rclone` release with retry logic so deployment does not drift onto older distro packages that lack newer flags such as `--name-transform`.

## Release Order

## AI Search OCR

OCR for AI indexing is disabled by default and is controlled separately from the
main AI search and indexing rollout flags.

| Variable                       | Default     | Purpose                                                                             |
| ------------------------------ | ----------- | ----------------------------------------------------------------------------------- |
| `AI_SEARCH_OCR_ENABLED`        | `false`     | Enables OCR routing for standalone PNG/JPEG AI indexing inputs.                     |
| `AI_SEARCH_OCR_PROVIDER`       | `tesseract` | Selects the OCR provider. Phase 6A implements only local Tesseract.js.              |
| `AI_SEARCH_OCR_TIMEOUT_MS`     | `60000`     | Bounds OCR work per extraction attempt.                                             |
| `AI_SEARCH_OCR_MIN_CONFIDENCE` | `0`         | Skips OCR output below this provider confidence when confidence is available.       |
| `AI_SEARCH_OCR_MAX_PAGES`      | `10`        | Reserved page cap for future PDF OCR; PDF rendering is not implemented in Phase 6A. |
| `AI_SEARCH_OCR_MAX_IMAGES`     | `20`        | Bounds image inputs per OCR extraction attempt.                                     |

The implemented local provider uses `tesseract.js` 7.x and WebAssembly. It does
not require system Tesseract binaries, WSL, or Fly.io rclone changes. PDF page
OCR is intentionally deferred because `pdfjs-dist` page rendering in Node uses
`@napi-rs/canvas`, which would add a native dependency and needs separate
Windows, Vercel, and Fly.io validation before scanned PDFs can be enabled.
Tesseract.js may fetch/cache language data on first use unless deployment
pre-seeds or configures that data path; validate this behavior manually before
enabling OCR outside internal environments.

When promoting environment-sensitive changes, keep the verification order strict:

1. Run local checks and CI for the exact candidate SHA.
2. Build the candidate with **Local VM Release**, deploy it through the local
   VM's drain-gated `deploy.sh`, and verify the local Compose/TLS stack.
3. For an approved rehearsal window, manually dispatch **VM Deploy** with
   `target=staging` and the same exact `release_sha`. The workflow builds a
   distinct `<sha>-staging` app image and rclone image, starts isolated app and
   worker containers, and routes only `vm-staging.stratofusion.io` to them.
   Worker control and signed browser-stream paths use that same rehearsal
   origin, so the shared production worker never needs to admit the Stage
   origin. Database-backed worker operation persistence and Stage worker error
   reporting are disabled for this read-only rehearsal. The workflow verifies
   that both production container IDs remain unchanged and restores the
   canonical Caddy route automatically when candidate deployment fails.
4. Keep staging validation read-only because the temporary candidate uses the
   authoritative VM's existing backing services. Do not launch transfers,
   billing writes, OAuth reconnects, migrations, or scheduled work from it.
5. Dispatch `target=cleanup-staging` after the rehearsal to restore the
   canonical production-only route and remove both candidate containers. The
   worker is removed only after its active-operation drain gate reaches zero.
   Cleanup fails unless production remains healthy and the public staging
   endpoint is unavailable (a transport failure, HTTP 404, or HTTP 421). HTTP
   5xx is a failure because it can indicate that the staging alias still
   targets a removed container. Rollback and cleanup
   read the canonical Caddy configuration from the exact candidate SHA without
   resetting the production checkout, so a pre-merge rehearsal cannot restore
   stale production routing.
6. Merge the reviewed release to `main`; successful push-triggered CI then
   promotes that exact SHA through the normal production workflow.

Do not skip directly to production for OAuth, billing, database, or rclone changes.

## Production Workflow

Use PowerShell for Git and GitHub operations on Windows. Do not use WSL.

1. Merge the reviewed release to `main`.
2. Confirm `CI Checks` succeeds for the `main` push. Its completed run triggers
   `VM Deploy`; failed, cancelled, manual, and pull-request CI runs cannot enter
   the privileged build job.
3. Confirm `VM Deploy` checks out `workflow_run.head_sha`, uses that exact SHA
   for both image tags and OCI revision labels, and passes it to the VM as
   `TAG=<sha>`. The deploy workflow emits maximum provenance plus image SBOMs
   and remains non-cancelling.
4. Verify `/opt/stratofusion` on the VM and both OCI revision labels equal the
   CI-tested commit SHA.
5. Verify application, rclone, PostgreSQL, public TLS, queue state, backups,
   observability, and exactly one cron replica.
6. Do not deploy `stratofusion-prod` on Vercel as part of normal production
   release work.

Manual `workflow_dispatch` requires an explicit target. `target=production`
remains the recovery path and deploys the supplied exact `release_sha` (or the
selected ref's `github.sha`). `target=staging` and `target=cleanup-staging` are
time-bounded rehearsal operations and never replace the production app
container. Production recovery follows the same exact-SHA checkout, image tag,
OCI label, and VM `TAG` path without requiring an upstream CI event.

## Local And CI Workflow

Develop on Windows with the local app, local PostgreSQL, local rclone worker,
local Stripe listener, and explicit local cron replay. Use the documented
local-VM stack when full Compose/TLS rehearsal is useful. Pull requests and
long-lived branch pushes receive CI validation; a branch is not a deployment
environment. A public staging target may exist only for a separately approved,
time-bounded migration or rehearsal and must be removed afterward.

## Domain Notes

- `stratofusion.io`, `www.stratofusion.io`, `rclone.stratofusion.io`,
  `grafana.stratofusion.io`, and `glitchtip.stratofusion.io` point to the
  OVHcloud VM.
- The retired `dev.stratofusion.io` assignment is external cleanup state, not
  an application environment. Follow the retirement runbook; do not repoint it
  to the production VM or another permanent shared deployment.
- DNS for `vm-staging.stratofusion.io` may remain pointed at the self-hosted VM
  for a controlled Phase-B rehearsal, but the canonical Caddy configuration
  does not serve it. The deployment workflow adds the route only for an
  explicitly approved window and cleanup must make the endpoint unavailable.
- The domain zone is delegated to Vercel nameservers.

## Preview Deployments

- Automatic Vercel preview deployments are disabled. Use local app and
  Storybook review instead.
- Do not use arbitrary `*.vercel.app` previews for OAuth validation, Clerk validation, or provider callback testing.
- Provider redirect URIs and popup return flows are intentionally pinned to
  localhost and `stratofusion.io`, plus the exact local-VM or
  `vm-staging.stratofusion.io` rehearsal origin only while
  `OAUTH_REHEARSAL_ORIGIN=https://vm-staging.stratofusion.io` is set for Phase B.
- The rehearsal opt-in accepts no wildcard, alternate subdomain, path, port, or
  non-HTTPS value.
- OAuth routes treat the direct `Host`/request URL as authoritative and use
  `X-Forwarded-Host` only as a same-authority consistency check. Conflicting
  hosts or non-default ports are rejected before provider or Clerk work.
- OAuth callback returns use the validated configured site origin rather than
  forwarded host or protocol values.
- Local Storybook review remains useful because it does not depend on the full OAuth callback chain.

## Verification Checklist

- VM repository HEAD and app/rclone OCI labels equal the intended `main` SHA.
- VM application, rclone, PostgreSQL, observability, backups, and one cron
  replica are healthy.
- Local `.env.local` points to `stratofusion-local`.
- `https://stratofusion.io` and the production service hostnames resolve to the
  OVHcloud VM.
- The retired shared-development host is rejected by OAuth, CORS, and cron
  policy. Its external domain state matches the approved stage of the
  retirement runbook (`410 Gone` or no DNS after removal).
- Both Fly apps report all Machines stopped; do not probe their public health
  endpoints during normal operation.
- Stable-host cron routes return `200` when called with `CRON_SECRET`.
- Preview or unknown-host cron calls return a skipped `200`.
- Local and local-VM testing use test Clerk, provider, and Stripe resources.
- Production uses live Clerk and live Stripe.
- Phase B uses the exact staging origin for both `NEXT_PUBLIC_SITE_URL` and
  `NEXT_PUBLIC_APP_URL`, and for the Google, OneDrive, and Dropbox callbacks.
- Clerk, Google, Microsoft, Dropbox, the Next.js runtime, and the rclone CORS
  policy all agree on the exact rehearsal origin before OAuth testing starts.
- Arbitrary preview hosts remain OAuth-disabled.
- `OAUTH_REHEARSAL_ORIGIN` is absent from normal production and the apex
  app/callback URLs are authoritative.
- `next.config.js` and `vercel.json` security headers are aligned before release.
- Public docs are available under `/docs`; all `/dev`, `/api/dev`,
  `/api/debug`, debug, demo, and test surfaces return a non-cacheable `404` on
  every non-loopback host.
- `pnpm audit --prod` reports no known production vulnerabilities before CASA submission.

For the completed rollout validation record, see [ENVIRONMENT_SEPARATION_VALIDATION_2026-03-24.md](./ENVIRONMENT_SEPARATION_VALIDATION_2026-03-24.md).

## Operational Notes

- The VM cron container is the sole production scheduler and exactly one
  replica must run.
- Vercel project relinking or deployment is a legacy recovery/cost exception,
  not a normal release step.
- If a custom domain is stuck on the wrong project, move the project-domain assignment before recreating the alias.
- For Stripe redirect flows, keep `NEXT_PUBLIC_APP_URL` aligned with `NEXT_PUBLIC_SITE_URL` for the target environment.