# Bootstrap Private Production Access Through CI

Use this procedure when the production SSH credential is available only to
GitHub Actions. `VM_SSH_HOST`, `VM_SSH_USER`, and `VM_SSH_KEY` stay in GitHub
secrets; do not try to export the deployment key to a workstation.

The manually dispatched **Bootstrap Production Admin Access** workflow uses
the existing deployment identity to inspect the VM, install the official
Tailscale package, and append a separate operator public key to the `deploy`
account. An explicitly confirmed `harden-ssh` action can first configure
key-only OpenSSH access. Tailnet enrollment stays interactive, and UFW,
Caddy, Compose, and application configuration remain unchanged. The
workstation private key never enters GitHub or the VM.

## Prerequisites

- The workflow must be reviewed and present on `main` before dispatch.
- Merging the workflow follows normal CI and can trigger the existing
  production deployment pipeline. Schedule that separately from bootstrap.
- `VM_SSH_USER` must be `deploy`, with working non-interactive sudo.
- The target must be Ubuntu 24.04 with the production checkout at
  `/opt/stratofusion`, `DEPLOY_ENV=prod`, and the exact production site origin.
- Installation requires key-only SSH. Inspection reports the actual CI SSH
  connection's settings before rejecting unsafe authentication. UFW must be
  active, and the app, rclone, and PostgreSQL containers must be healthy.
- Keep OVHcloud console access available during setup.

## Inspect First

From Windows PowerShell:

```powershell
gh workflow run bootstrap-production-admin-access.yml --ref main -f action=inspect -f confirmation=bootstrap-production-admin-access
```

Read that run's output. Record its `hostname` and the reported
`ssh_host_ed25519_fingerprint`, `ssh_host_ecdsa_fingerprint`, and
`ssh_host_rsa_fingerprint` values (when present). Inspection uses the same initial host-trust
model as the existing deployment action; confirm the fingerprint against the
OVHcloud console or a previously trusted session when available. Installation
then requires a reported fingerprint and the exact inspected hostname.

The CI SSH client can negotiate a different host-key type from workstation
OpenSSH. Use the inspected fingerprint for the negotiated type. If a pinned
connection fails with a fingerprint mismatch before running commands, retry
only with another fingerprint from the trusted inspection; never remove the
pin. Keep the ED25519 fingerprint for workstation verification when selecting
`HostKeyAlgorithms=ssh-ed25519`.

An inspection failure requires correcting the named prerequisite. Do not
weaken its checks or redirect it to the local/rehearsal VM.

## Correct SSH Authentication When Required

If inspection reports password, keyboard-interactive, or root login enabled,
review the actual CI-connection settings and explicitly approve changing the
production login policy before using `harden-ssh`. The initial SSH connection
must already succeed with the existing CI private key. Keep OVHcloud console
access available throughout this change.

Run the workflow with `action=harden-ssh`,
`confirmation=harden-production-ssh`, the inspected `expected_hostname` and
`ssh_host_fingerprint`, and an empty `operator_public_key`.

The action writes `/etc/ssh/sshd_config.d/00-stratofusion-key-only.conf` with
password and keyboard-interactive authentication disabled, root login
disabled, and public-key authentication enabled. It preserves authorized
keys and ports. It validates SSH syntax and the effective global and current
CI-connection policy before reloading `ssh`. A failed validation, config
precedence check, or reload restores the previous drop-in. If a drop-in
already existed, successful runs report a retained backup path.

A separate workflow step proves a fresh SSH connection using the existing CI
key after the reload. If this final connection fails, use the OVHcloud
console: restore the reported backup, or remove the new drop-in when the run
reported `ssh_dropin_created=yes`; validate with `sudo /usr/sbin/sshd -t`
before `sudo systemctl reload ssh`. This final connection failure does not
automatically roll back a successfully applied configuration.

Inspect again after hardening, then perform the normal install action.

## Register The Workstation And Install Tailscale

Create a dedicated ED25519 key using Windows OpenSSH, keeping its private file
under `%USERPROFILE%\.ssh` with access restricted to the Windows account.
A passphrase and the Windows SSH agent can additionally protect interactive
operator access. Never reuse or download the CI private key.

```powershell
ssh-keygen -t ed25519 -f "$env:USERPROFILE\.ssh\stratofusion_prod_ed25519" -C stratofusion-operator
```

In the GitHub workflow form, select `main`, then supply:

| Input                  | Value                                   |
| ---------------------- | --------------------------------------- |
| `action`               | `install`                               |
| `confirmation`         | `bootstrap-production-admin-access`     |
| `expected_hostname`    | Exact hostname from inspection          |
| `ssh_host_fingerprint` | SHA256 fingerprint from inspection      |
| `operator_public_key`  | Contents of the workstation `.pub` file |

The workflow accepts one ED25519 public key with an optional simple comment.
Installation preserves all existing authorized keys, backs up the original
file before adding a key, and does not broaden an existing restricted entry
for the same key. APT verifies Tailscale packages using the official repository
signing key. The command has bounded network/package timeouts and suppresses
automatic restarts of unrelated daemons. Post-checks verify unchanged healthy
app/worker/database container IDs and unchanged UFW rules.

The job uses the shared `vm-deploy` concurrency group. Inspect its result
before retrying a failed or timed-out installation: a failure can leave
Tailscale installed or the operator key registered. Re-running with the same
key is idempotent.

## Join And Verify

Connect using the new key to the verified production public hostname. Compare
the SSH host fingerprint with the inspected value before accepting it. The
public application hostname is usable only if its DNS points directly to the
same OVHcloud VM. Do not disable host-key checking.

From that private terminal, run `sudo tailscale up` and open the authorization
URL yourself. Keep this URL out of GitHub Actions logs, tickets, and commits.
Complete the least-privilege tailnet policy and test key-based SSH over the
VM's private MagicDNS name as described in
[Private VM Administration](./VM_ADMIN_ACCESS.md#join-the-tailnet).

Keep public SSH available for CI. Do not enable Tailscale SSH, routes, an exit
node, or Funnel as part of this workflow. Tailscale provides the private
network path; OpenSSH continues to authenticate the `deploy` public key.

After testing, add a `stratofusion-prod` host entry to the workstation SSH
config using the verified MagicDNS name, `User deploy`, the dedicated
`IdentityFile`, and `IdentitiesOnly yes`.

## Rollback

Verify another trusted SSH session or OVHcloud console works first. To revoke
this workstation, remove only its exact public-key entry from
`/home/deploy/.ssh/authorized_keys`; do not overwrite newer keys by blindly
restoring the backup. Use the parent runbook's Tailscale rollback procedure
only if the VM was subsequently enrolled. The bootstrap does not alter the
existing CI key or public SSH path.