Admin Infrastructure Dashboard: Read-Only First Slice Prompt
Status: Ready for implementation
Task category: New feature
Scope: Admin-only, read-only infrastructure observability for the authoritative production VM
Prompt
You are working on StratoFusion in C:\code\stratofusion. Implement the first slice of an admin infrastructure dashboard.
This slice is a read-only operational overview for the single authoritative OVHcloud production VM. Shape the domain model as a small fleet so more VMs can be added later, but do not add VM, Docker, deployment, scheduler, backup, database, or rclone mutations in this task.
Before coding:
1. Run `git status --short` and inspect relevant diffs. Preserve unrelated user changes. Do not revert, clean, stage, commit, or overwrite them.
2. Run `pnpm env:guard`.
3. Read:
- AGENTS.md
- README.md
- public/docs/developer/AI_OPERATING_PROTOCOL.md
- public/docs/ARCHITECTURE.md
- public/docs/API_REFERENCE.md
- public/docs/DEPLOYMENT.md
- public/docs/RCLONE_SERVICE.md
- public/docs/TESTING.md
- public/docs/UI_DESIGN_GUIDELINES.md
- public/docs/UX_PATTERNS.md
- public/docs/operations/VM_PRODUCTION_RUNBOOK.md
- public/docs/operations/VM_ADMIN_ACCESS.md
- docs/ai-skills/01-architecture.md
- docs/ai-skills/03-rclone-data-plane.md
- docs/ai-skills/06-api-development.md
- docs/ai-skills/07-ui-development.md
- docs/ai-skills/08-testing.md
- docs/ai-skills/11-deployment.md
- docs/ai-skills/12-feature-development-workflow.md
4. Inspect the current implementation before deciding file placement:
- src/app/admin/monitoring/page.tsx
- src/app/admin/users-roles/page.tsx
- src/components/SettingsMenu.tsx
- src/components/__tests__/SettingsMenu.test.tsx
- src/app/api/health/route.ts
- src/app/api/rclone/health/route.ts
- src/lib/rclone/core/flyio-client.ts
- src/lib/auth/roles.ts
- src/lib/api-response.ts
- deploy/docker-compose.prod.yml
- deploy/prometheus/prometheus.yml
- deploy/.env.production.example
- Dockerfile
- fly-rclone/src/services/queueMetrics.js
5. Write a short implementation plan before editing. Identify the canonical models, ports, adapters, service, API route, hook, UI components, tests, and docs you will change.
Environment rules:
- Use Windows 11 PowerShell for normal repository work.
- Use PNPM only.
- Do not use WSL or bare `bash`.
- Invoke `C:\Program Files\Git\bin\bash.exe` explicitly only if a `.sh` script must be run.
Goal
Create an admin-only `/admin/infrastructure` page that lets an operator answer, at a glance:
- Is the production node healthy?
- Are application/database readiness, the rclone worker, and Prometheus scrapes healthy?
- What are current CPU, memory, root-filesystem, network, uptime, and rclone queue readings?
- Which application release is running?
- When was the snapshot sampled, and which data is unavailable or stale?
The page must remain useful when one dependency is unavailable. Partial observability must degrade individual sections rather than fail the whole page.
Non-negotiable safety boundary
This first slice is read-only.
- Do not add POST, PATCH, PUT, or DELETE infrastructure endpoints.
- Do not add restart, reboot, deploy, rollback, drain, pause-cron, restore, shell, terminal, log-tail, or power controls.
- Do not run Docker, Compose, SSH, systemctl, rclone, OVHcloud, GitHub Actions, or host commands from the Next.js application.
- Do not mount, proxy, or access `/var/run/docker.sock`. A read-only filesystem mount does not make the Docker API least-privilege.
- Do not add OVHcloud API credentials, SSH keys, Tailscale credentials, Cockpit credentials, Prometheus credentials, or other infrastructure secrets to client code or API responses.
- Do not expose the private Cockpit tailnet URL. Keep Cockpit access governed by public/docs/operations/VM_ADMIN_ACCESS.md.
- Do not probe stopped legacy Fly.io apps. The VM rclone worker is authoritative; legacy Fly resources remain recovery-only.
- Do not infer container health, backup freshness, cron replica count, VM region, or provider state when there is no authoritative data source. Return an explicit `unknown` or omit the unsupported field.
- Do not change transfer behavior. If any manual rclone verification becomes necessary, state source, destination, direction, operation, and flags first and use `--dry-run`; however, no manual rclone command should be needed for this feature.
Architecture
Implement interface-first and follow local patterns. Keep UI render-only and API routes thin.
Define provider-neutral canonical models for a fleet-shaped response. Suggested concepts, with names adjusted only when existing conventions justify it:
- `InfrastructureOverview`
- `nodes: InfrastructureNodeSummary[]`
- `sampledAt: string`
- `overallStatus: "healthy" | "degraded" | "unavailable" | "unknown"`
- `warnings: InfrastructureWarning[]`
- `InfrastructureNodeSummary`
- stable canonical `id`; do not expose a provider-native service name or public IP
- display name, environment, provider label, and region only when configured
- status and last successful sample time
- release SHA when available from `GIT_COMMIT_SHA` or the established release source
- resource metrics
- observable service health
- rclone capacity/queue metrics
- Metric values must distinguish `value: 0` from unavailable data. Use nullable values or an explicit availability/status shape.
- Every external section should carry enough source status or freshness information for the UI to show partial failure accurately.
Introduce stable ports rather than importing Prometheus transport logic into routes or components. At minimum separate:
- infrastructure metrics/query behavior;
- application/rclone health collection;
- overview aggregation.
Implement a Prometheus HTTP API adapter that is server-only, uses bounded timeouts, validates response shapes, sanitizes errors, and supports dependency injection in tests. Do not add a large Prometheus client dependency if a small typed `fetch` adapter is sufficient.
Configuration
- Add a server-only optional `PROMETHEUS_BASE_URL` configuration. In the production Compose network it may default to or be documented as `http://prometheus:9090`; do not expose it through a `NEXT_PUBLIC_*` variable.
- Add only non-secret optional display configuration if needed, for example node display name, provider label, and region. Use safe defaults such as `Production VM`, `production`, and `unknown`; do not fabricate an OVH product, datacenter, region, hostname, public IP, or service ID.
- Reuse `GIT_COMMIT_SHA` for the release revision. Display a short SHA in the UI while retaining the validated full value in the server response if useful. Treat missing or malformed values as unavailable.
- Update `deploy/.env.production.example` and canonical deployment documentation for new environment variables. Never include real hostnames, service IDs, private URLs, or secrets.
- Local development must work without Prometheus. It should render an explicit unavailable/partial state and remain testable with an injected adapter.
Data sources and required metrics
Use the existing Prometheus HTTP query API and existing health/service boundaries. Do not query Prometheus directly from the browser.
Collect the following when their source is available:
1. Node state from the existing `node-exporter` scrape:
- target `up` status;
- CPU utilization over a bounded recent window;
- memory used and total bytes, plus utilization percentage;
- root-filesystem used and total bytes, plus utilization percentage, excluding pseudo/temporary filesystems;
- receive and transmit bytes per second;
- uptime derived from node boot time and the sample timestamp.
2. Rclone worker state from its existing health client and Prometheus metrics:
- health status;
- `rclone_queue_depth`;
- `rclone_running_operations`;
- `rclone_max_concurrent_operations`;
- `rclone_max_queue_size`;
- rclone Prometheus target `up` status.
3. Application/database readiness:
- reuse or extract the same bounded database readiness behavior as `/api/health` without making a public HTTP round trip back into the same Next.js process;
- do not leak database connection details or raw errors.
4. Prometheus self-scrape status.
5. Release metadata from server runtime configuration.
Keep PromQL in a cohesive adapter/module, not scattered through the service, route, hook, and component layers. Validate finite numeric values and clamp percentages only where mathematically appropriate. Treat absent, stale, malformed, NaN, or infinite samples as unavailable rather than zero.
Do not claim to report the health of Caddy, PostgreSQL as a container, Weaviate, Redis, GlitchTip, Grafana, cron, backup, or every Compose service in this slice. Only report the application/database readiness and Prometheus targets that have an authoritative implemented source. Clearly label the section `Observable services` or equivalent.
API
Add:
`GET /api/admin/infrastructure/overview`
Requirements:
- Require a signed-in admin using the existing server-side role mechanism.
- Return 401 when unauthenticated and 403 when authenticated without admin access.
- Use the repository's established response helpers and user-safe errors.
- Set dynamic/no-store behavior appropriate for an operational snapshot.
- Keep the route thin: authenticate, authorize, call the overview service, map the response.
- Aggregate independent dependencies with fail-soft semantics. A Prometheus outage should normally return a successful partial snapshot with warnings, not erase healthy application/rclone information or become an unhandled 500.
- Bound each external call with a timeout and avoid unbounded retries.
- Do not return internal URLs, credentials, raw PromQL, stack traces, raw upstream bodies, hostnames, IP addresses, or provider-native VM identifiers.
- Add route tests for unauthenticated, unauthorized, full success, partial Prometheus failure, health-source failure, and safe unexpected-error handling.
- Document the response shape and fail-soft semantics in `public/docs/API_REFERENCE.md`.
UI
Add an admin-only `/admin/infrastructure` page and make the existing `/admin/monitoring` route redirect to it for compatibility.
- Follow the existing admin auth and layout patterns, improving separation where practical rather than copying large client pages.
- Add an `Infrastructure` entry to the Admin section of `SettingsMenu`; regular users and dev-only users must not see it.
- Use existing shadcn/UI primitives, Tailwind conventions, Lucide icons, breadcrumbs, accessible names, and dark-mode styles.
- Keep the view dense and operational rather than marketing-style or excessively card-heavy.
- Do not add a charting dependency in this slice. Use compact metric rows, badges, progress bars, and tables. Historical trend charts are deferred.
- Present one node initially, but render `nodes[]` so the layout can support multiple rows later.
- Include:
- page title, concise description, manual Refresh button, and sampled-at timestamp;
- overall status banner;
- fleet/node summary table with node, environment, status, uptime, release, and alert/warning count;
- selected/single-node resource section for CPU, memory, root disk, and network;
- observable-service health list;
- rclone queue/capacity section;
- external links to the already public Grafana and GlitchTip origins only when configured through safe server-provided link identifiers or an allowlisted server mapping. Open external links safely. Do not include Cockpit.
- Use semantic status text and icons in addition to color.
- Format bytes, rates, percentages, duration, and timestamps through tested presentation helpers. Display `Unavailable`, not `0`, for missing values.
- Ensure small screens can scan the content without clipped tables or overflowing metric values.
Client orchestration
- Put fetching and refresh behavior in a focused hook, not in presentational components.
- Fetch immediately, support manual refresh, and poll no more frequently than every 30 seconds.
- Pause automatic polling while `document.visibilityState === "hidden"`, resume with an immediate refresh when visible, and clean up timers and requests on unmount.
- Prevent overlapping requests and stale responses. Use `AbortController` or an equivalent established pattern.
- Preserve the latest successful snapshot during a transient refresh failure and show a stale/degraded warning rather than blanking the dashboard.
- Model initial loading, partial success, total error, refresh-in-progress, stale data, and success explicitly.
Expected tests
Add focused deterministic tests at the lowest useful layers:
1. Prometheus adapter/service tests:
- maps valid vector/scalar query responses;
- computes or maps CPU, memory, disk, network, uptime, scrape health, and rclone metrics correctly;
- preserves real zero values;
- treats missing, stale, malformed, NaN, and infinite samples as unavailable;
- times out safely and sanitizes upstream failures;
- produces a partial overview when one query/source fails.
2. API route tests:
- 401 unauthenticated;
- 403 non-admin;
- successful full snapshot;
- successful partial snapshot with warnings;
- sanitized unexpected failure.
3. Hook tests with fake timers:
- initial fetch;
- 30-second-or-slower polling;
- hidden-tab pause and visible-tab refresh;
- no overlapping requests;
- abort/cleanup on unmount;
- last-known-good data retained after refresh failure.
4. Component/page tests:
- loading, full success, partial, unavailable, and stale states;
- zero is rendered as zero while missing is rendered as `Unavailable`;
- statuses are not color-only;
- no privileged action controls or private infrastructure values appear.
5. Navigation tests:
- admin sees `/admin/infrastructure`;
- regular user and dev-only user do not;
- `/admin/monitoring` redirects to the new page for an admin and remains protected by the existing middleware/page rules.
Acceptance criteria
- [ ] An admin can open `/admin/infrastructure` from the Admin menu.
- [ ] Non-admin callers cannot access the page or API.
- [ ] The page renders a fleet-shaped overview containing the one configured production node.
- [ ] Current node resources, observable-service status, rclone queue/capacity, release SHA, freshness, and warnings render from canonical types.
- [ ] Missing Prometheus or health data degrades only the affected section and is never displayed as a healthy zero.
- [ ] Local development without Prometheus renders a useful unavailable/partial state.
- [ ] No infrastructure mutation endpoint or privileged host/Docker access is introduced.
- [ ] No secret, internal URL, hostname, IP, raw upstream error, provider-native VM ID, or private Cockpit URL reaches the browser.
- [ ] UI components remain render-only; collection and aggregation live behind services/ports/adapters, and the API route remains thin.
- [ ] Files remain under 500 LOC.
- [ ] Focused tests, typecheck, lint, and appropriate build verification pass.
- [ ] Canonical architecture, API, deployment, testing, and operations documentation is updated where behavior changed.
Out of scope / follow-up candidates
- VM discovery through the OVHcloud API.
- Multiple configured VM targets.
- Docker/Compose container inventory and restart controls.
- VM power controls, deployment, rollback, terminal access, or log streaming.
- Cron replica controls or invariant reporting without a safe authoritative metric.
- Backup freshness without an authoritative exporter/metric.
- Historical graphs, alert management, notifications, and incident acknowledgement.
- Public Cockpit access or embedding Grafana, GlitchTip, or Cockpit.
Suggested implementation order
1. Define canonical response types and port contracts.
2. Add typed Prometheus transport/query adapter tests and implementation.
3. Add health collectors and overview aggregation tests/implementation.
4. Add the authenticated admin API route and route tests.
5. Add formatting helpers and their tests.
6. Add the polling hook and fake-timer tests.
7. Add page/view components and state tests.
8. Add navigation and compatibility redirect.
9. Update canonical docs and environment template.
10. Run targeted verification, then broaden based on affected surface.
Verification
Use the actual test paths created by the implementation. At minimum run:
pnpm env:guard
pnpm exec vitest run <infrastructure adapter/service tests>
pnpm exec vitest run <admin infrastructure route tests>
pnpm exec vitest run <infrastructure hook/component/navigation tests>
pnpm typecheck
pnpm lint
pnpm build
If the repository does not expose `pnpm lint`, use its documented targeted ESLint command. Do not silently skip a failed verification command; diagnose it or report the exact blocker.
Documentation
Update only the canonical documents affected by the implementation:
- public/docs/ARCHITECTURE.md for the infrastructure overview boundary, ports/adapters, and read-only security boundary;
- public/docs/API_REFERENCE.md for `GET /api/admin/infrastructure/overview`;
- public/docs/DEPLOYMENT.md and deploy/.env.production.example for new non-secret server configuration;
- public/docs/TESTING.md if the new adapter/hook testing pattern is material;
- public/docs/operations/VM_PRODUCTION_RUNBOOK.md to identify the dashboard as a read-only convenience, while keeping CLI/runbook checks authoritative for unsupported services and maintenance;
- public/docs/operations/VM_ADMIN_ACCESS.md only if the dashboard changes operator entry-point wording; do not weaken the Tailscale/Cockpit boundary;
- README.md only if the top-level product/runtime summary materially changes;
- relevant docs/ai-skills files if the reusable architecture or deployment workflow changes.
Final report
- Do not commit or open a PR unless explicitly requested.
- Lead with the implemented outcome.
- List changed files grouped by domain/API/UI/tests/docs.
- State the exact metrics and health sources implemented.
- State how partial failures, stale data, timeouts, and missing values behave.
- Confirm that no mutation endpoints, Docker socket, SSH, OVH credentials, or private Cockpit details were introduced.
- Report exact verification commands and results.
- Call out remaining risks and follow-ups, especially the absence of authoritative container, cron-replica, and backup-freshness metrics.