A lightweight Express service that runs rclone operations for StratoFusion. It
currently runs in the authoritative OVHcloud VM Compose stack. The stopped
Fly.io production app preserves an approval-gated recovery input; the stopped
development app is pending separately approved external retirement. The
Next.js app prepares full, service-specific paths
and an rclone config per request; the worker executes transfers, tracks
progress, and exposes operation status.
Stable names such as FLYIO_RCLONE_SERVICE_URL, FlyioRcloneClient, and
fly-rclone/ are compatibility names and do not imply that production traffic
currently reaches Fly.io.
Recent UI-facing progress updates include surfacing file counts (x / y files) and folder scan counts from rclone's Listed N stats output when byte counters lag.
How it actually works
Receives pre-resolved paths from the Next.js app. No ID-to-path resolution happens here.
Writes the provided rclone config to a secure temporary file for each operation and deletes it after completion.
Builds rclone commands.
Single file copy uses copyto and requires a destination including the filename. If fileName is omitted, it is derived from sourcePath.
Single file move uses move with destination as a directory path (no filename appended).
Folder copy/move use copy/move with directory paths; recursion is handled by rclone. Folder moves with preserveStructure target the destination parent plus the moved folder name so moving /parent/source to /target becomes /target/source instead of moving the source contents into /target.
Queues operations with priorities (folder ops and moves are prioritized). Concurrency is limited (default 3) and configurable at runtime.
Parses rclone output to compute rich progress (percentage, transferred/total, speed, ETA). Server‑side copy and skipped files are detected and set to 100%.
Operations are tracked in memory with IDs; terminal snapshots are also kept in a lightweight archive map so recently-finished operations remain queryable after active cleanup.
Operation output is bounded in memory and status responses return only recent, truncated output lines, so long-running folder copies and backups cannot inflate /api/operations/:id payloads into multi-megabyte poll responses.
Direct sync/bisync stdout and stderr buffers also retain only recent diagnostics, bounded by the operation output line/length budget, so multi-day runs do not retain their full process log history in memory.
Successful operations are finalized from child-process exit as well as close, which prevents Windows hosts from leaving 100%-complete jobs stuck in running while the UI keeps polling.
Backup/sync job lists trigger throttled foreground reconciliation in addition to the scheduled cron, so terminal rclone children can close their parent jobs during local development where cron is not running.
Live stats that include Errors: N while the process is still active are treated as retry/finalization states rather than terminal completion, so the UI does not sit on a misleading completed-looking 100%.
Client transfer monitoring now falls back when SSE streams go stale or emit server-side error events, and active fallback polling is protected from generic tab-visibility cleanup until the next successful snapshot.
Copy and move transfers are foreground operations. The transfer dialog registers active rclone operation IDs with the session timeout guard so auto-logout does not interrupt an in-progress copy or move. Uploads and browser downloads use the same foreground-operation guard.
Folder copies to OneDrive apply reliability flags (retry/throttle plus --disable Copy) to reduce transient destination processing failures.
When a OneDrive-targeted folder copy ends with exactly one failed file, the service attempts a single-file fallback retry before returning a partial failure.
Running operations that stop making meaningful transfer progress are cancelled automatically as timeouts to avoid indefinite hangs.
Watchdog timeout cancellations now carry a timeout error message so parent job reconciliation reports a failed timeout instead of a user cancellation.
Folder copy requests can now carry per-operation watchdog metadata (operationTimeoutMs, stallTimeoutMs, operationProfile) so large backup launches can run for hours without being capped by the queue default.
Sync and bisync requests also accept per-operation watchdog metadata (operationTimeoutMs, stallTimeoutMs, operationProfile, syncMode, first-time/resync hints), allowing sync jobs to use workload-specific runtime protection. Large one-way sync defaults to operationTimeoutMs: 0 (no absolute cap) and stallTimeoutMs: 1800000. Listing, checking, and copying can continue beyond 12 or 24 hours while meaningful progress advances. The stall window starts at process launch, including when no initial progress arrives. Positive explicit runtime limits remain hard caps; two-way sync, bisync initialization, and backup runtime budgets are unchanged.
The production VM rclone container serializes heavy child operations and has a 4 GB memory limit, reducing resource contention during metadata-heavy first-time bisync and large backup runs. The retained Fly production config mirrors those limits for recovery.
Parent sync reconciliation treats recoverable rclone credential expiry as a background-token refresh problem, not a permanent user-action failure. When refresh succeeds, the app clears operation IDs, resets sync items to pending, and schedules the next two-way attempt with --resync so bisync state is rebuilt safely.
Parent sync launch/reconciliation treats transfer-quota failures as pause-and-retry states. The job is scheduled for the reported reset window instead of remaining stuck as running or failing permanently, and two-way jobs are marked for --resync before retry.
Google Drive <-> OneDrive sync and bisync commands add --metadata-exclude content-type=inode/directory and exclude StratoFusion *.stratofusion-duplicate-* quarantine names, including recursive quarantined folders, alongside --create-empty-src-dirs; this skips Google zero-byte directory marker files without losing real empty folder creation or propagating earlier repair artifacts.
Google Drive to non-Google bisync also adds --drive-skip-gdocs so native Google Workspace documents are not treated as 0-byte reverse-copy candidates during first-time resync.
Google Drive <-> OneDrive bisync also uses conservative resource flags (--buffer-size 4M, --transfers 2, --checkers 4, --tpslimit 8) to reduce worker memory pressure and provider throttling during large first-time comparisons.
Two-way sync conflict policies are forwarded to rclone as --conflict-resolve and, during first-time or recovery resync, --resync-mode.
Manual force-resync for a scheduled two-way sync is one-shot. After a successful reconciliation/completion, the persisted job payload is reset from resyncBehavior: "force-resync" to resyncBehavior: "auto" so future scheduled runs do not rebuild the bisync baseline again.
First-time/recovery bisync resync launches disable --fast-list so provider traversal can produce incremental progress instead of waiting on a full recursive listing before useful output.
First-time/recovery bisync resync launches request rclone debug log output and treat Fetching metadata lines as scan activity. The jobs page surfaces this as Scanning metadata without exposing individual file paths, maps rclone debug result records to Comparing metadata, and the stall watchdog treats those phases as meaningful activity.
The worker parser sanitizes metadata-only debug chunks before storing them as rich progress, and avoids logging full zero-byte debug chunks during parser checks, so long first-time bisync scans do not overload logs or health checks with per-file comparison JSON.
Metadata-heavy progress now counts as liveness: Checks, standalone Listed: N, inline Listed N, transfer speed, current-file changes, and fresh rclone output all refresh the stall watchdog even when transferred bytes are temporarily flat during long listing/checking phases.
Sync/bisync and move commands use multiline stats rather than --stats-one-line, which omits rclone's Listed counter. Advancing listing/checking counters refresh meaningful progress; repeated identical zero-byte stats still time out. Copy/move and sync snapshots retain observed counters and byte totals across separate output chunks, including metadata-only chunks.
Copy/move dialogs and running job cards share progress presentation: entries listed, files checked, and an indeterminate bar during scanning/comparison when byte totals are unknown. Listing counts are not a percentage or a unique source-file total. Known byte totals enable transfer percentage; active finalization stays indeterminate, and terminal/paused states never animate. The percentage covers discovered transfer bytes, not the entire source folder while enumeration continues. Transfer/deletion semantics are unchanged.
Additional bisync metadata (Deleted, Renamed, Copied, Path1/Path2 resync phase changes, and directory modtime updates) is treated as meaningful progress, while repeated zero-byte stats stay as liveness only so a job reporting 0 B / 0 B forever can still be cancelled by the stall watchdog. This applies to both queued rclone operations and the direct sync/bisync route.
Parent job reconciliation waits for shared operation state to mark stale heartbeats terminal before rescheduling or failing the job, avoiding false failure from a stale heartbeat snapshot. Reconciled activity logs fall back to stored job summaries when best-effort provider path resolution returns raw file IDs.
Cancellation semantics
Frontend cancel now aborts in-flight batch/folder HTTP requests using AbortSignal.
Batch submission checks abort state between operations so cancel can stop additional starts mid-batch.
The Next.js batch copy route forwards request abort state into shared copy execution and exits early when aborted.
On Windows development hosts, process cancellation uses taskkill /F /T /PID <pid> so rclone child processes are terminated with the parent.
Rclone binary resolution: the service will use RCLONE_PATH/FLY_RCLONE_PATH when the override is a bare command name or points to an existing executable. If the override points to a missing file, the service logs a warning and falls back to a local binary next to the server (e.g. rclone/rclone.exe) or the system PATH.
On Windows development hosts, the recommended workflow is now a pinned repo-local binary at fly-rclone/rclone.exe. Use:
pnpm rclone:bootstrap
pnpm rclone:check
The bootstrap script reads ARG RCLONE_VERSION from fly-rclone/Dockerfile so local Windows stays aligned with the Fly image version instead of drifting with a globally installed rclone.exe.
API
Health.
GET /health → { status, timestamp, checks }.
Copy.
POST /api/copy → start single file copy (uses copyto).
POST /api/copy-folder → start folder copy (uses copy).
POST /api/batch-copy → start multiple file copies.
Move.
POST /api/move → start single file move (destination is directory).
POST /api/move-folder → start folder move.
POST /api/batch-move → start multiple file moves.
Download streaming.
POST /api/google-download/session → create a short-lived encrypted stream token for Google file streaming.
GET /api/google-download/stream?token=... → stream Google file bytes via Fly (native media or Workspace export).
POST /api/dropbox-download/session → create a short-lived encrypted stream token for Dropbox file streaming.
GET /api/dropbox-download/stream?token=... → stream Dropbox file bytes via Fly (temporary-link restricted fallback).
Download streaming note: Google and Dropbox session tokens are stateless and encrypted so stream URLs work across Fly load-balanced instances for the lifetime of the token.
Operations.
GET /api/operations/:id → operation status (includes progress and command echo without secrets).
DELETE /api/operations/:id → cancel.
GET /api/operations → list (safe fields only).
Queue.
GET /api/queue/status → size, running ops, stats, next up.
POST /api/queue/config → { maxConcurrent?, maxQueueSize? }.
Caching (internal metrics).
GET /api/cache/status | POST /api/cache/clear.
Shutdown (admin).
GET /api/shutdown/status | POST /api/shutdown.
Utilities.
POST /test/progress → parse a raw rclone output line (for debugging).
Metrics.
Internal-only GET /metrics on METRICS_PORT (default 9091) → Prometheus-style queue and capacity metrics for Fly scraping.
Request bodies
Common fields (per operation):
config string (required): full rclone config content for the remotes involved.
Copy (single file): also accepts fileName (optional) and fileSizeBytes (optional, helps progress).
Copy (folder): accepts folderName (optional), estimatedFiles, estimatedSize, estimatedFolders, operationTimeoutMs, stallTimeoutMs, operationProfile, operationProfileReason (used for progress + watchdog profile selection).
Sync/Bisync: accepts operationTimeoutMs (zero disables the absolute watchdog; positive values impose a hard cap), stallTimeoutMs, operationProfile, operationProfileReason, syncMode, estimatedFiles, estimatedSize, estimatedFolders, isFirstTimeBisyncInitialization, isAutomaticRecoveryRun, and debugProgress.
Move (single file): destination path is treated as a directory; filename comes from sourcePath.
Batch operations sent directly to the Fly rclone gateway use { operations: [ { ...same fields as above... } ] } and remain capped at 50 operations per request as a gateway safety limit.
The Next.js streamed app endpoints (/api/rclone/batch-copy/stream and /api/rclone/batch-move/stream) can launch larger logical user batches. They run aggregate auth, subscription, transfer-quota, filtering, and conflict-policy preflight before launching child rclone operations, then emit per-operation IDs as they are accepted. If a stream launch fails before any operation starts, logical batches over 50 are not sent to the legacy JSON fallback because that fallback still targets the 50-operation request path.
Status endpoint returns operation info including progress, bounded recent output, and a richProgress snapshot when available (speed, ETA, transferred/total).
Strong input validation for remote names, paths, UUIDs, and config size/content.
Path validation performs bounded repeated percent-decoding, NFKC
normalization, and slash canonicalization before rejecting traversal
segments, Windows drive paths, UNC paths, nulls, and dangerous system paths.
Validation happens before any rclone process is launched.
Secure temp files (owner‑read/write) and automatic cleanup on exit and periodically.
Protected Fly routes require service-to-service authentication via FLYIO_RCLONE_API_KEY.
Public routes remain limited to health, docs, and streaming endpoints that use signed or encrypted tokens.
Production CORS allows the apex origins by default. During an approved
temporary rehearsal, a distinct exact-SHA worker is routed beneath
https://vm-staging.stratofusion.io; its CORS policy admits that exact origin
only when OAUTH_REHEARSAL_ORIGIN, both app URLs, and DEPLOY_ENV=prod
match atomically. The authoritative production worker remains unchanged and
does not need the Stage origin. Arbitrary values, wildcards, paths, ports,
non-HTTPS origins, and other preview hosts remain blocked.
After VM cutover, the browser-facing worker is
https://rclone.stratofusion.io; vm-rclone.stratofusion.io remains an
alias during rollback. Caddy keeps streaming unbuffered with
flush_interval -1. Before the final dump or any SHA-tagged deployment,
verify both legacy Fly and VM operation counts are zero and that restored
public.active_operations exists so the drain gate cannot silently pass on
an empty schema.
Local VM CORS allows only https://local-vm.stratofusion.io when
DEPLOY_ENV=local-vm and both app URLs plus OAUTH_REHEARSAL_ORIGIN exactly
match that origin.
The rehearsal origin widens only the browser-origin allowlist; it does not
bypass FLYIO_RCLONE_API_KEY, signed stream tokens, user authorization, or
operation validation.
Concurrency & Queueing
Defaults: maxConcurrent=3, maxQueueSize=100.
Runtime config via POST /api/queue/config.
Priorities: folder ops and moves enqueue with higher priority; singles default to normal.
Local development
Port: 3001.
Local URL: http://127.0.0.1:3001.
Commands (from this directory).
pnpm install.
pnpm rclone:bootstrap from the repo root on Windows to install the pinned fly-rclone/rclone.exe.
pnpm rclone:check from the repo root or pnpm rclone:check in fly-rclone/ to verify the pinned binary is active.
pnpm dev (nodemon) or pnpm start.
Docker: docker-compose -f docker-compose.dev.yml up --build.
Deployment (VM production and legacy Fly rollback)
Config.
Retired development evidence: fly.toml (not deployable through the script).
Production recovery: fly.prod.toml.
The OVHcloud VM compose service is authoritative production.
Fly configs are retained only for an explicitly approved rollback or
recovery; automatic push deployment is disabled.
Both legacy apps are intentionally stopped. Do not probe their public
endpoints. Only the production app may be started for an approved recovery;
retire the development app through the shared-development runbook.
fly-rclone/deploy.sh refuses to deploy while /api/operations reports active rclone work unless --wait-for-idle seconds drains successfully or --force-active-deploy is used for an intentional recovery deploy.
Active operation polling and remote cancellation requests now dual-write to PostgreSQL for cross-machine lookup.
ZIP download endpoints have been removed; browser-direct downloads no longer rely on Fly-side ZIP assembly or ZIP session state.
Fly Proxy cannot auto-start retained Machines; recovery is an explicit
operator action.
Queue depth and running-operation metrics are exposed on an internal metrics port (METRICS_PORT, default 9091) so a separate fly-autoscaler app can be added later if queue backlog becomes the real bottleneck.
Required for protected Fly routes: FLYIO_RCLONE_API_KEY.
Required for shared Fly state: DATABASE_URL.
Temporary Stage only:
OAUTH_REHEARSAL_ORIGIN=https://vm-staging.stratofusion.io. Set the same
value on the isolated Stage app and worker containers. Do not set it on the
authoritative production worker or an unrelated Fly/preview deployment.
Development mode ignores this production-only CORS opt-in.
Local VM only: OAUTH_REHEARSAL_ORIGIN=https://local-vm.stratofusion.io
with DEPLOY_ENV=local-vm.
Optional: RCLONE_PATH or FLY_RCLONE_PATH to point to a specific rclone binary; path-like overrides must reference an existing executable.
Optional: MAX_CONCURRENT_OPERATIONS, MAX_QUEUE_SIZE for the queue.
Optional: OPERATION_TIMEOUT_MS for absolute per-operation timeout (default 30 minutes).
Optional: OPERATION_PROGRESS_STALL_TIMEOUT_MS for progress-stall timeout (default 2 minutes).
Retention is adaptive: folder, batch, and large operations are retained longer than simple operations, capped by OPERATION_RETENTION_MAX_MINUTES.
Retry is selective: only retryable non-partial failures (for example transient network/provider errors) are retried with exponential backoff.
Troubleshooting
rclone not found → include a binary on PATH or set RCLONE_PATH/FLY_RCLONE_PATH to a valid executable path.
On Windows local dev, prefer pnpm rclone:bootstrap over a global install. The deterministic path is fly-rclone/rclone.exe.
If pnpm rclone:check reports that RCLONE_PATH or FLY_RCLONE_PATH is bypassing the repo-local binary, clear that override unless you intentionally want a custom executable.
If Fly logs show spawn ... ENOENT, check for a stale RCLONE_PATH or FLY_RCLONE_PATH secret that no longer matches the image layout.
Progress shows 0% → ensure options.progress=true or provide fileSizeBytes/estimatedSize when known.
Destination name issues on copy → remember single-file copy uses copyto and needs the filename in the destination path.
View logs.
flyctl logs --app stratofusion-rclone-prod.
Health: GET /health should return { status: "healthy", checks: { serviceAuthConfigured: true } }.
Metrics: GET http://localhost:9091/metrics should include rclone_queue_depth, rclone_running_operations, rclone_max_concurrent_operations, and rclone_max_queue_size.
If FLYIO_RCLONE_API_KEY is missing, /health now returns 503 with status: "unhealthy" so deploy checks fail fast instead of surfacing the problem during uploads.
If a Phase-B browser request is blocked by CORS, verify the origin is exactly
https://vm-staging.stratofusion.io, the rclone container received the exact
OAUTH_REHEARSAL_ORIGIN value, and no path, slash, port, or alternate host
was included. Do not broaden the allowlist to a wildcard.
Sync implementation boundaries
The sync route delegates command preparation to syncCommand, lifecycle and watchdog outcomes to syncLifecycle, and stream/process events to syncProcessEvents. The refactor preserves existing flags, timeouts, cancellation, terminal persistence, and bisync state upload behavior.