Runtime status: the backup feature is current, but production hosting has
moved to the OVHcloud VM. One Compose cron replica invokes the same Next.js
routes, the rclone worker runs in Compose, and Compose PostgreSQL is
authoritative. Sections that prescribe Vercel Cron, active Fly Machines, or
Neon production are historical implementation records unless explicitly
labeled as legacy recovery. See Architecture Evolution,
Deployment, and the
Production VM Runbook.
References to the former shared-development hostname, Vercel project, or Fly
dev worker are also historical and must not be executed; use local tooling or
the shared-development retirement runbook instead.
The Backup Feature enables users to schedule automated backups of files and folders from one cloud storage service to another. Backups can be executed immediately or scheduled with daily/weekly/monthly frequencies, plus minutely intervals (every 5/10/15/20/30 minutes) for testing.
Key Features
ā Immediate Backups - Execute backups right away with "Run now" option.
Users can now fully manage their backup jobs through the Jobs page, including editing scheduled jobs and permanently deleting jobs from the database.
Edit Scheduled Backups
Capabilities:
Users can edit scheduled backup jobs to modify:
Schedule frequency (daily, weekly, monthly, or minutely intervals).
Scheduled time (HH:mm in 24-hour format).
Timezone (IANA timezone identifier).
Restrictions:
Only scheduled jobs can be edited (not running, completed, failed, or cancelled).
Source files and destination cannot be changed (create a new backup job instead).
Users can only edit their own jobs (enforced via userId check).
UI Location:
Jobs page (/user/jobs).
Edit button appears next to scheduled jobs.
Opens EditBackupDialog component.
User Experience:
Form pre-populated with current job settings.
Unsaved changes warning when closing without saving.
Auto-refresh pauses while dialog is open (prevents interference).
Success message after saving changes.
Jobs list refreshes automatically after edit.
Delete Backup Jobs
Capabilities:
Users can permanently delete backup jobs from the database.
Restrictions:
Cannot delete running jobs (must cancel first).
Can delete: completed, failed, cancelled, or scheduled jobs.
Requires confirmation dialog before deletion.
Users can only delete their own jobs (enforced via userId check).
UI Location:
Jobs page (/user/jobs).
Delete button appears next to completed, failed, cancelled, or scheduled jobs.
Opens DeleteBackupConfirmDialog component.
User Experience:
Confirmation dialog shows job details before deletion.
Auto-refresh pauses while dialog is open.
Success message after deletion.
Jobs list refreshes automatically after delete.
Implementation Details
Database Layer
File:src/lib/database/backup-jobs.ts
New Functions:
// Fetch a single job with its items for editingexportasyncfunctiongetBackupJobById( userId:string, jobId:string):Promise<{ job:BackupJob; items:BackupJobItem[]}|null>// Update an existing scheduled jobexportasyncfunctionupdateBackupJob( userId:string, jobId:string, input:UpdateBackupJobInput):Promise<{ job:BackupJob; items:BackupJobItem[]}|null>// Permanently delete a job and its itemsexportasyncfunctiondeleteBackupJob( userId:string, jobId:string):Promise<boolean>
Key Implementation Details:
updateBackupJob() regenerates summaries and recalculates nextRunAt based on new schedule.
updateBackupJob() deletes old job items and creates new ones (maintains referential integrity).
deleteBackupJob() prevents deletion of running jobs.
Both functions enforce userId ownership checks.
API Routes
File:src/app/api/jobs/[id]/route.ts
Endpoints:
GET /api/jobs/[id] - Get job details for editing
PATCH /api/jobs/[id] - Update job
DELETE /api/jobs/[id] - Delete job
Authentication:
All endpoints require user authentication via getCurrentUserId().
All operations are scoped to the authenticated user's jobs only.
Error Handling:
Returns appropriate error messages for invalid operations.
Validates job ownership before any modifications.
Returns 404 for non-existent jobs.
Returns 400 for invalid requests (e.g., trying to delete running job).
# Run the investigation scriptnode scripts/investigate-failed-backup.mjs
# This will show:# - Error message from the failure# - Job configuration and timing# - Source and destination details# - Recommendations for fixes
ā Timeout errors ā Job took longer than expected.
ā "Failed to create folder" ā OAuth token expired or service API issue.
ā "Batch copy failed" ā Rclone service issue.
3. Verify Environment Variables (1 minute)
# Check if all required variables are setvercel envls# Required variables:# ā CRON_SECRET# ā NEXT_PUBLIC_SITE_URL (should be https://dev.stratofusion.io)# ā DATABASE_URL# ā FLYIO_RCLONE_SERVICE_URL
4. Test Rclone Service (30 seconds)
# Check if rclone service is healthycurl https://stratofusion-rclone-dev.fly.dev/health
# Expected: {"status":"ok","timestamp":"..."}# If error: Service is down, needs restart
Common Issues & Quick Fixes
Issue 1: Missing CRON_SECRET ā ļø CRITICAL
How to identify:
Vercel logs show: "Unauthorized cron request".
HTTP 401 error in cron endpoint.
Quick fix:
# Generate a secure secretopenssl rand -base64 32# Add to Vercelvercel envadd CRON_SECRET production
# Paste the generated secret# Redeployvercel --prod
Issue 2: OAuth Token Expired š
How to identify:
Error message contains: "Authentication failed" or "Invalid credentials".
Problem: "Backup Items" button remained enabled after scheduling, allowing users to accidentally register the same schedule multiple times. Dialog stayed open after successful registration.
Solution Implemented:
Added automatic dialog close after successful schedule creation.
Button already properly disabled during API call via isLoading state.
Error cases still keep dialog open for user to retry.
Files Modified:
src/components/BatchFileTransferDialog.tsx - Added onClose() call after successful schedule creation.
Benefits:
Prevents duplicate schedule registrations.
Better UX - dialog closes after successful action.
Problem: Scheduled backup cron jobs were executing successfully but creating empty directories instead of copying files.
Root Cause:process.env.NEXT_PUBLIC_SITE_URL is undefined in server-side code. The NEXT_PUBLIC_ prefix means the variable is for client-side code only. This caused fetch URLs to be invalid, resulting in 404 HTML error pages instead of JSON responses.
Solution Implemented:
Created server-side URL utility (src/lib/server-url.ts):
getServerApiUrl(path) - Async function to construct full API URLs for server-side fetch calls.
getServerBaseUrl() - Determines the correct base URL for the current environment.
URL Construction Priority:
VERCEL_URL environment variable (automatically set by Vercel)
x-forwarded-host header (from incoming request)
host header (from incoming request)
Fallback: http://localhost:3000 (for local development)
Files Modified:
src/lib/server-url.ts - Created server-side URL utility.
src/app/api/jobs/[id]/execute/route.ts - Updated to use getServerApiUrl().
src/app/api/cron/execute-backups/route.ts - Updated to use getServerApiUrl().
Problem: Folder backups failed with "User not authenticated" error.
Root Cause: The FolderCopyOperationRequest interface was missing the userId field. Without userId, the buildConfigAndRemotes() function couldn't retrieve OAuth tokens from the database.
Solution Implemented:
Added userId field to FolderCopyOperationRequest interface.
Updated executeFolderCopyOperation() to pass userId to buildConfigAndRemotes().
Ensured consistent userId handling across file and folder operations.
Files Modified:
src/lib/rclone/services/folder-copy-service.ts - Added userId field and parameter passing.
Problem 1: Backup jobs completed without errors but created empty folders - no files or folder contents were copied.
Root Cause: The backup execution logic was treating ALL items (files and folders) as individual files and using the file copy service. The executeBatchCopyOperations() service uses rclone copyto which copies a single file, not folder contents.
Problem 2: Backup folder timestamps showed time 1 hour ahead of actual execution.
Root Cause: Using new Date().toISOString() which always returns UTC time instead of user's timezone.
Solution Implemented:
Separate File and Folder Operations
Check isFolder field in backup job items.
Use executeCopyOperation() for files.
Use executeFolderCopyOperation() for folders.
Properly handle both operation types in parallel.
Fix Timestamp Timezone
Use formatInTimeZone() from date-fns-tz library.
Format timestamp in user's configured timezone.
Store timezone in backup job metadata.
Files Modified:
src/lib/backup/execute-backup-job.ts - Separated file and folder operations, fixed timezone handling.
Status: ā Fixed - Commit: f39ad9d3
6. Folder Backup File ID Resolution Error (October 18, 2025)
Problem: Folder backups failed with misleading "User not authenticated" error. Real error was "directory not found" because rclone was given an ID instead of a path.
Root Cause:resolveFileId() was called without userId parameter. Without userId, couldn't authenticate to query cloud service API. Resolution failed silently, and rclone received raw ID instead of resolved path.
Solution Implemented:
Added userId parameter to resolveSourceAndDestPaths() function.
Ensured userId is passed through the entire resolution chain.
Updated all callers to provide userId parameter.
Files Modified:
src/lib/rclone/services/transfer-utils.ts - Added userId to MinimalTransferOp interface and resolveSourceAndDestPaths function.
src/lib/rclone/services/folder-copy-service.ts - Updated to pass userId.
src/lib/rclone/services/copy-service.ts - Updated to pass userId.
Cron Endpoint ā executeBackupJob() ā executeBatchCopyOperations()
(direct function call) (direct function call)
Implementation:
Created shared backup execution logic in src/lib/backup/execute-backup-job.ts.
Cron endpoint calls executeBackupJob() directly.
Execute endpoint also calls executeBackupJob() directly.
No HTTP requests between server-side endpoints.
Files Modified:
src/lib/backup/execute-backup-job.ts - Created shared execution logic.
src/app/api/cron/execute-backups/route.ts - Updated to call executeBackupJob directly.
src/app/api/jobs/[id]/execute/route.ts - Updated to call executeBackupJob directly.
Benefits:
No HTTP requests = No deployment protection issues.
Reusable logic for both cron and manual execution.
Easier to test (no need to mock HTTP calls).
Better performance (no HTTP overhead).
Status: ā Fixed - Commit: a0cc8120
8. React State Timing Issue - Files Copied to Wrong Location (October 27, 2025)
Problem: During "Run Once" backups, timestamped backup folders (e.g., backup-2025-10-27_10-22-31) were created correctly, but files were being copied adjacent to (next to) the timestamped folder instead of inside it.
Root Cause: React state update timing issue. The backup operation flow was:
Create timestamped folder with ID id:2V5kn5RRfnAAAAAAAADx3A
Call setDestinationFolderId(folderId) to update state
Immediately call handleTransferBatchInternal() to start file transfer
Problem: React state updates are asynchronous, so handleTransferBatchInternal() was still using the OLD destinationFolderId value ("root") instead of the new timestamped folder ID
Files were copied to Dropbox root instead of inside the timestamped folder
Evidence from Logs:
[DEBUG] [FILE-ID-RESOLVER] Resolved Dropbox ID id:2V5kn5RRfnAAAAAAAADx3A to path: /backup-2025-10-27_10-22-31
[INFO] [COPY-SERVICE] ā Resolved paths for copy operation:
[INFO] [COPY-SERVICE] Source: google_google_rhounslow_1761257642198_996:test-files/create_all_formats.py
[INFO] [COPY-SERVICE] Dest: dropbox_default:
[INFO] [COPY-SERVICE] FileName: create_all_formats.py
The destination path was empty (dropbox_default:) because the state hadn't updated yet.
Solution Implemented:
Architecture Change: Pass folder ID directly instead of relying on state updates.
Before:
const backupResult =awaitcreateRunNowBackup({...});// State update happens in callbackawaithandleTransferBatchInternal();// Uses stale state value
After:
const backupResult =awaitcreateRunNowBackup({...});// Validate folder ID existsif(!backupResult?.folderId){thrownewError("Failed to create timestamped backup folder");}// Pass folder ID directlyawaithandleTransferBatchInternal(backupResult.folderId);
For Questions or Issues: Check the FAQ or contact support.
Large backup mode and watchdog behavior (2026-04)
Backup root launches now evaluate a shared profile helper using estimatedSize, estimatedFiles, estimatedFolders, source-count, and root/unknown-estimate fallback.
When the large profile is active (large-backup), launches pass explicit operationTimeoutMs (24h) and longer stallTimeoutMs to fly-rclone.
fly-rclone stall detection now treats metadata activity (Checks, Listed, current file transitions, output churn, and nonzero speed) as meaningful liveness.
This prevents valid metadata-heavy transfers (hundreds of thousands of files/folders) from being cancelled around the previous 30-minute absolute timeout or short stall windows.
Timeout cancellations are still terminal failures (cancelReason: "timeout") so parent reconciliation distinguishes watchdog timeouts from user-requested cancellations.