Cloud storage services (Google Drive, OneDrive, Dropbox) use internal file/folder IDs that are not compatible with rclone's path-based operations. These IDs need to be resolved to actual file paths before being passed to rclone.
Previously, file ID resolution was handled inconsistently:
Google Drive: Resolution in rclone service.
OneDrive: Resolution in both Next.js API route AND rclone service (redundant).
Dropbox: No resolution (causing failures).
Decision
All file and folder ID resolution will be handled in the Next.js application layer, not in the rclone service.
Rationale
Separation of Concerns: Next.js handles business logic, rclone handles file operations
Better Architecture: Service registry and authentication properly managed
Easier Testing: Can test resolution logic without rclone dependencies
Better Error Handling: Can return proper HTTP responses with detailed errors
No Duplication: Single place for resolution logic per service
Consistency: All services follow the same pattern
Implementation Pattern
Next.js API Route Responsibilities:
Detect service-specific file/folder IDs.
Use service registry to get appropriate service instance.
Resolve IDs to file/folder paths using service APIs.
Pass resolved paths to rclone service.
Rclone Service Responsibilities:
Accept resolved file/folder paths.
Execute rclone operations with paths.
Handle rclone-specific logic and progress tracking.
Example Flow:
Frontend Request: sourceFileId = "id:2V5kn5RRfnAAAAAAAADsgw"
↓
Next.js API Route: Detects Dropbox file ID
↓
Service Resolution: dropboxService.getFileMetadata() → "/Documents/file.pdf"
↓
Rclone Service: rclone copy dropbox:/Documents/file.pdf dest:
Services Affected
✅ Dropbox: Implemented via the unified resolver using direct Dropbox API calls.
✅ Google Drive: Handled by the unified resolver (internal resolver at src/lib/rclone/resolvers/google-drive.ts).
✅ OneDrive: Handled by the unified resolver (internal resolver at src/lib/rclone/resolvers/onedrive.ts).
Files Modified
src/app/api/rclone/batch-copy/route.ts - File ID resolution logic.
src/app/api/rclone/copy/route.ts - File ID resolution logic.
src/app/api/rclone/copy-folder/route.ts - Folder ID resolution logic.
Dropbox transfer launches normalize to remote-relative rclone paths and strip legacy Business /<homePath>/... prefixes before copy/move/sync.
Standardized error handling with throwOperationError(prefix, operation, err, ErrorCtor).
Progress normalization with normalizeProgressFromOperation(operation) returning a consistent structure consumed by UI.
src/lib/rclone/services/folder-copy-service.ts.
Centralizes recursive folder copy setup and delegates to the rclone client. The Fly client name is retained for compatibility, while production targets the VM worker.
Reuses precomputed manifest/size preflight results from /api/rclone/copy-folder when available so large folder launches avoid duplicate enumeration.
OneDrive folder sources prefer Graph driveItem.size for byte estimation and only fall back to recursive enumeration when provider metadata does not include a folder size.
Transfer preflight manifest resolution is metadata-only and skips per-file download URL generation, so Dropbox folder copy/move/sync launches do not burn /files/get_temporary_link requests before the rclone operation starts.
Empty folders are treated as valid zero-file manifests for folder copy/move preflight so users can transfer empty directories without a startup error.
Deferred manifest validation is driven by a provider fast-launch policy table so each service can opt into different thresholds and estimate strategies; OneDrive is currently enabled, while other providers remain explicitly disabled until they expose a cheap trustworthy estimate or an approved alternate policy.
Google Drive to Dropbox folder copies now run a destination-compatibility preflight that auto-sanitizes safe path mismatches and blocks only when two Google paths would collapse to the same Dropbox destination name.
Used by /api/rclone/copy-folder route.
Provider Path Compatibility
Google Drive to Dropbox
Dropbox rejects some names that Google Drive allows, especially folder segments ending in _, ., or spaces, and extensionless file names that arrive as ambiguous upload targets.
Stratofusion now resolves those safe mismatches at the destination layer instead of mutating Google Drive.
Folder segments are trimmed only at the Dropbox destination.
Extensionless files get a .txt suffix only at the Dropbox destination.
Every applied rename is recorded in rclone operation metadata as pathSanitization.
Before launch, folder-copy preflight checks the resolved manifest for unsafe collapses. If sanitizing Folder_ and Folder would produce the same Dropbox path, the copy is blocked and the user must rename the source items manually.
Root folder names are sanitized with the same rules when the transfer target is Dropbox, including empty-folder copies that bypass rclone copy.
Unified File ID Resolver (src/lib/rclone/services/file-resolver.ts)
The unified resolver provides a single interface for resolving file IDs across all cloud storage services:
// Google Drive: 32+ character alphanumeric stringsfunctionisGoogleDriveFileId(id:string):boolean// OneDrive: Contains "!" separator or specific patternsfunctionisOneDriveFileId(id:string):boolean// Dropbox: Starts with "id:" prefixfunctionisDropboxFileId(id:string):boolean
Error Handling:
Graceful degradation when resolution fails.
Fallback to original ID for rclone to handle.
Detailed error metadata for debugging.
Consistent error response format.
Folder Structure Preservation
Overview
Stratofusion supports uploading entire folders while maintaining their directory structure in cloud storage services. When users drag and drop folders, the system automatically creates the necessary folder hierarchy.
Implementation
Client-Side Detection
The FileUploader component uses the browser's webkitRelativePath property:
The upload-session API routes accept metadata only, resolve folder hierarchies, and return provider upload URLs so the browser can stream bytes directly without proxying through Vercel or Fly.io:
// Extract folder path from relative pathconst folderPath = relativePath.substring(0, relativePath.lastIndexOf('/'));const fileName = relativePath.substring(relativePath.lastIndexOf('/')+1);// Create folder hierarchy if needed before returning the upload sessionif(folderPath){const folderId =awaitcreateFolderHierarchy(service, accountId, parentFolderId, folderPath);// Return uploadUrl + headers to the browser}
Browser Upload Flow
Browser sends metadata only to /api/{service}/upload-session
Server validates auth/quota and resolves destination folders
Server returns an upload URL and any provider headers required for the transfer
Browser uploads file bytes to the provider-native session endpoint, except Google which streams through the Stratofusion Fly upload gateway because Google blocks browser resumable CORS in practice
Folder Creation Strategy
Hierarchical Creation:
Split folder path into individual folder names
Create folders sequentially from root to leaf
Cache created folder IDs to prevent duplicates
Handle race conditions with proper locking
Caching Mechanism:
In-memory cache for folder creation during upload session.
Prevents duplicate API calls for the same folder path.
Single file upload when folder upload not supported.
Clear user feedback about browser limitations.
Progressive enhancement approach.
Consistent UI across all browsers.
Shared Copy Service Refactor
Problem Addressed
Inconsistent destination folder ID resolution across copy routes.
Significant code duplication between single‑file and batch copy endpoints.
Solution Overview
Introduced a shared copy service that centralizes validation, ID→path resolution for source and destination, rclone config creation, and error handling.
All copy routes now delegate to this shared logic, removing duplication and aligning behavior.
Key Outcomes
Destination ID resolution works consistently for single and batch copy.
Reduced route sizes and a single source of truth for copy behavior.
Easier maintenance, testing, and logging of copy operations.
Files Touched (high level)
Shared service module for copy operations (new).
File and batch copy API routes refactored to delegate.
Dedicated test suite for the shared service.
Transfer Conflict Resolution
Overview
Copy/move transfer flows now support explicit conflict policies end-to-end, instead of relying on implicit provider behavior.
Policies
rename (default for copy and move): preserve both files by generating name (N).ext.
skip: preserve destination file and pass --ignore-existing to rclone.
overwrite: keep default replacement semantics.
ask: normalized to rename for non-interactive/batch APIs.
Key Components
src/types/transfer-options.ts.
canonical policy types, guards, and normalization.
src/lib/utils/file-naming.ts.
deterministic unique name generation and batch-safe rename planning.
src/lib/conflicts/conflict-detector.ts.
destination conflict detection for incoming names.
src/lib/conflicts/conflict-detector-factory.ts.
provider-agnostic detector adapter via ServiceRegistry.
src/lib/rclone/services/copy-service.ts.
single + batch copy policy enforcement (rename/skip/overwrite).
src/lib/rclone/services/move-service.ts.
single + batch move policy enforcement (rename/skip/overwrite).
Batch rename is pre-resolved per destination (service, account, folder) to avoid in-batch duplicate collisions.
Backup launchers default to skip for file and folder copy operations.
Move operations use filename-aware destination behavior in the rclone worker (moveto when a target filename is supplied), enabling rename-on-conflict semantics.
Folder move operations with preserved structure launch rclone against the destination parent plus the moved folder name. This avoids ancestor-overlap failures when moving a folder from a nested path back up to one of its parent folders.
Batch copy launch streams emit operation-started as each file operation is accepted by the rclone worker, and the client opens /api/ops?id=... monitoring immediately for that operation instead of waiting for the entire batch launch to finish.
This streaming behavior does not serialize transfer completion: later rclone operations may run while earlier operations are still transferring.
Dropbox Batch Copy Fix
Problem
When copying files from Dropbox, the batch copy operation was failing with "directory not found" errors because Dropbox file IDs were not being resolved to paths before being passed to rclone.
Root Cause
The batch copy operation had file ID resolution logic for:
✅ Google Drive files: Using resolveGoogleDriveFilePath().
✅ OneDrive files: Using resolveOneDriveFilePath().
❌ Dropbox files: File IDs were passed directly to rclone without resolution.
Solution
Added missing Dropbox file ID resolution logic to ensure all file IDs are properly resolved to paths before rclone operations.
Enhanced Refresh Implementation
Features
Manual refresh with loading states.
Selection persists across manual refresh and browser reload per current view context.
The selection indicator surfaces the global count across current and saved folder scopes, while batch actions remain scoped to the current view.
Periodic auto-refresh and tab focus refresh.
Keyboard shortcuts (Ctrl+R, F5) support.
Proper cleanup and memory management.
Error handling with retry mechanisms.
Implementation
Enhanced useDriveItems hook with:
refreshCurrentFolder(): Manual refresh function with loading state.
isRefreshing: Boolean state indicating refresh in progress.
Automatic refresh triggers on tab focus and periodic intervals.
OneDrive Search Enhancement
Hybrid Search Approach
OneDrive implements an intelligent hybrid search that automatically chooses between:
Fast Basic Search (for simple queries):
API: /me/drive/root/search(q='query').
Use case: Simple filename matching.
Performance: Optimal for basic searches.
Advanced Microsoft Search (for complex queries):
API: /search/query with Microsoft Search API.
Use case: Full-text content search and KQL support.
Features: Advanced search capabilities.
Automatic Query Detection
The system analyzes query complexity to determine which search method provides the best balance of speed and capability.
Testing Strategy
Unit Tests:
Path validation functions.
Folder hierarchy creation logic.
Cache management operations.
Error handling scenarios.
Integration Tests:
End-to-end folder upload workflows.
Cross-service folder structure preservation.
Large folder upload performance.
Concurrent upload handling.
Browser Tests:
webkitRelativePath detection.
Drag and drop functionality.
Progress tracking accuracy.
Error state handling.
Search Results Path Display
Overview
Search results include a "Path" column that displays the full folder location for each file or folder, helping users identify files across different folders and cloud services.
Path Resolution by Service
Google Drive:
Resolves parent folder paths using the Google Drive API.
Builds folder path cache for efficient batch resolution.
Format: /Documents/Work/file.pdf.
Requires additional API calls (optimized with memoized parallel fetching).
OneDrive:
Uses parentReference.path from Microsoft Graph API.
Uses cached parent lookups when Microsoft Search results omit parent paths.
Format: /Documents/Work/file.pdf.
Dropbox:
Uses path_display from Dropbox API.
No additional API calls needed (path included in response).
Long: Very Long Fold... / ... / Final Folder (segments and folders truncated).
User Experience
Normal Folder Browsing:
Path column is hidden.
Standard table layout.
Search Results View:
Path column appears automatically.
Shows full folder location for each result.
Long paths truncated with "..." in the middle.
Hover to see complete path in tooltip.
Performance Considerations
Google Drive: Batched folder fetches with Promise.all.
OneDrive/Dropbox: Zero additional API calls.
Client-side path formatting and truncation.
Tooltip rendering on-demand.
Hybrid Download Strategy
Status: Updated (February 2026)
Overview
The download pipeline now uses browser-native direct-to-disk writes on Chrome/Edge via the File System Access API for both files and folders. The browser fetches provider URLs or Fly tokenized stream URLs and writes them into the user-selected local folder.
Problem Statement
The original download architecture streamed all files through the Fly.io rclone service, resulting in:
Bandwidth costs: $0.02/GB (North America/Europe) for all downloads.
Example cost: 100 users downloading 1GB each = $2.00 in Fly.io bandwidth.
Solution Architecture
Scenario
Method
Bandwidth Cost
Maintains Structure
Single file
Browser direct-to-disk (showDirectoryPicker) using provider direct/Fly stream URL
$0 for OneDrive/direct Dropbox paths
Yes
Multiple files
Browser direct-to-disk (showDirectoryPicker) with provider manifests
$0 for OneDrive/direct Dropbox paths
Yes
Folders
Browser manifest + recursive provider listing
$0 for OneDrive/direct Dropbox paths
Yes
Google files
Fly.io stream (/api/google-download/stream) when browser-readable direct URLs are unavailable
~$0.02/GB (Fly.io bandwidth only)
Yes
Dropbox restricted links
Fly.io stream (/api/dropbox-download/stream) when temporary links are blocked
~$0.02/GB only for those fallback bytes
Yes
Implementation Components
Manifest resolution:
src/app/api/download/resolve/route.ts.
src/server/download/*-resolver.ts (launches Google direct/Fly URLs and Dropbox temporary links).
Fly.io download streaming:
fly-rclone/src/routes/googleDownloadRoutes.js.
fly-rclone/src/routes/dropboxDownloadRoutes.js.
fly-rclone/src/utils/downloadToken.js.
Updated browser flow:
src/components/DriveTable.tsx.
src/components/DownloadDialog.tsx.
src/hooks/useDownloadRunner.ts.
src/lib/download/browser-write.ts.
src/lib/download/resolve-stream.ts.
src/lib/download/path-safety.ts.
Notes
/api/download/proxy is retired and now returns 410 Gone. Downloads call provider URLs or Fly.io streams from the browser instead of streaming through Vercel.
Google and Dropbox Fly download session tokens are stateless and encrypted so browser download streams survive Fly load balancing across instances.
Google and Dropbox Fly-browser downloads now mint a fresh session immediately before each file attempt so long-running batches do not rely on manifest-time session or access tokens.
OneDrive direct URLs and Dropbox temporary links are refreshed on retry instead of every file so long batches can recover from stale links without adding unnecessary provider calls to healthy first attempts.
User Experience
Files/folders (Chrome/Edge):
User clicks Download
Same click opens one destination-folder picker (showDirectoryPicker)
Dialog resolves the full manifest via /api/download/resolve
Browser streams provider URLs directly to local disk
Files that hit timeout/network/write failures are queued for a sequential retry phase with exponential backoff, a slow-throughput watchdog cuts off dribbling streams, tiny-file handling is provider-aware so provider-direct downloads can fail fast without over-triggering Google proxy downloads, tokenized Google/Dropbox stream paths refresh immediately before each attempt, OneDrive and Dropbox direct links refresh on retry, and retry timeouts are only extended for non-timeout failures
Progress is tracked locally per file/folder, including retry-phase status for failed files
Cancel/close aborts resolve + transfer requests immediately
Service Behavior
Google Drive:
Uses Fly tokenized streaming endpoints for browser-write flows when browser-readable direct URLs are unavailable or blocked by CORS.
Session creation happens server-side from Next.js resolver code (src/server/download/google-fly-download.ts).
Browser download attempts call /api/download/session just-in-time so each file starts with a fresh Google access token and Fly stream token.
Google Workspace files are exported to compatible formats (.docx, .xlsx, .pptx, etc.).
OneDrive:
Uses Graph @microsoft.graph.downloadUrl in the resolve manifest.
Retry attempts call /api/download/session to mint a fresh direct download URL instead of reusing a stale manifest-time URL.
Browser fetches URL directly and writes to local disk.
Dropbox:
Uses Dropbox temporary links resolved server-side per file.
Retry attempts can refresh Dropbox temporary links through /api/download/session, while Dropbox Fly fallback streams mint a fresh tokenized session for every attempt.
Browser fetches returned CDN URL and writes to local disk.
When temporary links are blocked (not_allowed, email_not_verified), falls back to Fly tokenized streaming (/api/dropbox-download/stream).
Stream tokens are stateless and encrypted so the browser stream can continue even if Fly routes the stream request to a different instance than the session request.
Relative paths are deconflicted before writing so folders like _tika_corpus/gnarly can download successfully even when providers expose both a file and a folder with the same stem.
Performance Impact
Positive impacts:
Reduced Fly.io bandwidth and compute for download operations.
No server-side ZIP assembly latency for browser-direct multi-file/folder downloads.
Maintains folder structure without temporary archive storage.
Future Enhancements
Persistent directory permission reuse where browser policy allows
Optional bounded concurrency for faster large manifests
Per-file retry with backoff for transient provider/CDN failures
Conclusion
The current browser-direct strategy removes server-side ZIP assembly for modern browser multi-file and folder downloads while preserving provider-specific direct/stream exceptions where required. Unsupported browsers are explicitly blocked from download flows.