# File Management Architecture

## ADR-001: File ID Resolution Architecture

### Status: Accepted

### Context
Cloud storage services (Google Drive, OneDrive, Dropbox) use internal file/folder IDs that are not compatible with rclone's path-based operations. These IDs need to be resolved to actual file paths before being passed to rclone.

Previously, file ID resolution was handled inconsistently:
- **Google Drive**: Resolution in rclone service.
- **OneDrive**: Resolution in both Next.js API route AND rclone service (redundant).
- **Dropbox**: No resolution (causing failures).

### Decision
**All file and folder ID resolution will be handled in the Next.js application layer, not in the rclone service.**

### Rationale
1. **Separation of Concerns**: Next.js handles business logic, rclone handles file operations
2. **Better Architecture**: Service registry and authentication properly managed
3. **Easier Testing**: Can test resolution logic without rclone dependencies
4. **Better Error Handling**: Can return proper HTTP responses with detailed errors
5. **No Duplication**: Single place for resolution logic per service
6. **Consistency**: All services follow the same pattern

### Implementation Pattern

#### Next.js API Route Responsibilities:
- Detect service-specific file/folder IDs.
- Use service registry to get appropriate service instance.
- Resolve IDs to file/folder paths using service APIs.
- Pass resolved paths to rclone service.

#### Rclone Service Responsibilities:
- Accept resolved file/folder paths.
- Execute rclone operations with paths.
- Handle rclone-specific logic and progress tracking.

#### Example Flow:
```
Frontend Request: sourceFileId = "id:2V5kn5RRfnAAAAAAAADsgw"
    ↓
Next.js API Route: Detects Dropbox file ID
    ↓
Service Resolution: dropboxService.getFileMetadata() → "/Documents/file.pdf"
    ↓
Rclone Service: rclone copy dropbox:/Documents/file.pdf dest:
```

### Services Affected
- ✅ **Dropbox**: Implemented via the unified resolver using direct Dropbox API calls.
- ✅ **Google Drive**: Handled by the unified resolver (internal resolver at `src/lib/rclone/resolvers/google-drive.ts`).
- ✅ **OneDrive**: Handled by the unified resolver (internal resolver at `src/lib/rclone/resolvers/onedrive.ts`).

### Files Modified
- `src/app/api/rclone/batch-copy/route.ts` - File ID resolution logic.
- `src/app/api/rclone/copy/route.ts` - File ID resolution logic.
- `src/app/api/rclone/copy-folder/route.ts` - Folder ID resolution logic.
- `fly-rclone/server.js` - Remove redundant resolution logic.

### Utilities
- `src/lib/rclone/services/file-resolver.ts` - Unified resolution utility (single entry point).
- `src/lib/rclone/resolvers/*` - Service-specific resolvers (INTERNAL). Do NOT import directly; use the unified resolver.
- Dropbox resolution uses direct Dropbox API calls internally via the unified resolver.

## File ID Resolution Implementation

### Transfer Utilities and Progress Normalization

- `src/lib/rclone/services/transfer-utils.ts`.
  - Shared helpers for: path resolution (source/dest), rclone config/remote construction, and basic validation.
  - Dropbox transfer launches normalize to remote-relative rclone paths and strip legacy Business `/<homePath>/...` prefixes before copy/move/sync.
  - Standardized error handling with `throwOperationError(prefix, operation, err, ErrorCtor)`.
  - Progress normalization with `normalizeProgressFromOperation(operation)` returning a consistent structure consumed by UI.
- `src/lib/rclone/services/folder-copy-service.ts`.
  - Centralizes recursive folder copy setup and delegates to the rclone client. The `Fly` client name is retained for compatibility, while production targets the VM worker.
  - Reuses precomputed manifest/size preflight results from `/api/rclone/copy-folder` when available so large folder launches avoid duplicate enumeration.
  - OneDrive folder sources prefer Graph `driveItem.size` for byte estimation and only fall back to recursive enumeration when provider metadata does not include a folder size.
  - Transfer preflight manifest resolution is metadata-only and skips per-file download URL generation, so Dropbox folder copy/move/sync launches do not burn `/files/get_temporary_link` requests before the rclone operation starts.
  - Empty folders are treated as valid zero-file manifests for folder copy/move preflight so users can transfer empty directories without a startup error.
  - Deferred manifest validation is driven by a provider fast-launch policy table so each service can opt into different thresholds and estimate strategies; OneDrive is currently enabled, while other providers remain explicitly disabled until they expose a cheap trustworthy estimate or an approved alternate policy.
  - Google Drive to Dropbox folder copies now run a destination-compatibility preflight that auto-sanitizes safe path mismatches and blocks only when two Google paths would collapse to the same Dropbox destination name.
  - Used by `/api/rclone/copy-folder` route.

## Provider Path Compatibility

### Google Drive to Dropbox

- Dropbox rejects some names that Google Drive allows, especially folder segments ending in `_`, `.`, or spaces, and extensionless file names that arrive as ambiguous upload targets.
- Stratofusion now resolves those safe mismatches at the destination layer instead of mutating Google Drive.
  - Folder segments are trimmed only at the Dropbox destination.
  - Extensionless files get a `.txt` suffix only at the Dropbox destination.
  - Every applied rename is recorded in rclone operation metadata as `pathSanitization`.
- Before launch, folder-copy preflight checks the resolved manifest for unsafe collapses. If sanitizing `Folder_` and `Folder` would produce the same Dropbox path, the copy is blocked and the user must rename the source items manually.
- Root folder names are sanitized with the same rules when the transfer target is Dropbox, including empty-folder copies that bypass `rclone copy`.


### Unified File ID Resolver (`src/lib/rclone/services/file-resolver.ts`)

The unified resolver provides a single interface for resolving file IDs across all cloud storage services:

```typescript
interface FileIdResolutionResult {
  resolvedPath: string;
  wasResolved: boolean;
  originalId: string;
  metadata?: {
    fileName?: string;
    isFolder?: boolean;
    error?: string;
  };
}

// Main resolution function
async function resolveFileId(
  service: ServiceType,
  accountId: string,
  fileId: string
): Promise<FileIdResolutionResult>
```

### Service-Specific Resolvers (INTERNAL — use unified API)

#### Google Drive Resolver (`src/lib/rclone/resolvers/google-drive.ts`)
- Resolves Google Drive file IDs to full paths.
- Handles both file and folder IDs.
- Uses Google Drive API v3 for metadata retrieval.
- Supports shared drives and regular drives.

#### OneDrive Resolver (`src/lib/rclone/resolvers/onedrive.ts`)
- Resolves OneDrive item IDs to paths.
- Handles composite IDs for both personal (`driveId!itemId`) and business (`b!driveId!itemId`) OneDrive items.
- Uses Microsoft Graph API for metadata retrieval.
- Supports both personal and business accounts.

#### Dropbox Resolver (`src/lib/rclone/resolvers/dropbox.ts`)
- Converts Dropbox file IDs to paths.
- Handles "id:" prefixed identifiers.
- Uses Dropbox API v2 for metadata retrieval.
- Graceful fallback for permission issues.

### Resolution Patterns

#### ID Detection:
```typescript
// Google Drive: 32+ character alphanumeric strings
function isGoogleDriveFileId(id: string): boolean

// OneDrive: Contains "!" separator or specific patterns
function isOneDriveFileId(id: string): boolean

// Dropbox: Starts with "id:" prefix
function isDropboxFileId(id: string): boolean
```

#### Error Handling:
- Graceful degradation when resolution fails.
- Fallback to original ID for rclone to handle.
- Detailed error metadata for debugging.
- Consistent error response format.

## Folder Structure Preservation

### Overview
Stratofusion supports uploading entire folders while maintaining their directory structure in cloud storage services. When users drag and drop folders, the system automatically creates the necessary folder hierarchy.

### Implementation

#### Client-Side Detection
The `FileUploader` component uses the browser's `webkitRelativePath` property:

```typescript
// In FileUploader.tsx
onBeforeFileAdded: (currentFile) => {
  if (currentFile.source === "Dashboard" && currentFile.data instanceof File) {
    const file = currentFile.data as File & { webkitRelativePath?: string };
    if (file.webkitRelativePath) {
      currentFile.meta = {
        ...currentFile.meta,
        relativePath: file.webkitRelativePath,
      };
    }
  }
  return currentFile;
}
```

#### Session-Based Upload Processing
The upload-session API routes accept metadata only, resolve folder hierarchies, and return provider upload URLs so the browser can stream bytes directly without proxying through Vercel or Fly.io:

```typescript
// Extract folder path from relative path
const folderPath = relativePath.substring(0, relativePath.lastIndexOf('/'));
const fileName = relativePath.substring(relativePath.lastIndexOf('/') + 1);

// Create folder hierarchy if needed before returning the upload session
if (folderPath) {
  const folderId = await createFolderHierarchy(service, accountId, parentFolderId, folderPath);
  // Return uploadUrl + headers to the browser
}
```

#### Browser Upload Flow
1. Browser sends metadata only to `/api/{service}/upload-session`
2. Server validates auth/quota and resolves destination folders
3. Server returns an upload URL and any provider headers required for the transfer
4. Browser uploads file bytes to the provider-native session endpoint, except Google which streams through the Stratofusion Fly upload gateway because Google blocks browser resumable CORS in practice

### Folder Creation Strategy

#### Hierarchical Creation:
1. Split folder path into individual folder names
2. Create folders sequentially from root to leaf
3. Cache created folder IDs to prevent duplicates
4. Handle race conditions with proper locking

#### Caching Mechanism:
- In-memory cache for folder creation during upload session.
- Prevents duplicate API calls for the same folder path.
- Automatic cleanup after upload completion.
- Thread-safe operations for concurrent uploads.

#### Error Handling:
- Retry logic for folder creation failures.
- Graceful handling of existing folders.
- Detailed error reporting for debugging.
- Rollback mechanism for partial failures.

### Security Considerations

#### Path Validation:
```typescript
function isPathSafe(path: string): boolean {
  // Prevent directory traversal attacks
  if (path.includes('..') || path.includes('//')) {
    return false;
  }

  // Check for invalid characters
  const invalidChars = ['<', '>', ':', '"', '|', '?', '*'];
  return !invalidChars.some(char => path.includes(char));
}
```

#### Upload Limits:
- Maximum file size: 5GB per file.
- Maximum batch size: 50GB total.
- Maximum file count: 1,000 files per batch.
- Path length validation: 260 characters maximum.

### Performance Optimizations

#### Concurrent Operations:
- Parallel folder creation where possible.
- Batched API requests for efficiency.
- Progress tracking per file and folder.
- Memory usage monitoring for large uploads.

#### Caching Strategy:
- Folder ID caching during upload session.
- Metadata caching for recently created folders.
- Automatic cache invalidation after operations.
- Memory-efficient data structures.

### Browser Compatibility

#### webkitRelativePath Support:
- Chrome: Full support.
- Firefox: Full support.
- Safari: Full support.
- Edge: Full support.
- IE: Not supported (graceful degradation).

#### Fallback Behavior:
- Single file upload when folder upload not supported.
- Clear user feedback about browser limitations.
- Progressive enhancement approach.
- Consistent UI across all browsers.

## Shared Copy Service Refactor

### Problem Addressed
- Inconsistent destination folder ID resolution across copy routes.
- Significant code duplication between single‑file and batch copy endpoints.

### Solution Overview
- Introduced a shared copy service that centralizes validation, ID→path resolution for source and destination, rclone config creation, and error handling.
- All copy routes now delegate to this shared logic, removing duplication and aligning behavior.

### Key Outcomes
- Destination ID resolution works consistently for single and batch copy.
- Reduced route sizes and a single source of truth for copy behavior.
- Easier maintenance, testing, and logging of copy operations.

### Files Touched (high level)
- Shared service module for copy operations (new).
- File and batch copy API routes refactored to delegate.
- Dedicated test suite for the shared service.

## Transfer Conflict Resolution

### Overview
Copy/move transfer flows now support explicit conflict policies end-to-end, instead of relying on implicit provider behavior.

### Policies
- `rename` (default for copy and move): preserve both files by generating `name (N).ext`.
- `skip`: preserve destination file and pass `--ignore-existing` to rclone.
- `overwrite`: keep default replacement semantics.
- `ask`: normalized to `rename` for non-interactive/batch APIs.

### Key Components
- `src/types/transfer-options.ts`.
  - canonical policy types, guards, and normalization.
- `src/lib/utils/file-naming.ts`.
  - deterministic unique name generation and batch-safe rename planning.
- `src/lib/conflicts/conflict-detector.ts`.
  - destination conflict detection for incoming names.
- `src/lib/conflicts/conflict-detector-factory.ts`.
  - provider-agnostic detector adapter via `ServiceRegistry`.
- `src/lib/rclone/services/copy-service.ts`.
  - single + batch copy policy enforcement (rename/skip/overwrite).
- `src/lib/rclone/services/move-service.ts`.
  - single + batch move policy enforcement (rename/skip/overwrite).
- `src/components/BatchFileTransferDialog.tsx`.
  - user-facing conflict selection (copy/move dialog).

### Runtime Behavior
- Batch rename is pre-resolved per destination `(service, account, folder)` to avoid in-batch duplicate collisions.
- Backup launchers default to `skip` for file and folder copy operations.
- Move operations use filename-aware destination behavior in the rclone worker (`moveto` when a target filename is supplied), enabling rename-on-conflict semantics.
- Folder move operations with preserved structure launch rclone against the destination parent plus the moved folder name. This avoids ancestor-overlap failures when moving a folder from a nested path back up to one of its parent folders.
- Batch copy launch streams emit `operation-started` as each file operation is accepted by the rclone worker, and the client opens `/api/ops?id=...` monitoring immediately for that operation instead of waiting for the entire batch launch to finish.
- This streaming behavior does not serialize transfer completion: later rclone operations may run while earlier operations are still transferring.

### Dropbox Batch Copy Fix

#### Problem
When copying files from Dropbox, the batch copy operation was failing with "directory not found" errors because Dropbox file IDs were not being resolved to paths before being passed to rclone.

#### Root Cause
The batch copy operation had file ID resolution logic for:
- ✅ **Google Drive files**: Using `resolveGoogleDriveFilePath()`.
- ✅ **OneDrive files**: Using `resolveOneDriveFilePath()`.
- ❌ **Dropbox files**: File IDs were passed directly to rclone without resolution.

#### Solution
Added missing Dropbox file ID resolution logic to ensure all file IDs are properly resolved to paths before rclone operations.

### Enhanced Refresh Implementation

#### Features
- Manual refresh with loading states.
- Selection persists across manual refresh and browser reload per current view context.
- The selection indicator surfaces the global count across current and saved folder scopes, while batch actions remain scoped to the current view.
- Periodic auto-refresh and tab focus refresh.
- Keyboard shortcuts (Ctrl+R, F5) support.
- Proper cleanup and memory management.
- Error handling with retry mechanisms.

#### Implementation
Enhanced `useDriveItems` hook with:
- `refreshCurrentFolder()`: Manual refresh function with loading state.
- `isRefreshing`: Boolean state indicating refresh in progress.
- Automatic refresh triggers on tab focus and periodic intervals.

### OneDrive Search Enhancement

#### Hybrid Search Approach
OneDrive implements an intelligent hybrid search that automatically chooses between:

**Fast Basic Search** (for simple queries):
- API: `/me/drive/root/search(q='query')`.
- Use case: Simple filename matching.
- Performance: Optimal for basic searches.

**Advanced Microsoft Search** (for complex queries):
- API: `/search/query` with Microsoft Search API.
- Use case: Full-text content search and KQL support.
- Features: Advanced search capabilities.

#### Automatic Query Detection
The system analyzes query complexity to determine which search method provides the best balance of speed and capability.

### Testing Strategy

#### Unit Tests:
- Path validation functions.
- Folder hierarchy creation logic.
- Cache management operations.
- Error handling scenarios.

#### Integration Tests:
- End-to-end folder upload workflows.
- Cross-service folder structure preservation.
- Large folder upload performance.
- Concurrent upload handling.

#### Browser Tests:
- webkitRelativePath detection.
- Drag and drop functionality.
- Progress tracking accuracy.
- Error state handling.

---

## Search Results Path Display

### Overview
Search results include a "Path" column that displays the full folder location for each file or folder, helping users identify files across different folders and cloud services.

### Path Resolution by Service

**Google Drive:**
- Resolves parent folder paths using the Google Drive API.
- Builds folder path cache for efficient batch resolution.
- Format: `/Documents/Work/file.pdf`.
- Requires additional API calls (optimized with memoized parallel fetching).

**OneDrive:**
- Uses `parentReference.path` from Microsoft Graph API.
- Uses cached parent lookups when Microsoft Search results omit parent paths.
- Format: `/Documents/Work/file.pdf`.

**Dropbox:**
- Uses `path_display` from Dropbox API.
- No additional API calls needed (path included in response).
- Format: `/Documents/Work/file.pdf`.
- Zero performance impact.

### Path Formatting

**Utilities** (`src/lib/utils/path-utils.ts`):
- `formatPath(path, maxLength)` - Intelligent path truncation.
- `getFullPathForTooltip(path)` - Full path for hover tooltips.
- `getParentPath(path)` - Extract parent folder.
- `getFilenameFromPath(path)` - Extract filename.

**Features:**
- Mobile-first design with configurable max lengths.
- Handles Windows and Unix path separators.
- Graceful handling of edge cases (root paths, missing paths).
- Intelligent truncation preserving first and last segments.

**Example Truncations:**
- Short: `Documents / Work` (no truncation).
- Medium: `Documents / ... / Projects` (middle folders hidden).
- Long: `Very Long Fold... / ... / Final Folder` (segments and folders truncated).

### User Experience

**Normal Folder Browsing:**
- Path column is hidden.
- Standard table layout.

**Search Results View:**
- Path column appears automatically.
- Shows full folder location for each result.
- Long paths truncated with "..." in the middle.
- Hover to see complete path in tooltip.

### Performance Considerations

- Google Drive: Batched folder fetches with Promise.all.
- OneDrive/Dropbox: Zero additional API calls.
- Client-side path formatting and truncation.
- Tooltip rendering on-demand.

---

## Hybrid Download Strategy

### Status: Updated (February 2026)

### Overview

The download pipeline now uses browser-native direct-to-disk writes on Chrome/Edge via the File System Access API for both files and folders. The browser fetches provider URLs or Fly tokenized stream URLs and writes them into the user-selected local folder.

### Problem Statement

The original download architecture streamed all files through the Fly.io rclone service, resulting in:
- 2x bandwidth consumption: Cloud Service -> Fly.io (inbound, free) + Fly.io -> Client (outbound, charged).
- Bandwidth costs: $0.02/GB (North America/Europe) for all downloads.
- Example cost: 100 users downloading 1GB each = $2.00 in Fly.io bandwidth.

### Solution Architecture

| Scenario | Method | Bandwidth Cost | Maintains Structure |
|----------|--------|----------------|---------------------|
| Single file | Browser direct-to-disk (`showDirectoryPicker`) using provider direct/Fly stream URL | $0 for OneDrive/direct Dropbox paths | Yes |
| Multiple files | Browser direct-to-disk (`showDirectoryPicker`) with provider manifests | $0 for OneDrive/direct Dropbox paths | Yes |
| Folders | Browser manifest + recursive provider listing | $0 for OneDrive/direct Dropbox paths | Yes |
| Google files | Fly.io stream (`/api/google-download/stream`) when browser-readable direct URLs are unavailable | ~$0.02/GB (Fly.io bandwidth only) | Yes |
| Dropbox restricted links | Fly.io stream (`/api/dropbox-download/stream`) when temporary links are blocked | ~$0.02/GB only for those fallback bytes | Yes |

### Implementation Components

1. Manifest resolution:
   - src/app/api/download/resolve/route.ts.
   - src/server/download/*-resolver.ts (launches Google direct/Fly URLs and Dropbox temporary links).
2. Fly.io download streaming:
   - fly-rclone/src/routes/googleDownloadRoutes.js.
   - fly-rclone/src/routes/dropboxDownloadRoutes.js.
   - fly-rclone/src/utils/downloadToken.js.
3. Updated browser flow:
   - src/components/DriveTable.tsx.
   - src/components/DownloadDialog.tsx.
   - src/hooks/useDownloadRunner.ts.
   - src/lib/download/browser-write.ts.
   - src/lib/download/resolve-stream.ts.
   - src/lib/download/path-safety.ts.

### Notes

- `/api/download/proxy` is retired and now returns 410 Gone. Downloads call provider URLs or Fly.io streams from the browser instead of streaming through Vercel.
- Google and Dropbox Fly download session tokens are stateless and encrypted so browser download streams survive Fly load balancing across instances.
- Google and Dropbox Fly-browser downloads now mint a fresh session immediately before each file attempt so long-running batches do not rely on manifest-time session or access tokens.
- OneDrive direct URLs and Dropbox temporary links are refreshed on retry instead of every file so long batches can recover from stale links without adding unnecessary provider calls to healthy first attempts.

### User Experience

Files/folders (Chrome/Edge):
1. User clicks Download
2. Same click opens one destination-folder picker (showDirectoryPicker)
3. Dialog resolves the full manifest via /api/download/resolve
4. Browser streams provider URLs directly to local disk
5. Files that hit timeout/network/write failures are queued for a sequential retry phase with exponential backoff, a slow-throughput watchdog cuts off dribbling streams, tiny-file handling is provider-aware so provider-direct downloads can fail fast without over-triggering Google proxy downloads, tokenized Google/Dropbox stream paths refresh immediately before each attempt, OneDrive and Dropbox direct links refresh on retry, and retry timeouts are only extended for non-timeout failures
6. Progress is tracked locally per file/folder, including retry-phase status for failed files
7. Cancel/close aborts resolve + transfer requests immediately

### Service Behavior

Google Drive:
- Uses Fly tokenized streaming endpoints for browser-write flows when browser-readable direct URLs are unavailable or blocked by CORS.
- Session creation happens server-side from Next.js resolver code (`src/server/download/google-fly-download.ts`).
- Browser download attempts call `/api/download/session` just-in-time so each file starts with a fresh Google access token and Fly stream token.
- Google Workspace files are exported to compatible formats (.docx, .xlsx, .pptx, etc.).

OneDrive:
- Uses Graph @microsoft.graph.downloadUrl in the resolve manifest.
- Retry attempts call `/api/download/session` to mint a fresh direct download URL instead of reusing a stale manifest-time URL.
- Browser fetches URL directly and writes to local disk.

Dropbox:
- Uses Dropbox temporary links resolved server-side per file.
- Retry attempts can refresh Dropbox temporary links through `/api/download/session`, while Dropbox Fly fallback streams mint a fresh tokenized session for every attempt.
- Browser fetches returned CDN URL and writes to local disk.
- When temporary links are blocked (`not_allowed`, `email_not_verified`), falls back to Fly tokenized streaming (`/api/dropbox-download/stream`).
- Stream tokens are stateless and encrypted so the browser stream can continue even if Fly routes the stream request to a different instance than the session request.
- Relative paths are deconflicted before writing so folders like `_tika_corpus/gnarly` can download successfully even when providers expose both a file and a folder with the same stem.

### Performance Impact

Positive impacts:
- Reduced Fly.io bandwidth and compute for download operations.
- No server-side ZIP assembly latency for browser-direct multi-file/folder downloads.
- Maintains folder structure without temporary archive storage.

### Future Enhancements

1. Persistent directory permission reuse where browser policy allows
2. Optional bounded concurrency for faster large manifests
3. Per-file retry with backoff for transient provider/CDN failures

### Conclusion

The current browser-direct strategy removes server-side ZIP assembly for modern browser multi-file and folder downloads while preserving provider-specific direct/stream exceptions where required. Unsupported browsers are explicitly blocked from download flows.