# Sync Job Production Fixes

**Date:** November 10, 2025
**Commit:** 492134a6
**Status:** ✅ Fixed

## Overview

Fixed two critical issues preventing sync jobs from executing successfully on the production server (Fly.io):

1. **Rclone Binary Not Found** - `/bin/sh: rclone.exe: not found`
2. **Google Drive Folder ID Resolution Failing** - 404 errors causing sync operations to fail silently

---

## Issue 1: Rclone Binary Not Found (CRITICAL)

### Problem

Sync jobs were failing on production with error:
```
/bin/sh: rclone.exe: not found
```

### Root Cause

The rclone executable name was hardcoded to `"rclone.exe"` in `fly-rclone/src/constants.js`:

```javascript
const RCLONE = {
  EXECUTABLE: "rclone.exe",  // ❌ Windows-only executable name
  // ...
};
```

**Environment Mismatch:**
- **Development (Windows):** Uses `rclone.exe`.
- **Production (Fly.io/Linux):** Uses `rclone` (installed at `/usr/bin/rclone`).

The code was trying to execute `rclone.exe` on a Linux server, which doesn't exist.

### Solution

Created OS-aware rclone path resolution:

**1. New Utility File:** `fly-rclone/src/utils/rclonePath.js`
```javascript
function resolveRclonePath() {
  // 1) Check environment variables first
  const envPath = process.env.RCLONE_PATH || process.env.FLY_RCLONE_PATH;
  if (envPath) return envPath;

  // 2) Check for local binary (OS-aware)
  const defaultExec = process.platform === "win32" ? "rclone.exe" : "rclone";
  const localPath = join(__dirname, "..", "..", defaultExec);
  if (existsSync(localPath)) return localPath;

  // 3) Fall back to system PATH
  return defaultExec;
}

const RCLONE_PATH = resolveRclonePath();
module.exports = { resolveRclonePath, RCLONE_PATH };
```

**2. Updated Constants:** `fly-rclone/src/constants.js`
```javascript
const { RCLONE_PATH } = require("./utils/rclonePath");

const RCLONE = {
  EXECUTABLE: RCLONE_PATH,  // ✅ OS-aware: rclone.exe on Windows, rclone on Linux
  // ...
};
```

**3. Updated Server:** `fly-rclone/server.js`
```javascript
// Import from shared utility instead of duplicating logic
const { RCLONE_PATH } = require("./src/utils/rclonePath");
```

### Impact

- ✅ Rclone now works on both Windows (development) and Linux (production).
- ✅ Eliminates "rclone.exe: not found" errors on Fly.io.
- ✅ Supports environment variable override via `RCLONE_PATH` or `FLY_RCLONE_PATH`.
- ✅ Centralized path resolution logic (DRY principle).

---

## Issue 2: Google Drive Folder ID Resolution Failing

### Problem

Sync jobs were reporting success even when folder IDs couldn't be resolved:

**Activity Log:**
```
Syncing: 1TDpy53fMpSeNj8iBiI7227Vu_4Sb27c8 → root
Status: 1 succeeded, 0 failed
```

**Rclone Service Log:**
```
NOTICE: Failed to sync: directory not found: 1TDpy53fMpSeNj8iBiI7227Vu_4Sb27c8
```

**Google Drive API Response:**
```json
{
  "error": {
    "code": 404,
    "message": "File not found: 1TDpy53fMpSeNj8iBiI7227Vu_4Sb27c8"
  }
}
```

### Root Cause

**Path Resolution Failure:**
1. Sync job attempts to resolve folder ID `1TDpy53fMpSeNj8iBiI7227Vu_4Sb27c8` to a path
2. Google Drive API returns 404 (folder deleted/moved/no access)
3. Path resolver catches error and returns `wasResolved: false`
4. Sync job logs a warning but **proceeds anyway** with the raw folder ID
5. Rclone receives invalid folder ID and fails
6. Error was already caught in previous fix, but path validation was missing

**Why 404 Happens:**
- Folder was deleted from Google Drive.
- Folder was moved to trash.
- OAuth token doesn't have permission to access the folder.
- Folder is in a shared drive requiring different API calls.

### Solution

Added early validation to fail sync jobs when folder IDs cannot be resolved:

**Updated:** `src/lib/sync/execute-sync-job.ts`

```typescript
// Validate path resolution - fail early if folders don't exist
const sourceResolutionFailed = !resolvedSource?.resolvedPath &&
                               item.sourceId !== "root" &&
                               item.sourceId !== "";
const destResolutionFailed = !resolvedDestination?.resolvedPath &&
                             item.destinationFolderId !== "root" &&
                             item.destinationFolderId !== "";

if (sourceResolutionFailed || destResolutionFailed) {
  const errors: string[] = [];

  if (sourceResolutionFailed) {
    errors.push(
      `Source folder not found or inaccessible: ${item.sourceService} folder ID "${item.sourceId}". ` +
      `The folder may have been deleted, moved, or you may no longer have access to it.`
    );
    logger.error(`[executeSyncJob] Source path resolution failed for item ${item.id}`, {
      jobId, userId, service: item.sourceService,
      accountId: item.sourceAccountId, folderId: item.sourceId,
    });
  }

  if (destResolutionFailed) {
    errors.push(
      `Destination folder not found or inaccessible: ${item.destinationService} folder ID "${item.destinationFolderId}". ` +
      `The folder may have been deleted, moved, or you may no longer have access to it.`
    );
    logger.error(`[executeSyncJob] Destination path resolution failed for item ${item.id}`, {
      jobId, userId, service: item.destinationService,
      accountId: item.destinationAccountId, folderId: item.destinationFolderId,
    });
  }

  // Throw error to skip this item and mark it as failed
  throw new Error(
    `Path resolution failed: ${errors.join(" ")} ` +
    `Please update the sync job configuration with valid folder IDs.`
  );
}
```

### Impact

**Before:**
- ❌ Sync jobs proceeded with invalid folder IDs.
- ❌ Rclone failed with cryptic "directory not found" errors.
- ❌ Activity logs showed success when operations actually failed.
- ❌ Users had no idea what went wrong.

**After:**
- ✅ Sync jobs fail early with clear error messages.
- ✅ Activity logs show detailed failure reasons.
- ✅ Users know exactly what's wrong: "folder may have been deleted, moved, or you may no longer have access".
- ✅ Users get actionable guidance: "Please update the sync job configuration with valid folder IDs".
- ✅ Enhanced error logging with full context for debugging.

---

## Testing

### Test Case 1: Rclone Binary Path

**Development (Windows):**
```bash
# Should use rclone.exe
node fly-rclone/server.js
# ✅ Uses rclone.exe from local directory or PATH
```

**Production (Fly.io/Linux):**
```bash
# Should use /usr/local/bin/rclone or the system PATH entry for rclone
node server.js
# ✅ Uses the rclone binary installed by Dockerfile
```

### Test Case 2: Folder ID Validation

**Valid Folder IDs:**
```typescript
// Should succeed
sourceId: "1ABC123xyz" (exists in Google Drive)
destinationId: "root" (always valid)
// ✅ Sync proceeds normally
```

**Invalid Folder IDs:**
```typescript
// Should fail with clear error
sourceId: "1TDpy53fMpSeNj8iBiI7227Vu_4Sb27c8" (404 from Google Drive)
// ❌ Sync fails with: "Source folder not found or inaccessible..."
```

---

## Deployment

### Rclone Service (Fly.io)

The rclone service needs to be redeployed to pick up the new path resolution logic:

```bash
cd fly-rclone
./deploy.sh --env prod
```

**Verification:**
```bash
# Check logs for successful startup
flyctl logs --app stratofusion-rclone-prod

# Should see:
# ✅ "Rclone path resolved to: rclone" (not rclone.exe)
```

### Next.js Application

The Next.js application needs to be redeployed for the sync job validation changes:

```bash
# Deploy to Vercel or your hosting platform
vercel --prod
```

---

## Monitoring

### Key Metrics to Watch

1. **Sync Job Success Rate**
   - Monitor `syncJobs` table for `status = 'failed'`.
   - Check `syncJobItems` for failure patterns.

2. **Rclone Operation Status**
   - Monitor Fly.io logs for "rclone.exe: not found" errors (should be zero).
   - Check operation completion rates.

3. **Path Resolution Failures**
   - Monitor activity logs for "Path resolution failed" messages.
   - Track which folder IDs are failing most often.

### Alerts to Set Up

```sql
-- Alert on high sync job failure rate
SELECT COUNT(*)
FROM sync_jobs
WHERE status = 'failed'
  AND created_at > NOW() - INTERVAL '1 hour'
HAVING COUNT(*) > 5;

-- Alert on path resolution failures
SELECT *
FROM activity_logs
WHERE message LIKE '%Path resolution failed%'
  AND created_at > NOW() - INTERVAL '1 hour';
```

---

## Future Improvements

### 1. Folder ID Validation UI

Add a validation step in the sync job creation UI:
- Check if folder IDs exist before saving the job.
- Show warning if folders are inaccessible.
- Suggest alternative folders.

### 2. Automatic Folder ID Updates

Implement a background job to:
- Detect when folder IDs become invalid.
- Attempt to find the folder by name in the new location.
- Update sync job configuration automatically.
- Notify users of changes.

### 3. Better Error Recovery

Add retry logic for transient errors:
- Distinguish between permanent (404) and temporary (503) errors.
- Retry temporary errors with exponential backoff.
- Only fail permanently for 404/403 errors.

---

## Related Files

- `fly-rclone/src/utils/rclonePath.js` - OS-aware rclone path resolution.
- `fly-rclone/src/constants.js` - Rclone configuration constants.
- `fly-rclone/server.js` - Main rclone service server.
- `src/lib/sync/execute-sync-job.ts` - Sync job execution logic.
- `src/lib/logging/path-resolver.ts` - Path resolution for logging.
- `src/lib/rclone/services/file-resolver.ts` - File ID resolution.
- `src/lib/rclone/resolvers/google-drive.ts` - Google Drive path resolution.

---

## References

- [Rclone Documentation](https://rclone.org/docs/).
- [Fly.io Deployment Guide](https://fly.io/docs/).
- [Google Drive API - Files.get](https://developers.google.com/drive/api/v3/reference/files/get).