# Backup Feature Documentation

> **Runtime status:** the backup feature is current, but production hosting has
> moved to the OVHcloud VM. One Compose cron replica invokes the same Next.js
> routes, the rclone worker runs in Compose, and Compose PostgreSQL is
> authoritative. Sections that prescribe Vercel Cron, active Fly Machines, or
> Neon production are historical implementation records unless explicitly
> labeled as legacy recovery. See [Architecture Evolution](./ARCHITECTURE_EVOLUTION.md),
> [Deployment](./DEPLOYMENT.md), and the
> [Production VM Runbook](./operations/VM_PRODUCTION_RUNBOOK.md).
> References to the former shared-development hostname, Vercel project, or Fly
> dev worker are also historical and must not be executed; use local tooling or
> the shared-development retirement runbook instead.

**Last Updated:** 2025-10-27 (React State Timing Issue fix added - Bug #8)
**Status:** ✅ Phases 1-3 Complete | ✅ Code Quality Improvements (Phase 1-2 Refactorings) | 🚧 Phases 4-8 In Progress

---

## Table of Contents

1. [Overview](#overview)
2. [Current Implementation Status](#current-implementation-status)
3. [Architecture](#architecture)
4. [User Guide](#user-guide)
5. [Implementation Phases](#implementation-phases)
6. [Technical Details](#technical-details)
7. [Troubleshooting](#troubleshooting)
8. [Future Enhancements](#future-enhancements)

---

## Overview

The Backup Feature enables users to schedule automated backups of files and folders from one cloud storage service to another. Backups can be executed immediately or scheduled with daily/weekly/monthly frequencies, plus minutely intervals (every 5/10/15/20/30 minutes) for testing.

### Key Features

- ✅ **Immediate Backups** - Execute backups right away with "Run now" option.
- ✅ **Scheduled Backups** - Daily, weekly, monthly, or minutely recurring backups.
- ✅ **Time-of-Day Selection** - Choose specific times for scheduled backups.
- ✅ **Timezone Support** - Schedule backups in your local timezone.
- ✅ **Timestamped Folders** - Each backup creates a timestamped folder (e.g., `backup-2025-10-14_14-30-00`).
- ✅ **Automatic Execution** - Vercel cron jobs trigger scheduled backups every 5 minutes.
- ✅ **Job Management** - View, retry, and cancel backup jobs from the Jobs page.
- 🚧 **Progress Tracking** - Per-item progress tracking (Phase 4 - In Progress).
- 🚧 **Error Notifications** - Email notifications for failed jobs (Phase 5 - Planned).

---

## Current Implementation Status

### Phase 1: Foundation ✅ COMPLETE

**Database Schema:**
- `backup_jobs` table with job metadata.
- `backup_job_items` table for individual file/folder tracking.
- Indexes for efficient querying.

**UI Components:**
- `BatchFileTransferDialog` with backup mode.
- Schedule selector (Run now, Daily, Weekly, Monthly, Every 5/10/15/20/30 minutes).
- Jobs page at `/user/jobs` for monitoring.

**API Routes:**
- `POST /api/jobs/backup` - Create backup job.
- `GET /api/jobs` - List user's jobs.
- `POST /api/jobs/[id]/cancel` - Cancel job.
- `POST /api/jobs/[id]/retry` - Retry failed job by cloning it and queuing the launch outside the response path.

### Phase 2: Time-of-Day Selection UI ✅ COMPLETE

**UI Components Created:**
- `TimePicker` component (`src/components/ui/time-picker.tsx`).
  - 24-hour time selection.
  - Hour and minute dropdowns.
  - 15-minute intervals for minutes.

- `TimezonePicker` component (`src/components/ui/timezone-picker.tsx`).
  - Common timezone selector.
  - Defaults to browser timezone.
  - 16 common timezones included.

**Database Changes:**
- Added `scheduled_time` column (TEXT, format: "HH:mm").
- Added `timezone` column (TEXT, IANA timezone).
- Added `backup_jobs_next_run_idx` index for efficient scheduler queries.

**Integration:**
- Time picker shown when schedule !== 'none'.
- Display format: "Daily at 14:30 America/New_York".
- Responsive layout (stacks on mobile).

### Phase 3: Scheduler System ✅ COMPLETE

**Vercel Cron Configuration:**
```json
{
  "crons": [{
    "path": "/api/cron/execute-backups",
    "schedule": "*/5 * * * *"  // Every 5 minutes
  }],
  "functions": {
    "src/app/api/cron/execute-backups/route.ts": {
      "maxDuration": 300
    }
  }
}
```

**Cron API Endpoint:**
- File: `src/app/api/cron/execute-backups/route.ts`.
- Runs every 5 minutes via Vercel cron.
- Authenticated with `CRON_SECRET` environment variable.
- Queries for jobs where `nextRunAt <= now` and `status = 'scheduled'`.
- Processes max 10 jobs per run to avoid timeouts.
- Updates job status to 'running'.
- Queues `executeBackupJob()` in `after()` for each due job.
- Uses a 300-second launch budget so broad selected-item backups can persist all operation IDs before reconciliation evaluates completeness.

**Job Execution Endpoint:**
- File: `src/app/api/jobs/[id]/execute/route.ts`.
- Executes a backup job by ID.
- Creates timestamped backup folder.
- Uses existing rclone batch-copy service.
- Handles both one-time and recurring jobs.
- Updates job status on completion/failure.

### Phase 4: Progress Tracking 🚧 IN PROGRESS

**Planned Features:**
- Add progress fields to `backup_job_items` table.
- Create progress update API endpoints.
- Enhance Jobs page with expandable detail view.
- Implement real-time updates for running jobs.

### Phase 5-8: Future Phases 📋 PLANNED

See [Implementation Phases](#implementation-phases) section for details.

---

## Architecture

### System Overview

```
┌─────────────────────────────────────────────────────────────┐
│  Vercel Cron (every 5 minutes)                               │
│  Triggers: /api/cron/execute-backups                         │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│  Cron Endpoint                                               │
│  1. Query for due jobs (nextRunAt <= now, status=scheduled) │
│  2. Mark jobs as 'running'                                   │
│  3. Call /api/jobs/[id]/execute for each                    │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│  Execute Endpoint                                            │
│  1. Create timestamped backup folder                         │
│  2. Call /api/rclone/batch-copy                             │
│  3. Mark job complete/failed                                 │
│  4. Compute next run time (if recurring)                     │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│  Rclone Service (Fly.io)                                     │
│  Executes file transfers                                     │
└─────────────────────────────────────────────────────────────┘
```

### Job Lifecycle

```
User creates scheduled backup
         ↓
Status: 'scheduled', nextRunAt computed
         ↓
Cron finds due job (nextRunAt <= now)
         ↓
Status: 'running', startedAt set
         ↓
Execute endpoint runs transfers
         ↓
    ┌────────────┐
    │  Success?  │
    └─────┬──────┘
          │
    ┌─────┴─────┐
    │           │
   Yes         No
    │           │
    ▼           ▼
Recurring?   Status: 'failed'
    │        lastError set
┌───┴───┐
│       │
Yes    No
│       │
▼       ▼
Status:  Status:
'scheduled' 'completed'
nextRunAt   finishedAt
computed    set
```

For detailed architecture diagrams, see [Backup Scheduler Architecture](./diagrams/backup-scheduler-architecture.md).

---

## User Guide

### Creating a Backup

1. **Select Files/Folders:**
   - Navigate to the source service (e.g., Google Drive).
   - Select files and/or folders you want to backup.
   - Click the "Backup" button in the toolbar.

2. **Configure Backup:**
   - **Destination:** Choose target service, account, and folder.
   - **Schedule:** Select backup frequency.
     - **Run now:** Execute immediately (one-time).
     - **Daily:** Run every day at specified time.
     - **Weekly:** Run every week at specified time.
     - **Monthly:** Run every month at specified time.
   - **Time (for scheduled backups):** Select hour, minute, and timezone.
   - **Options**.
     - Maintain folder structure (recommended).
     - Apply transfer filters (e.g., exclude Personal Vault).

3. **Start Backup:**
   - Click "Start Backup" to create the job.
   - For immediate backups, transfer begins right away.
   - For scheduled backups, job is queued for execution at specified time.
   - Job creation does not recursively enumerate every selected folder; quota, auth, compatibility, and path checks run when the backup launches.

### Managing Backup Jobs

**Jobs Page:** Navigate to User > Jobs

**Job Actions:**
- **View Details:** Click on a job to see detailed information.
- **Cancel:** Stop a running or scheduled job.
- **Retry:** Re-run a failed job.
- **Delete:** Remove completed or failed jobs from history.

**Job Status:**
- `scheduled` - Waiting for next run time.
- `running` - Currently executing.
- `completed` - Successfully finished.
- `failed` - Encountered an error.
- `cancelled` - Manually stopped by user.

---

## Job Management Features

**Date:** 2025-10-21
**Status:** ✅ Complete
**Branch:** feature/111-backup-tidyup

### Overview

Users can now fully manage their backup jobs through the Jobs page, including editing scheduled jobs and permanently deleting jobs from the database.

### Edit Scheduled Backups

**Capabilities:**

Users can edit scheduled backup jobs to modify:
- **Schedule frequency** (daily, weekly, monthly, or minutely intervals).
- **Scheduled time** (HH:mm in 24-hour format).
- **Timezone** (IANA timezone identifier).

**Restrictions:**
- Only **scheduled** jobs can be edited (not running, completed, failed, or cancelled).
- Source files and destination cannot be changed (create a new backup job instead).
- Users can only edit their own jobs (enforced via userId check).

**UI Location:**
- Jobs page (`/user/jobs`).
- Edit button appears next to scheduled jobs.
- Opens `EditBackupDialog` component.

**User Experience:**
- Form pre-populated with current job settings.
- Unsaved changes warning when closing without saving.
- Auto-refresh pauses while dialog is open (prevents interference).
- Success message after saving changes.
- Jobs list refreshes automatically after edit.

### Delete Backup Jobs

**Capabilities:**

Users can permanently delete backup jobs from the database.

**Restrictions:**
- Cannot delete **running** jobs (must cancel first).
- Can delete: completed, failed, cancelled, or scheduled jobs.
- Requires confirmation dialog before deletion.
- Users can only delete their own jobs (enforced via userId check).

**UI Location:**
- Jobs page (`/user/jobs`).
- Delete button appears next to completed, failed, cancelled, or scheduled jobs.
- Opens `DeleteBackupConfirmDialog` component.

**User Experience:**
- Confirmation dialog shows job details before deletion.
- Auto-refresh pauses while dialog is open.
- Success message after deletion.
- Jobs list refreshes automatically after delete.

### Implementation Details

#### Database Layer

**File:** `src/lib/database/backup-jobs.ts`

**New Functions:**

```typescript
// Fetch a single job with its items for editing
export async function getBackupJobById(
  userId: string,
  jobId: string
): Promise<{ job: BackupJob; items: BackupJobItem[] } | null>

// Update an existing scheduled job
export async function updateBackupJob(
  userId: string,
  jobId: string,
  input: UpdateBackupJobInput
): Promise<{ job: BackupJob; items: BackupJobItem[] } | null>

// Permanently delete a job and its items
export async function deleteBackupJob(
  userId: string,
  jobId: string
): Promise<boolean>
```

**Key Implementation Details:**
- `updateBackupJob()` regenerates summaries and recalculates `nextRunAt` based on new schedule.
- `updateBackupJob()` deletes old job items and creates new ones (maintains referential integrity).
- `deleteBackupJob()` prevents deletion of running jobs.
- Both functions enforce userId ownership checks.

#### API Routes

**File:** `src/app/api/jobs/[id]/route.ts`

**Endpoints:**

```
GET    /api/jobs/[id]     - Get job details for editing
PATCH  /api/jobs/[id]     - Update job
DELETE /api/jobs/[id]     - Delete job
```

**Authentication:**
- All endpoints require user authentication via `getCurrentUserId()`.
- All operations are scoped to the authenticated user's jobs only.

**Error Handling:**
- Returns appropriate error messages for invalid operations.
- Validates job ownership before any modifications.
- Returns 404 for non-existent jobs.
- Returns 400 for invalid requests (e.g., trying to delete running job).

#### Types

**File:** `src/types/backup.ts`

**New Interfaces:**

```typescript
interface UpdateBackupJobRequest {
  schedule: BackupSchedule;
  scheduledTime?: string;
  timezone?: string;
  sources: BackupSourceItem[];
  destination: BackupDestination;
  maintainStructure?: boolean;
  applyFilters?: boolean;
}

interface UpdateBackupJobInput {
  schedule: BackupSchedule;
  scheduledTime?: string;
  timezone?: string;
  sources: BackupSourceItem[];
  destination: BackupDestination;
  maintainStructure?: boolean;
  applyFilters?: boolean;
}
```

#### UI Components

**EditBackupDialog Component**
- **File:** `src/components/EditBackupDialog.tsx`.
- **Features**.
  - Fetches job data on open.
  - Pre-populates form with current settings.
  - Tracks unsaved changes with `useFormDirtyState` hook.
  - Shows warning dialog when closing with unsaved changes.
  - Notifies parent when dialog opens/closes (for auto-refresh pause).
  - Validates form before submission.
  - Shows success/error messages.

**DeleteBackupConfirmDialog Component**
- **File:** `src/components/DeleteBackupConfirmDialog.tsx`.
- **Features**.
  - Shows job details in confirmation dialog.
  - Prevents deletion of running jobs.
  - Notifies parent when dialog opens/closes (for auto-refresh pause).
  - Shows success/error messages.
  - Refreshes jobs list after deletion.

**Jobs Page Integration**
- **File:** `src/app/user/jobs/page.tsx`.
- **Features**.
  - Edit button for scheduled jobs.
  - Delete button for completed/failed/cancelled/scheduled jobs.
  - Auto-refresh pauses when dialogs are open.
  - Consolidated success handlers for edit/delete operations.
  - Memoized dialog state change handler for performance.

### UX Improvements

**Auto-Refresh Pause Pattern:**
- Jobs page auto-refreshes every 8 seconds.
- Auto-refresh pauses when edit or delete dialog is open.
- Prevents interference with user workflow.
- Resumes auto-refresh when dialog closes.

**Unsaved Changes Warning:**
- Edit dialog tracks form dirty state.
- Shows warning dialog when closing with unsaved changes.
- Prevents accidental data loss.
- Uses consistent shadcn/ui dialog (not browser confirm).

**Reusable Hooks:**
- `useDialogStateNotification` - Notifies parent of dialog state changes.
- `useFormDirtyState` - Tracks unsaved changes in forms.
- Both hooks are reusable across the application.

For detailed UX pattern documentation, see [UX Patterns](/dev/docs/ux-patterns).

### Testing

**Unit Tests:**
- Database layer functions (getBackupJobById, updateBackupJob, deleteBackupJob).
- API route handlers (GET, PATCH, DELETE).
- Hook functionality (useDialogStateNotification, useFormDirtyState).
- Component behavior (EditBackupDialog, DeleteBackupConfirmDialog, UnsavedChangesDialog).

**Test Coverage:**
- 42 comprehensive tests for hooks and components.
- 100% test pass rate.
- TypeScript compilation successful.
- ESLint validation successful.

**Storybook Stories:**
- 7 interactive stories for UnsavedChangesDialog.
- Visual documentation for all states.
- Responsive design testing.
- Dark mode support verification.

For detailed testing documentation, see [Testing Guide](/dev/docs/testing#reusable-hooks-and-components-testing).

### Security

**Authorization:**
- All operations require user authentication.
- Users can only edit/delete their own jobs.
- userId checks enforced at database layer.
- API routes validate ownership before modifications.

**Validation:**
- Cannot delete running jobs.
- Cannot edit non-scheduled jobs.
- Schedule and time validation.
- Timezone validation (IANA format).

### Future Enhancements

**Planned Features:**
- Bulk edit operations (edit multiple jobs at once).
- Job templates (save job configurations for reuse).
- Job history (view past executions and results).
- Advanced scheduling (custom cron expressions).
- Job dependencies (run jobs in sequence).

---

## Implementation Phases

### Phase 1: Foundation ✅ COMPLETE
- Database schema (backup_jobs, backup_job_items).
- Basic UI for backup configuration.
- API routes for job management.
- Jobs page for monitoring.
- Immediate backups with timestamped folders.

### Phase 2: Time-of-Day Selection UI ✅ COMPLETE
- TimePicker component.
- TimezonePicker component.
- Database columns for scheduled_time and timezone.
- UI integration in BatchFileTransferDialog.
- Timezone-aware scheduling logic.

### Phase 3: Scheduler System ✅ COMPLETE
- Vercel cron configuration.
- Cron API endpoint (/api/cron/execute-backups).
- Job execution endpoint (/api/jobs/[id]/execute).
- Automatic job execution every 5 minutes.
- Recurring job rescheduling.

### Phase 4: Progress Tracking 🚧 IN PROGRESS
- Add progress fields to backup_job_items table.
- Create progress update API endpoints.
- Enhance Jobs page with expandable detail view.
- Implement real-time updates for running jobs.

### Phase 5: Error Handling & Notifications 📋 PLANNED
- Email notifications for failed jobs.
- Retry logic with exponential backoff.
- Better error messages and recovery suggestions.

### Phase 6: Subscription Tier Enforcement 📋 PLANNED
- Limit backup frequency by tier.
- Limit backup size by tier.
- Enforce storage quotas.

### Phase 7: Testing & Documentation 📋 PLANNED
- Unit tests for time picker components.
- Unit tests for computeNextRunAt with timezones.
- Integration tests for cron execution.
- Update README.md with backup documentation.

### Phase 8: Polish & Optimization 📋 PLANNED
- Performance optimizations.
- UI/UX improvements.
- Advanced features (backup versioning, incremental backups).

---

## Technical Details

### Environment Variables

```bash
# Cron Job Configuration
# Secret token for authenticating Vercel cron jobs
# Generate a random string for production: openssl rand -base64 32
CRON_SECRET=your-cron-secret-token-here
```

**Setup for Local Development:**
```bash
# Add to .env.local
CRON_SECRET=local-dev-secret-123
```

**Setup for Production (Vercel):**
1. Generate secure secret: `openssl rand -base64 32`
2. Add to Vercel environment variables
3. Vercel will automatically use this for cron authentication

### Database Schema

**backup_jobs table:**
```sql
CREATE TABLE backup_jobs (
  id TEXT PRIMARY KEY,
  user_id TEXT NOT NULL,
  status TEXT NOT NULL,
  schedule TEXT NOT NULL,
  scheduled_time TEXT,
  timezone TEXT DEFAULT 'UTC',
  source_summary TEXT NOT NULL,
  destination_summary TEXT NOT NULL,
  payload TEXT NOT NULL,
  created_at TIMESTAMP NOT NULL,
  updated_at TIMESTAMP NOT NULL,
  started_at TIMESTAMP,
  finished_at TIMESTAMP,
  next_run_at TIMESTAMP,
  last_run_at TIMESTAMP,
  last_error TEXT
);

CREATE INDEX backup_jobs_next_run_idx
  ON backup_jobs(next_run_at)
  WHERE status = 'scheduled';
```

### Key Functions

**computeNextRunAt()** - Timezone-aware scheduling
```typescript
function computeNextRunAt(
  from: Date,
  schedule: BackupSchedule,
  scheduledTime?: string,
  timezone?: string
): Date | null
```

**createBackupJob()** - Create new backup job
```typescript
export async function createBackupJob(input: CreateBackupJobInput)
```

**completeBackupJob()** - Mark job as complete and reschedule if recurring
```typescript
export async function completeBackupJob(userId: string, jobId: string)
```

**failBackupJob()** - Mark job as failed with error message
```typescript
export async function failBackupJob(userId: string, jobId: string, error: string)
```

---

## Troubleshooting

### Quick Reference - Immediate Investigation Steps

#### 1. Get Failed Job Details (30 seconds)

```bash
# Run the investigation script
node scripts/investigate-failed-backup.mjs

# This will show:
# - Error message from the failure
# - Job configuration and timing
# - Source and destination details
# - Recommendations for fixes
```

#### 2. Check Vercel Logs (2 minutes)

**Go to:** https://vercel.com/dashboard → Your Project → Logs

**Filter by:**
- Time: Around the scheduled backup time.
- Path: `/api/cron/execute-backups` OR `/api/jobs/[id]/execute`.

**Look for:**
- ❌ 401/403 errors → Missing/incorrect CRON_SECRET.
- ❌ 500 errors → Check error message in logs.
- ❌ Timeout errors → Job took longer than expected.
- ❌ "Failed to create folder" → OAuth token expired or service API issue.
- ❌ "Batch copy failed" → Rclone service issue.

#### 3. Verify Environment Variables (1 minute)

```bash
# Check if all required variables are set
vercel env ls

# Required variables:
# ✅ CRON_SECRET
# ✅ NEXT_PUBLIC_SITE_URL (should be https://dev.stratofusion.io)
# ✅ DATABASE_URL
# ✅ FLYIO_RCLONE_SERVICE_URL
```

#### 4. Test Rclone Service (30 seconds)

```bash
# Check if rclone service is healthy
curl https://stratofusion-rclone-dev.fly.dev/health

# Expected: {"status":"ok","timestamp":"..."}
# If error: Service is down, needs restart
```

---

### Common Issues & Quick Fixes

#### Issue 1: Missing CRON_SECRET ⚠️ CRITICAL

**How to identify:**
- Vercel logs show: "Unauthorized cron request".
- HTTP 401 error in cron endpoint.

**Quick fix:**
```bash
# Generate a secure secret
openssl rand -base64 32

# Add to Vercel
vercel env add CRON_SECRET production
# Paste the generated secret

# Redeploy
vercel --prod
```

#### Issue 2: OAuth Token Expired 🔑

**How to identify:**
- Error message contains: "Authentication failed" or "Invalid credentials".
- Folder creation fails with 401/403.

**Quick fix:**
1. Go to https://dev.stratofusion.io/settings
2. Find the connected service (Google Drive, OneDrive, etc.)
3. Click "Disconnect"
4. Click "Connect" and re-authorize
5. Retry the failed backup job

#### Issue 3: Rclone Service Down 🚨

**How to identify:**
- Error message: "Batch copy failed: Service Unavailable".
- Health check fails: `curl https://stratofusion-rclone-dev.fly.dev/health`.

**Quick fix:**
```bash
# Check status
flyctl status --app stratofusion-rclone-dev

# Restart if needed
flyctl apps restart stratofusion-rclone-dev

# Check logs
flyctl logs --app stratofusion-rclone-dev
```

#### Issue 4: Scheduled Backups Not Running

**Symptom:** Jobs remain in 'scheduled' status past their nextRunAt time

**Possible Causes:**
1. Missing `CRON_SECRET` environment variable
2. Vercel cron not configured correctly
3. Job execution endpoint failing

**Solutions:**
1. Verify `CRON_SECRET` is set in environment variables
2. Check `vercel.json` has correct cron configuration
3. Review Vercel logs for cron execution errors
4. Test cron endpoint manually:
   ```bash
   curl -X GET http://localhost:3000/api/cron/execute-backups \
     -H "Authorization: Bearer your-cron-secret"
   ```

#### Issue 5: Timezone Issues

**Symptom:** Backups run at wrong time

**Possible Causes:**
1. Incorrect timezone selected
2. Timezone conversion logic error
3. Server time vs. user time mismatch

**Solutions:**
1. Verify timezone is correctly saved in database
2. Check `computeNextRunAt()` uses `date-fns-tz` correctly
3. Test with different timezones to verify conversion

#### Issue 6: Job Execution Failures

**Symptom:** Jobs fail with errors

**Possible Causes:**
1. Rclone service unavailable
2. Invalid source/destination paths
3. Authentication token expired
4. Network connectivity issues

**Solutions:**
1. Check Fly.io rclone service status
2. Verify file/folder IDs are valid
3. Refresh OAuth tokens
4. Review error logs for specific failure reasons

---

### Cron Verification Quick Reference

#### Essential Vercel CLI Commands

```powershell
# Login to Vercel
vercel login

# List deployments
vercel ls --prod

# View production logs (real-time)
vercel logs --prod --follow

# View logs for last hour
vercel logs --prod --since 1h

# Filter logs for cron
vercel logs --prod | Select-String -Pattern "cron"

# List environment variables
vercel env ls

# Inspect deployment
vercel inspect dev.stratofusion.io
```

#### Quick Verification Checklist

**Configuration:**
- [ ] `vercel.json` exists and has cron configuration.
- [ ] Cron path is `/api/cron/execute-backups`.
- [ ] Cron schedule is `*/5 * * * *`.

**Deployment:**
- [ ] API route file exists: `src/app/api/cron/execute-backups/route.ts`.
- [ ] Cron job shows in Vercel Dashboard → Cron Jobs tab.
- [ ] Cron job status is "Enabled".

**Environment Variables:**
- [ ] `CRON_SECRET` is set in production.
- [ ] `NEXT_PUBLIC_SITE_URL` is set (NO trailing slash).
- [ ] `NEXT_PUBLIC_FLYIO_RCLONE_SERVICE_URL` is set.
- [ ] `DATABASE_URL` is set.

**Execution:**
- [ ] Manual test returns status 200.
- [ ] Response is JSON (not HTML).
- [ ] Response has `success: true`.
- [ ] Cron logs show automatic triggers every 5 minutes.

---

### Comprehensive Investigation Guide

For detailed investigation of backup failures, follow these steps:

#### System Architecture Overview

```
┌─────────────────────────────────────────────────────────────┐
│  Vercel Cron (every 5 minutes)                               │
│  Triggers: /api/cron/execute-backups                         │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│  Cron Endpoint (src/app/api/cron/execute-backups/route.ts) │
│  1. Authenticates with CRON_SECRET                          │
│  2. Queries: WHERE status='scheduled' AND nextRunAt <= now  │
│  3. Updates status to 'running'                             │
│  4. Calls /api/jobs/[id]/execute for each job              │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│  Execute Endpoint (src/app/api/jobs/[id]/execute/route.ts) │
│  1. Verifies authentication                                  │
│  2. Creates timestamped backup folder                        │
│  3. Calls rclone batch-copy service                         │
│  4. Marks job as 'completed' or 'failed'                    │
└─────────────────────────────────────────────────────────────┘
```

#### Investigation Steps

**Step 1: Check Failed Job Details**

Run the investigation script to get detailed information:

```bash
# Check all failed jobs
node scripts/investigate-failed-backup.mjs

# Check specific job by ID
node scripts/investigate-failed-backup.mjs <job-id>
```

This will show:
- Job configuration and scheduling details.
- Error message from the failure.
- Job items and their status.
- Timing analysis (when it was supposed to run vs. when it failed).

**Step 2: Review Vercel Deployment Logs**

1. Access Vercel Dashboard
   - Go to https://vercel.com/dashboard.
   - Select your project.
   - Navigate to "Logs" tab.

2. Filter for Cron Execution
   - Time range: Around scheduled backup time.
   - Filter by path: `/api/cron/execute-backups`.
   - Look for authentication errors, database query errors, timeout errors.

3. Filter for Job Execution
   - Filter by path: `/api/jobs/[id]/execute`.
   - Look for folder creation failures, rclone service errors, token expiration.

**Step 3: Verify Environment Variables**

Check that all required environment variables are set in Vercel:

```bash
vercel env ls

# Required variables:
# - CRON_SECRET
# - NEXT_PUBLIC_SITE_URL
# - DATABASE_URL
# - FLYIO_RCLONE_SERVICE_URL
# - OAuth credentials for each service
```

**Step 4: Test Rclone Service**

```bash
# Health check
curl https://stratofusion-rclone-dev.fly.dev/health

# Production equivalent
curl https://stratofusion-rclone-prod.fly.dev/health

# Check service status
flyctl status --app stratofusion-rclone-dev
flyctl status --app stratofusion-rclone-prod

# View recent logs
flyctl logs --app stratofusion-rclone-dev
flyctl logs --app stratofusion-rclone-prod
```

**Step 5: Manual Cron Endpoint Test**

```bash
# Test the cron endpoint manually
curl -X POST https://dev.stratofusion.io/api/cron/execute-backups \
  -H "Authorization: Bearer your-cron-secret" \
  -H "Content-Type: application/json"
```

Expected response:
```json
{
  "success": true,
  "message": "Executed X backup jobs",
  "executedJobs": [...]
}
```

---

## Code Quality Improvements

### Phase 1 & 2 Refactorings (2025-10-25) ✅ COMPLETE

**Objective:** Improve code quality, eliminate duplication, and enhance maintainability of backup duplicate prevention logic.

#### Phase 1: Critical Refactorings

**1. useSubmissionGuard Hook** (Commit: bb15423b)
- **Purpose:** Reusable hook to prevent duplicate submissions with automatic cleanup.
- **Location:** `src/hooks/useSubmissionGuard.ts`.
- **Features**.
  - `isSubmitting` state management.
  - `startSubmission()` - returns false if already submitting.
  - `endSubmission()` - resets state.
  - `withSubmissionGuard()` - wrapper function for async operations.
  - Automatic cleanup on unmount via `isMountedRef`.
  - Optional callbacks: `onSubmissionStart`, `onSubmissionEnd`.
  - Debug logging option.
- **Testing:** 13 unit tests passing.
- **Impact:** Eliminated 6 manual `setIsSubmitting` calls in BatchFileTransferDialog.

**2. Backup Flow Refactoring** (Commit: 437dfa12)
- **Changes**.
  - Created `handleBackupOperation` function for backup-specific logic.
  - Created `handleTransferBatch` wrapper for operation routing.
  - Renamed internal transfer logic to `handleTransferBatchInternal`.
  - Removed problematic useEffect for isSubmitting reset.
- **Benefits**.
  - Clear separation of concerns.
  - Eliminated race conditions.
  - Improved code readability.

#### Phase 2: High Priority Refactorings

**1. useBackupOperation Hook** (Commit: 58f003d2)
- **Purpose:** Custom hook for managing backup operations (scheduled and run-now).
- **Location:** `src/hooks/useBackupOperation.ts`.
- **Features**.
  - Validation of backup parameters.
  - Creating scheduled backups.
  - Creating run-now backups with timestamped folders.
  - Error handling and state management.
  - Callbacks for success/error events.
- **Testing:** 18 unit tests passing.
- **Impact:** Reduced `handleBackupOperation` from 116 lines to 58 lines (50% reduction).

**2. Database Optimization** (Commit: 8c4dc62b)
- **Function:** `findDuplicateScheduledBackup()` in `src/lib/database/backup-jobs.ts`.
- **Optimization:** Reduced complexity from O(n²) to O(n).
- **Changes**.
  - Use `backupJobItems` table for source matching instead of JSON parsing.
  - Use Set data structures for O(1) lookups.
  - Filter candidates by schedule type before detailed comparison.
- **Benefits**.
  - Faster duplicate detection.
  - Reduced database load.
  - Less JavaScript processing and memory usage.

**3. Validation Utilities** (Commit: 74a45475)
- **Purpose:** Reusable validation functions for transfer operations.
- **Location:** `src/lib/validation/transfer-validation.ts`.
- **Functions**.
  - `validateTransferRequirements` - Basic transfer validation.
  - `validateDifferentLocation` - Prevent copying to same location.
  - `validateFileSize` - File size limit validation.
  - `validateBatchLimits` - Batch item count validation.
  - `validateItemMetadata` - Required metadata validation.
  - `validateTransferOperation` - Comprehensive validation combining all checks.
- **Testing:** 33 unit tests passing.
- **Benefits**.
  - Consistent validation across all transfer operations.
  - Reusable validation logic.
  - Better error messages.

#### E2E Testing Validation (2025-10-25)

**Test Scope:** Scheduled backup job creation and verification
- ✅ User authentication and navigation.
- ✅ File/folder selection from Google Drive.
- ✅ Backup dialog configuration (schedule, time, timezone, destination).
- ✅ Job submission and success notification.
- ✅ Job verification in Jobs page with correct details.
- ✅ All refactored code validated in production-like environment.

**Test Results:**
- All Phase 1 and Phase 2 refactorings working correctly.
- useSubmissionGuard prevents duplicate submissions.
- useBackupOperation handles backup creation properly.
- Database optimization allows efficient duplicate detection.
- Validation utilities ensure data integrity.
- User experience is smooth with proper feedback.

**Test Environment:**
- Application: http://localhost:3000.
- User: stratofusion002@gmail.com.
- Browser: Chrome (via Playwright MCP).
- Duration: ~3 minutes.

#### Benefits Summary

**Code Quality:**
- Eliminated code duplication (6 instances of manual submission guard).
- Improved separation of concerns.
- Better error handling and validation.
- Consistent patterns across codebase.

**Performance:**
- Database query optimization (O(n²) → O(n)).
- Reduced memory usage.
- Faster duplicate detection.

**Maintainability:**
- Reusable hooks and utilities.
- Comprehensive test coverage (64 unit tests total).
- Clear documentation.
- Easier to extend and modify.

**User Experience:**
- Prevents duplicate submissions.
- Better error messages.
- Consistent validation.
- Reliable backup operations.

---

## Bug Fixes & Solutions

This section documents critical bug fixes and solutions implemented for the backup system, organized chronologically.

### October 2025 Bug Fixes

#### 1. Backup Job Cancellation Fix (October 21, 2025)

**Problem:** Backup jobs continued running after cancellation. Clicking "Cancel" updated the UI status but didn't stop the actual rclone operations.

**Root Cause:** Backup jobs didn't track rclone operation IDs, making it impossible to cancel the actual operations running on the Fly.io service.

**Solution Implemented:**

1. **Database Schema Update**
   - Added `operationIds` field to `backup_jobs` table.
   - Stores JSON array of rclone operation IDs for cancellation.
   - Migration: `drizzle/0007_add_operation_ids_to_backup_jobs.sql`.

2. **Track Operation IDs During Execution**
   - Modified `executeBackupJob` to collect all rclone operation IDs.
   - Stores them in the database as a JSON array.
   - Logs the number of operations tracked.

3. **Cancel Rclone Operations**
   - Enhanced `cancelBackupJob` to parse stored operation IDs.
   - Calls `client.cancelOperation()` for each operation ID.
   - Sends DELETE requests to Fly.io rclone service.
   - Kills rclone processes (SIGTERM, then SIGKILL if needed).

**Files Modified:**
- `src/lib/database/schema.ts` - Added operationIds field.
- `src/lib/database/backup-jobs.ts` - Enhanced cancelBackupJob function.
- `src/lib/backup/execute-backup-job.ts` - Operation ID tracking.
- `drizzle/0007_add_operation_ids_to_backup_jobs.sql` - Database migration.

**Status:** ✅ Fixed - Commit: See TASKS.md

---

#### 2. Backup Dialog Duplicate Prevention (October 21, 2025)

**Problem:** "Backup Items" button remained enabled after scheduling, allowing users to accidentally register the same schedule multiple times. Dialog stayed open after successful registration.

**Solution Implemented:**
- Added automatic dialog close after successful schedule creation.
- Button already properly disabled during API call via `isLoading` state.
- Error cases still keep dialog open for user to retry.

**Files Modified:**
- `src/components/BatchFileTransferDialog.tsx` - Added `onClose()` call after successful schedule creation.

**Benefits:**
- Prevents duplicate schedule registrations.
- Better UX - dialog closes after successful action.
- Consistent with other dialog patterns in the app.

**Status:** ✅ Fixed

---

#### 3. Scheduled Backups Creating Empty Directories (October 18, 2025)

**Problem:** Scheduled backup cron jobs were executing successfully but creating empty directories instead of copying files.

**Root Cause:** `process.env.NEXT_PUBLIC_SITE_URL` is undefined in server-side code. The `NEXT_PUBLIC_` prefix means the variable is for client-side code only. This caused fetch URLs to be invalid, resulting in 404 HTML error pages instead of JSON responses.

**Solution Implemented:**

Created server-side URL utility (`src/lib/server-url.ts`):
- `getServerApiUrl(path)` - Async function to construct full API URLs for server-side fetch calls.
- `getServerBaseUrl()` - Determines the correct base URL for the current environment.

**URL Construction Priority:**
1. VERCEL_URL environment variable (automatically set by Vercel)
2. x-forwarded-host header (from incoming request)
3. host header (from incoming request)
4. Fallback: `http://localhost:3000` (for local development)

**Files Modified:**
- `src/lib/server-url.ts` - Created server-side URL utility.
- `src/app/api/jobs/[id]/execute/route.ts` - Updated to use `getServerApiUrl()`.
- `src/app/api/cron/execute-backups/route.ts` - Updated to use `getServerApiUrl()`.

**Status:** ✅ Fixed - Commit: ee9a0302

---

#### 4. Folder Backup Authentication Error (October 18, 2025)

**Problem:** Folder backups failed with "User not authenticated" error.

**Root Cause:** The `FolderCopyOperationRequest` interface was missing the `userId` field. Without `userId`, the `buildConfigAndRemotes()` function couldn't retrieve OAuth tokens from the database.

**Solution Implemented:**
- Added `userId` field to `FolderCopyOperationRequest` interface.
- Updated `executeFolderCopyOperation()` to pass `userId` to `buildConfigAndRemotes()`.
- Ensured consistent userId handling across file and folder operations.

**Files Modified:**
- `src/lib/rclone/services/folder-copy-service.ts` - Added userId field and parameter passing.

**Status:** ✅ Fixed - Commit: 4a1dafdc

---

#### 5. Backup Folders Empty and Timestamp Timezone Issues (October 18, 2025)

**Problem 1:** Backup jobs completed without errors but created empty folders - no files or folder contents were copied.

**Root Cause:** The backup execution logic was treating ALL items (files and folders) as individual files and using the file copy service. The `executeBatchCopyOperations()` service uses `rclone copyto` which copies a single file, not folder contents.

**Problem 2:** Backup folder timestamps showed time 1 hour ahead of actual execution.

**Root Cause:** Using `new Date().toISOString()` which always returns UTC time instead of user's timezone.

**Solution Implemented:**

1. **Separate File and Folder Operations**
   - Check `isFolder` field in backup job items.
   - Use `executeCopyOperation()` for files.
   - Use `executeFolderCopyOperation()` for folders.
   - Properly handle both operation types in parallel.

2. **Fix Timestamp Timezone**
   - Use `formatInTimeZone()` from `date-fns-tz` library.
   - Format timestamp in user's configured timezone.
   - Store timezone in backup job metadata.

**Files Modified:**
- `src/lib/backup/execute-backup-job.ts` - Separated file and folder operations, fixed timezone handling.

**Status:** ✅ Fixed - Commit: f39ad9d3

---

#### 6. Folder Backup File ID Resolution Error (October 18, 2025)

**Problem:** Folder backups failed with misleading "User not authenticated" error. Real error was "directory not found" because rclone was given an ID instead of a path.

**Root Cause:** `resolveFileId()` was called without `userId` parameter. Without `userId`, couldn't authenticate to query cloud service API. Resolution failed silently, and rclone received raw ID instead of resolved path.

**Solution Implemented:**
- Added `userId` parameter to `resolveSourceAndDestPaths()` function.
- Ensured `userId` is passed through the entire resolution chain.
- Updated all callers to provide `userId` parameter.

**Files Modified:**
- `src/lib/rclone/services/transfer-utils.ts` - Added userId to MinimalTransferOp interface and resolveSourceAndDestPaths function.
- `src/lib/rclone/services/folder-copy-service.ts` - Updated to pass userId.
- `src/lib/rclone/services/copy-service.ts` - Updated to pass userId.

**Status:** ✅ Fixed - Commit: 51a002a4

---

#### 7. Vercel Deployment Protection Blocking Internal API Calls (October 18, 2025)

**Problem:** Scheduled backups failed with 401 Authentication Required error. Vercel Deployment Protection was blocking internal server-to-server HTTP calls.

**Root Cause:** Vercel's deployment protection intercepts requests before they reach Next.js middleware, even for internal server-to-server fetch() calls.

**Solution Implemented:**

**Architecture Change:** Direct function calls instead of HTTP requests.

**Before:**
```
Cron Endpoint → HTTP fetch() → Execute Endpoint → HTTP fetch() → Batch Copy Endpoint
                     ↓ 401                              ↓ 401
              Vercel Protection              Vercel Protection
```

**After:**
```
Cron Endpoint → executeBackupJob() → executeBatchCopyOperations()
                (direct function call)   (direct function call)
```

**Implementation:**
- Created shared backup execution logic in `src/lib/backup/execute-backup-job.ts`.
- Cron endpoint calls `executeBackupJob()` directly.
- Execute endpoint also calls `executeBackupJob()` directly.
- No HTTP requests between server-side endpoints.

**Files Modified:**
- `src/lib/backup/execute-backup-job.ts` - Created shared execution logic.
- `src/app/api/cron/execute-backups/route.ts` - Updated to call executeBackupJob directly.
- `src/app/api/jobs/[id]/execute/route.ts` - Updated to call executeBackupJob directly.

**Benefits:**
- No HTTP requests = No deployment protection issues.
- Reusable logic for both cron and manual execution.
- Easier to test (no need to mock HTTP calls).
- Better performance (no HTTP overhead).

**Status:** ✅ Fixed - Commit: a0cc8120

---

#### 8. React State Timing Issue - Files Copied to Wrong Location (October 27, 2025)

**Problem:** During "Run Once" backups, timestamped backup folders (e.g., `backup-2025-10-27_10-22-31`) were created correctly, but files were being copied adjacent to (next to) the timestamped folder instead of inside it.

**Root Cause:** React state update timing issue. The backup operation flow was:
1. Create timestamped folder with ID `id:2V5kn5RRfnAAAAAAAADx3A`
2. Call `setDestinationFolderId(folderId)` to update state
3. Immediately call `handleTransferBatchInternal()` to start file transfer
4. **Problem:** React state updates are asynchronous, so `handleTransferBatchInternal()` was still using the OLD `destinationFolderId` value (`"root"`) instead of the new timestamped folder ID
5. Files were copied to Dropbox root instead of inside the timestamped folder

**Evidence from Logs:**
```
[DEBUG] [FILE-ID-RESOLVER] Resolved Dropbox ID id:2V5kn5RRfnAAAAAAAADx3A to path: /backup-2025-10-27_10-22-31
[INFO] [COPY-SERVICE] ✅ Resolved paths for copy operation:
[INFO] [COPY-SERVICE]   Source: google_google_rhounslow_1761257642198_996:test-files/create_all_formats.py
[INFO] [COPY-SERVICE]   Dest: dropbox_default:
[INFO] [COPY-SERVICE]   FileName: create_all_formats.py
```

The destination path was empty (`dropbox_default:`) because the state hadn't updated yet.

**Solution Implemented:**

**Architecture Change:** Pass folder ID directly instead of relying on state updates.

**Before:**
```typescript
const backupResult = await createRunNowBackup({...});
// State update happens in callback
await handleTransferBatchInternal(); // Uses stale state value
```

**After:**
```typescript
const backupResult = await createRunNowBackup({...});
// Validate folder ID exists
if (!backupResult?.folderId) {
  throw new Error("Failed to create timestamped backup folder");
}
// Pass folder ID directly
await handleTransferBatchInternal(backupResult.folderId);
```

**Implementation Details:**

1. **Modified `handleTransferBatchInternal` function:**
   - Added optional parameter: `overrideDestinationFolderId?: string`.
   - Uses override if provided, otherwise falls back to state value.
   - All references to `destinationFolderId` now use `effectiveDestinationFolderId`.

2. **Updated file transfer requests:**
   ```typescript
   destFolderId: effectiveDestinationFolderId, // Instead of destinationFolderId
   ```

3. **Updated folder transfer requests:**
   ```typescript
   destFolderId: effectiveDestinationFolderId, // Instead of destinationFolderId
   ```

4. **Modified `handleBackupOperation` function:**
   - Added validation to ensure `backupResult?.folderId` exists.
   - Passes timestamped folder ID directly to transfer function.
   - Throws descriptive error if folder creation fails.

5. **Updated misleading comment:**
   - Changed from: `// Update destination folder ID for the transfer`.
   - To: `// Update destination folder ID for UI display (not used for transfer - passed directly)`.

**Files Modified:**
- `src/components/BatchFileTransferDialog.tsx` - Added override parameter to handleTransferBatchInternal, updated all transfer requests, added validation.

**Benefits:**
- ✅ Files now copied inside timestamped backup folder as intended.
- ✅ No dependency on React state update timing.
- ✅ Clearer separation between UI state and business logic.
- ✅ Better error handling with explicit validation.
- ✅ Extensible pattern for future transfer types.

**Code Review Findings:**
- ✅ DRY: Pattern applied consistently across file and folder transfers.
- ✅ Separation of Concerns: Backup logic separate from transfer logic.
- ✅ SOLID Principles: Single responsibility maintained, extensible design.
- ✅ Readability: Self-documenting variable names, clear comments.
- ✅ No similar timing issues found elsewhere in codebase.

**Status:** ✅ Fixed - Commit: 069fa5cc

---

### Common Patterns in Bug Fixes

**Authentication Issues:**
- Always pass `userId` through the entire call chain.
- Server-side code needs explicit userId parameter.
- Client-side code can use Clerk's `useUser()` hook.

**Server-Side URL Construction:**
- Never use `NEXT_PUBLIC_*` variables in server-side code.
- Use `getServerApiUrl()` utility for server-side fetch calls.
- Prefer direct function calls over HTTP requests for internal communication.

**File vs Folder Operations:**
- Always check `isFolder` field when processing items.
- Use appropriate service for each type (copy-service vs folder-copy-service).
- Handle both operation types in parallel for better performance.

**Timezone Handling:**
- Always use user's configured timezone for timestamps.
- Use `date-fns-tz` library for timezone-aware formatting.
- Store timezone in job metadata for consistency.

**React State Management:**
- Never rely on state updates completing before subsequent function calls.
- Pass values directly as parameters when immediate execution is needed.
- Use state for UI rendering, parameters for business logic.
- Add validation for critical values before proceeding with operations.
- Log both state and override values for debugging timing issues.

---

## Future Enhancements

### Planned Features

1. **Incremental Backups** - Only backup changed files
2. **Backup Versioning** - Keep multiple versions of backups
3. **Backup Verification** - Verify backup integrity after completion
4. **Backup Compression** - Compress files before backup
5. **Backup Encryption** - Encrypt backups for security
6. **Backup Retention Policies** - Automatically delete old backups
7. **Backup Templates** - Save backup configurations as templates
8. **Backup Analytics** - Track backup history and statistics

### Performance Optimizations

1. **Parallel Transfers** - Transfer multiple files simultaneously
2. **Chunked Uploads** - Split large files into chunks
3. **Resume Support** - Resume interrupted backups
4. **Bandwidth Throttling** - Limit bandwidth usage
5. **Smart Scheduling** - Avoid peak usage times

---

## Related Documentation

- [Backup Scheduler Architecture](./diagrams/backup-scheduler-architecture.md) - Detailed architecture diagrams.
- [Rclone Usage](./RCLONE_USAGE.md) - Rclone service documentation.
- [File Management](./FILE_MANAGEMENT.md) - File operations documentation.
- [Testing](./TESTING.md) - Testing strategies and guidelines.

---

**For Questions or Issues:** Check the FAQ or contact support.



## Large backup mode and watchdog behavior (2026-04)

- Backup root launches now evaluate a shared profile helper using `estimatedSize`, `estimatedFiles`, `estimatedFolders`, source-count, and root/unknown-estimate fallback.
- When the large profile is active (`large-backup`), launches pass explicit `operationTimeoutMs` (24h) and longer `stallTimeoutMs` to fly-rclone.
- fly-rclone stall detection now treats metadata activity (`Checks`, `Listed`, current file transitions, output churn, and nonzero speed) as meaningful liveness.
- This prevents valid metadata-heavy transfers (hundreds of thousands of files/folders) from being cancelled around the previous 30-minute absolute timeout or short stall windows.
- Timeout cancellations are still terminal failures (`cancelReason: "timeout"`) so parent reconciliation distinguishes watchdog timeouts from user-requested cancellations.