This document describes the automatic OAuth token refresh functionality implemented to prevent sync jobs from silently failing due to expired authentication tokens.
Problem Statement
Sync jobs were failing because OAuth tokens expired and the automatic token refresh was not working properly. The issues were:
Silent Failures: Sync jobs completed with status "completed" but files weren't synced
No Token Validation: No token validation or refresh before sync operations
Errors Not Surfaced: Errors were caught but not surfaced to users
No User Notification: No mechanism to notify users of authentication failures
Solution Architecture
1. Unified Token Refresh Utility
File: src/lib/auth/unified-token-refresh.ts
This module provides automatic OAuth token refresh for all cloud storage services (Google Drive, OneDrive, Dropbox) and works in both session-based (user-initiated) and non-session (cron job) contexts.
Key Functions:
areTokensExpired(tokens): Check if tokens are expired or expiring soon (within 5 minutes).
getTokensWithAutoRefresh(options): Get tokens with automatic refresh if expired.
getTokensWithAutoRefresh({ forceRefresh: true, ... }): Refresh even when the access token has not reached the expiry buffer, used by background job launch preflight.
getTokensWithAutoRefreshOrThrow(options): Convenience wrapper that throws on failure.
batchRefreshTokens(requests): Batch refresh tokens for multiple services/accounts.
Token Refresh Flow:
1. Get current tokens from database or session
2. Check if tokens are expired (within 5-minute buffer), unless forceRefresh is set
3. If expired or forceRefresh is set:
a. Check if refresh token is available
b. Call service-specific refresh function
c. If refresh succeeds, return new tokens
d. If refresh fails, return error with needsReauth flag
4. If not expired, return current tokens
2. Background Job Token Preflight
File: src/lib/jobs/background-token-preflight.ts
Scheduled backups and syncs now proactively refresh every unique source and
destination account before provider validation, folder creation, or Fly rclone
launch. This keeps background jobs running with fresh access tokens and persists
rotated refresh tokens in Neon before rclone receives its temporary config.
The preflight:
Deduplicates repeated accounts across multi-source jobs.
Calls getTokensWithAutoRefresh({ forceRefresh: true }) in cron context.
Continues with the existing unexpired access token if a proactive refresh has a transient provider failure.
Treats invalid_grant, interaction_required, and consent-required provider errors as reconnect-required.
Uses non-reconnect wording for transient provider/configuration failures so the UI does not send users through OAuth unnecessarily.
3. Integration with Rclone Config Generation
File: src/lib/rclone/core/flyio-client.ts
Updated createRcloneConfigForFlyIo() to use the new unified token refresh:
Before:
// Get tokensconst sourceTokens =awaitgetTokensOrThrow({...});// Check if expiredif(!isTokenValid(sourceTokens)){// Attempt refresh using session-serverconst refreshedTokens =awaitrefreshTokens(...);// ...}
After:
// Get tokens with automatic refreshconst sourceTokens =awaitgetTokensWithAutoRefreshOrThrow({ service: sourceService, accountId: finalSourceAccountId, userId,});
This ensures that:
Tokens are validated before every sync operation.
Expired tokens are automatically refreshed.
Refresh failures throw descriptive errors.
Works for both source and destination services.
4. Enhanced Error Handling in Sync Jobs
File: src/lib/sync/execute-sync-job.ts
Updated error handling to detect and surface authentication errors:
catch(error){const errorMsg = error instanceofError? error.message:String(error);// Check if error is due to token expiration/authenticationconst isAuthError = errorMsg.toLowerCase().includes('token')|| errorMsg.toLowerCase().includes('auth')|| errorMsg.toLowerCase().includes('expired')|| errorMsg.toLowerCase().includes('disconnect and reconnect');// Update item status with errorawait db.update(syncJobItems).set({ status:"failed", error: errorMsg, updatedAt: now,});// If it's an auth error, also update the job's last_error to surface itif(isAuthError){await db.update(syncJobs).set({ lastError: errorMsg, updatedAt: now,});}}
This ensures that:
Authentication errors are detected.
Errors are stored in both syncJobItems.error and syncJobs.lastError.
Errors can be queried and surfaced to users.
5. User Notification System
API Endpoint
File: src/app/api/sync/check-auth-errors/route.ts
Created an API endpoint to check for sync jobs with authentication errors:
Periodically checks for sync jobs with authentication errors (default: every 60 seconds).
Shows toast notifications for each affected service.
Only shows each notification once per session.
Provides guidance to disconnect and reconnect accounts.
Usage:
import{ useSyncAuthErrorNotifications }from'~/hooks/useSyncAuthErrorNotifications';functionMyComponent(){// Check for auth errors every minuteuseSyncAuthErrorNotifications();return<div>...</div>;}
Jobs Page Feedback
Files:
src/lib/jobs/job-error-feedback.ts.
src/lib/mappers/job-mappers.ts.
src/components/jobs/JobErrorNotice.tsx.
src/app/user/jobs/page.tsx.
The Jobs API now maps stored raw launch errors into human-safe feedback before
returning list/detail responses. Authentication failures such as raw
[SYNC-SERVICE] launch errors are displayed as provider-specific reconnect
prompts with a link to /user/accounts/connect?service=.... The raw database
error remains available server-side for operators, but the Jobs page does not
show internal service tags or provider item IDs to users.
If a raw rclone auth error does not name the provider, the Jobs API falls back to
the job payload and offers reconnect actions for the candidate providers instead
of showing a generic "Storage account" prompt.
Service-Specific Token Refresh
The implementation uses existing service-specific refresh functions from src/lib/service-token-refresh.ts:
Google Drive: Uses Google OAuth2 client to refresh tokens.
OneDrive: Uses Microsoft OAuth endpoint to refresh tokens.
Dropbox: Uses Dropbox OAuth endpoint to refresh tokens.
Each refresh function:
Takes the current refresh token
Calls the service's OAuth endpoint
Returns new access token and expiry date
Updates tokens in the database
Returns { success, tokens, error, needsReauth } result
Error Messages
User-friendly error messages are generated for token refresh failures:
// When refresh token is invalid/expired"Google Drive authentication has expired. Please disconnect and reconnect your Google Drive account to continue.";// When refresh fails for other reasons"Failed to refresh authentication for google:default. Please try again.";
Testing
To test the implementation:
Simulate Token Expiration:
Manually set token expiry date to past in database.
Trigger a sync operation.
Verify automatic refresh occurs.
Simulate Refresh Failure:
Manually invalidate refresh token in database.
Trigger a sync operation.
Verify error is caught and surfaced.
Verify User Notification:
Create sync job with expired tokens.
Wait for cron job to execute.
Verify toast notification appears in UI.
Benefits
Automatic Recovery: Tokens are automatically refreshed before operations and proactively force-refreshed before scheduled backup/sync launches
No Silent Failures: Authentication errors are detected and surfaced
Human Intervention Only When Required: Users are asked to reconnect only when the provider rejects the refresh token or requires interactive consent
Multi-Service Support: Works for all cloud storage services
Context-Aware: Works in both user-initiated and cron job contexts
Consistent Error Handling: Standardized error detection and reporting
Future Enhancements
Email Notifications: Send email when manual reconnection is required
Auto-Pause Jobs: Automatically pause sync jobs with auth errors
Retry Logic: Implement exponential backoff for transient failures