StratoFusion uses two HTTP streaming formats. Server-Sent Events (SSE) carries
server-to-browser updates, usually over a longer-lived GET. Newline-delimited
JSON (NDJSON) carries a finite sequence of records while a POST validates or
starts a batch operation.
Streaming changes when the client receives results. It does not relax route
authentication, ownership checks, quotas, validation, or cancellation rules.
Choosing a format
Question
SSE
NDJSON
Wire format
event: and data: fields terminated by a blank line
One complete JSON value per line
Typical request
Long-lived or progressively consumed GET
Finite, progressively consumed POST
Current purpose
Job snapshots, operation progress, and search results
Batch launch, delete progress, and download-manifest resolution
Browser client
EventSource when automatic reconnection fits; streaming fetch when explicit abort/parsing is needed
Streaming fetch with a ReadableStream reader
Completion
Explicit terminal event, rollover, abort, or disconnect
complete/error record followed by stream close
Use ordinary JSON when a request has one response and incremental feedback does
not improve the workflow.
Current SSE endpoints
Endpoint
Producer
Browser consumer
Events and lifecycle
GET /api/jobs/events
src/app/api/jobs/events/route.ts
src/hooks/useJobUpdates.ts
Authenticated, user-scoped update, error, and reconnect events. The server checks for changed job snapshots every 2 seconds, sends a heartbeat every 15 seconds, and rolls the stream at 270 seconds.
GET /api/search/indexing/jobs/events
src/app/api/search/indexing/jobs/events/route.ts
src/hooks/useAiIndexingJobUpdates.ts
Authenticated, user-scoped update, error, and reconnect events for recent AI indexing jobs. It shares the job-stream timing constants and only sends changed snapshots.
GET /api/ops?id={operationId}
src/app/api/ops/route.ts
src/lib/rclone/services/transfer-service.ts
Requires authentication and verifies that the operation belongs to the user before streaming update, done, or error. The route samples worker state every 3 seconds and emits a heartbeat every 15 seconds.
The job and operation clients use EventSource. Streaming search uses fetch
because the hook explicitly owns request construction, cancellation, response
validation, and parsing.
SSE encoding
The job and operation routes use named events:
event: update
data: {"jobs":[]}
Streaming search uses an unnamed SSE message whose JSON body contains its own
event type:
In both forms, the blank line terminates one event. A client must retain an
incomplete trailing chunk until the next read instead of assuming that network
chunks align with event boundaries.
SSE fallback and cleanup
The shared job constants live in src/lib/jobs/job-events.ts:
job snapshot interval: 2 seconds;
heartbeat interval: 15 seconds;
initial connection timeout: 3 seconds;
reconnect delay: 1 second;
server rollover: 270 seconds, before the routes' 300-second maximum.
useJobUpdates falls back to GET /api/jobs every 8 seconds when a fresh SSE
connection cannot open. useAiIndexingJobUpdates falls back to
GET /api/search/indexing/jobs/recent?limit=20 every 5 seconds. After an
established job stream rolls over or disconnects, the hooks schedule a new SSE
connection rather than permanently switching to polling.
Operation monitoring falls back to the existing operation-status polling path
when EventSource is unavailable, the stream does not open within 3 seconds,
the browser receives an error, or no SSE activity is observed for 15 seconds.
Streaming search reports a request error and does not have a polling fallback.
Every producer and consumer must clear intervals, readers, event sources, and
abort listeners when the request completes, fails, is replaced, or unmounts.
Current NDJSON endpoints
Endpoint
Producer
Consumer
Records and lifecycle
POST /api/rclone/batch-copy/stream
src/app/api/rclone/batch-copy/stream/route.ts
src/lib/rclone/services/transfer-service.ts
Authenticates, validates, filters, checks operation and transfer quotas, then emits launch progress such as started, validated, filtered, operation-launching, operation-started, operation-failed, complete, or error.
POST /api/rclone/batch-move/stream
src/app/api/rclone/batch-move/stream/route.ts
src/lib/rclone/services/transfer-service.ts
Uses the same finite launch-stream contract for move operations. Move remains potentially destructive; streaming does not change that safety boundary.
POST /api/delete/stream
src/app/api/delete/stream/route.ts
src/application/files/delete-files.ts
Authenticates before opening the stream, then emits start, heartbeat/progress, item_complete, complete, or error records while processing the requested delete targets.
POST /api/download/resolve?stream=1
src/app/api/download/resolve/route.ts
src/lib/download/resolve-manifest-client.ts
Resolves a finite download manifest with start, item_start, item_complete, progress, complete, or error records while preserving browser, rate-limit, quota, and provider checks.
The shared batch-transfer encoder and response headers are in
src/app/api/rclone/_shared/ndjson.ts. NDJSON responses use
Content-Type: application/x-ndjson; charset=utf-8 and Cache-Control: no-store.
The client must buffer a partial final line, parse only newline-terminated
records during each read, and flush a non-empty buffered record at end of
stream. It must also inspect streamed error records. Some streaming routes
return an HTTP 200 after the stream is established and carry the effective
error status in the record's httpStatus field.
Security and architecture boundaries
Authenticate and validate ownership before emitting user or operation data.
Keep provider-specific IDs, paths, and errors behind existing services and
canonical response types; a streaming transport is not a domain boundary.
Apply operation limits, transfer quotas, provider quotas, rate limits, and
destructive-operation policy before costly work starts.
Use Cache-Control: no-cache, no-transform and disable proxy buffering for
SSE; use Cache-Control: no-store for the current NDJSON streams.
Treat request abort as cancellation of remaining server work where the route
supports it. Closing a browser stream does not by itself undo operations that
have already started.
Do not include credentials, tokens, raw provider responses, customer data, or
private logs in event payloads or diagnostics.
Implementation checklist
When adding or changing a stream:
Define the event union and terminal/error behavior before the route.
Keep authentication, validation, ownership, quota, and service orchestration
outside the encoder.
Set the exact streaming content type and cache headers.
Buffer incomplete chunks in the client parser.
Handle request abort, stream close, timers, and duplicate connection cleanup.
Test authorization failure, partial chunks, event errors, terminal events,
cancellation, and fallback behavior where applicable.
Update this inventory and public/docs/API_REFERENCE.md if the API contract
changes.
Focused verification
Representative deterministic tests are:
pnpm test src/app/api/jobs/events/route.test.ts `
src/app/api/search/indexing/jobs/events/route.test.ts `
src/hooks/__tests__/useJobUpdates.test.tsx `
src/hooks/__tests__/useAiIndexingJobUpdates.test.tsx
pnpm test src/app/api/delete/stream/__tests__/route.test.ts `
src/application/files/delete-files.test.ts `
src/lib/__tests__/resolve-stream.test.ts `
src/lib/download/__tests__/resolve-manifest-client.test.ts
Add the focused route, hook, service, or parser test for the stream being
changed. Broaden to pnpm check and the relevant integration or browser loop
when behavior—not only documentation—changes.