# Streaming with SSE and NDJSON StratoFusion uses two HTTP streaming formats. Server-Sent Events (SSE) carries server-to-browser updates, usually over a longer-lived `GET`. Newline-delimited JSON (NDJSON) carries a finite sequence of records while a `POST` validates or starts a batch operation. Streaming changes when the client receives results. It does not relax route authentication, ownership checks, quotas, validation, or cancellation rules. ## Choosing a format | Question | SSE | NDJSON | | --------------- | ------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | | Wire format | `event:` and `data:` fields terminated by a blank line | One complete JSON value per line | | Typical request | Long-lived or progressively consumed `GET` | Finite, progressively consumed `POST` | | Current purpose | Job snapshots, operation progress, and search results | Batch launch, delete progress, and download-manifest resolution | | Browser client | `EventSource` when automatic reconnection fits; streaming `fetch` when explicit abort/parsing is needed | Streaming `fetch` with a `ReadableStream` reader | | Completion | Explicit terminal event, rollover, abort, or disconnect | `complete`/`error` record followed by stream close | Use ordinary JSON when a request has one response and incremental feedback does not improve the workflow. ## Current SSE endpoints | Endpoint | Producer | Browser consumer | Events and lifecycle | | -------------------------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GET /api/jobs/events` | `src/app/api/jobs/events/route.ts` | `src/hooks/useJobUpdates.ts` | Authenticated, user-scoped `update`, `error`, and `reconnect` events. The server checks for changed job snapshots every 2 seconds, sends a heartbeat every 15 seconds, and rolls the stream at 270 seconds. | | `GET /api/search/indexing/jobs/events` | `src/app/api/search/indexing/jobs/events/route.ts` | `src/hooks/useAiIndexingJobUpdates.ts` | Authenticated, user-scoped `update`, `error`, and `reconnect` events for recent AI indexing jobs. It shares the job-stream timing constants and only sends changed snapshots. | | `GET /api/ops?id={operationId}` | `src/app/api/ops/route.ts` | `src/lib/rclone/services/transfer-service.ts` | Requires authentication and verifies that the operation belongs to the user before streaming `update`, `done`, or `error`. The route samples worker state every 3 seconds and emits a heartbeat every 15 seconds. | | `GET /api/search/stream?...` | `src/app/api/search/stream/route.ts` re-exports `src/lib/search/orchestrators/streaming-search-orchestrator.ts` | `src/hooks/useStreamingSearch.ts` and `src/contexts/ServiceManagerContext.tsx` | Authenticated finite search stream. JSON payloads carry `search_started`, `service_started`, `service_results`, `service_error`, `service_complete`, and `search_complete` types. | The job and operation clients use `EventSource`. Streaming search uses `fetch` because the hook explicitly owns request construction, cancellation, response validation, and parsing. ### SSE encoding The job and operation routes use named events: ```text event: update data: {"jobs":[]} ``` Streaming search uses an unnamed SSE message whose JSON body contains its own event type: ```text data: {"type":"search_started","data":{},"timestamp":0} ``` In both forms, the blank line terminates one event. A client must retain an incomplete trailing chunk until the next read instead of assuming that network chunks align with event boundaries. ## SSE fallback and cleanup The shared job constants live in `src/lib/jobs/job-events.ts`: - job snapshot interval: 2 seconds; - heartbeat interval: 15 seconds; - initial connection timeout: 3 seconds; - reconnect delay: 1 second; - server rollover: 270 seconds, before the routes' 300-second maximum. `useJobUpdates` falls back to `GET /api/jobs` every 8 seconds when a fresh SSE connection cannot open. `useAiIndexingJobUpdates` falls back to `GET /api/search/indexing/jobs/recent?limit=20` every 5 seconds. After an established job stream rolls over or disconnects, the hooks schedule a new SSE connection rather than permanently switching to polling. Operation monitoring falls back to the existing operation-status polling path when `EventSource` is unavailable, the stream does not open within 3 seconds, the browser receives an error, or no SSE activity is observed for 15 seconds. Streaming search reports a request error and does not have a polling fallback. Every producer and consumer must clear intervals, readers, event sources, and abort listeners when the request completes, fails, is replaced, or unmounts. ## Current NDJSON endpoints | Endpoint | Producer | Consumer | Records and lifecycle | | ------------------------------------- | ----------------------------------------------- | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `POST /api/rclone/batch-copy/stream` | `src/app/api/rclone/batch-copy/stream/route.ts` | `src/lib/rclone/services/transfer-service.ts` | Authenticates, validates, filters, checks operation and transfer quotas, then emits launch progress such as `started`, `validated`, `filtered`, `operation-launching`, `operation-started`, `operation-failed`, `complete`, or `error`. | | `POST /api/rclone/batch-move/stream` | `src/app/api/rclone/batch-move/stream/route.ts` | `src/lib/rclone/services/transfer-service.ts` | Uses the same finite launch-stream contract for move operations. Move remains potentially destructive; streaming does not change that safety boundary. | | `POST /api/delete/stream` | `src/app/api/delete/stream/route.ts` | `src/application/files/delete-files.ts` | Authenticates before opening the stream, then emits `start`, heartbeat/progress, `item_complete`, `complete`, or `error` records while processing the requested delete targets. | | `POST /api/download/resolve?stream=1` | `src/app/api/download/resolve/route.ts` | `src/lib/download/resolve-manifest-client.ts` | Resolves a finite download manifest with `start`, `item_start`, `item_complete`, progress, `complete`, or `error` records while preserving browser, rate-limit, quota, and provider checks. | The shared batch-transfer encoder and response headers are in `src/app/api/rclone/_shared/ndjson.ts`. NDJSON responses use `Content-Type: application/x-ndjson; charset=utf-8` and `Cache-Control: no-store`. ### NDJSON encoding Each line is independently parseable JSON: ```text {"type":"started","batchId":"batch_example","totalOperations":2} {"type":"operation-started","index":0,"operationId":"op_example"} {"type":"complete","data":{"operationIds":["op_example"]}} ``` The client must buffer a partial final line, parse only newline-terminated records during each read, and flush a non-empty buffered record at end of stream. It must also inspect streamed `error` records. Some streaming routes return an HTTP 200 after the stream is established and carry the effective error status in the record's `httpStatus` field. ## Security and architecture boundaries - Authenticate and validate ownership before emitting user or operation data. - Keep provider-specific IDs, paths, and errors behind existing services and canonical response types; a streaming transport is not a domain boundary. - Apply operation limits, transfer quotas, provider quotas, rate limits, and destructive-operation policy before costly work starts. - Use `Cache-Control: no-cache, no-transform` and disable proxy buffering for SSE; use `Cache-Control: no-store` for the current NDJSON streams. - Treat request abort as cancellation of remaining server work where the route supports it. Closing a browser stream does not by itself undo operations that have already started. - Do not include credentials, tokens, raw provider responses, customer data, or private logs in event payloads or diagnostics. ## Implementation checklist When adding or changing a stream: 1. Define the event union and terminal/error behavior before the route. 2. Keep authentication, validation, ownership, quota, and service orchestration outside the encoder. 3. Set the exact streaming content type and cache headers. 4. Buffer incomplete chunks in the client parser. 5. Handle request abort, stream close, timers, and duplicate connection cleanup. 6. Test authorization failure, partial chunks, event errors, terminal events, cancellation, and fallback behavior where applicable. 7. Update this inventory and `public/docs/API_REFERENCE.md` if the API contract changes. ## Focused verification Representative deterministic tests are: ```powershell pnpm test src/app/api/jobs/events/route.test.ts ` src/app/api/search/indexing/jobs/events/route.test.ts ` src/hooks/__tests__/useJobUpdates.test.tsx ` src/hooks/__tests__/useAiIndexingJobUpdates.test.tsx pnpm test src/app/api/delete/stream/__tests__/route.test.ts ` src/application/files/delete-files.test.ts ` src/lib/__tests__/resolve-stream.test.ts ` src/lib/download/__tests__/resolve-manifest-client.test.ts ``` Add the focused route, hook, service, or parser test for the stream being changed. Broaden to `pnpm check` and the relevant integration or browser loop when behavior—not only documentation—changes. ## References - [API reference](API_REFERENCE.md) - [Architecture](ARCHITECTURE.md) - [Rclone service](RCLONE_SERVICE.md) - [Testing](TESTING.md) - [MDN: Using server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) - [NDJSON specification](https://github.com/ndjson/ndjson-spec)