# AI Search Phase 7: Indexing Observability and Diagnostics Prompt **Status:** Ready for implementation **Priority:** High **Dependencies:** Phase 1 PDF, Phase 2 DOCX, Phase 3 XLSX, Phase 4 PPTX, Phase 5 Google Workspace Export, Phase 6A OCR Foundation --- ## Review Notes Incorporated - Uses the current `/api/search/indexing/...` route namespace, not `/api/ai/indexing/...`. - Builds on the existing AI indexing tables: `ai_indexing_jobs`, `ai_indexing_job_items`, `ai_indexed_files`, `ai_indexing_dirty_scopes`, and `ai_indexing_reconciliation_cursors`. - Builds on the existing `/user/ai-indexing` UI, SSE/polling job updates, failure dialog, cancel, retry, dismiss, and batch-dismiss controls. - Treats telemetry as privacy-sensitive: do not log file contents, extracted text, raw provider paths, raw provider errors, OAuth tokens, or user document names in structured application logs. - Keeps UI render-focused. Business logic belongs in `src/lib/search/ai/indexing`, database modules, API routes, and hooks. - Narrows Phase 7 to diagnostics, durable telemetry, error taxonomy, and user-facing explanations. Alerts, budgets, dynamic tuning, multi-tenant analytics, cloud OCR providers, and PDF OCR remain future work. --- ## Prompt ```text You are working in C:\code\stratofusion on Windows 11 PowerShell. Do not use WSL or bare bash. Use pnpm. First: - Run `git status --short` and inspect relevant `git diff`. - If there are unrelated uncommitted changes, summarize them and do not overwrite, revert, clean, commit, or stage them unless explicitly approved. - Run `pnpm env:guard`. - Read README.md and the relevant canonical docs under public/docs before coding: - public/docs/developer/AI_OPERATING_PROTOCOL.md - public/docs/SEARCH_FEATURES.md - public/docs/API_REFERENCE.md - public/docs/ARCHITECTURE.md - public/docs/PRIVACY_ARCHITECTURE.md - public/docs/TESTING.md - public/docs/DEPLOYMENT.md if environment behavior changes - For this non-trivial feature, use the feature intake guidance in public/docs/developer/AI_OPERATING_PROTOCOL.md. - Review the existing AI indexing implementation before editing: - src/lib/database/schema/ai-indexing.ts - src/lib/database/ai-indexing-jobs.ts - src/lib/database/ai-indexed-files.ts - src/lib/database/ai-indexed-file-state.ts - src/lib/search/ai/indexing/backend-indexing-runner.ts - src/lib/search/ai/indexing/backend-indexing-runner-support.ts - src/lib/search/ai/indexing/database-progress-writer.ts - src/lib/search/ai/indexing/job-service.ts - src/lib/search/ai/indexing/job-list.ts - src/lib/search/ai/indexing/activity-log.ts - src/lib/search/ai/extraction/extract-document.ts - src/lib/search/ai/extraction/types.ts - src/lib/search/ai/extraction/pdf-text.ts - src/lib/search/ai/extraction/docx-text.ts - src/lib/search/ai/extraction/xlsx-text.ts - src/lib/search/ai/extraction/pptx-text.ts - src/lib/search/ai/extraction/google-workspace-export.ts - src/lib/search/ai/extraction/ocr/ocr-error.ts - src/app/api/search/indexing/jobs/[jobId]/route.ts - src/app/api/search/indexing/jobs/events/route.ts - src/app/user/ai-indexing/page.tsx - src/components/jobs/AiIndexingJobsSection.tsx - src/components/jobs/AiIndexingJobCard.tsx - src/components/jobs/AiIndexingJobFailureDialog.tsx - src/hooks/useAiIndexingJobUpdates.ts Goal: implement AI Search Phase 7, adding indexing observability and diagnostics so users can understand indexing progress, failures, extraction coverage, and likely remediation without support intervention. Important current-state constraints: - AI indexing endpoints live under `/api/search/indexing`. - Existing job controls already include cancel, retry, dismiss, and completed-job batch dismiss. Do not rebuild those workflows. - Existing durable scopes are `account` and `folder`. Do not invent a single-file scope unless the implementation deliberately extends the shared `IndexingScope` type and all affected services. - Provider/account/resource identity is the boundary. Persist and query by `service`, `accountId`, and `resourceId`; do not rely on provider paths as business truth. - UI may display authorized file names already stored in user-owned database rows, but structured application logs must avoid file names, raw provider paths, extracted text, snippets, and raw provider error payloads. - Do not add cloud OCR providers, scanned PDF OCR, cost budgets, dynamic batch tuning, job scheduling, Slack/email alerts, or multi-tenant admin analytics in this phase. Scope: 1. Durable telemetry fields - Extend Drizzle schema and generate a migration for the minimum fields needed to diagnose a job without parsing logs. - Prefer durable item-level telemetry on `ai_indexing_job_items` for run-specific facts: - `processing_stage` or `failure_stage` - `extraction_method` - `extraction_duration_ms` - `extraction_error_code` - `extraction_error_message` - `extraction_retryable` - `text_length_chars` - `token_count` - `chunk_count` - `embedding_duration_ms` - `semantic_upsert_duration_ms` - `ocr_provider` - `ocr_confidence` - `ocr_image_count` - Add durable latest-file telemetry to `ai_indexed_files` only where it is needed for file status and search coverage diagnostics: - latest `extraction_method` - latest `extraction_error_code` - latest sanitized `extraction_error_message` - latest `extraction_retryable` - latest `extraction_duration_ms` - latest OCR metadata - latest `text_length_chars` - Avoid denormalized JSON summaries on `ai_indexing_jobs` unless query cost proves they are needed. Derive method breakdown, error breakdown, latency, and throughput from item rows first. - Add indexes only for real diagnostics queries, for example `(user_id, job_id, status)`, `(user_id, service, account_id, resource_id)`, and `(user_id, extraction_error_code)`. 2. Error taxonomy - Add a focused taxonomy module, for example: - src/lib/search/ai/indexing/error-taxonomy.ts - src/lib/search/ai/indexing/error-taxonomy.test.ts - Define stable user-facing failure codes such as: - `pdf_encrypted`, `pdf_parse_failed`, `pdf_timeout`, `pdf_no_text` - `docx_encrypted`, `docx_parse_failed`, `docx_timeout`, `docx_no_text` - `extraction_xlsx_encrypted`, `extraction_xlsx_parse_failed`, `extraction_xlsx_no_text` - keep legacy `xlsx_encrypted`, `xlsx_parse_failed`, `xlsx_timeout`, `xlsx_no_text` readable for stored records - `extraction_pptx_encrypted`, `extraction_pptx_parse_failed`, `extraction_pptx_no_text` - keep legacy `pptx_encrypted`, `pptx_parse_failed`, `pptx_timeout`, `pptx_no_text` readable for stored records - `google_export_quota_or_permission`, `google_export_network`, `google_export_timeout`, `google_export_unsupported_type` - `ocr_timeout`, `ocr_quota_or_cost_limit`, `ocr_provider_unavailable`, `ocr_invalid_provider_config`, `ocr_network_or_provider_failure`, `ocr_no_text`, `ocr_low_confidence` - `unsupported_file_type`, `file_too_large`, `download_failed`, `embedding_failed`, `semantic_index_failed`, `job_cancelled`, `unknown_error` - Map existing typed extraction errors to the taxonomy: - PdfExtractionError, DocxExtractionError, XlsxExtractionError, PptxExtractionError - GoogleWorkspaceExportError - OcrExtractionError - Convert unsupported extraction results and known skip reasons into taxonomy codes where possible. - Include `retryable` and `recommendation` in the taxonomy. Retryable examples: provider timeout, temporary network failure, quota/rate limit after cooldown, provider unavailable. Non-retryable examples: unsupported type, encrypted file, parser-corrupt file, file too large, no extractable text unless OCR/config changes could help. - Sanitize all messages. Never expose raw provider responses, access tokens, file contents, extracted text, or stack traces to users or logs. 3. Runner instrumentation - Instrument the existing BackendIndexingRunner without changing successful indexing semantics. - Capture per-item timings around: - provider download - extraction/export/OCR - chunking - embedding - semantic index upsert - Derive `extraction_method` from the extraction policy and metadata: - `plain_text`, `pdf`, `docx`, `xlsx`, `pptx`, `google_workspace_export`, `ocr_image`, `unsupported`, or `unknown` - Persist telemetry through the existing progress writer/database module path. Keep database writes cohesive so item status and telemetry do not drift apart. - When an item is skipped, failed, indexed, or cancelled, persist its taxonomy code, retryability, sanitized message, and stage. - Add structured logs only for aggregate events and safe identifiers: - job id - user id only where existing log policy allows it - service - account id - status - counts - method/error codes - durations Do not log file names, raw provider paths, document text, OCR text, snippets, or raw provider errors. 4. Diagnostics services - Add service-level builders under `src/lib/search/ai/indexing` rather than putting aggregation logic in API routes. - Suggested modules: - `diagnostics.ts` for job diagnostics and file status builders - `health.ts` for account/folder health metrics if it stays small, otherwise split by responsibility - `diagnostics.test.ts` and `health.test.ts` - Job diagnostics should return: - current job snapshot and progress - processed/remaining counts - estimated completion only when throughput is meaningful; otherwise return null - extraction method breakdown - failure/skip breakdown by taxonomy code - retryable vs non-retryable counts - average, p50, p95, and p99 extraction duration for completed item attempts - slowest safe item summaries using authorized resource names from DB, not logs - failed/skipped item samples capped to a small limit - Health metrics should support account and folder scope and return: - indexed, failed, unsupported/skipped, pending, stale/deleted counts - coverage percent based on current reconciled state where possible - success rate by MIME family or extraction method - OCR count, average confidence, and low-confidence count when OCR metadata exists - recent jobs for the requested scope - File status diagnostics should accept `service`, `accountId`, and `resourceId` as query parameters, then return: - current indexed-file state if present - latest job item state if present - last indexed timestamp - extraction/index status - taxonomy code, message, retryability, and recommendation - OCR metadata - chunk count and text length when available - whether reindex is recommended due to dirty/reconciliation state 5. API endpoints - Keep API routes thin: auth, validation, service call, response mapping. - Use existing response helpers and error conventions. - Add endpoints under the current namespace: - `GET /api/search/indexing/jobs/[jobId]/diagnostics` - `GET /api/search/indexing/health?service=...&accountId=...&scopeType=account|folder&resourceId=...` - `GET /api/search/indexing/files/status?service=...&accountId=...&resourceId=...` - Do not add `/api/ai/indexing`. - Do not add a single-file reindex endpoint unless you extend `IndexingScope` intentionally. For Phase 7, prefer a reindex recommendation that points users to existing account/folder retry or re-index controls. - Validate that the requested job, account, folder, or resource belongs to the authenticated user. - Add route tests for authentication failure, missing/invalid parameters, not found, service failures, and success. 6. UI diagnostics - Build on the existing `/user/ai-indexing` surface and job cards. - Prefer extending `AiIndexingJobFailureDialog` into a diagnostics dialog or adding a sibling `AiIndexingJobDiagnosticsDialog` under `src/components/jobs`. - Keep UI render-only. Fetching belongs in hooks; aggregation belongs in services/API. - Add: - job diagnostics dialog with progress, ETA when reliable, method breakdown, error groups, retryability, and failed/skipped samples - compact health summary on `/user/ai-indexing` - file indexing status display only where the file browser already has service/account/resource identity available - Use existing UI primitives and lucide icons. Keep panels dense and operational; do not build a marketing-style dashboard. - Avoid card nesting. Use tables, compact lists, progress bars, badges, and dialogs. - Ensure copy is user-safe and actionable: - "This file is unsupported" with supported-type guidance - "OCR is disabled" with configuration guidance only for admins/developers where appropriate - "Google export hit a quota or permission limit" with reconnect/retry guidance - "This file has no extractable text" with OCR/scanned PDF limitation guidance - Do not expose provider-specific IDs, raw provider paths, stack traces, or raw provider errors in UI. 7. Documentation - Update canonical docs for behavior changes: - public/docs/SEARCH_FEATURES.md: user-visible diagnostics, coverage, limitations - public/docs/API_REFERENCE.md: new diagnostics endpoints and response summaries - public/docs/ARCHITECTURE.md: telemetry/diagnostics service boundaries - public/docs/PRIVACY_ARCHITECTURE.md: telemetry redaction and no-content logging guarantees - public/docs/TESTING.md: targeted diagnostics test coverage if a new testing pattern is added - public/docs/DEPLOYMENT.md only if env vars or operational config change - Update README.md only if the top-level AI indexing description changes. - Keep docs concise and consistent with existing AI search language. Suggested implementation order: 1. Add taxonomy tests, then taxonomy implementation. 2. Add schema fields and migration. 3. Extend database update types/functions so telemetry can be persisted with item and indexed-file state transitions. 4. Instrument BackendIndexingRunner with timing and taxonomy persistence. 5. Add diagnostics service tests and implementation. 6. Add API route tests and routes. 7. Add UI hooks and components. 8. Update docs. Testing requirements: - Unit tests: - taxonomy maps typed extraction/export/OCR errors correctly - taxonomy sanitizes messages and marks retryability correctly - diagnostics aggregations compute counts, breakdowns, p50/p95/p99, and ETA correctly - Integration/database-oriented tests: - telemetry fields persist on indexed, skipped, failed, and cancelled items - latest indexed-file diagnostics update correctly - diagnostics queries are user-scoped - Runner tests: - PDF/DOCX/XLSX/PPTX/Google Workspace/OCR success records method and duration - unsupported file records unsupported taxonomy and retryability false - timeout/provider failures record retryable taxonomy where appropriate - embedding and semantic-index failures record the correct stage - API tests: - diagnostics, health, and file-status endpoints handle auth, validation, not found, and success - Component tests: - diagnostics dialog renders method breakdown, error groups, retry guidance, and empty states - health summary renders loading/error/success states - status display does not show raw IDs or stack traces Verification: Run the smallest useful feedback loop first, then broaden as needed. At minimum: pnpm env:guard pnpm vitest run src/lib/search/ai/indexing/error-taxonomy.test.ts pnpm vitest run src/lib/search/ai/indexing/diagnostics.test.ts pnpm vitest run src/lib/search/ai/indexing/backend-indexing-runner.test.ts pnpm vitest run src/app/api/search/indexing pnpm vitest run src/components/jobs pnpm typecheck pnpm exec eslint --config .eslintrc.json src/lib/search/ai src/app/api/search/indexing src/components/jobs src/hooks If schema generation, bundling, or UI changes are broad, also run: pnpm check pnpm build Deliverable: - Do not commit unless the user explicitly asks for a commit or PR. - Summarize changed files. - Summarize the exact telemetry fields added and where they are persisted. - Summarize new diagnostics endpoints and UI entry points. - Summarize privacy safeguards for logs and API/UI messages. - Include verification command results. - Call out remaining risks and follow-ups, especially: - derived metrics may be approximate for active in-process jobs - ETA is unreliable for small jobs or bursty provider throttling - file-level status depends on current resource identity availability - OCR confidence is provider-dependent - cost estimates require explicit pricing/config data and are not part of Phase 7 unless approved ``` --- ## Next Phase Candidates - Real-time operational alerts for quota, timeout, and regression thresholds. - Explicit OCR and embedding budgets with hard limits. - Dynamic batch size and concurrency tuning. - Off-peak indexing schedules. - Admin-only multi-tenant indexing analytics. - Single-file reindex scope, if product requirements justify extending `IndexingScope`.