Daniel Han e8945cab46 Whole-document context for RAG chat attachments (#6693) ... * Add whole-document context mode to RAG chat attachments Thread-attached files are injected in full when they fit a token budget, instead of only top-K retrieved chunks, so the model reads the entire file for summarize/reason-over-document requests. Oversized files fall back to top-K retrieval so the context window is never blown. KB and project corpora are unchanged (still retrieval). - core/rag/store.py: all_chunks_for_scope returns every completed-document chunk for a scope, ordered document-then-index, joined with filename. - core/rag/tool.py: whole_document_context renders the chunks as the same <chunk> blocks + citation source-map retrieval produces, returns None when empty or over budget. - core/inference/tools.py: build_rag_autoinject tries whole-document first for thread scopes, falls through to search_for_autoinject otherwise. - core/rag/config.py: THREAD_WHOLE_DOC + WHOLE_DOC_MAX_TOKENS (env-tunable). - tests/test_rag_whole_document.py: store ordering, whole-doc render + budget cutoff, auto-inject whole-doc vs top-K fallback, KB never whole-doc. * Add scanned-PDF OCR fallback to RAG ingestion A PDF page with no extractable text layer (a scanned or image-only page) previously ingested as empty, so image PDFs were invisible to retrieval and whole-document context. Such pages are now rendered and transcribed by the loaded vision model during ingestion, so they become searchable and readable like any other page. This restores OCR for the RAG document flow without a separate extraction pipeline. - core/rag/parsers.py: render_pdf_pages renders whole pages (1-based) to PNG. - core/rag/captioner.py: factor the shared vision call into _vision_complete; add _ocr_one + ocr_pages (transcribe rendered pages, OCR_MAX_PAGES bound). - core/rag/ingestion.py: _ocr_scanned_pages runs right after parse, replacing text on near-empty PDF pages. No-op when OCR is off, no page is scanned, or no vision model is loaded (degrades like figure captioning). - core/rag/config.py: OCR_SCANNED, OCR_MIN_CHARS, OCR_MAX_PAGES, OCR_DPI, OCR_TIMEOUT_S, OCR_MAX_TOKENS (env-tunable). - tests/test_rag_ocr_fallback.py: page render, ocr_pages gating + cap, scanned PDF end-to-end OCR into chunks + whole-doc, born-digital skips OCR, disabled leaves the page empty. * Broaden OCR prompt to figures/tables and guard against repetition runaway The OCR prompt now asks the vision model to also transcribe text inside figures, diagrams, charts and tables, so labels and table cells on scanned pages are indexed rather than skipped. Verified on real documents that this does not regress plain-text transcription. Some vision models loop on sparse images (e.g. a title-only cover) and emit the same line hundreds of times. _collapse_runaway caps any run of identical consecutive lines so a pathological page cannot flood the index; legitimate short repeats (a label appearing a few times) survive. Applied in ocr_pages. * Restrict whole-document injection to thread attachments only whole_document_context resolved the combined project+thread scope, so a project chat (the frontend sends both thread_id and project_id) injected the entire project corpus in full, contradicting the design that project and KB corpora stay retrieval-only. A large project corpus could also push the total over budget and drop a small thread attachment back to top-K. Resolve the thread scope alone in whole_document_context, and in build_rag_autoinject only enter whole-doc mode when a thread attachment is present and no KB is selected (a KB pick is exclusive: search that corpus). Project sources and KBs keep top-K retrieval. Adds regression tests for the mixed project+thread payload, the budget isolation, and KB precedence. * Address review: keep project retrieval, harden budget + OCR guards Follow-up to the 8-reviewer pass on the whole-document + OCR work. - Preserve project grounding in project chats. The thread-scope-only fix made whole-doc exclusive of retrieval, so a thread attachment silently dropped the project corpus for that turn. build_rag_autoinject now whole-docs the thread attachment AND retrieves the project sources top-K, merged under one citation numbering via tool.render_sources. KB selection stays exclusive. - Budget: a NULL/zero token_count no longer bypasses the cap (length-based fallback in _row_token_count), so a malformed huge doc can't inject in full. - OCR runaway guard: _collapse_runaway now also caps each distinct line at a generous total across the page (not just consecutive), bounding the interleaved/alternating loops weak models emit; blank-line floods collapse too. - OCR: warn when a scanned PDF exceeds OCR_MAX_PAGES (pages past the cap stay untranscribed) instead of silently dropping them. - Document the known limits: OCR'd pages have no PDF highlight regions; vision models need a micro-batch >= image tokens (Gemma-family) or the server aborts. - Tests for project-retrieval composition, NULL-token budget, and interleaved runaway; drop the now-superseded exclude-project test. * Add OCR toggle to RAG retrieval settings Make scanned-PDF OCR user-controllable per upload instead of only via the RAG_OCR_SCANNED config default. The retrieval settings panel gains an OCR scanned pages switch (persisted in localStorage, on by default); the chosen value is read fresh at upload time and sent with each document upload. Backend: the three upload routes accept an optional ocr form field and pass it through start_ingestion to _ocr_scanned_pages, which now treats None as use the config default and an explicit bool as an override. The on/off policy lives only in _ocr_scanned_pages now, so ocr_pages no longer re-checks the config (that double gate would have blocked a per-upload ocr=True while the default was off). Tests cover both override directions (force on while config off, force off while config on). * Add "Describe figures & charts" toggle with chart-aware captions Surface RAG figure captioning as a user control and make it actually useful for graphs and plots. The figure detection already clustered vector drawings and raster images into regions and rendered them, but captioning was off by default, had no UI, and used a thin generic prompt. Accuracy: the caption prompt now asks for chart type, axis titles and units, legend or series, salient trends and readable values, and table columns, while forbidding invented numbers. The token budget is configurable (CAPTION_MAX_TOKENS) and captions pass through the same runaway guard as OCR so a looping vision model cannot flood the index. Control: a per-upload caption override threads from the three upload routes through start_ingestion and _run, with the on/off policy single-sourced in _run (caption self-gating removed from caption_images, mirroring the OCR change) so a force-on override works when the config default is off. The frontend adds a "Describe figures & charts" switch in the retrieval settings, persisted in localStorage and sent with each upload. Default on; it is a no-op without a vision model and bounded to CAPTION_MAX_IMAGES figures per document. Tests cover the new caption_images contract, the runaway guard on captions, the chart-aware prompt and token budget (and that OCR keeps its own prompt and budget), and both override directions end to end through ingestion. * Generalize figure understanding: transcribe-first prompt + high-DPI tiling Make figure/chart description work across any visual and any model strength, not just a strong VLM on simple figures. Two changes, validated by a recall benchmark on authoritative documents (ResNet/Attention papers, USDA, UN UDHR). 1. Transcribe-first caption prompt. The caption now asks the model to transcribe every visible label verbatim (titles, axis labels and units, legends, every box/node/arrow label, table cells, equations) and then add a one-line summary, instead of only describing the figure. Transcription is the most model-robust visual task, so weak models that cannot reason about a chart still recover its labels. 2. High-DPI tiling of figure pages. Figure-bearing pages are rendered as an overlapping grid of high-DPI tiles (plus a full-page pass for context); each tile is transcribed, then merged and de-duplicated. This keeps small diagram labels legible and covers every sub-figure without relying on exact region detection, which previously missed sub-figures and small labels. Supporting changes: figure render DPI 130 -> 200 with a clip margin so edge labels are not lost; vision calls are deterministic (temperature 0) so transcription does not randomly drop labels; the repetition guard now applies to captions too. New config knobs: FIGURE_DPI, FIGURE_MARGIN_FRAC, FIGURE_TILE_ROWS/COLS, FIGURE_TILE_ OVERLAP, FIGURE_FULLPAGE, CAPTION_MAX_PAGES, larger CAPTION_MAX_TOKENS, and CAPTION_MAX_IMAGES as a per-document tile budget. Measured figure context recall (per-label, dense academic figures): Qwen2.5-VL: 0.50 -> 0.83 (overall 0.81 -> 0.94) Gemma-4-E2B (weak): ~0 with loops -> 0.83 (overall 0.91) Born-digital text and scanned-page recall are unchanged (no regression). parsers gains _figure_boxes (shared detection), pages_with_figures, and render_pdf_figure_tiles; captioner gains merge_page_captions and a temperature parameter; ingestion routes figure captioning through the tiled path. * Fix RAG review issues: whole-doc budget pre-check, figure gating, empty re-ingest, vision auth Whole-document context now runs a cheap token-sum pre-check (store.scope_token_estimate) before hydrating every chunk's text, so an attachment that cannot fit the budget is rejected without loading the whole corpus into memory. The estimate mirrors all_chunks_for_scope's filter and the per-row token-count fallback exactly. Ingestion skips all figure work (PDF rasterization and detection, not just the caption call) unless a vision model is loaded, so a text-only deployment pays nothing. When OCR is enabled, scanned/image-only pages are excluded from figure tiling since OCR already transcribes them whole, avoiding double vision work and overlapping index entries; a scanned figure page is still tiled when OCR is off. start_ingestion no longer dedupes forever to a prior ingest that produced zero chunks (e.g. a scanned PDF uploaded before a vision model was loaded): the empty record is dropped and the content is re-ingested. Vision OCR and caption requests now send the backend Authorization header, so they match the chat endpoint and do not 401 under direct-stream (--api-key) mode. Adds tests for the budget estimate, scanned-page exclusion, the vision-model gate, the empty re-ingest path, and the auth-header passthrough. * Trim RAG vision-ingestion comments and docstrings Tighten the verbose multi-line docstrings and comments added across the RAG vision ingestion work (captioner, config, parsers, ingestion, store, tool, build_rag_autoinject, the RAG tests, and the chat-store/upload-hook frontend toggles) to one or two lines while keeping their intent. No code changed: verified comment/docstring-only against the prior commit, and the RAG test suite still passes. * Fix figure-tiling exclusion and client dedupe for re-ingestable docs Figure tiling now excludes only the pages OCR actually transcribed, not every text-less page. _ocr_scanned_pages returns the set of pages it OCR'd, and _run passes that to pages_with_figures as exclude_pages (replacing the ocr_on-keyed min_text_chars heuristic). A scanned page that OCR skipped (past OCR_MAX_PAGES, or whose OCR returned empty) is no longer dropped from captioning, so a chart on such a page still gets a caption. The document panel's upload dedupe no longer skips re-selecting a file whose only matching doc completed with zero chunks. Such a doc is re-ingestable (e.g. a scan attached before a vision model loaded), and the backend re-ingests on the same content hash, so the client must let it reach the backend; healthy or still-indexing docs are still skipped. The SSE complete frame's chunk count is recorded on the doc so the check is exact. Adds a regression test for the un-OCR'd scanned figure page and updates the pages_with_figures test to the exclude_pages interface. * Address review findings: whole-doc budget guard, job numChunks, dead code, upload cap whole_document_context now treats a non-positive max_tokens as "never inject" instead of injecting the whole corpus unbounded, so RAG_WHOLE_DOC_MAX_TOKENS=0 tightens rather than disables the budget (the real off switch stays RAG_THREAD_WHOLE_DOC=0). The job-status endpoint and get_job_status now expose num_chunks (joined from the document), and the upload hook threads it through the SSE-fallback completion paths (reconcile + poll). Previously a document that finished via the connection-cap fallback had no chunk count client-side, so the re-ingest dedupe wrongly treated it as empty and re-uploaded it. IndexJob/JobEvent gain the field and the untyped cast is dropped. Removes the dead render_pdf_figures function (superseded by the tiling path), its test, and the unused FIGURE_MARGIN_FRAC config knob. Adds an upload size cap (RAG_MAX_UPLOAD_BYTES, default 200 MB; 413 on exceed with the partial file cleaned up) so a pathological file can't drive unbounded parse + vision work. render_pdf_figure_tiles clamps rows/cols to >= 1 (no ZeroDivisionError on a misconfigured grid). Captioning progress is reported after OCR so the bar is monotonic. sqlite connections set busy_timeout=5000 so a long figure/scan ingest holding its connection doesn't make a concurrent ingest/read fail with "database is locked". Adds tests for the non-positive budget, the zero-grid clamp, job-status num_chunks, and the oversize-upload rejection. * Extract PDF text as layout-aware Markdown via pymupdf4llm parsers._pdf now extracts each PDF page as Markdown with pymupdf4llm.to_markdown (page_chunks=True) instead of flat page.get_text("text"), so tables, headings and lists keep their structure in the indexed chunks and retrieve far better (a table's cells stay associated with their row instead of flattening into a token stream). Gated by RAG_PDF_MARKDOWN (default on); falls back to plain PyMuPDF text when the toggle is off, pymupdf4llm is missing, extraction fails, or a page yields no Markdown. The scanned-page OCR and figure-tiling passes operate on rendered pixels and are unaffected; docx/html/txt keep their existing extractors. The preview-highlight locator already strips Markdown punctuation when building anchors; it now also splits anchor tokens on pipes so a Markdown table row still anchors to the raw PDF word stream. Declares pymupdf4llm as a studio/RAG dependency (was only transitively present via the data-designer plugin). Adds parser tests (Markdown table reaches the page text, the plain-text fallback, the missing-lib fallback) and a locator test for table-pipe anchoring. * Pin pymupdf4llm to 0.3.4 so the package scan does not pull onnxruntime The lockstep pymupdf4llm 1.27.x line makes pymupdf-layout a hard dependency, which in turn pulls onnxruntime (plus numpy/networkx/protobuf). The security-audit pip scan-packages job resolves requirements --with-deps, so adding pymupdf4llm to no-torch-runtime.txt and studio.txt surfaced onnxruntime's un-baselined CRITICAL finding and flipped the hf-stack shard from pass to fail. pymupdf4llm 0.3.x keeps pymupdf-layout behind an optional [layout] extra, so a plain install resolves to pymupdf + tabulate only and never touches onnxruntime. 0.3.4 requires pymupdf>=1.27.1, satisfied by our pinned pymupdf==1.27.2.3, and to_markdown (page_chunks=True) produces equivalent layout-aware Markdown on real PDFs (verified on the Attention, ResNet and USDA documents). Production already installs these files --no-deps, so onnxruntime was never shipped at runtime; this only fixes the scanner. The parser test now asserts Markdown markup (heading or table pipes) rather than table pipes specifically, since 0.3.4 emits a heading but not a pipe table on the tiny borderless synthetic fixture; both markers are absent from the plain-text fallback. * Fix RAG whole-doc review findings * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address RAG whole-doc review follow-ups * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address RAG review follow-up edge cases * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reserve image budget for whole-document RAG * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: wasimysaid <wasimysdev@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>		2026-06-30 15:55:23 +02:00
..
backend	Whole-document context for RAG chat attachments (#6693)	2026-06-30 15:55:23 +02:00
frontend	Whole-document context for RAG chat attachments (#6693)	2026-06-30 15:55:23 +02:00
src-tauri	Speed up Studio desktop startup (#6742)	2026-06-29 15:27:39 +02:00
__init__.py	Final cleanup	2026-03-12 18:28:04 +00:00
install_llama_prebuilt.py	Studio: stop handing CI/user secrets to downloaded llama.cpp binaries (#6696)	2026-06-27 05:21:05 -07:00
install_node_prebuilt.py	Pin isolated Node.js installer to committed sha256 digests (#6625)	2026-06-24 05:47:58 -07:00
install_python_stack.py	Make _uv_safe_path space-safe on macOS/Linux (#6503) (#6534)	2026-06-24 04:02:24 -07:00
LICENSE.AGPL-3.0	Add AGPL-3.0 license to studio folder	2026-03-09 19:36:25 +00:00
node_prebuilt_pins.json	Pin isolated Node.js installer to committed sha256 digests (#6625)	2026-06-24 05:47:58 -07:00
package-lock.json	ci: advisory lockfile supply-chain audit (no install-script changes) (#5604)	2026-05-19 05:56:56 -07:00
package.json	ci: advisory lockfile supply-chain audit (no install-script changes) (#5604)	2026-05-19 05:56:56 -07:00
setup.bat	Final cleanup	2026-03-12 18:28:04 +00:00
setup.ps1	Studio: UNSLOTH_NPM_REGISTRY opt-in for corporate npm mirrors (#6491) (#6663)	2026-06-25 04:01:43 -07:00
setup.sh	Studio: UNSLOTH_NPM_REGISTRY opt-in for corporate npm mirrors (#6491) (#6663)	2026-06-25 04:01:43 -07:00
Unsloth_Studio_Colab.ipynb	Studio Colab: opt-in shareable Cloudflare tunnel link (#6684)	2026-06-26 00:56:23 -07:00