mirror of https://github.com/MODSetter/SurfSense.git synced 2026-10-21 05:05:36 +00:00

$DESKTOP-RTLN3BA\$punk$ DESKTOP-RTLN3BA\$punk 6c2e2df61e perf(local): read born-digital PDFs from their text layer

Every PDF went through Docling's layout, OCR and TableFormer models, about
3 s a page (5.5 on table pages) plus an 11 s cold start and 1 GB of RSS.
A born-digital PDF already carries its text: pdfium reads it in about
7 ms a page with the page check.

A PDF is now read from its text layer when every page passes a per-page
check (no scans, scanner OCR layers, garbled or margin-only text, rotated
pages or filled form fields); otherwise Docling reads it as before. Pages
are joined with a blank line, soft hyphens are rejoined. Text-read PDFs
keep their figures through the figures-only pass: on first request, and at
idle priority when the file has images.

Measured through the upload route on an idle Ryzen 5800X: Executive
Summary (12 pp) 16.7 s -> 1.4 s, Operating Budget (46 pp) 125.6 s -> 9.0 s,
a 33-page spec 73 s -> 3.8 s. In a Basic-mode A/B on the Boston budget
corpus the shipped text scored 0.70 strict against 0.60 for Docling's
markdown; a second, smaller corpus is 2.9 points behind on table cells.

SURFSENSE_LOCAL_PDF_TEXT_LAYER=off restores Docling for every PDF.
ADR 0059.

2026-10-16 03:02:17 -07:00

4.6 KiB

Raw Permalink Blame History

status

code

proposed

surfsense_local/backend/worker/ingestion/parsing.py

surfsense_local/backend/scripts/fetch_docling_models.py

surfsense_local/backend/pyproject.toml

A GPU for the document parser

Owns: whether Docling ingest uses a GPU, and through which runtime. Today it never does (documents). This proposal records the evidence so the choice is made once, with numbers, rather than by adding a wheel.

Why the parser is on the CPU today

pyproject.toml takes torch from PyPI. The Windows wheel is CPU-only (2.14.0+cpu), and Linux is pinned to the CPU index on purpose. Docling's device="auto" asks torch, finds no CUDA and picks cpu, even on an RTX card.
ADR 0012 ships no CUDA payload, and the Windows installer must stay under makensis's 2 GB ceiling.
The GPU is the chat model's. audio.cpp is pinned to the CPU for that reason, and nothing coordinates VRAM between worker-ingest and llama-server, whose fit reads live free memory (fit).

Where the time goes on a CPU

Docling 2.125, Ryzen 7 5800X, with the parser settings from #2096:

First document of a session: the import and model load, about 12 s from a warm disk cache and 34 s in the first frozen run we logged. This is not GPU work.
Per page, 8 threads, generated corpus: 3.0 s born-digital, 5.2 s scanned, 6.0 s photographed. Layout is about 0.7 to 1 s a page and TableFormer about 2 s a table. The born-digital figure includes the OCR stage, which changed nothing on born-digital budget pages: with it off, Docling's Markdown was byte-identical and 1.44 to 1.59 times as fast. A born-digital PDF whose pages all pass the text-layer triage no longer reaches Docling (ADR 0059).
Scans and photos: OCR dominates, and within it text recognition: 3.5 s of 4.5 s on one full-page scan. The OCR stage is one thread and does not overlap pages.

So a GPU helps scans and table-heavy files. It does nothing for the first-job wait.

Options

Route	Covers	Payload	For	Against
CUDA torch	NVIDIA	+1.9 to 2.6 GB (Windows `cu130`/`cu126` wheels)	Every Docling stage; Docling's published 3.5 to 5x on layout and tables with OCR off	Breaks ADR 0012 and the NSIS ceiling; NVIDIA only; silent CPU fallback when the driver is older than the wheel
`onnxruntime-gpu`	NVIDIA	+0.6 to 1 GB with cuDNN	OCR and an ONNX layout model	Same ADR conflict; RapidOCR's cuDNN default re-tunes per line shape and ran recognition 4 to 7x slower than CPU until it is set to `DEFAULT`
`onnxruntime-directml`	Windows: NVIDIA, AMD, Intel	about +11 MB over the CPU wheel; +171 MB for the heron ONNX layout model	Vendor-neutral, DX12 only; Docling already asks RapidOCR for DML	Maintenance mode; stuck at 1.24.4 while we run 1.29, and it replaces the `onnxruntime` package the embedder also uses; TableFormer stays on the CPU
ONNX Runtime WebGPU plugin	D3D12 or Vulkan, Metal	5 to 13 MB	Closest match to ADR 0012: one package for every vendor	Needs a hook, since neither Docling's ONNX layout engine nor RapidOCR can register a plugin EP; open DETR-family bugs; unmeasured on heron and PP-OCR shapes
CoreML (macOS)	Apple Silicon	none, in the stock wheel	RapidOCR's `use_coreml` is reachable through `rapidocr_params`	CoreML EP was slower than CPU in RapidOCR's own test; layout already uses MPS
VLM through llama.cpp	every GPU the chat runtime sees	granite-docling 258M GGUF, about 133 MB	Uses the Vulkan runtime we already ship; the biggest accuracy lever on hard scans and forms	A second model resident beside chat; hallucination risk; only worth it for pages Docling scores low

Docling behaviour to know first

Docling sets RapidOCR's per-stage Det/Cls/Rec.use_dml and use_cuda keys, but RapidOCR reads only EngineConfig.onnxruntime.*, so GPU OCR has to be asked for through RapidOcrOptions.rapidocr_params.
The ONNX layout engine passes an explicit providers list straight to onnxruntime; its automatic mode only ever picks CUDA or CPU.
Only the layout stage batches pages, so batch sizes do nothing on a CPU.

Next step

Measure before choosing. On one NVIDIA, one AMD and one Intel machine, time OCR and heron-onnx on DirectML and on the WebGPU plugin, with a chat model loaded, against the CPU numbers above. Decide the VRAM rule in the same pass: for example, the parser uses the GPU only while no chat model is resident.

4.6 KiB Raw Permalink Blame History