docs(local): ADR 0059 and documents.md match the merged figures pass, mixed PDFs and re-measured answers

docs/adr/0059-a-born-digital-pdf-is-read-from-its-text-layer.md

@@ -17,7 +17,7 @@ What the text layer cannot see: a page that is a picture, such as a scan or a co
 - A `.pdf` is read from its text layer when every page passes a triage that takes milliseconds: a page of running text (`ok:text`), a page with under 20 characters and almost no image (`ok:sparse`), an empty page (`ok:blank`), or at least 20 characters over images covering most of the page, as on a section cover (`mixed:big-image`). At least one page must be `ok:text`. The rules are in [documents](../architecture/documents.md#pdfs-with-a-text-layer), and the code in [`pdf_text_layer/`](../../surfsense_local/backend/worker/ingestion/pdf_text_layer/).
 - Any other page, a turned page, a filled form field, an empty text layer or a pdfium error sends the whole file to Docling, which reads it as before and keeps its figures. The choice is made for the whole document, for safety in a release: no source's text is spliced from two readers, and a PDF that needs Docling anywhere reads exactly as it did before. Splicing Docling's pages in by page range can follow once it is measured. Amended: a PDF with some failed pages, at most half of them, now has only those pages read by Docling, by page range, and the rest from its text layer, while a scan, a filled form, a file pdfium cannot read and one with no `ok:text` page still go to Docling whole ([documents](../architecture/documents.md#pdfs-with-a-text-layer)).
 - The text is the A/B's text arm: each page's text as pdfium lays it out, and pages a blank line apart. Three changes: a word hyphenated at a line's end is joined again, control codes are dropped, and a blank line is added where a paragraph ends, found from the gaps between its lines, so the chunker cuts there and not at an arbitrary line. Only `\n` is added; the A/B's characters are kept as they were. It has no Markdown headings and no pipe tables.
-- Ingest keeps no figures for a text-read PDF. By the maintainer's decision they come from the existing figures-only pass, a full Docling conversion, when the agent or Studio first asks for them. A pass after every upload would cost about 32 CPU-s a page, give back the time saved and compete with local chat. For a PDF whose pages hold image objects, ingest also queues the pass at the lowest priority, so it runs once nothing else waits.
+- Ingest keeps no figures for a text-read PDF. By the maintainer's decision they come from the figures-only pass when the agent or Studio first asks for them. That pass reads born-digital pages with Docling's layout model alone, about 6 CPU-s a page against 32 for a full conversion, and gives way to waiting sources between page runs. For a PDF of at most 40 pages whose pages hold image objects, ingest also queues the pass at the lowest priority, so it runs once nothing else waits.
 - `SURFSENSE_LOCAL_PDF_TEXT_LAYER=off` sends every PDF to Docling, as before. A value it cannot read also means off.
 - **Rejected: a Docling pass afterwards** over pages that have a text layer, to replace their text with Docling's. Docling's text answered worse, and the pass would double the CPU and change chunk ids under existing citations.
 - **Rejected: LiteParse** as the reader. Its tables were the best of the fast tools, but it gave empty Markdown on a turned page and 76 false code blocks in 30 pages, dropped a column, bundles a second PDFium, and downloads Tesseract data by default.
@@ -30,9 +30,9 @@ What the text layer cannot see: a page that is a picture, such as a scan or a co
 - 32 of the 36 PDFs measured are read from their text layer. Of the 19 Boston budget PDFs, two go to Docling for a cover photo whose title is under 20 characters, and one for a raster org chart whose page holds only its running header. A filled form goes to Docling too.
 - Through the shipped code, on the A/B's questions with Gemma 3 4B, Qwen3.5-9B and Haiku 4.5, the text layer scored 105/150 = 0.70 strict against Docling's 90/150 = 0.60, and 35/69 table cells against 24/69. The second corpus was six PDFs: a two-column paper, a deck exported to PDF, the spec and three reports. There it scored 52/69 against 54/69: better on charts (6/9 against 2/9), level on slides, and worse on tables (19/30 against 27/30). Most of that loss is one paper's Table 1, whose caption and header the chunker cut from its rows; the rest is a table whose rows begin with a number.
 - Content that exists only as pixels on a page with running text is not in the text. That covers a raster chart or org chart under 65% of the page, and a picture under a cover's title. Docling's Markdown had lost these as well. A wrong ToUnicode map gives fluent but wrong letters, which are indexed as they are; no reader tested caught it.
-- One page that needs Docling sends every page of the file to it, so a long report with one scanned page gains nothing.
+- Docling reads only the runs of pages that need it, when they are at most half the file; a scan, a filled form and a file pdfium cannot read are still read whole, so a scan gains only the embedder's speed-up.
 - A text-read PDF's figures arrive after it is ready. Until the pass has run, the agent hears that its images are being extracted, and Studio's first draft places none from it.
 
 ## Where the code stands
 
-- Measured on one machine, and the stress PDFs are generated. The table loss on the second corpus was in chunking and line geometry, not the triage. Paragraph breaks keep Table 1's caption, header and rows in one passage, and put the answer in the top 5 passages for 22 of 23 questions there (20 before) and 44 of 50 on Boston (41); the answers have not been measured again. Rows that begin with a number are still open.
+- Measured on one machine, and the stress PDFs are generated. The table loss on the second corpus was in chunking and line geometry, not the triage. Paragraph breaks keep Table 1's caption, header and rows in one passage, and put the answer in the top 5 passages for 22 of 23 questions there (20 before) and 44 of 50 on Boston (41). Measured again, answers scored 0.83 to Docling's 0.78 on the second corpus, table cells 28 of 30 to 27, and 0.70 to 0.60 on Boston (Gemma 3 4B and Qwen3.5 9B), where Gemma lost three table cells to passages that now pack across a page break. Rows that begin with a number are still open.

docs/architecture/documents.md

@@ -293,7 +293,7 @@ The upload is accepted. Finding no speech model, the job fails without retry wit
 - A wrong ToUnicode map is not detected. A generated PDF whose map shifts every letter by one reads as fluent `ok:text` and is indexed as it is; Docling read it no better.
 - A filled form field sends the whole file to Docling, and so do failed pages over half the file. Text spliced from the two readers meets at a page break: a paragraph running from a text-layer page into a page Docling read is cut there, and only Docling's pages carry Markdown headings and pipe tables.
 - A cover photo whose title is under 20 characters, or a picture page whose only live text is its running header, is read by Docling, which adds OCR of the photo. Education's and Capital Planning's covers are (titles of 11 and 18 characters), and Budget Organization's org chart on page 3, which Docling's OCR reads as 272 mostly misspelled words.
-- A text-read table is lines of values with no column boundaries. On a second corpus the text layer answered 19 of 30 table questions to Docling's 27. Most misses came where the chunker cut a paper's Table 1 from its caption and header, which paragraph breaks now keep in one passage; the answers have not been measured again. The rest came where a table's rows begin with a number, as in the spec's roadmap, `5 Dispatcher …` on one line and `6 Gateway slash command …` on the next, which the breaks do not change.
+- A text-read table is lines of values with no column boundaries. On a second corpus the text layer answered 19 of 30 table questions to Docling's 27. Most misses came where the chunker cut a paper's Table 1 from its caption and header, which paragraph breaks now keep in one passage: measured again, it answered 28 of 30 to Docling's 27. The rest came where a table's rows begin with a number, as in the spec's roadmap, `5 Dispatcher …` on one line and `6 Gateway slash command …` on the next, which the breaks do not change.
 - A table whose rows are spaced like paragraphs, as when its cells wrap to two lines, gets a blank line between rows, so a long one can still be cut between rows. A sentence that wraps so that `Table 2:` starts a line is taken for a caption.
 - The figures-only pass gives way to a waiting source, but not to a chat turn, so a local model shares the CPU with it.
 - `documents` has no `AUTOINCREMENT`, so a deleted source's id goes to the next document. Upload and the figures pass remove what a delete leaves ([Retry, cancel and delete](#retry-cancel-and-delete)), but a file another process holds open through both, such as an original streaming to a download, can still be found in the new document's directory. `sqlite_autoincrement` needs a migration that rebuilds the table.