mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-05-19 07:55:35 +00:00
9bcd50164d
Adds the full parser_compare experiment for the multimodal_doc suite:
six arms compared on 30 PDFs / 171 questions from MMLongBench-Doc with
anthropic/claude-sonnet-4.5 across the board.
Source code:
- core/parsers/{azure_di,llamacloud,pdf_pages}.py: direct parser SDK
callers (Azure Document Intelligence prebuilt-read/layout, LlamaParse
parse_page_with_llm/parse_page_with_agent) used by the LC arms,
bypassing the SurfSense backend so each (basic/premium) extraction
is a clean A/B independent of backend ETL routing.
- suites/multimodal_doc/parser_compare/{ingest,runner,prompt}.py:
six-arm benchmark (native_pdf, azure_basic_lc, azure_premium_lc,
llamacloud_basic_lc, llamacloud_premium_lc, surfsense_agentic) with
byte-identical prompts per question, deterministic grader, Wilson
CIs, and the per-page preprocessing tariff cost overlay.
Reproducibility:
- pyproject.toml + uv.lock pin pypdf, azure-ai-documentintelligence,
llama-cloud-services as new deps.
- .env.example documents the AZURE_DI_* and LLAMA_CLOUD_API_KEY env
vars now required for parser_compare.
- 12 analysis scripts under scripts/: retry pass with exponential
backoff, post-retry accuracy merge, McNemar / latency / per-PDF
stats, context-overflow hypothesis test, etc. Each produces one
number cited by the blog report.
Citation surface:
- reports/blog/multimodal_doc_parser_compare_n171_report.md: 1219-line
technical writeup (16 sections) covering headline accuracy, per-format
accuracy, McNemar pairwise significance, latency / token / per-PDF
distributions, error analysis, retry experiment, post-retry final
accuracy, cost amortization model with closed-form derivation, threats
to validity, and reproducibility appendix.
- data/multimodal_doc/runs/2026-05-14T00-53-19Z/parser_compare/{raw,
raw_retries,raw_post_retry}.jsonl + run_artifact.json + retry summary
whitelisted via data/.gitignore as the verifiable numbers source.
Gitignore:
- ignore logs_*.txt + retry_run.log; structured artifacts cover the
citation surface, debug logs are noise.
- data/.gitignore default-ignores everything, whitelists the n=171 run
artifacts only (parser manifest left ignored to avoid leaking local
Windows usernames in absolute paths; manifest is fully regenerable
via 'ingest multimodal_doc parser_compare').
- reports/.gitignore now whitelists hand-curated reports/blog/.
Also retires the abandoned CRAG Task 3 implementation (download script,
streaming Task 3 ingest, CragTask3Benchmark + tests) and trims the
runner / ingest module APIs to match.
Co-authored-by: Cursor <cursoragent@cursor.com>
100 lines
3.3 KiB
Python
100 lines
3.3 KiB
Python
"""Stub the mmlongbench manifest so parser_compare can extract in parallel.
|
|
|
|
The mmlongbench Surfsense ingest writes its manifest only at the very
|
|
end of the upload pipeline (~hours of celery work). parser_compare's
|
|
ingest, on the other hand, just needs a list of (doc_id, pdf_path)
|
|
tuples to know which PDFs to extract — it doesn't care about the
|
|
SurfSense ``document_id`` (the runner does, later, after a refresh).
|
|
|
|
This script extends the existing manifest with the *additional* PDFs
|
|
that mmlongbench has already cached on disk (i.e. all 30 PDFs in
|
|
``data/multimodal_doc/mmlongbench/pdfs/`` even though only 5 have
|
|
SurfSense ``document_id``s yet) so parser_compare can run all four
|
|
extractions for them in parallel with the SurfSense ingest.
|
|
|
|
After mmlongbench finishes, re-run::
|
|
|
|
python -m surfsense_evals ingest multimodal_doc parser_compare \
|
|
--max-docs 30
|
|
|
|
…to refresh ``parser_compare_doc_map.jsonl`` with the now-populated
|
|
``document_id`` values for the 25 new PDFs. The extractions
|
|
themselves are cached on disk so the second pass is essentially free.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from pathlib import Path
|
|
|
|
|
|
REPO = Path(__file__).resolve().parents[1]
|
|
MAP_PATH = REPO / "data" / "multimodal_doc" / "maps" / "mmlongbench_doc_map.jsonl"
|
|
PDF_DIR = REPO / "data" / "multimodal_doc" / "mmlongbench" / "pdfs"
|
|
QUESTIONS = REPO / "data" / "multimodal_doc" / "mmlongbench" / "questions.jsonl"
|
|
|
|
|
|
def _question_count_per_doc() -> dict[str, int]:
|
|
counts: dict[str, int] = {}
|
|
with QUESTIONS.open("r", encoding="utf-8") as fh:
|
|
for line in fh:
|
|
line = line.strip()
|
|
if not line:
|
|
continue
|
|
row = json.loads(line)
|
|
counts[row["doc_id"]] = counts.get(row["doc_id"], 0) + 1
|
|
return counts
|
|
|
|
|
|
def main() -> None:
|
|
if not MAP_PATH.exists():
|
|
raise SystemExit(
|
|
f"manifest not found at {MAP_PATH} — "
|
|
"run `surfsense_evals ingest multimodal_doc mmlongbench` first."
|
|
)
|
|
|
|
existing_lines = MAP_PATH.read_text(encoding="utf-8").splitlines()
|
|
existing_rows: list[dict] = []
|
|
settings_line = None
|
|
for line in existing_lines:
|
|
line = line.strip()
|
|
if not line:
|
|
continue
|
|
row = json.loads(line)
|
|
if "__settings__" in row:
|
|
settings_line = line
|
|
else:
|
|
existing_rows.append(row)
|
|
|
|
by_doc_id = {r["doc_id"]: r for r in existing_rows}
|
|
counts = _question_count_per_doc()
|
|
|
|
cached_pdfs = sorted(p for p in PDF_DIR.glob("*.pdf"))
|
|
print(f"existing manifest entries: {len(existing_rows)}")
|
|
print(f"cached PDFs on disk: {len(cached_pdfs)}")
|
|
|
|
added = 0
|
|
for pdf in cached_pdfs:
|
|
if pdf.name in by_doc_id:
|
|
continue
|
|
by_doc_id[pdf.name] = {
|
|
"doc_id": pdf.name,
|
|
"document_id": None,
|
|
"pdf_path": str(pdf),
|
|
"n_questions": counts.get(pdf.name, 0),
|
|
}
|
|
added += 1
|
|
|
|
out_lines: list[str] = []
|
|
if settings_line:
|
|
out_lines.append(settings_line)
|
|
for doc_id in sorted(by_doc_id):
|
|
out_lines.append(json.dumps(by_doc_id[doc_id]))
|
|
MAP_PATH.write_text("\n".join(out_lines) + "\n", encoding="utf-8")
|
|
|
|
print(f"added {added} stub rows; manifest now has {len(by_doc_id)} PDFs")
|
|
print(f"wrote: {MAP_PATH}")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|