Refine document query image routing

Prefer vision_load for images, screenshots, scans, charts, photos, and diagrams when vision tools are available.

Keep document_query focused on PDFs, documents, large text-heavy files, code-file Q&A, and fallback OCR when vision cannot read the needed text.

Update bundled prompt, skill guidance, DOX notes, and regression coverage for the routing contract.
This commit is contained in:
Alessandro 2026-06-19 01:49:50 +02:00
parent afdc3aeb44
commit 77552b9394
6 changed files with 27 additions and 15 deletions

View file

@ -20,6 +20,7 @@
## Work Guidance
- Coordinate parser changes with timeout handling and fallback behavior.
- Keep prompt and skill guidance clear that image files, screenshots, scans, charts, photos, and diagrams should go to vision tools first when available; `document_query` image OCR is a text-focused fallback when vision is unavailable or cannot read the needed text.
## Verification

View file

@ -1,7 +1,9 @@
### document_query
read, extract, summarize, compare, OCR, or answer questions over local/remote documents, code files, and text-heavy document images/scans.
read, extract, summarize, compare, or answer questions over local/remote documents, PDFs, Office files, HTML/text/code files, large text-heavy files, and fallback OCR when vision tools cannot read a document image/scan.
For document Q&A, document/code analysis, multi-document comparison, or OCR/text extraction from images/scans, first load the `document-query` skill with `skills_tool:load`, then call this tool using the loaded instructions.
For document Q&A, document/code analysis, multi-document comparison, PDF or large-file extraction, or fallback OCR after vision tools are unavailable or insufficient, first load the `document-query` skill with `skills_tool:load`, then call this tool using the loaded instructions.
Use vision tools first for image files, screenshots, scans, charts, photos, diagrams, and other visual inputs when available. Call `document_query` for images only as document-style OCR fallback when vision cannot read the needed text.
Minimal args after loading the skill:
- `document`: one local path/URL or a list of paths/URLs

View file

@ -1,6 +1,6 @@
---
name: document-query
description: Use when reading, extracting, summarizing, comparing, OCRing, or answering questions over local or remote documents, code files, PDFs, Office files, HTML/text files, and text-heavy document images or scans with the document_query tool.
description: Use when reading, extracting, summarizing, comparing, or answering questions over local or remote documents, code files, PDFs, Office files, HTML/text files, large text-heavy files, and fallback OCR for document images or scans when vision tools are unavailable or insufficient.
version: 1.0.0
author: Agent Zero Team
tags: ["documents", "ocr", "qa", "pdf", "code", "analysis"]
@ -10,8 +10,8 @@ trigger_patterns:
- ask questions about a document
- summarize document
- compare documents
- extract text from image
- OCR document
- fallback image OCR
- OCR document fallback
- analyze code file
---
@ -24,13 +24,15 @@ Use the `document_query` tool to read, extract, summarize, compare, or answer qu
Use `document_query` for:
- Local files and URLs containing document text: PDF, HTML, Office files, plain text, Markdown, CSV/TSV, XML/JSON, logs, code files, and similar content.
- Large text-heavy files where ordinary reading/search would be too broad and the user needs Q&A, summarization, comparison, or extraction.
- Q&A over one or more documents.
- Summaries, comparisons, entity extraction, key-point extraction, and table/text extraction.
- Code-file Q&A when the user points to specific files or URLs and wants answers from their contents.
- Text-heavy images, scans, screenshots, and document images when the task is OCR/text extraction or Q&A over visible text.
- Image OCR when vision tools are unavailable, when the main chat model is not multimodal, or when the user wants document text rather than visual scene understanding.
- Fallback OCR over text-heavy document images, scans, or screenshots only when vision tools are unavailable, insufficient, or unable to read the needed text.
Do not use `document_query` for purely visual questions that require spatial/visual reasoning beyond document text; use vision tools when available for those cases.
Use vision tools first for image files, screenshots, scans, charts, photos, diagrams, and other visual inputs when available. Do not route images to `document_query` as a first-pass reader.
Do not use `document_query` for purely visual questions, first-pass screenshot inspection, or tasks that require spatial/visual reasoning beyond document text; use vision tools when available for those cases.
## Inputs
@ -111,14 +113,14 @@ For directories or codebases, first identify the relevant files with file/search
}
```
### OCR Or Q&A Over A Document Image
### Fallback OCR After Vision Cannot Read A Document Image
```json
{
"thoughts": [
"The user wants text from a scanned document image."
"Vision tools are unavailable or could not read the scanned document text, so document_query is the fallback OCR path."
],
"headline": "Reading text from the scanned document",
"headline": "Reading document text with fallback OCR",
"tool_name": "document_query",
"tool_args": {
"document": "/a0/usr/workdir/scan.png",

View file

@ -23,6 +23,7 @@
- Read the rendering path before changing placeholders or filenames.
- Prefer small prompt additions over broad rewrites when fixing a specific behavior.
- Keep document/OCR routing explicit: image files, screenshots, scans, charts, photos, and diagrams should prefer vision tools when available, while `document_query` is for documents, large text-heavy files, and fallback OCR.
- Update tests or snapshots when prompt budget, required sections, or generated system content changes.
## Verification

View file

@ -24,8 +24,8 @@ always use specialized subordinate agents for specialized tasks matching their p
## Documents and OCR
use document_query to read, extract, summarize, compare, or answer questions about documents from local paths or URLs
use document_query to read, extract, summarize, compare, or answer questions about documents from local paths or URLs, especially PDFs, Office files, HTML/text files, logs, code files, and large files that need Q&A
use document_query for Q&A, summaries, comparisons, or extraction over specific code files when the user asks about file contents rather than asking to edit or search the codebase
use document_query for document images, screenshots, scans, and other image files when the task is text extraction/OCR or Q&A over document content
when vision tools are unavailable or the main chat model is not multimodal, use document_query for image OCR instead of asking the user to switch models
use vision_load first for image files, screenshots, scans, charts, photos, diagrams, and other visual inputs when vision tools are available
use document_query for image OCR only when vision tools cannot read the image, vision tools are unavailable, or the user specifically needs document-style fallback OCR over visible text
keep parser/runtime details internal; users only need the document answer

View file

@ -559,9 +559,15 @@ def test_document_query_prompt_uses_progressive_skill_disclosure():
assert skill is not None
assert "document_query for Q&A" in main_prompt
assert "specific code files" in main_prompt
assert "use vision_load first for image files" in main_prompt
assert "document_query for image OCR only when vision tools cannot read" in main_prompt
assert "skills_tool:load" in prompt
assert "document-query" in prompt
assert "document_query" in prompt
assert "Use vision tools first" in prompt
assert "fallback OCR" in prompt
assert "answering questions over local or remote documents" in skill.description
assert "fallback OCR" in skill.description
assert "### Answer Questions Over A Document" in skill.content
assert "### OCR Or Q&A Over A Document Image" in skill.content
assert "Use vision tools first" in skill.content
assert "### Fallback OCR After Vision Cannot Read A Document Image" in skill.content