mirror of
https://github.com/agent0ai/agent-zero.git
synced 2026-07-09 17:08:29 +00:00
Refine document query image routing
Prefer vision_load for images, screenshots, scans, charts, photos, and diagrams when vision tools are available. Keep document_query focused on PDFs, documents, large text-heavy files, code-file Q&A, and fallback OCR when vision cannot read the needed text. Update bundled prompt, skill guidance, DOX notes, and regression coverage for the routing contract.
This commit is contained in:
parent
afdc3aeb44
commit
77552b9394
6 changed files with 27 additions and 15 deletions
|
|
@ -20,6 +20,7 @@
|
|||
## Work Guidance
|
||||
|
||||
- Coordinate parser changes with timeout handling and fallback behavior.
|
||||
- Keep prompt and skill guidance clear that image files, screenshots, scans, charts, photos, and diagrams should go to vision tools first when available; `document_query` image OCR is a text-focused fallback when vision is unavailable or cannot read the needed text.
|
||||
|
||||
## Verification
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,9 @@
|
|||
### document_query
|
||||
read, extract, summarize, compare, OCR, or answer questions over local/remote documents, code files, and text-heavy document images/scans.
|
||||
read, extract, summarize, compare, or answer questions over local/remote documents, PDFs, Office files, HTML/text/code files, large text-heavy files, and fallback OCR when vision tools cannot read a document image/scan.
|
||||
|
||||
For document Q&A, document/code analysis, multi-document comparison, or OCR/text extraction from images/scans, first load the `document-query` skill with `skills_tool:load`, then call this tool using the loaded instructions.
|
||||
For document Q&A, document/code analysis, multi-document comparison, PDF or large-file extraction, or fallback OCR after vision tools are unavailable or insufficient, first load the `document-query` skill with `skills_tool:load`, then call this tool using the loaded instructions.
|
||||
|
||||
Use vision tools first for image files, screenshots, scans, charts, photos, diagrams, and other visual inputs when available. Call `document_query` for images only as document-style OCR fallback when vision cannot read the needed text.
|
||||
|
||||
Minimal args after loading the skill:
|
||||
- `document`: one local path/URL or a list of paths/URLs
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
name: document-query
|
||||
description: Use when reading, extracting, summarizing, comparing, OCRing, or answering questions over local or remote documents, code files, PDFs, Office files, HTML/text files, and text-heavy document images or scans with the document_query tool.
|
||||
description: Use when reading, extracting, summarizing, comparing, or answering questions over local or remote documents, code files, PDFs, Office files, HTML/text files, large text-heavy files, and fallback OCR for document images or scans when vision tools are unavailable or insufficient.
|
||||
version: 1.0.0
|
||||
author: Agent Zero Team
|
||||
tags: ["documents", "ocr", "qa", "pdf", "code", "analysis"]
|
||||
|
|
@ -10,8 +10,8 @@ trigger_patterns:
|
|||
- ask questions about a document
|
||||
- summarize document
|
||||
- compare documents
|
||||
- extract text from image
|
||||
- OCR document
|
||||
- fallback image OCR
|
||||
- OCR document fallback
|
||||
- analyze code file
|
||||
---
|
||||
|
||||
|
|
@ -24,13 +24,15 @@ Use the `document_query` tool to read, extract, summarize, compare, or answer qu
|
|||
Use `document_query` for:
|
||||
|
||||
- Local files and URLs containing document text: PDF, HTML, Office files, plain text, Markdown, CSV/TSV, XML/JSON, logs, code files, and similar content.
|
||||
- Large text-heavy files where ordinary reading/search would be too broad and the user needs Q&A, summarization, comparison, or extraction.
|
||||
- Q&A over one or more documents.
|
||||
- Summaries, comparisons, entity extraction, key-point extraction, and table/text extraction.
|
||||
- Code-file Q&A when the user points to specific files or URLs and wants answers from their contents.
|
||||
- Text-heavy images, scans, screenshots, and document images when the task is OCR/text extraction or Q&A over visible text.
|
||||
- Image OCR when vision tools are unavailable, when the main chat model is not multimodal, or when the user wants document text rather than visual scene understanding.
|
||||
- Fallback OCR over text-heavy document images, scans, or screenshots only when vision tools are unavailable, insufficient, or unable to read the needed text.
|
||||
|
||||
Do not use `document_query` for purely visual questions that require spatial/visual reasoning beyond document text; use vision tools when available for those cases.
|
||||
Use vision tools first for image files, screenshots, scans, charts, photos, diagrams, and other visual inputs when available. Do not route images to `document_query` as a first-pass reader.
|
||||
|
||||
Do not use `document_query` for purely visual questions, first-pass screenshot inspection, or tasks that require spatial/visual reasoning beyond document text; use vision tools when available for those cases.
|
||||
|
||||
## Inputs
|
||||
|
||||
|
|
@ -111,14 +113,14 @@ For directories or codebases, first identify the relevant files with file/search
|
|||
}
|
||||
```
|
||||
|
||||
### OCR Or Q&A Over A Document Image
|
||||
### Fallback OCR After Vision Cannot Read A Document Image
|
||||
|
||||
```json
|
||||
{
|
||||
"thoughts": [
|
||||
"The user wants text from a scanned document image."
|
||||
"Vision tools are unavailable or could not read the scanned document text, so document_query is the fallback OCR path."
|
||||
],
|
||||
"headline": "Reading text from the scanned document",
|
||||
"headline": "Reading document text with fallback OCR",
|
||||
"tool_name": "document_query",
|
||||
"tool_args": {
|
||||
"document": "/a0/usr/workdir/scan.png",
|
||||
|
|
|
|||
|
|
@ -23,6 +23,7 @@
|
|||
|
||||
- Read the rendering path before changing placeholders or filenames.
|
||||
- Prefer small prompt additions over broad rewrites when fixing a specific behavior.
|
||||
- Keep document/OCR routing explicit: image files, screenshots, scans, charts, photos, and diagrams should prefer vision tools when available, while `document_query` is for documents, large text-heavy files, and fallback OCR.
|
||||
- Update tests or snapshots when prompt budget, required sections, or generated system content changes.
|
||||
|
||||
## Verification
|
||||
|
|
|
|||
|
|
@ -24,8 +24,8 @@ always use specialized subordinate agents for specialized tasks matching their p
|
|||
|
||||
## Documents and OCR
|
||||
|
||||
use document_query to read, extract, summarize, compare, or answer questions about documents from local paths or URLs
|
||||
use document_query to read, extract, summarize, compare, or answer questions about documents from local paths or URLs, especially PDFs, Office files, HTML/text files, logs, code files, and large files that need Q&A
|
||||
use document_query for Q&A, summaries, comparisons, or extraction over specific code files when the user asks about file contents rather than asking to edit or search the codebase
|
||||
use document_query for document images, screenshots, scans, and other image files when the task is text extraction/OCR or Q&A over document content
|
||||
when vision tools are unavailable or the main chat model is not multimodal, use document_query for image OCR instead of asking the user to switch models
|
||||
use vision_load first for image files, screenshots, scans, charts, photos, diagrams, and other visual inputs when vision tools are available
|
||||
use document_query for image OCR only when vision tools cannot read the image, vision tools are unavailable, or the user specifically needs document-style fallback OCR over visible text
|
||||
keep parser/runtime details internal; users only need the document answer
|
||||
|
|
|
|||
|
|
@ -559,9 +559,15 @@ def test_document_query_prompt_uses_progressive_skill_disclosure():
|
|||
assert skill is not None
|
||||
assert "document_query for Q&A" in main_prompt
|
||||
assert "specific code files" in main_prompt
|
||||
assert "use vision_load first for image files" in main_prompt
|
||||
assert "document_query for image OCR only when vision tools cannot read" in main_prompt
|
||||
assert "skills_tool:load" in prompt
|
||||
assert "document-query" in prompt
|
||||
assert "document_query" in prompt
|
||||
assert "Use vision tools first" in prompt
|
||||
assert "fallback OCR" in prompt
|
||||
assert "answering questions over local or remote documents" in skill.description
|
||||
assert "fallback OCR" in skill.description
|
||||
assert "### Answer Questions Over A Document" in skill.content
|
||||
assert "### OCR Or Q&A Over A Document Image" in skill.content
|
||||
assert "Use vision tools first" in skill.content
|
||||
assert "### Fallback OCR After Vision Cannot Read A Document Image" in skill.content
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue