+
+
+ Vision override
+ /
+
+
/
@@ -86,6 +98,16 @@
border-top: 1px solid var(--color-border);
}
+ .model-preset-row-nested {
+ border-top: 0 !important;
+ padding-top: var(--spacing-xs);
+ font-size: 0.76rem;
+ }
+
+ .model-preset-identity-vision {
+ grid-column: 3;
+ }
+
.model-preset-icon,
.model-preset-provider,
.model-preset-separator {
diff --git a/prompts/AGENTS.md b/prompts/AGENTS.md
index a3b9fb85e..3119674eb 100644
--- a/prompts/AGENTS.md
+++ b/prompts/AGENTS.md
@@ -25,6 +25,7 @@
- Read the rendering path before changing placeholders or filenames.
- Prefer small prompt additions over broad rewrites when fixing a specific behavior.
- Keep document/OCR routing explicit: image files, screenshots, scans, charts, photos, and diagrams should prefer vision tools when available, while `document_query` is for documents, large text-heavy files, and fallback OCR.
+- Keep native and sidecar vision prompts synchronized with `vision_load`: related images belong in one call, while a sidecar result is text-only unless Main explicitly requests its native raw route.
- Update tests or snapshots when prompt budget, required sections, or generated system content changes.
## Verification
diff --git a/prompts/agent.system.tools_vision_sidecar.md b/prompts/agent.system.tools_vision_sidecar.md
new file mode 100644
index 000000000..73a84ee23
--- /dev/null
+++ b/prompts/agent.system.tools_vision_sidecar.md
@@ -0,0 +1,48 @@
+## multimodal vision tools
+
+### vision_load
+analyze images with the separate Vision Model and return a text result
+args: `paths` list of absolute image paths or ephemeral image refs, `query` optional focused instruction, `raw` optional boolean
+Input schema for tool_args:
+```json
+{
+ "type": "object",
+ "properties": {
+ "paths": {
+ "type": "array",
+ "items": {"type": "string"},
+ "description": "Absolute image paths or ephemeral image refs."
+ },
+ "query": {
+ "type": "string",
+ "description": "What the Vision Model should inspect, compare, locate, or read."
+ },
+ "raw": {
+ "type": "boolean",
+ "description": "Use the Main model's native vision instead, when Main supports vision."
+ }
+ },
+ "required": ["paths"],
+ "additionalProperties": false
+}
+```
+rules:
+- put all images needed for one comparison or visual task in the same `paths` array; they are sent in one Vision Model call
+- use a focused `query`; if omitted, the Vision Model returns a concise general description
+- the result is a text capsule; the Main model does not receive the raw images
+- use `raw=true` only when Main supports vision and must inspect the pixels itself
+- only bitmaps are supported
+example:
+```json
+{
+ "thoughts": [
+ "I need to compare both screenshots before answering."
+ ],
+ "headline": "Comparing screenshots",
+ "tool_name": "vision_load",
+ "tool_args": {
+ "paths": ["/path/to/before.png", "/path/to/after.png"],
+ "query": "Compare the error banners and describe what changed."
+ }
+}
+```
diff --git a/tests/test_browser_agent_regressions.py b/tests/test_browser_agent_regressions.py
index 7fba2ba48..a924ec00b 100644
--- a/tests/test_browser_agent_regressions.py
+++ b/tests/test_browser_agent_regressions.py
@@ -36,8 +36,13 @@ class _TestAgentContextType:
class _TestResponse(SimpleNamespace):
- def __init__(self, message="", break_loop=False, **kwargs):
- super().__init__(message=message, break_loop=break_loop, **kwargs)
+ def __init__(self, message="", break_loop=False, additional=None, **kwargs):
+ super().__init__(
+ message=message,
+ break_loop=break_loop,
+ additional=additional,
+ **kwargs,
+ )
class _TestTool:
@@ -96,6 +101,9 @@ _model_config_stub = ModuleType("plugins._model_config.helpers.model_config")
_model_config_stub.get_presets = lambda: []
_model_config_stub.get_preset_by_name = lambda name: None
_model_config_stub.get_chat_model_config = lambda agent=None: {}
+_model_config_stub.get_vision_model_config = lambda agent=None: {}
+_model_config_stub.use_vision_sidecar = lambda agent=None: False
+_model_config_stub.build_vision_model = lambda agent=None: None
sys.modules.setdefault("plugins._model_config.helpers.model_config", _model_config_stub)
@@ -4297,11 +4305,9 @@ async def test_vision_load_materializes_ephemeral_browser_refs(monkeypatch, tmp_
monkeypatch.setattr(vision_load_module.chat_media.files, "get_abs_path", fake_get_abs_path)
monkeypatch.setattr(vision_load_module.chat_media.files, "normalize_a0_path", fake_normalize_a0_path)
- monkeypatch.setattr(
- vision_load_module.plugins,
- "get_plugin_config",
- lambda *args, **kwargs: {"chat_model": {"max_embeds": 10}},
- )
+ monkeypatch.setattr(vision_load_module, "get_chat_model_config", lambda _agent: {"vision": True, "max_embeds": 10})
+ monkeypatch.setattr(vision_load_module, "get_vision_model_config", lambda _agent: {})
+ monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: False)
tool_results = []
messages = []
diff --git a/tests/test_model_config_api_keys.py b/tests/test_model_config_api_keys.py
index eb779aba1..a52f5583a 100644
--- a/tests/test_model_config_api_keys.py
+++ b/tests/test_model_config_api_keys.py
@@ -149,7 +149,7 @@ def test_model_config_frontend_tracks_provider_api_key_edits():
assert "/plugins/_model_config/missing_api_key_status" not in model_gate_content
assert '@input="$store.modelConfig.setApiKeyValue(_prov, $el.value)"' in config_content
assert "apiKeyMode: 'none'" not in preset_modal_content
- assert preset_modal_content.count("apiKeyMode: 'store'") == 3
+ assert preset_modal_content.count("apiKeyMode: 'store'") == 4
assert "$store.modelConfig.resetApiKeyDrafts();" in preset_modal_content
assert "await $store.modelConfig.refreshApiKeyStatus();" in preset_modal_content
assert "await store.persistAllDirtyApiKeys();" in store_content
diff --git a/tests/test_model_config_project_presets.py b/tests/test_model_config_project_presets.py
index bb8c67045..1c972b43c 100644
--- a/tests/test_model_config_project_presets.py
+++ b/tests/test_model_config_project_presets.py
@@ -1092,6 +1092,44 @@ def test_preset_application_inherits_optional_slots(monkeypatch, tmp_path):
assert config["embedding_model"] == base_config["embedding_model"]
+def test_preset_vision_slot_is_optional_and_never_inherited(monkeypatch, tmp_path):
+ _prepare_a0_tree(monkeypatch, tmp_path)
+
+ from plugins._model_config.helpers import model_config
+
+ base_config = {
+ "chat_model": {"provider": "openrouter", "name": "main", "vision": False},
+ "vision_model": {
+ "provider": "openrouter",
+ "name": "default-vision",
+ "max_embeds": 2,
+ },
+ }
+
+ without_sidecar = model_config.build_config_from_preset(
+ {"name": "Text only", "chat": {"provider": "openrouter", "name": "text"}},
+ base_config,
+ )
+ with_sidecar = model_config.build_config_from_preset(
+ {
+ "name": "Visual",
+ "chat": {"provider": "openrouter", "name": "text"},
+ "vision": {
+ "provider": "anthropic",
+ "name": "visual",
+ "max_embeds": 5,
+ },
+ },
+ base_config,
+ )
+
+ assert without_sidecar["vision_model"] == {}
+ assert with_sidecar["vision_model"]["provider"] == "anthropic"
+ assert with_sidecar["vision_model"]["name"] == "visual"
+ assert with_sidecar["vision_model"]["max_embeds"] == 5
+ assert "default-vision" not in str(with_sidecar["vision_model"])
+
+
def test_legacy_utility_preset_defaults_preserve_tuning_but_clear_kwargs(
monkeypatch,
tmp_path,
diff --git a/tests/test_model_config_ui.py b/tests/test_model_config_ui.py
index 1feb61061..5c6d54aa8 100644
--- a/tests/test_model_config_ui.py
+++ b/tests/test_model_config_ui.py
@@ -85,6 +85,49 @@ def test_preset_editor_uses_standard_modal_footer_buttons() -> None:
assert "preset-editor-footer" not in preset_modal
+def test_preset_editor_nests_one_conditional_vision_selector_in_main() -> None:
+ preset_modal = read("plugins", "_model_config", "webui", "main.html")
+ model_field = read("plugins", "_model_config", "webui", "model-field.html")
+ preset_overview = read("plugins", "_model_config", "webui", "preset-overview.html")
+ preset_store = read("plugins", "_model_config", "webui", "model-config-store.js")
+
+ main_start = preset_modal.index('Main Model
')
+ utility_start = preset_modal.index('Utility Model
')
+ selector_start = preset_modal.index('class="vision-sidecar-selector"')
+ supports_start = model_field.index('Supports Vision
')
+ override_start = model_field.index('Use separate Vision Model
')
+ context_start = model_field.index('Context window size
')
+ advanced_start = model_field.index('')
+
+ assert 'Vision Model
' not in preset_modal
+ assert preset_modal.count('class="vision-sidecar-selector"') == 1
+ assert main_start < selector_start < utility_start
+ assert '!selectedPreset.chat.vision || selectedPreset.vision.override_main' in preset_modal
+ assert "get visionModel() { return selectedPreset.vision; }" in preset_modal
+ assert "modelType: 'vision'" in preset_modal
+ assert preset_modal.count("apiKeyMode: 'store'") == 4
+ assert "margin: 0.75rem 0 0;" in preset_modal
+ assert "padding: 0.25rem 0 0;" in preset_modal
+ assert "border-left: 2px solid var(--color-border);" not in preset_modal
+ assert "Use separate Vision Model" in model_field
+ assert supports_start < override_start < context_start < advanced_start
+ assert "When disabled, vision_load uses this model's native vision." in model_field
+ assert "When enabled, vision_load uses the preset's Vision Model" not in model_field
+ assert 'x-model="visionModel.override_main"' in model_field
+ assert '' in model_field
+ assert "model-preset-row-nested" in preset_overview
+ assert "model.title === 'Vision'" in preset_overview
+ assert preset_overview.count("model.title !== 'Vision'") == 2
+ assert 'Vision override' in preset_overview
+ assert "'model-preset-identity-vision': model.title === 'Vision'" in preset_overview
+ assert "grid-column: 3;" in preset_overview
+ assert "padding-top: var(--spacing-xs);" in preset_overview
+ assert "margin-left: 2rem;" not in preset_overview
+ assert "border-left: 1px solid var(--color-border);" not in preset_overview
+ assert "if (slotKey === 'vision') config[sectionKey] = {};" in preset_store
+ assert "['chat', 'vision', 'utility']" in preset_store
+
+
def test_plugin_settings_reset_is_explicit_and_does_not_capture_toast_early() -> None:
settings_modal = read("webui", "components", "plugins", "plugin-settings.html")
settings_store = read("webui", "components", "plugins", "plugin-settings-store.js")
diff --git a/tests/test_parallel_tool.py b/tests/test_parallel_tool.py
index dc1a91199..575f21f51 100644
--- a/tests/test_parallel_tool.py
+++ b/tests/test_parallel_tool.py
@@ -160,6 +160,42 @@ def test_normalize_parallel_tool_calls_accepts_json_string_array() -> None:
assert calls[1].tool_args["message"] == "Research nuclear fusion news in Italian."
+def test_parallel_keeps_mixed_tools_and_batched_vision_paths() -> None:
+ calls = parallel_tools.normalize_parallel_tool_calls(
+ [
+ {"tool_name": "search_a", "tool_args": {"query": "one"}},
+ {"tool_name": "search_b", "tool_args": {"query": "two"}},
+ {"tool_name": "browser_agent", "tool_args": {"message": "open page"}},
+ {
+ "tool_name": "vision_load",
+ "tool_args": {
+ "paths": ["/tmp/before.png", "/tmp/after.png"],
+ "query": "compare",
+ },
+ },
+ ]
+ )
+
+ assert [call.tool_name for call in calls] == [
+ "search_a",
+ "search_b",
+ "browser_agent",
+ "vision_load",
+ ]
+ assert calls[3].tool_args["paths"] == ["/tmp/before.png", "/tmp/after.png"]
+
+
+def test_parallel_allows_multiple_independent_vision_calls() -> None:
+ calls = parallel_tools.normalize_parallel_tool_calls(
+ [
+ {"tool_name": "vision_load", "tool_args": {"paths": ["/tmp/a.png"]}},
+ {"tool_name": "vision_load", "tool_args": {"paths": ["/tmp/b.png"]}},
+ ]
+ )
+
+ assert [call.tool_name for call in calls] == ["vision_load", "vision_load"]
+
+
def test_subordinate_prompts_share_reusable_tree_contract() -> None:
call_prompt = (PROJECT_ROOT / "prompts/agent.system.tool.call_sub.md").read_text(
encoding="utf-8"
diff --git a/tests/test_tool_policy.py b/tests/test_tool_policy.py
index 2db20350c..3d3447325 100644
--- a/tests/test_tool_policy.py
+++ b/tests/test_tool_policy.py
@@ -546,6 +546,59 @@ async def test_vision_tool_follows_chat_config_not_profile_policy(
assert tool_policy.resolve_tool(agent, "vision_load").source == "runtime-config"
+@pytest.mark.asyncio
+async def test_vision_sidecar_prompt_replaces_native_vision_prompt(
+ monkeypatch, tmp_path: Path
+) -> None:
+ _write_prompt(tmp_path, "agent.system.tools.md", "TOOLS\n{{tools}}")
+ _write_prompt(
+ tmp_path,
+ "agent.system.tools_vision.md",
+ "### vision_load\nnative pixels\nargs: `paths`",
+ )
+ _write_prompt(
+ tmp_path,
+ "agent.system.tools_vision_sidecar.md",
+ "### vision_load\nsidecar capsule\nargs: `paths`, `query`",
+ )
+ monkeypatch.setattr(tool_policy.subagents, "get_paths", _prompt_paths(tmp_path))
+ monkeypatch.setattr(
+ "plugins._model_config.helpers.model_config.get_chat_model_config",
+ lambda agent: {"vision": True},
+ )
+ monkeypatch.setattr(
+ "plugins._model_config.helpers.model_config.use_vision_sidecar",
+ lambda agent: True,
+ )
+ monkeypatch.setattr(responses_tools, "_mcp_tools", lambda agent: [])
+ agent = _Agent(tmp_path)
+
+ prompt = await _11_tools_prompt.build_prompt(agent)
+ schemas, _name_map = responses_tools.build_responses_function_tools(agent)
+
+ assert "sidecar capsule" in prompt
+ assert "native pixels" not in prompt
+ assert schemas[0]["name"] == "vision_load"
+ assert schemas[0]["description"] == "sidecar capsule"
+
+
+def test_vision_sidecar_prompt_declares_multi_image_native_schema() -> None:
+ prompt = (
+ Path(__file__).resolve().parents[1]
+ / "prompts"
+ / "agent.system.tools_vision_sidecar.md"
+ ).read_text(encoding="utf-8")
+ schema = responses_tools._schema_from_prompt(prompt)
+
+ assert schema["required"] == ["paths"]
+ assert schema["properties"]["paths"] == {
+ "type": "array",
+ "items": {"type": "string"},
+ "description": "Absolute image paths or ephemeral image refs.",
+ }
+ assert {"query", "raw"} <= schema["properties"].keys()
+
+
def test_mcp_prompt_and_native_schema_omit_blocked_tool(
monkeypatch, tmp_path: Path
) -> None:
diff --git a/tests/test_vision_load_image_refs.py b/tests/test_vision_load_image_refs.py
index 2842a5e1e..619901aa0 100644
--- a/tests/test_vision_load_image_refs.py
+++ b/tests/test_vision_load_image_refs.py
@@ -1,3 +1,4 @@
+import asyncio
import types
from types import SimpleNamespace
import sys
@@ -13,8 +14,13 @@ from helpers import images
class _TestResponse(SimpleNamespace):
- def __init__(self, message="", break_loop=False, **kwargs):
- super().__init__(message=message, break_loop=break_loop, **kwargs)
+ def __init__(self, message="", break_loop=False, additional=None, **kwargs):
+ super().__init__(
+ message=message,
+ break_loop=break_loop,
+ additional=additional,
+ **kwargs,
+ )
class _TestTool:
@@ -74,11 +80,9 @@ async def test_vision_load_materializes_local_image_to_chat_artifact(monkeypatch
monkeypatch.setattr(vision_load_module.chat_media.files, "get_abs_path", fake_get_abs_path)
monkeypatch.setattr(vision_load_module.chat_media.files, "normalize_a0_path", fake_normalize_a0_path)
- monkeypatch.setattr(
- vision_load_module.plugins,
- "get_plugin_config",
- lambda *args, **kwargs: {"chat_model": {"max_embeds": 10}},
- )
+ monkeypatch.setattr(vision_load_module, "get_chat_model_config", lambda _agent: {"vision": True, "max_embeds": 10})
+ monkeypatch.setattr(vision_load_module, "get_vision_model_config", lambda _agent: {})
+ monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: False)
async def direct_call(func, *args, **kwargs):
return func(*args, **kwargs)
@@ -121,3 +125,215 @@ async def test_vision_load_materializes_local_image_to_chat_artifact(monkeypatch
stored_path = tmp_path / stored_ref.removeprefix("/a0/")
assert stored_path.read_bytes() == b"png-data"
assert updates[-1]["result"] == "1 images loaded, 0 skipped"
+
+
+def test_vision_sidecar_route_matrix_prefers_main_native_vision(monkeypatch):
+ from plugins._model_config.helpers import model_config
+
+ cases = [
+ ({"vision": False}, {}, False),
+ ({"vision": True}, {"provider": "p", "name": "v"}, False),
+ ({"vision": False}, {"provider": "p", "name": "v"}, True),
+ (
+ {"vision": True},
+ {"provider": "p", "name": "v", "override_main": True},
+ True,
+ ),
+ ]
+ for chat, vision, expected in cases:
+ monkeypatch.setattr(
+ model_config,
+ "get_effective_config",
+ lambda _agent=None, chat=chat, vision=vision: {
+ "chat_model": chat,
+ "vision_model": vision,
+ },
+ )
+ assert model_config.use_vision_sidecar() is expected
+
+
+@pytest.mark.anyio
+async def test_vision_sidecar_sends_multiple_images_once_and_keeps_history_text_only(
+ monkeypatch,
+ tmp_path,
+):
+ _install_tool_stub(monkeypatch)
+ import tools.vision_load as vision_load_module
+
+ async def direct_call(func, *args, **kwargs):
+ return func(*args, **kwargs)
+
+ calls = []
+
+ class FakeVisionModel:
+ async def unified_call(self, **kwargs):
+ calls.append(kwargs)
+ return "The second screenshot fixes the red login error.", ""
+
+ monkeypatch.setattr(vision_load_module.runtime, "call_development_function", direct_call)
+ monkeypatch.setattr(vision_load_module, "build_vision_model", lambda _agent: FakeVisionModel())
+ monkeypatch.setattr(
+ vision_load_module,
+ "get_chat_model_config",
+ lambda _agent: {"vision": True, "max_embeds": 1},
+ )
+ monkeypatch.setattr(
+ vision_load_module,
+ "get_vision_model_config",
+ lambda _agent: {"provider": "test", "name": "vision", "max_embeds": 5},
+ )
+ monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: True)
+
+ image_paths = [tmp_path / "before.png", tmp_path / "after.png"]
+ for path in image_paths:
+ path.write_bytes(b"png-data")
+
+ tool_results = []
+ raw_messages = []
+ agent = SimpleNamespace(
+ context=SimpleNamespace(id=""),
+ agent_name="Agent 0",
+ hist_add_tool_result=lambda *args, **kwargs: tool_results.append((args, kwargs)),
+ hist_add_message=lambda *args, **kwargs: raw_messages.append((args, kwargs)),
+ )
+ tool = vision_load_module.VisionLoad(
+ agent=agent,
+ name="vision_load",
+ method=None,
+ args={"paths": [str(path) for path in image_paths]},
+ message="",
+ loop_data=None,
+ )
+ tool.log = SimpleNamespace(id="vision-log", update=lambda **kwargs: None)
+
+ response = await tool.execute(
+ paths=[str(path) for path in image_paths],
+ query="Compare the login errors.",
+ )
+ response.additional = {"_responses_output_item": {"output": response.message}}
+ await tool.after_execution(response)
+
+ assert len(calls) == 1
+ content = calls[0]["messages"][1].content
+ assert content[0] == {"type": "text", "text": "Compare the login errors."}
+ assert [item["type"] for item in content].count("image_url") == 2
+ assert "fixes the red login error" in response.message
+ assert response.message != "dummy"
+ assert raw_messages == []
+ assert tool.loaded_paths == [str(path) for path in image_paths]
+ assert tool_results[0][1]["_responses_output_item"]["output"] == response.message
+
+
+@pytest.mark.anyio
+async def test_parallel_worker_consumes_parent_ephemeral_image(monkeypatch, tmp_path):
+ _install_tool_stub(monkeypatch)
+ import tools.vision_load as vision_load_module
+
+ def fake_get_abs_path(*parts):
+ return str(tmp_path.joinpath(*parts))
+
+ def fake_normalize_a0_path(path):
+ return "/a0/" + str(Path(path).relative_to(tmp_path)).replace("\\", "/")
+
+ monkeypatch.setattr(vision_load_module.chat_media.files, "get_abs_path", fake_get_abs_path)
+ monkeypatch.setattr(vision_load_module.chat_media.files, "normalize_a0_path", fake_normalize_a0_path)
+ monkeypatch.setattr(vision_load_module.VisionLoad, "_config_agent", lambda self: self.agent)
+ monkeypatch.setattr(
+ vision_load_module,
+ "get_chat_model_config",
+ lambda _agent: {"vision": True, "max_embeds": 10},
+ )
+ monkeypatch.setattr(vision_load_module, "get_vision_model_config", lambda _agent: {})
+ monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: False)
+
+ parent_id = "parent-vision"
+ ref = vision_load_module.ephemeral_images.put_image_bytes(
+ context_id=parent_id,
+ mime="image/png",
+ payload=b"png-data",
+ name="shot.png",
+ )
+ context = SimpleNamespace(
+ id="parallel-worker",
+ get_data=lambda key: parent_id
+ if key == vision_load_module.PARALLEL_WORKER_PARENT_CONTEXT_KEY
+ else None,
+ )
+ agent = SimpleNamespace(context=context, agent_name="Agent 0")
+ tool = vision_load_module.VisionLoad(
+ agent=agent,
+ name="vision_load",
+ method=None,
+ args={"paths": [ref]},
+ message="",
+ loop_data=None,
+ )
+
+ await tool.execute(paths=[ref])
+
+ assert tool.loaded_paths == ["shot.png"]
+ assert vision_load_module.ephemeral_images.get_image(ref, context_id=parent_id) is None
+ stored_ref = tool.images_dict["shot.png"]
+ assert stored_ref.startswith("/a0/usr/chats/parent-vision/images/vision-load/shot-")
+
+
+@pytest.mark.anyio
+async def test_independent_vision_sidecar_calls_can_run_concurrently(monkeypatch, tmp_path):
+ _install_tool_stub(monkeypatch)
+ import tools.vision_load as vision_load_module
+
+ active = 0
+ max_active = 0
+ call_count = 0
+
+ class FakeVisionModel:
+ async def unified_call(self, **kwargs):
+ nonlocal active, max_active, call_count
+ active += 1
+ call_count += 1
+ max_active = max(max_active, active)
+ await asyncio.sleep(0.02)
+ active -= 1
+ return "done", ""
+
+ async def direct_call(func, *args, **kwargs):
+ return func(*args, **kwargs)
+
+ monkeypatch.setattr(vision_load_module.runtime, "call_development_function", direct_call)
+ monkeypatch.setattr(vision_load_module, "build_vision_model", lambda _agent: FakeVisionModel())
+ monkeypatch.setattr(vision_load_module, "get_chat_model_config", lambda _agent: {"vision": False})
+ monkeypatch.setattr(
+ vision_load_module,
+ "get_vision_model_config",
+ lambda _agent: {"provider": "test", "name": "vision", "max_embeds": 10},
+ )
+ monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: True)
+
+ image_paths = [tmp_path / "one.png", tmp_path / "two.png"]
+ for path in image_paths:
+ path.write_bytes(b"png-data")
+
+ def make_tool(index):
+ agent = SimpleNamespace(context=SimpleNamespace(id=""), agent_name=f"Agent {index}")
+ return vision_load_module.VisionLoad(
+ agent=agent,
+ name="vision_load",
+ method=None,
+ args={"paths": [str(path) for path in image_paths]},
+ message="",
+ loop_data=None,
+ )
+
+ responses = await asyncio.gather(
+ *(
+ make_tool(index).execute(
+ paths=[str(path) for path in image_paths],
+ query=f"inspection {index}",
+ )
+ for index in range(4)
+ )
+ )
+
+ assert call_count == 4
+ assert max_active == 4
+ assert all("done" in response.message for response in responses)
diff --git a/tools/vision_load.py b/tools/vision_load.py
index 7358afbde..b29ff931f 100644
--- a/tools/vision_load.py
+++ b/tools/vision_load.py
@@ -1,89 +1,219 @@
-from helpers.print_style import PrintStyle
-from helpers.tool import Tool, Response
-from helpers import runtime, files, plugins, ephemeral_images, images, chat_media
+import asyncio
+import json
from mimetypes import guess_type
-from helpers import history
-# image token estimation for context window
+from langchain_core.messages import HumanMessage, SystemMessage
+
+from helpers import chat_media, ephemeral_images, files, history, images, runtime
+from helpers.parallel_tools import PARALLEL_WORKER_PARENT_CONTEXT_KEY, coerce_bool
+from helpers.print_style import PrintStyle
+from helpers.tool import Response, Tool
+from plugins._model_config.helpers.model_config import (
+ build_vision_model,
+ get_chat_model_config,
+ get_vision_model_config,
+ use_vision_sidecar,
+)
+
TOKENS_ESTIMATE = 1500
+VISION_TIMEOUT_SECONDS = 300
+VISION_SYSTEM_PROMPT = (
+ "You are a precise vision analyst. Answer only what was asked about the images. "
+ "Be concise and factual. Preserve exact visible text when asked to read it."
+)
+DEFAULT_VISION_QUERY = (
+ "Describe the images precisely, including the key objects, visible text, and layout."
+)
class VisionLoad(Tool):
- async def execute(self, paths: list[str] = [], **kwargs) -> Response:
-
- self.images_dict = {}
+ async def execute(
+ self,
+ paths: list[str] | str | None = None,
+ query: str = "",
+ raw: bool = False,
+ **kwargs,
+ ) -> Response:
+ self.images_dict: dict[str, str] = {}
self.loaded_paths: list[str] = []
self.skipped_paths: list[str] = []
-
- max_embeds = self._get_max_embeds()
- requested = [
- (str(path or "").strip(), self._display_input_path(str(path or "").strip(), idx + 1))
- for idx, path in enumerate(paths)
- ]
- limited_paths = requested if max_embeds <= 0 else requested[-max_embeds:]
- self.skipped_paths = (
- [display for _, display in requested[:-max_embeds]]
- if max_embeds > 0 and len(requested) > max_embeds
- else []
+ self._config_owner = self._config_agent()
+ self._main_has_vision = bool(
+ get_chat_model_config(self._config_owner).get("vision", False)
)
+ self._delegated = use_vision_sidecar(self._config_owner) and not (
+ coerce_bool(raw, False) and self._main_has_vision
+ )
+ self._max_embeds = self._get_max_embeds()
- for idx, (path, display_path) in enumerate(limited_paths):
+ normalized = self._normalize_paths(paths)
+ if isinstance(normalized, str):
+ self._history_result = normalized
+ return Response(message=normalized, break_loop=False)
+
+ requested = [
+ (path.strip(), self._display_input_path(path.strip(), index + 1))
+ for index, path in enumerate(normalized)
+ ]
+ limited = requested if self._max_embeds <= 0 else requested[-self._max_embeds :]
+ if self._max_embeds > 0 and len(requested) > self._max_embeds:
+ self.skipped_paths = [display for _, display in requested[: -self._max_embeds]]
+
+ for index, (path, display_path) in enumerate(limited):
if not path:
continue
if ephemeral_images.is_ref(path):
- image = ephemeral_images.consume_image(
- path,
- context_id=self._context_id(),
- )
+ image = ephemeral_images.consume_image(path, context_id=self._context_id())
if image is None:
continue
- display = image.display_name or display_path
+ display_path = image.display_name or display_path
stored_ref = self._store_ephemeral_image(image)
- if stored_ref:
- self.images_dict[display] = stored_ref
- self.loaded_paths.append(display)
- continue
- if self._is_data_image_url(path):
- stored_ref = self._store_data_url(path, preferred_name=f"vision-load-{idx + 1}.png")
- if stored_ref:
- self.images_dict[display_path] = stored_ref
- self.loaded_paths.append(display_path)
- continue
- if not await runtime.call_development_function(files.exists, str(path)):
+ elif self._is_data_image_url(path):
+ stored_ref = self._store_data_url(
+ path, preferred_name=f"vision-load-{index + 1}.png"
+ )
+ elif await runtime.call_development_function(files.exists, path):
+ mime_type, _ = guess_type(path)
+ if not mime_type or not mime_type.startswith("image/"):
+ continue
+ try:
+ stored_ref = self._store_local_image(
+ path, preferred_name=files.basename(path)
+ )
+ except (FileNotFoundError, OSError, ValueError):
+ continue
+ else:
continue
- if path not in self.images_dict:
- mime_type, _ = guess_type(str(path))
- if mime_type and mime_type.startswith("image/"):
- try:
- stored_ref = self._store_local_image(path, preferred_name=files.basename(path))
- self.images_dict[display_path] = stored_ref
- self.loaded_paths.append(display_path)
- except (FileNotFoundError, OSError, ValueError):
- continue
+ if stored_ref:
+ self.images_dict[display_path] = stored_ref
+ self.loaded_paths.append(display_path)
- return Response(message="dummy", break_loop=False)
+ summary = self._summary()
+ if self._delegated and self.images_dict:
+ try:
+ capsule = await self._call_vision_model(
+ list(self.images_dict.values()), self._query(query, kwargs)
+ )
+ message = (
+ f"Vision Model analyzed {len(self.images_dict)} image(s)"
+ f"; {len(self.skipped_paths)} skipped.\n\n{capsule.strip()}"
+ )
+ self._history_result = message
+ return Response(message=message, break_loop=False)
+ except Exception as exc:
+ message = f"Vision Model error: {str(exc)[:1000]}"
+ self._history_result = f"{summary}\n\n{message}"
+ return Response(message=message, break_loop=False)
+
+ if self.images_dict and not self._main_has_vision:
+ summary += (
+ "\n\nImages were not injected because neither Main native vision nor "
+ "a usable Vision Model is active."
+ )
+ self._history_result = (
+ summary if self.images_dict or self.skipped_paths else "No images processed"
+ )
+ message = (
+ "No images processed"
+ if not self.images_dict and not self.skipped_paths
+ else f"{len(self.images_dict)} images loaded, {len(self.skipped_paths)} skipped"
+ )
+ return Response(message=message, break_loop=False)
+
+ async def after_execution(self, response: Response, **kwargs):
+ log_id = str(getattr(getattr(self, "log", None), "id", "") or "")
+ self.agent.hist_add_tool_result(
+ self.name,
+ self._history_result,
+ id=log_id,
+ **(response.additional or {}),
+ )
+
+ if self.images_dict and self._main_has_vision and not self._delegated:
+ content = [
+ {"type": "image_url", "image_url": {"url": image_path}}
+ for image_path in self.images_dict.values()
+ ]
+ self.agent.hist_add_message(
+ False,
+ content=history.RawMessage(
+ raw_content=content,
+ preview="",
+ ),
+ tokens=TOKENS_ESTIMATE * len(content),
+ )
+
+ PrintStyle(
+ font_color="#1B4F72", background_color="white", padding=True, bold=True
+ ).print(f"{self.agent.agent_name}: Response from tool '{self.name}'")
+ PrintStyle(font_color="#85C1E9").print(response.message)
+ if getattr(self, "log", None):
+ self.log.update(result=response.message)
def _get_max_embeds(self) -> int:
- cfg = plugins.get_plugin_config("_model_config", agent=self.agent) or {}
- chat_cfg = cfg.get("chat_model", {})
- max_embeds = chat_cfg.get("max_embeds", 10)
- return int(max_embeds or 0)
+ cfg = (
+ get_vision_model_config(self._config_owner)
+ if self._delegated
+ else get_chat_model_config(self._config_owner)
+ )
+ try:
+ return int(cfg.get("max_embeds", 10) or 0)
+ except (TypeError, ValueError):
+ return 10
def _context_id(self) -> str:
- return str(getattr(getattr(self.agent, "context", None), "id", "") or "").strip()
+ context = getattr(self.agent, "context", None)
+ if not context:
+ return ""
+ get_data = getattr(context, "get_data", None)
+ parent_id = get_data(PARALLEL_WORKER_PARENT_CONTEXT_KEY) if get_data else ""
+ return str(parent_id or getattr(context, "id", "") or "").strip()
+
+ def _config_agent(self):
+ context = getattr(self.agent, "context", None)
+ get_data = getattr(context, "get_data", None)
+ parent_id = get_data(PARALLEL_WORKER_PARENT_CONTEXT_KEY) if get_data else ""
+ if parent_id:
+ from agent import AgentContext
+
+ parent = AgentContext.get(str(parent_id))
+ if parent:
+ return parent.agent0
+ return self.agent
+
+ async def _call_vision_model(self, image_paths: list[str], query: str) -> str:
+ model = build_vision_model(self._config_owner)
+ content = [{"type": "text", "text": query or DEFAULT_VISION_QUERY}]
+ content.extend(
+ {"type": "image_url", "image_url": {"url": path}}
+ for path in image_paths
+ )
+ response, _ = await asyncio.wait_for(
+ model.unified_call(
+ messages=[
+ SystemMessage(content=VISION_SYSTEM_PROMPT),
+ HumanMessage(content=content),
+ ],
+ explicit_caching=False,
+ max_tokens=2000,
+ ),
+ timeout=VISION_TIMEOUT_SECONDS,
+ )
+ if not str(response or "").strip():
+ raise RuntimeError("Vision Model returned an empty response.")
+ return str(response)
def _store_ephemeral_image(self, image: ephemeral_images.EphemeralImage) -> str:
context_id = self._context_id()
if not context_id:
return image.data_url
source = chat_media.infer_source(image.ref, image.display_name)
- category = chat_media.category_for_source(source)
saved = chat_media.save_image_base64(
context_id=context_id,
data=image.data,
mime_type=image.mime,
- category=category,
+ category=chat_media.category_for_source(source),
source=source,
preferred_name=image.display_name,
)
@@ -94,11 +224,10 @@ class VisionLoad(Tool):
if not context_id:
return data_url
source = chat_media.infer_source(data_url, preferred_name)
- category = chat_media.category_for_source(source)
saved = chat_media.save_image_data_url(
context_id=context_id,
data_url=data_url,
- category=category,
+ category=chat_media.category_for_source(source),
source=source,
preferred_name=preferred_name,
)
@@ -115,6 +244,37 @@ class VisionLoad(Tool):
preferred_name=preferred_name,
)
+ def _summary(self) -> str:
+ loaded = "\n".join(self.loaded_paths) if self.loaded_paths else "none"
+ skipped = "\n".join(self.skipped_paths) if self.skipped_paths else "none"
+ return (
+ f"Loaded images ({len(self.loaded_paths)}):\n{loaded}\n\n"
+ f"Skipped images ({len(self.skipped_paths)}, max {self._max_embeds}):\n{skipped}"
+ )
+
+ @staticmethod
+ def _normalize_paths(paths: list[str] | str | None) -> list[str] | str:
+ if isinstance(paths, str):
+ try:
+ decoded = json.loads(paths)
+ except json.JSONDecodeError:
+ decoded = paths
+ paths = decoded if isinstance(decoded, list) else [paths]
+ if paths is None:
+ return []
+ if not isinstance(paths, (list, tuple)):
+ return "vision_load error: `paths` must be an array of image paths."
+ return [str(path or "").strip() for path in paths]
+
+ @staticmethod
+ def _query(query: str, kwargs: dict) -> str:
+ if str(query or "").strip():
+ return str(query).strip()
+ for key in ("prompt", "question", "instruction", "focus", "request"):
+ if str(kwargs.get(key) or "").strip():
+ return str(kwargs[key]).strip()
+ return DEFAULT_VISION_QUERY
+
@staticmethod
def _is_data_image_url(value: str) -> bool:
normalized = str(value or "").strip().lower()
@@ -125,57 +285,5 @@ class VisionLoad(Tool):
if ephemeral_images.is_ref(value):
return ephemeral_images.display_ref(value)
if cls._is_data_image_url(value):
- prefix = value.split(",", 1)[0]
- return f"{prefix},"
+ return f"{value.split(',', 1)[0]},"
return value
-
- async def after_execution(self, response: Response, **kwargs):
-
- # build image data messages for LLMs, or error message
- content = []
- loaded_count = len(self.loaded_paths)
- skipped_count = len(self.skipped_paths)
- loaded_summary = "\n".join(self.loaded_paths) if self.loaded_paths else "none"
- skipped_summary = "\n".join(self.skipped_paths) if self.skipped_paths else "none"
- summary = (
- f"Loaded images: {loaded_count}\n"
- f"Loaded images:\n{loaded_summary}\n\n"
- f"Skipped images: {skipped_count}\n"
- f"Skipped images (max {self._get_max_embeds()} loaded at a time according to model configuration):\n{skipped_summary}"
- )
- if self.images_dict:
- self.agent.hist_add_tool_result(self.name, summary, id=self.log.id if self.log else "")
- for path, image_path in self.images_dict.items():
- if image_path:
- content.append(
- {
- "type": "image_url",
- "image_url": {"url": image_path},
- }
- )
- else:
- content.append(
- {
- "type": "text",
- "text": "Error processing image " + path,
- }
- )
- # append as raw message content for LLMs with vision tokens estimate
- msg = history.RawMessage(raw_content=content, preview="")
- self.agent.hist_add_message(
- False, content=msg, tokens=TOKENS_ESTIMATE * len(content)
- )
- else:
- self.agent.hist_add_tool_result(self.name, summary if self.skipped_paths else "No images processed", id=self.log.id if self.log else "")
-
- # print and log short version
- message = (
- "No images processed"
- if not self.images_dict and not self.skipped_paths
- else f"{loaded_count} images loaded, {skipped_count} skipped"
- )
- PrintStyle(
- font_color="#1B4F72", background_color="white", padding=True, bold=True
- ).print(f"{self.agent.agent_name}: Response from tool '{self.name}'")
- PrintStyle(font_color="#85C1E9").print(message)
- self.log.update(result=message)
diff --git a/tools/vision_load.py.dox.md b/tools/vision_load.py.dox.md
index da9531242..2269eab56 100644
--- a/tools/vision_load.py.dox.md
+++ b/tools/vision_load.py.dox.md
@@ -3,7 +3,7 @@
## Purpose
- Own the `vision_load.py` agent tool.
-- This module loads images into model-visible content for vision-capable models.
+- This module routes images either into Main model-visible content or through the preset's optional Vision Model.
- Keep this file-level DOX profile synchronized with `vision_load.py` because this directory is intentionally flat.
## Ownership
@@ -12,13 +12,19 @@
- `vision_load.py.dox.md` owns durable notes about responsibilities, contracts, side effects, and verification for that implementation.
- Classes:
- `VisionLoad` (`Tool`)
- - `async execute(self, paths: list[str]=..., **kwargs) -> Response`
+ - `async execute(self, paths, query="", raw=False, **kwargs) -> Response`
- `async after_execution(self, response: Response, **kwargs)`
- Notable constants/configuration names: `TOKENS_ESTIMATE`.
## Runtime Contracts
- Tool modules must define `helpers.tool.Tool` subclasses and return `helpers.tool.Response` from `execute(...)`.
+- One call may contain multiple paths. The delegated route sends every selected path in one Vision Model request and returns one textual capsule.
+- Main native vision wins unless the effective preset selects the sidecar route; `raw=true` returns to Main native vision only when Main supports it.
+- Delegation completes during `execute(...)` so native Responses function output contains the real capsule before `after_execution(...)` persists it.
+- Delegated history contains the text capsule only. Native history contains the tool result followed by one raw message holding all loaded image blocks.
+- Direct parallel workers resolve ephemeral refs, model routing, and durable chat-media storage against their recorded parent context; independent vision jobs remain generic parallel jobs.
+- `max_embeds` comes from the model that actually receives the images.
- Update this file whenever tool arguments, output shape, `break_loop` behavior, intervention handling, prompt instructions, or side effects change.
- `VisionLoad` is a `Tool`.
- `VisionLoad` defines `execute(...)`.
@@ -27,7 +33,7 @@
## Key Concepts
-- Important called helpers/classes observed in the source: `self._get_max_embeds`, `Response`, `str.strip`, `self._context_id`, `chat_media.infer_source`, `chat_media.category_for_source`, `chat_media.save_image_base64`, `chat_media.save_image_data_url`, `chat_media.materialize_image_ref`, `str.strip.lower`, `ephemeral_images.is_ref`, `cls._is_data_image_url`, `self._is_data_image_url`, `plugins.get_plugin_config`, `images.to_data_url`, `normalized.startswith`, `ephemeral_images.display_ref`, `join`, `self.agent.hist_add_tool_result`, `history.RawMessage`.
+- Important called helpers/classes observed in the source: `build_vision_model`, `use_vision_sidecar`, `self._get_max_embeds`, `Response`, `self._context_id`, `chat_media.save_image_base64`, `chat_media.save_image_data_url`, `chat_media.materialize_image_ref`, `ephemeral_images.consume_image`, `images.to_data_url`, `self.agent.hist_add_tool_result`, `history.RawMessage`.
- Keep request/response, tool, or helper semantics documented here at the same time as source changes.
## Work Guidance