mirror of
https://github.com/agent0ai/agent-zero.git
synced 2026-08-27 09:13:25 +00:00
Integrate vision sidecar routing with model presets
Add a strict per-preset sidecar slot, Main-first routing, one-call multi-image analysis, parallel-worker image resolution, and inline model settings with focused regression coverage.
This commit is contained in:
Alessandro
parent
80bcb3cf0d
commit
59f5ae03ea
21 changed files with 880 additions and 150 deletions
|
|
@ -51,10 +51,15 @@ async def build_prompt(agent: Agent) -> str:
|
|||
prompt = agent.read_prompt("agent.system.tools.md", tools=tools_str)
|
||||
|
||||
# vision support
|
||||
from plugins._model_config.helpers.model_config import get_chat_model_config
|
||||
from plugins._model_config.helpers.model_config import (
|
||||
get_chat_model_config,
|
||||
use_vision_sidecar,
|
||||
)
|
||||
|
||||
chat_cfg = get_chat_model_config(agent)
|
||||
if chat_cfg.get("vision", False):
|
||||
if use_vision_sidecar(agent):
|
||||
prompt += "\n\n" + agent.read_prompt("agent.system.tools_vision_sidecar.md")
|
||||
elif chat_cfg.get("vision", False):
|
||||
prompt += "\n\n" + agent.read_prompt("agent.system.tools_vision.md")
|
||||
|
||||
return prompt
|
||||
|
|
|
|||
|
|
@ -114,8 +114,13 @@ def _local_tool_prompts(agent: Any) -> list[tuple[str, str]]:
|
|||
|
||||
def _vision_tool_prompt(agent: Any) -> str:
|
||||
try:
|
||||
from plugins._model_config.helpers.model_config import get_chat_model_config
|
||||
from plugins._model_config.helpers.model_config import (
|
||||
get_chat_model_config,
|
||||
use_vision_sidecar,
|
||||
)
|
||||
|
||||
if use_vision_sidecar(agent):
|
||||
return agent.read_prompt("agent.system.tools_vision_sidecar.md")
|
||||
if not get_chat_model_config(agent).get("vision", False):
|
||||
return ""
|
||||
return agent.read_prompt("agent.system.tools_vision.md")
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@
|
|||
|
||||
## Local Contracts
|
||||
|
||||
- Build local function tools from enabled `agent.system.tool.*.md` prompt files and include `vision_load` only when the active chat model enables the matching vision prompt.
|
||||
- Build local function tools from enabled `agent.system.tool.*.md` prompt files and include `vision_load` when either Main native vision or the effective preset's Vision Model enables the matching prompt.
|
||||
- Discover local prompt files through `helpers.subagents.get_paths`; this module
|
||||
owns the Responses-specific prompt-name compatibility rules.
|
||||
- Local prompt-derived function names use existing bullet declarations that pair a backticked name with `arg` or `args` for multi-tool prompt files, otherwise prefer explicit `"tool_name"` examples, then the first prompt heading, and finally the prompt filename.
|
||||
|
|
@ -25,8 +25,8 @@
|
|||
- Preserve original Agent Zero tool names through the native Responses name map.
|
||||
- Keep MCP tool schemas merged after local prompt-derived tools.
|
||||
- Apply `helpers.tool_policy` before emitting local or MCP schemas; a blocked
|
||||
capability is absent from provider-native tool definitions. Vision remains
|
||||
controlled solely by the active chat model configuration.
|
||||
capability is absent from provider-native tool definitions. Vision routing
|
||||
is controlled by the effective model preset rather than Agent Editor.
|
||||
- Connector remote tools are advertised only when `_a0_connector` runtime metadata says the matching connected CLI capability is currently available.
|
||||
|
||||
## Work Guidance
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@
|
|||
|
||||
## Local Contracts
|
||||
|
||||
- `Default` is the first global preset and cannot be deleted or renamed. It owns the complete main, utility, and embedding baseline.
|
||||
- `Default` is the first global preset and cannot be deleted or renamed. It owns the complete main, utility, and embedding baseline; its Vision Model slot is optional.
|
||||
- Preset definitions are global. Global, project, agent-profile, and project/profile plugin configs persist only `model_preset`; chats may persist a preset reference as their explicit override.
|
||||
- Preserve scoped plugin resolution order and fall back invalid or missing scope/chat references to `Default`.
|
||||
- Project Settings `llm` payloads are owned here through the generic `helpers.projects` project extension-data hooks; keep project helper code agnostic to `_model_config` paths, presets, and inheritance rules.
|
||||
|
|
@ -22,7 +22,12 @@
|
|||
- Check API-key readiness only for the effective model configuration; unused global presets must not produce Welcome-screen warnings.
|
||||
- Coordinate OAuth-backed providers with `_oauth` instead of hardcoding provider-specific auth here.
|
||||
- `model_config_get` exposes `model_configured` as a derived chat-model readiness flag from provider, model name, and API-key availability.
|
||||
- Non-default presets may inherit omitted slots or durable tuning from `Default`, but must replace or clear per-slot `kwargs` so provider-specific extra params never leak across model providers.
|
||||
- Non-default presets may inherit omitted main, utility, or embedding slots and durable tuning from `Default`, but must replace or clear per-slot `kwargs` so provider-specific extra params never leak across model providers.
|
||||
- The optional `vision` slot is strictly per preset and never inherited from `Default`; an empty slot disables sidecar vision for that preset.
|
||||
- Main native vision wins by default. A configured Vision Model handles `vision_load` when Main lacks vision, or when that preset explicitly enables `override_main`.
|
||||
- Keep the optional Vision provider/model selector inside the Main Model card and flush with Main's field alignment, without a nested left inset. Show it only while Main vision is disabled or `override_main` is enabled; do not render a standalone Vision Model card.
|
||||
- Show `Use separate Vision Model` immediately below `Supports Vision` while Main vision is enabled, not inside Advanced Settings; describe the disabled state as using Main's native vision.
|
||||
- In model overviews, render the effective Vision Model as a text-only `Vision override / Provider / Model` child aligned with Main's provider column, not as an icon-bearing peer row.
|
||||
- Changing a model provider in the settings UI must clear `api_base` and `kwargs` because both may be provider-specific.
|
||||
- Repair provider-specific model-config aliases at the model-config read/build boundary; keep provider-specific repairs out of provider-agnostic core wrappers such as `models.py`.
|
||||
- `modelConfig.createPresetEditor()` owns local preset drafts, row actions, and stable UI-only row keys so deletion or renaming cannot rebind nested model fields.
|
||||
|
|
|
|||
|
|
@ -65,6 +65,16 @@
|
|||
</span>
|
||||
</div>
|
||||
</template>
|
||||
<template x-if="preset.vision?.name && (!preset.chat?.vision || preset.vision?.override_main)">
|
||||
<div class="model-switcher-model-row">
|
||||
<span class="model-switcher-model-label">Vision</span>
|
||||
<span class="model-switcher-model-value">
|
||||
<span x-text="preset.vision.provider" style="opacity:0.5;"></span>
|
||||
<span style="opacity:0.3; margin:0 3px;">/</span>
|
||||
<span x-text="preset.vision.name"></span>
|
||||
</span>
|
||||
</div>
|
||||
</template>
|
||||
<template x-if="preset.utility?.name">
|
||||
<div class="model-switcher-model-row">
|
||||
<span class="model-switcher-model-label">Utility</span>
|
||||
|
|
|
|||
|
|
@ -16,11 +16,21 @@ PRESET_SCOPE_GLOBAL = "global"
|
|||
PRESET_SCOPE_PROJECT = "project"
|
||||
PRESET_SLOT_CONFIG_SECTIONS = {
|
||||
"chat": "chat_model",
|
||||
"vision": "vision_model",
|
||||
"utility": "utility_model",
|
||||
"embedding": "embedding_model",
|
||||
}
|
||||
MODEL_SLOT_PRESET_REPLACE_FIELDS = {"kwargs"}
|
||||
IMPLICIT_PRESET_SLOT_DEFAULTS = {
|
||||
"vision": {
|
||||
"vision": True,
|
||||
"max_embeds": 10,
|
||||
"override_main": False,
|
||||
"rl_requests": 0,
|
||||
"rl_input": 0,
|
||||
"rl_output": 0,
|
||||
"kwargs": {},
|
||||
},
|
||||
"utility": {
|
||||
"ctx_length": 128000,
|
||||
"ctx_input": 0.7,
|
||||
|
|
@ -267,6 +277,9 @@ def _clean_preset_for_file(preset: dict) -> dict:
|
|||
slot_config = preset.get(slot)
|
||||
if isinstance(slot_config, dict):
|
||||
slot_clean = _strip_ui_fields(slot_config, strip_api_key=True)
|
||||
if slot == "vision" and _slot_has_identity(slot_clean):
|
||||
slot_clean["vision"] = True
|
||||
slot_clean.setdefault("max_embeds", 10)
|
||||
cleaned[slot] = (
|
||||
slot_clean
|
||||
if name == DEFAULT_PRESET_NAME
|
||||
|
|
@ -339,7 +352,12 @@ def validate_presets(presets: list, *, require_default: bool = True) -> list:
|
|||
def normalize_config_for_save(config: dict) -> dict:
|
||||
"""Remove UI-only fields and inline API keys before storing scoped config."""
|
||||
cleaned = deepcopy(config or {})
|
||||
for section_name in ("chat_model", "utility_model", "embedding_model"):
|
||||
for section_name in (
|
||||
"chat_model",
|
||||
"vision_model",
|
||||
"utility_model",
|
||||
"embedding_model",
|
||||
):
|
||||
section = cleaned.get(section_name)
|
||||
if isinstance(section, dict):
|
||||
cleaned[section_name] = _strip_ui_fields(section, strip_api_key=True)
|
||||
|
|
@ -659,10 +677,12 @@ def build_config_from_preset(
|
|||
continue
|
||||
slot_config = _get_preset_slot_config(preset, slot)
|
||||
if not _should_apply_preset_slot(slot, slot_config):
|
||||
if slot == "vision":
|
||||
config[section] = {}
|
||||
continue
|
||||
config[section] = _merge_model_slot(
|
||||
slot,
|
||||
config.get(section, {}),
|
||||
{} if slot == "vision" else config.get(section, {}),
|
||||
slot_config,
|
||||
strip_api_key=strip_api_key,
|
||||
)
|
||||
|
|
@ -738,6 +758,23 @@ def get_chat_model_config(agent=None) -> dict:
|
|||
return get_effective_config(agent).get("chat_model", {})
|
||||
|
||||
|
||||
def get_vision_model_config(agent=None) -> dict:
|
||||
"""Get the optional, strictly per-preset Vision Model config."""
|
||||
return get_effective_config(agent).get("vision_model", {})
|
||||
|
||||
|
||||
def use_vision_sidecar(agent=None) -> bool:
|
||||
"""Return whether vision_load should delegate to the preset's Vision Model."""
|
||||
vision_cfg = get_vision_model_config(agent)
|
||||
if not (
|
||||
str(vision_cfg.get("provider") or "").strip()
|
||||
and str(vision_cfg.get("name") or "").strip()
|
||||
):
|
||||
return False
|
||||
chat_cfg = get_chat_model_config(agent)
|
||||
return not bool(chat_cfg.get("vision")) or bool(vision_cfg.get("override_main"))
|
||||
|
||||
|
||||
def get_utility_model_config(agent=None) -> dict:
|
||||
"""Get utility model config, with per-chat override if active."""
|
||||
return get_effective_config(agent).get("utility_model", {})
|
||||
|
|
@ -835,6 +872,16 @@ def build_utility_model(agent=None):
|
|||
)
|
||||
|
||||
|
||||
def build_vision_model(agent=None):
|
||||
"""Build the optional Vision Model selected by the effective preset."""
|
||||
cfg = get_vision_model_config(agent)
|
||||
mc = build_model_config(cfg, models.ModelType.CHAT)
|
||||
mc.vision = True
|
||||
return models.get_chat_model(
|
||||
mc.provider, mc.name, model_config=mc, **mc.build_kwargs()
|
||||
)
|
||||
|
||||
|
||||
def build_embedding_model(agent=None):
|
||||
"""Build and return an embedding model wrapper."""
|
||||
cfg = get_embedding_model_config(agent)
|
||||
|
|
@ -881,6 +928,8 @@ def get_missing_api_key_providers(agent=None) -> list[dict]:
|
|||
("Utility Model", cfg.get("utility_model", {})),
|
||||
("Embedding Model", get_embedding_model_config(agent)),
|
||||
]
|
||||
if use_vision_sidecar(agent):
|
||||
checks.insert(1, ("Vision Model", cfg.get("vision_model", {})))
|
||||
|
||||
for label, model_cfg in checks:
|
||||
provider = model_cfg.get("provider", "")
|
||||
|
|
|
|||
|
|
@ -21,7 +21,7 @@
|
|||
<div class="preset-editor-intro">
|
||||
<div class="field-title">Model Presets</div>
|
||||
<div class="field-description">
|
||||
Each preset contains the complete main, utility, and embedding model setup. Default is always available.
|
||||
Each preset contains the main, utility, embedding, and optional vision-sidecar setup. Default is always available.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
|
@ -67,9 +67,21 @@
|
|||
<div class="section-title">Main Model</div>
|
||||
<div class="section-description">Primary model for conversations, reasoning, and tools.</div>
|
||||
</div>
|
||||
<div x-data="{ get model() { return selectedPreset.chat; }, modelType: 'chat', providers: $store.modelConfig.chatProviders, searchType: 'chat', apiKeyMode: 'store' }">
|
||||
<div x-data="{ get model() { return selectedPreset.chat; }, get visionModel() { return selectedPreset.vision; }, modelType: 'chat', providers: $store.modelConfig.chatProviders, searchType: 'chat', apiKeyMode: 'store' }">
|
||||
<x-component path="/plugins/_model_config/webui/model-field.html"></x-component>
|
||||
</div>
|
||||
<template x-if="!selectedPreset.chat.vision || selectedPreset.vision.override_main">
|
||||
<div class="vision-sidecar-selector">
|
||||
<div class="field-title">Vision sidecar</div>
|
||||
<div class="field-description"
|
||||
x-text="selectedPreset.chat.vision
|
||||
? 'Overrides Main model vision for vision_load.'
|
||||
: 'Used by vision_load while Main model vision is disabled. Leave empty to disable image analysis.'"></div>
|
||||
<div x-data="{ get model() { return selectedPreset.vision; }, modelType: 'vision', providers: $store.modelConfig.chatProviders, searchType: 'chat', apiKeyMode: 'store' }">
|
||||
<x-component path="/plugins/_model_config/webui/model-field.html"></x-component>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
</section>
|
||||
|
||||
<section class="preset-model-section">
|
||||
|
|
@ -166,6 +178,11 @@
|
|||
margin-top: 0;
|
||||
}
|
||||
|
||||
.vision-sidecar-selector {
|
||||
margin: 0.75rem 0 0;
|
||||
padding: 0.25rem 0 0;
|
||||
}
|
||||
|
||||
.preset-editor-secondary-actions {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import { switcherState, switcherMethods } from "/plugins/_model_config/webui/swi
|
|||
|
||||
export const MODEL_SECTIONS = [
|
||||
{ key: 'chat_model', title: 'Main Model', desc: 'Primary model for chat, reasoning, and browser tasks.' },
|
||||
{ key: 'vision_model', title: 'Vision Model', desc: 'Optional model used by vision_load when Main vision is unavailable or overridden.' },
|
||||
{ key: 'utility_model', title: 'Utility Model', desc: 'Lightweight model for background tasks: memory management, prompt preparation, summarization.' },
|
||||
{ key: 'embedding_model', title: 'Embedding Model', desc: 'Model for generating vector embeddings used in knowledge retrieval.' }
|
||||
];
|
||||
|
|
@ -59,6 +60,15 @@ function isBlankPresetValue(value) {
|
|||
}
|
||||
|
||||
const IMPLICIT_PRESET_SLOT_DEFAULTS = {
|
||||
vision: {
|
||||
vision: true,
|
||||
max_embeds: 10,
|
||||
override_main: false,
|
||||
rl_requests: 0,
|
||||
rl_input: 0,
|
||||
rl_output: 0,
|
||||
kwargs: {},
|
||||
},
|
||||
utility: {
|
||||
ctx_length: 128000,
|
||||
ctx_input: 0.7,
|
||||
|
|
@ -143,15 +153,21 @@ export function configFromPreset(preset, baseConfig, stripApiKey = true) {
|
|||
const config = clonePlain(baseConfig || {});
|
||||
const slots = [
|
||||
['chat', 'chat_model'],
|
||||
['vision', 'vision_model'],
|
||||
['utility', 'utility_model'],
|
||||
['embedding', 'embedding_model'],
|
||||
];
|
||||
|
||||
for (const [slotKey, sectionKey] of slots) {
|
||||
const slot = preset?.[slotKey];
|
||||
if (!slot || typeof slot !== 'object') continue;
|
||||
if (!hasModelIdentity(slot)) continue;
|
||||
config[sectionKey] = mergeModelSlot(config[sectionKey] || {}, slot, stripApiKey, slotKey);
|
||||
if (slotKey === 'vision') config[sectionKey] = {};
|
||||
if (!slot || typeof slot !== 'object' || !hasModelIdentity(slot)) continue;
|
||||
config[sectionKey] = mergeModelSlot(
|
||||
slotKey === 'vision' ? {} : (config[sectionKey] || {}),
|
||||
slot,
|
||||
stripApiKey,
|
||||
slotKey,
|
||||
);
|
||||
}
|
||||
|
||||
return config;
|
||||
|
|
@ -206,8 +222,10 @@ export const store = createStore("modelConfig", {
|
|||
const source = (rawPresets || []).filter(p => p && typeof p === 'object');
|
||||
const rawDefault = source.find(p => String(p.name || '').toLowerCase() === 'default') || {};
|
||||
const slot = value => ({ provider: '', name: '', api_key: '', api_base: '', kwargs: {}, ...(value || {}) });
|
||||
const visionSlot = value => ({ ...slot(value), vision: true, max_embeds: Number(value?.max_embeds ?? 10), override_main: !!value?.override_main });
|
||||
const defaultConfig = {
|
||||
chat_model: slot(rawDefault.chat),
|
||||
vision_model: hasModelIdentity(rawDefault.vision) ? visionSlot(rawDefault.vision) : {},
|
||||
utility_model: slot(rawDefault.utility),
|
||||
embedding_model: slot(rawDefault.embedding),
|
||||
};
|
||||
|
|
@ -219,6 +237,7 @@ export const store = createStore("modelConfig", {
|
|||
return {
|
||||
name: p.name || '',
|
||||
chat: { ...slot(effective.chat_model), _kwargs_text: kwargsToText(effective.chat_model?.kwargs) },
|
||||
vision: { ...visionSlot(effective.vision_model), _kwargs_text: kwargsToText(effective.vision_model?.kwargs) },
|
||||
utility: { ...slot(effective.utility_model), _kwargs_text: kwargsToText(effective.utility_model?.kwargs) },
|
||||
embedding: { ...slot(effective.embedding_model), _kwargs_text: kwargsToText(effective.embedding_model?.kwargs) },
|
||||
};
|
||||
|
|
@ -273,6 +292,7 @@ export const store = createStore("modelConfig", {
|
|||
// Config field initialization (converts kwargs dicts to editable text)
|
||||
initConfigFields(config) {
|
||||
if (config?.chat_model) config.chat_model._kwargs_text = kwargsToText(config.chat_model.kwargs);
|
||||
if (config?.vision_model) config.vision_model._kwargs_text = kwargsToText(config.vision_model.kwargs);
|
||||
if (config?.utility_model) config.utility_model._kwargs_text = kwargsToText(config.utility_model.kwargs);
|
||||
if (config?.embedding_model) config.embedding_model._kwargs_text = kwargsToText(config.embedding_model.kwargs);
|
||||
},
|
||||
|
|
@ -340,6 +360,7 @@ export const store = createStore("modelConfig", {
|
|||
|| this.presets[0]
|
||||
|| {
|
||||
chat: { provider: '', name: '', api_base: '', kwargs: {}, _kwargs_text: '' },
|
||||
vision: { provider: '', name: '', api_base: '', vision: true, max_embeds: 10, override_main: false, kwargs: {}, _kwargs_text: '' },
|
||||
utility: { provider: '', name: '', api_base: '', kwargs: {}, _kwargs_text: '' },
|
||||
embedding: { provider: '', name: '', api_base: '', kwargs: {}, _kwargs_text: '' },
|
||||
}
|
||||
|
|
@ -420,9 +441,10 @@ export const store = createStore("modelConfig", {
|
|||
const clean = presets.map(p => {
|
||||
const c = { name: p.name };
|
||||
const isDefault = p._originalName === DEFAULT_PRESET_NAME;
|
||||
for (const slot of ['chat', 'utility']) {
|
||||
for (const slot of ['chat', 'vision', 'utility']) {
|
||||
if (p[slot]) {
|
||||
const rest = cleanPresetSlot(p[slot], true, slot, isDefault);
|
||||
if (slot === 'vision' && hasModelIdentity(rest)) rest.vision = true;
|
||||
if (hasModelIdentity(rest)) c[slot] = rest;
|
||||
}
|
||||
}
|
||||
|
|
@ -608,9 +630,15 @@ export const store = createStore("modelConfig", {
|
|||
const label = (list, id) => (list.find(x => x.value === id) || {}).label || id || '\u2014';
|
||||
return [
|
||||
{ icon: 'chat', title: 'Main', cfg: preset?.chat, pList: chatP },
|
||||
{ icon: 'eye', title: 'Vision', cfg: preset?.vision, pList: chatP },
|
||||
{ icon: 'manufacturing', title: 'Utility', cfg: preset?.utility, pList: chatP },
|
||||
{ icon: 'database', title: 'Embedding', cfg: preset?.embedding, pList: embedP },
|
||||
].map(s => ({ icon: s.icon, title: s.title, provider: label(s.pList, s.cfg?.provider), name: s.cfg?.name || '\u2014' }));
|
||||
]
|
||||
.filter(s => s.title !== 'Vision' || (
|
||||
hasModelIdentity(s.cfg)
|
||||
&& (!preset?.chat?.vision || s.cfg?.override_main)
|
||||
))
|
||||
.map(s => ({ icon: s.icon, title: s.title, provider: label(s.pList, s.cfg?.provider), name: s.cfg?.name || '\u2014' }));
|
||||
},
|
||||
|
||||
getPreset(name) {
|
||||
|
|
@ -747,9 +775,15 @@ export const store = createStore("modelConfig", {
|
|||
const label = (list, id) => (list.find(x => x.value === id) || {}).label || id || '\u2014';
|
||||
return [
|
||||
{ icon: 'chat', title: 'Main', cfg: cfg.chat_model, pList: chatP },
|
||||
{ icon: 'eye', title: 'Vision', cfg: cfg.vision_model, pList: chatP },
|
||||
{ icon: 'manufacturing', title: 'Utility', cfg: cfg.utility_model, pList: chatP },
|
||||
{ icon: 'database', title: 'Embedding', cfg: cfg.embedding_model, pList: embedP },
|
||||
].map(s => ({ icon: s.icon, title: s.title, provider: label(s.pList, s.cfg?.provider), name: s.cfg?.name || '\u2014' }));
|
||||
]
|
||||
.filter(s => s.title !== 'Vision' || (
|
||||
hasModelIdentity(s.cfg)
|
||||
&& (!cfg.chat_model?.vision || s.cfg?.override_main)
|
||||
))
|
||||
.map(s => ({ icon: s.icon, title: s.title, provider: label(s.pList, s.cfg?.provider), name: s.cfg?.name || '\u2014' }));
|
||||
},
|
||||
|
||||
async refreshModelsSummary(contextId = '') {
|
||||
|
|
|
|||
|
|
@ -10,13 +10,14 @@
|
|||
Reusable model configuration field set.
|
||||
Parent x-data scope must provide:
|
||||
model — reactive object with provider, name, api_key, api_base, ctx_length, ctx_history, ctx_input, vision, max_embeds, rl_requests, rl_input, rl_output, kwargs, _kwargs_text
|
||||
modelType — 'chat' | 'utility' | 'embedding'
|
||||
modelType — 'chat' | 'vision' | 'utility' | 'embedding'
|
||||
providers — array of { value, label }
|
||||
searchType — 'chat' | 'embedding'
|
||||
apiKeyMode — 'store' (key lives in $store.modelConfig.apiKeyValues), 'inline' (key lives in model.api_key), or 'none'
|
||||
Optional:
|
||||
providerFallback — fallback provider string for search/API key status (e.g. preset.chat.provider for utility slot)
|
||||
apiBaseFallback — fallback api_base string for search (e.g. preset.chat.api_base for utility slot)
|
||||
visionModel — vision slot object, required by the chat model's sidecar switch
|
||||
-->
|
||||
<div x-data="{ get _prov() { return model.provider || (typeof providerFallback !== 'undefined' ? providerFallback : ''); }, get _apiBase() { return model.api_base || (typeof apiBaseFallback !== 'undefined' ? apiBaseFallback : ''); } }">
|
||||
<!-- Provider -->
|
||||
|
|
@ -146,8 +147,23 @@
|
|||
</div>
|
||||
</template>
|
||||
|
||||
<template x-if="modelType === 'chat' && model.vision">
|
||||
<div class="field">
|
||||
<div class="field-label">
|
||||
<div class="field-title">Use separate Vision Model</div>
|
||||
<div class="field-description">When disabled, vision_load uses this model's native vision.</div>
|
||||
</div>
|
||||
<div class="field-control">
|
||||
<label class="toggle">
|
||||
<input type="checkbox" x-model="visionModel.override_main" />
|
||||
<span class="toggler"></span>
|
||||
</label>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<!-- Context window size (main and utility only) -->
|
||||
<template x-if="modelType !== 'embedding'">
|
||||
<template x-if="modelType === 'chat' || modelType === 'utility'">
|
||||
<div class="field">
|
||||
<div class="field-label">
|
||||
<div class="field-title">Context window size</div>
|
||||
|
|
@ -207,6 +223,18 @@
|
|||
</div>
|
||||
</template>
|
||||
|
||||
<template x-if="modelType === 'vision'">
|
||||
<div class="field">
|
||||
<div class="field-label">
|
||||
<div class="field-title">Max embeds</div>
|
||||
<div class="field-description">Maximum number of images sent to the Vision Model in one call. Set to 0 for unlimited.</div>
|
||||
</div>
|
||||
<div class="field-control">
|
||||
<input type="number" min="0" x-model.number="model.max_embeds" />
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<!-- Utility-specific: ctx_input slider -->
|
||||
<template x-if="modelType === 'utility'">
|
||||
<div class="field">
|
||||
|
|
|
|||
|
|
@ -17,10 +17,22 @@
|
|||
|
||||
<div class="model-preset-rows" aria-live="polite">
|
||||
<template x-for="model in modelRows" :key="model.title">
|
||||
<div class="model-preset-row">
|
||||
<x-icon class="model-preset-icon" :name="model.icon"></x-icon>
|
||||
<span class="model-preset-role" x-text="model.title"></span>
|
||||
<span class="model-preset-identity">
|
||||
<div class="model-preset-row"
|
||||
:class="{ 'model-preset-row-nested': model.title === 'Vision' }">
|
||||
<template x-if="model.title !== 'Vision'">
|
||||
<x-icon class="model-preset-icon" :name="model.icon"></x-icon>
|
||||
</template>
|
||||
<template x-if="model.title !== 'Vision'">
|
||||
<span class="model-preset-role" x-text="model.title"></span>
|
||||
</template>
|
||||
<span class="model-preset-identity"
|
||||
:class="{ 'model-preset-identity-vision': model.title === 'Vision' }">
|
||||
<template x-if="model.title === 'Vision'">
|
||||
<span>
|
||||
<span class="model-preset-role">Vision override</span>
|
||||
<span class="model-preset-separator">/</span>
|
||||
</span>
|
||||
</template>
|
||||
<span class="model-preset-provider" x-text="model.provider"></span>
|
||||
<span class="model-preset-separator">/</span>
|
||||
<span x-text="model.name"></span>
|
||||
|
|
@ -86,6 +98,16 @@
|
|||
border-top: 1px solid var(--color-border);
|
||||
}
|
||||
|
||||
.model-preset-row-nested {
|
||||
border-top: 0 !important;
|
||||
padding-top: var(--spacing-xs);
|
||||
font-size: 0.76rem;
|
||||
}
|
||||
|
||||
.model-preset-identity-vision {
|
||||
grid-column: 3;
|
||||
}
|
||||
|
||||
.model-preset-icon,
|
||||
.model-preset-provider,
|
||||
.model-preset-separator {
|
||||
|
|
|
|||
|
|
@ -25,6 +25,7 @@
|
|||
- Read the rendering path before changing placeholders or filenames.
|
||||
- Prefer small prompt additions over broad rewrites when fixing a specific behavior.
|
||||
- Keep document/OCR routing explicit: image files, screenshots, scans, charts, photos, and diagrams should prefer vision tools when available, while `document_query` is for documents, large text-heavy files, and fallback OCR.
|
||||
- Keep native and sidecar vision prompts synchronized with `vision_load`: related images belong in one call, while a sidecar result is text-only unless Main explicitly requests its native raw route.
|
||||
- Update tests or snapshots when prompt budget, required sections, or generated system content changes.
|
||||
|
||||
## Verification
|
||||
|
|
|
|||
48
prompts/agent.system.tools_vision_sidecar.md
Normal file
48
prompts/agent.system.tools_vision_sidecar.md
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
## multimodal vision tools
|
||||
|
||||
### vision_load
|
||||
analyze images with the separate Vision Model and return a text result
|
||||
args: `paths` list of absolute image paths or ephemeral image refs, `query` optional focused instruction, `raw` optional boolean
|
||||
Input schema for tool_args:
|
||||
```json
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"paths": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
"description": "Absolute image paths or ephemeral image refs."
|
||||
},
|
||||
"query": {
|
||||
"type": "string",
|
||||
"description": "What the Vision Model should inspect, compare, locate, or read."
|
||||
},
|
||||
"raw": {
|
||||
"type": "boolean",
|
||||
"description": "Use the Main model's native vision instead, when Main supports vision."
|
||||
}
|
||||
},
|
||||
"required": ["paths"],
|
||||
"additionalProperties": false
|
||||
}
|
||||
```
|
||||
rules:
|
||||
- put all images needed for one comparison or visual task in the same `paths` array; they are sent in one Vision Model call
|
||||
- use a focused `query`; if omitted, the Vision Model returns a concise general description
|
||||
- the result is a text capsule; the Main model does not receive the raw images
|
||||
- use `raw=true` only when Main supports vision and must inspect the pixels itself
|
||||
- only bitmaps are supported
|
||||
example:
|
||||
```json
|
||||
{
|
||||
"thoughts": [
|
||||
"I need to compare both screenshots before answering."
|
||||
],
|
||||
"headline": "Comparing screenshots",
|
||||
"tool_name": "vision_load",
|
||||
"tool_args": {
|
||||
"paths": ["/path/to/before.png", "/path/to/after.png"],
|
||||
"query": "Compare the error banners and describe what changed."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
|
@ -36,8 +36,13 @@ class _TestAgentContextType:
|
|||
|
||||
|
||||
class _TestResponse(SimpleNamespace):
|
||||
def __init__(self, message="", break_loop=False, **kwargs):
|
||||
super().__init__(message=message, break_loop=break_loop, **kwargs)
|
||||
def __init__(self, message="", break_loop=False, additional=None, **kwargs):
|
||||
super().__init__(
|
||||
message=message,
|
||||
break_loop=break_loop,
|
||||
additional=additional,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
|
||||
class _TestTool:
|
||||
|
|
@ -96,6 +101,9 @@ _model_config_stub = ModuleType("plugins._model_config.helpers.model_config")
|
|||
_model_config_stub.get_presets = lambda: []
|
||||
_model_config_stub.get_preset_by_name = lambda name: None
|
||||
_model_config_stub.get_chat_model_config = lambda agent=None: {}
|
||||
_model_config_stub.get_vision_model_config = lambda agent=None: {}
|
||||
_model_config_stub.use_vision_sidecar = lambda agent=None: False
|
||||
_model_config_stub.build_vision_model = lambda agent=None: None
|
||||
sys.modules.setdefault("plugins._model_config.helpers.model_config", _model_config_stub)
|
||||
|
||||
|
||||
|
|
@ -4297,11 +4305,9 @@ async def test_vision_load_materializes_ephemeral_browser_refs(monkeypatch, tmp_
|
|||
|
||||
monkeypatch.setattr(vision_load_module.chat_media.files, "get_abs_path", fake_get_abs_path)
|
||||
monkeypatch.setattr(vision_load_module.chat_media.files, "normalize_a0_path", fake_normalize_a0_path)
|
||||
monkeypatch.setattr(
|
||||
vision_load_module.plugins,
|
||||
"get_plugin_config",
|
||||
lambda *args, **kwargs: {"chat_model": {"max_embeds": 10}},
|
||||
)
|
||||
monkeypatch.setattr(vision_load_module, "get_chat_model_config", lambda _agent: {"vision": True, "max_embeds": 10})
|
||||
monkeypatch.setattr(vision_load_module, "get_vision_model_config", lambda _agent: {})
|
||||
monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: False)
|
||||
|
||||
tool_results = []
|
||||
messages = []
|
||||
|
|
|
|||
|
|
@ -149,7 +149,7 @@ def test_model_config_frontend_tracks_provider_api_key_edits():
|
|||
assert "/plugins/_model_config/missing_api_key_status" not in model_gate_content
|
||||
assert '@input="$store.modelConfig.setApiKeyValue(_prov, $el.value)"' in config_content
|
||||
assert "apiKeyMode: 'none'" not in preset_modal_content
|
||||
assert preset_modal_content.count("apiKeyMode: 'store'") == 3
|
||||
assert preset_modal_content.count("apiKeyMode: 'store'") == 4
|
||||
assert "$store.modelConfig.resetApiKeyDrafts();" in preset_modal_content
|
||||
assert "await $store.modelConfig.refreshApiKeyStatus();" in preset_modal_content
|
||||
assert "await store.persistAllDirtyApiKeys();" in store_content
|
||||
|
|
|
|||
|
|
@ -1092,6 +1092,44 @@ def test_preset_application_inherits_optional_slots(monkeypatch, tmp_path):
|
|||
assert config["embedding_model"] == base_config["embedding_model"]
|
||||
|
||||
|
||||
def test_preset_vision_slot_is_optional_and_never_inherited(monkeypatch, tmp_path):
|
||||
_prepare_a0_tree(monkeypatch, tmp_path)
|
||||
|
||||
from plugins._model_config.helpers import model_config
|
||||
|
||||
base_config = {
|
||||
"chat_model": {"provider": "openrouter", "name": "main", "vision": False},
|
||||
"vision_model": {
|
||||
"provider": "openrouter",
|
||||
"name": "default-vision",
|
||||
"max_embeds": 2,
|
||||
},
|
||||
}
|
||||
|
||||
without_sidecar = model_config.build_config_from_preset(
|
||||
{"name": "Text only", "chat": {"provider": "openrouter", "name": "text"}},
|
||||
base_config,
|
||||
)
|
||||
with_sidecar = model_config.build_config_from_preset(
|
||||
{
|
||||
"name": "Visual",
|
||||
"chat": {"provider": "openrouter", "name": "text"},
|
||||
"vision": {
|
||||
"provider": "anthropic",
|
||||
"name": "visual",
|
||||
"max_embeds": 5,
|
||||
},
|
||||
},
|
||||
base_config,
|
||||
)
|
||||
|
||||
assert without_sidecar["vision_model"] == {}
|
||||
assert with_sidecar["vision_model"]["provider"] == "anthropic"
|
||||
assert with_sidecar["vision_model"]["name"] == "visual"
|
||||
assert with_sidecar["vision_model"]["max_embeds"] == 5
|
||||
assert "default-vision" not in str(with_sidecar["vision_model"])
|
||||
|
||||
|
||||
def test_legacy_utility_preset_defaults_preserve_tuning_but_clear_kwargs(
|
||||
monkeypatch,
|
||||
tmp_path,
|
||||
|
|
|
|||
|
|
@ -85,6 +85,49 @@ def test_preset_editor_uses_standard_modal_footer_buttons() -> None:
|
|||
assert "preset-editor-footer" not in preset_modal
|
||||
|
||||
|
||||
def test_preset_editor_nests_one_conditional_vision_selector_in_main() -> None:
|
||||
preset_modal = read("plugins", "_model_config", "webui", "main.html")
|
||||
model_field = read("plugins", "_model_config", "webui", "model-field.html")
|
||||
preset_overview = read("plugins", "_model_config", "webui", "preset-overview.html")
|
||||
preset_store = read("plugins", "_model_config", "webui", "model-config-store.js")
|
||||
|
||||
main_start = preset_modal.index('<div class="section-title">Main Model</div>')
|
||||
utility_start = preset_modal.index('<div class="section-title">Utility Model</div>')
|
||||
selector_start = preset_modal.index('class="vision-sidecar-selector"')
|
||||
supports_start = model_field.index('<div class="field-title">Supports Vision</div>')
|
||||
override_start = model_field.index('<div class="field-title">Use separate Vision Model</div>')
|
||||
context_start = model_field.index('<div class="field-title">Context window size</div>')
|
||||
advanced_start = model_field.index('<!-- Advanced Settings (collapsed by default) -->')
|
||||
|
||||
assert '<div class="section-title">Vision Model</div>' not in preset_modal
|
||||
assert preset_modal.count('class="vision-sidecar-selector"') == 1
|
||||
assert main_start < selector_start < utility_start
|
||||
assert '!selectedPreset.chat.vision || selectedPreset.vision.override_main' in preset_modal
|
||||
assert "get visionModel() { return selectedPreset.vision; }" in preset_modal
|
||||
assert "modelType: 'vision'" in preset_modal
|
||||
assert preset_modal.count("apiKeyMode: 'store'") == 4
|
||||
assert "margin: 0.75rem 0 0;" in preset_modal
|
||||
assert "padding: 0.25rem 0 0;" in preset_modal
|
||||
assert "border-left: 2px solid var(--color-border);" not in preset_modal
|
||||
assert "Use separate Vision Model" in model_field
|
||||
assert supports_start < override_start < context_start < advanced_start
|
||||
assert "When disabled, vision_load uses this model's native vision." in model_field
|
||||
assert "When enabled, vision_load uses the preset's Vision Model" not in model_field
|
||||
assert 'x-model="visionModel.override_main"' in model_field
|
||||
assert '<div class="advanced-section" x-data="{ advOpen: false }">' in model_field
|
||||
assert "model-preset-row-nested" in preset_overview
|
||||
assert "model.title === 'Vision'" in preset_overview
|
||||
assert preset_overview.count("model.title !== 'Vision'") == 2
|
||||
assert '<span class="model-preset-role">Vision override</span>' in preset_overview
|
||||
assert "'model-preset-identity-vision': model.title === 'Vision'" in preset_overview
|
||||
assert "grid-column: 3;" in preset_overview
|
||||
assert "padding-top: var(--spacing-xs);" in preset_overview
|
||||
assert "margin-left: 2rem;" not in preset_overview
|
||||
assert "border-left: 1px solid var(--color-border);" not in preset_overview
|
||||
assert "if (slotKey === 'vision') config[sectionKey] = {};" in preset_store
|
||||
assert "['chat', 'vision', 'utility']" in preset_store
|
||||
|
||||
|
||||
def test_plugin_settings_reset_is_explicit_and_does_not_capture_toast_early() -> None:
|
||||
settings_modal = read("webui", "components", "plugins", "plugin-settings.html")
|
||||
settings_store = read("webui", "components", "plugins", "plugin-settings-store.js")
|
||||
|
|
|
|||
|
|
@ -160,6 +160,42 @@ def test_normalize_parallel_tool_calls_accepts_json_string_array() -> None:
|
|||
assert calls[1].tool_args["message"] == "Research nuclear fusion news in Italian."
|
||||
|
||||
|
||||
def test_parallel_keeps_mixed_tools_and_batched_vision_paths() -> None:
|
||||
calls = parallel_tools.normalize_parallel_tool_calls(
|
||||
[
|
||||
{"tool_name": "search_a", "tool_args": {"query": "one"}},
|
||||
{"tool_name": "search_b", "tool_args": {"query": "two"}},
|
||||
{"tool_name": "browser_agent", "tool_args": {"message": "open page"}},
|
||||
{
|
||||
"tool_name": "vision_load",
|
||||
"tool_args": {
|
||||
"paths": ["/tmp/before.png", "/tmp/after.png"],
|
||||
"query": "compare",
|
||||
},
|
||||
},
|
||||
]
|
||||
)
|
||||
|
||||
assert [call.tool_name for call in calls] == [
|
||||
"search_a",
|
||||
"search_b",
|
||||
"browser_agent",
|
||||
"vision_load",
|
||||
]
|
||||
assert calls[3].tool_args["paths"] == ["/tmp/before.png", "/tmp/after.png"]
|
||||
|
||||
|
||||
def test_parallel_allows_multiple_independent_vision_calls() -> None:
|
||||
calls = parallel_tools.normalize_parallel_tool_calls(
|
||||
[
|
||||
{"tool_name": "vision_load", "tool_args": {"paths": ["/tmp/a.png"]}},
|
||||
{"tool_name": "vision_load", "tool_args": {"paths": ["/tmp/b.png"]}},
|
||||
]
|
||||
)
|
||||
|
||||
assert [call.tool_name for call in calls] == ["vision_load", "vision_load"]
|
||||
|
||||
|
||||
def test_subordinate_prompts_share_reusable_tree_contract() -> None:
|
||||
call_prompt = (PROJECT_ROOT / "prompts/agent.system.tool.call_sub.md").read_text(
|
||||
encoding="utf-8"
|
||||
|
|
|
|||
|
|
@ -546,6 +546,59 @@ async def test_vision_tool_follows_chat_config_not_profile_policy(
|
|||
assert tool_policy.resolve_tool(agent, "vision_load").source == "runtime-config"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_vision_sidecar_prompt_replaces_native_vision_prompt(
|
||||
monkeypatch, tmp_path: Path
|
||||
) -> None:
|
||||
_write_prompt(tmp_path, "agent.system.tools.md", "TOOLS\n{{tools}}")
|
||||
_write_prompt(
|
||||
tmp_path,
|
||||
"agent.system.tools_vision.md",
|
||||
"### vision_load\nnative pixels\nargs: `paths`",
|
||||
)
|
||||
_write_prompt(
|
||||
tmp_path,
|
||||
"agent.system.tools_vision_sidecar.md",
|
||||
"### vision_load\nsidecar capsule\nargs: `paths`, `query`",
|
||||
)
|
||||
monkeypatch.setattr(tool_policy.subagents, "get_paths", _prompt_paths(tmp_path))
|
||||
monkeypatch.setattr(
|
||||
"plugins._model_config.helpers.model_config.get_chat_model_config",
|
||||
lambda agent: {"vision": True},
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"plugins._model_config.helpers.model_config.use_vision_sidecar",
|
||||
lambda agent: True,
|
||||
)
|
||||
monkeypatch.setattr(responses_tools, "_mcp_tools", lambda agent: [])
|
||||
agent = _Agent(tmp_path)
|
||||
|
||||
prompt = await _11_tools_prompt.build_prompt(agent)
|
||||
schemas, _name_map = responses_tools.build_responses_function_tools(agent)
|
||||
|
||||
assert "sidecar capsule" in prompt
|
||||
assert "native pixels" not in prompt
|
||||
assert schemas[0]["name"] == "vision_load"
|
||||
assert schemas[0]["description"] == "sidecar capsule"
|
||||
|
||||
|
||||
def test_vision_sidecar_prompt_declares_multi_image_native_schema() -> None:
|
||||
prompt = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "prompts"
|
||||
/ "agent.system.tools_vision_sidecar.md"
|
||||
).read_text(encoding="utf-8")
|
||||
schema = responses_tools._schema_from_prompt(prompt)
|
||||
|
||||
assert schema["required"] == ["paths"]
|
||||
assert schema["properties"]["paths"] == {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
"description": "Absolute image paths or ephemeral image refs.",
|
||||
}
|
||||
assert {"query", "raw"} <= schema["properties"].keys()
|
||||
|
||||
|
||||
def test_mcp_prompt_and_native_schema_omit_blocked_tool(
|
||||
monkeypatch, tmp_path: Path
|
||||
) -> None:
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import asyncio
|
||||
import types
|
||||
from types import SimpleNamespace
|
||||
import sys
|
||||
|
|
@ -13,8 +14,13 @@ from helpers import images
|
|||
|
||||
|
||||
class _TestResponse(SimpleNamespace):
|
||||
def __init__(self, message="", break_loop=False, **kwargs):
|
||||
super().__init__(message=message, break_loop=break_loop, **kwargs)
|
||||
def __init__(self, message="", break_loop=False, additional=None, **kwargs):
|
||||
super().__init__(
|
||||
message=message,
|
||||
break_loop=break_loop,
|
||||
additional=additional,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
|
||||
class _TestTool:
|
||||
|
|
@ -74,11 +80,9 @@ async def test_vision_load_materializes_local_image_to_chat_artifact(monkeypatch
|
|||
|
||||
monkeypatch.setattr(vision_load_module.chat_media.files, "get_abs_path", fake_get_abs_path)
|
||||
monkeypatch.setattr(vision_load_module.chat_media.files, "normalize_a0_path", fake_normalize_a0_path)
|
||||
monkeypatch.setattr(
|
||||
vision_load_module.plugins,
|
||||
"get_plugin_config",
|
||||
lambda *args, **kwargs: {"chat_model": {"max_embeds": 10}},
|
||||
)
|
||||
monkeypatch.setattr(vision_load_module, "get_chat_model_config", lambda _agent: {"vision": True, "max_embeds": 10})
|
||||
monkeypatch.setattr(vision_load_module, "get_vision_model_config", lambda _agent: {})
|
||||
monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: False)
|
||||
|
||||
async def direct_call(func, *args, **kwargs):
|
||||
return func(*args, **kwargs)
|
||||
|
|
@ -121,3 +125,215 @@ async def test_vision_load_materializes_local_image_to_chat_artifact(monkeypatch
|
|||
stored_path = tmp_path / stored_ref.removeprefix("/a0/")
|
||||
assert stored_path.read_bytes() == b"png-data"
|
||||
assert updates[-1]["result"] == "1 images loaded, 0 skipped"
|
||||
|
||||
|
||||
def test_vision_sidecar_route_matrix_prefers_main_native_vision(monkeypatch):
|
||||
from plugins._model_config.helpers import model_config
|
||||
|
||||
cases = [
|
||||
({"vision": False}, {}, False),
|
||||
({"vision": True}, {"provider": "p", "name": "v"}, False),
|
||||
({"vision": False}, {"provider": "p", "name": "v"}, True),
|
||||
(
|
||||
{"vision": True},
|
||||
{"provider": "p", "name": "v", "override_main": True},
|
||||
True,
|
||||
),
|
||||
]
|
||||
for chat, vision, expected in cases:
|
||||
monkeypatch.setattr(
|
||||
model_config,
|
||||
"get_effective_config",
|
||||
lambda _agent=None, chat=chat, vision=vision: {
|
||||
"chat_model": chat,
|
||||
"vision_model": vision,
|
||||
},
|
||||
)
|
||||
assert model_config.use_vision_sidecar() is expected
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_vision_sidecar_sends_multiple_images_once_and_keeps_history_text_only(
|
||||
monkeypatch,
|
||||
tmp_path,
|
||||
):
|
||||
_install_tool_stub(monkeypatch)
|
||||
import tools.vision_load as vision_load_module
|
||||
|
||||
async def direct_call(func, *args, **kwargs):
|
||||
return func(*args, **kwargs)
|
||||
|
||||
calls = []
|
||||
|
||||
class FakeVisionModel:
|
||||
async def unified_call(self, **kwargs):
|
||||
calls.append(kwargs)
|
||||
return "The second screenshot fixes the red login error.", ""
|
||||
|
||||
monkeypatch.setattr(vision_load_module.runtime, "call_development_function", direct_call)
|
||||
monkeypatch.setattr(vision_load_module, "build_vision_model", lambda _agent: FakeVisionModel())
|
||||
monkeypatch.setattr(
|
||||
vision_load_module,
|
||||
"get_chat_model_config",
|
||||
lambda _agent: {"vision": True, "max_embeds": 1},
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
vision_load_module,
|
||||
"get_vision_model_config",
|
||||
lambda _agent: {"provider": "test", "name": "vision", "max_embeds": 5},
|
||||
)
|
||||
monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: True)
|
||||
|
||||
image_paths = [tmp_path / "before.png", tmp_path / "after.png"]
|
||||
for path in image_paths:
|
||||
path.write_bytes(b"png-data")
|
||||
|
||||
tool_results = []
|
||||
raw_messages = []
|
||||
agent = SimpleNamespace(
|
||||
context=SimpleNamespace(id=""),
|
||||
agent_name="Agent 0",
|
||||
hist_add_tool_result=lambda *args, **kwargs: tool_results.append((args, kwargs)),
|
||||
hist_add_message=lambda *args, **kwargs: raw_messages.append((args, kwargs)),
|
||||
)
|
||||
tool = vision_load_module.VisionLoad(
|
||||
agent=agent,
|
||||
name="vision_load",
|
||||
method=None,
|
||||
args={"paths": [str(path) for path in image_paths]},
|
||||
message="",
|
||||
loop_data=None,
|
||||
)
|
||||
tool.log = SimpleNamespace(id="vision-log", update=lambda **kwargs: None)
|
||||
|
||||
response = await tool.execute(
|
||||
paths=[str(path) for path in image_paths],
|
||||
query="Compare the login errors.",
|
||||
)
|
||||
response.additional = {"_responses_output_item": {"output": response.message}}
|
||||
await tool.after_execution(response)
|
||||
|
||||
assert len(calls) == 1
|
||||
content = calls[0]["messages"][1].content
|
||||
assert content[0] == {"type": "text", "text": "Compare the login errors."}
|
||||
assert [item["type"] for item in content].count("image_url") == 2
|
||||
assert "fixes the red login error" in response.message
|
||||
assert response.message != "dummy"
|
||||
assert raw_messages == []
|
||||
assert tool.loaded_paths == [str(path) for path in image_paths]
|
||||
assert tool_results[0][1]["_responses_output_item"]["output"] == response.message
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_parallel_worker_consumes_parent_ephemeral_image(monkeypatch, tmp_path):
|
||||
_install_tool_stub(monkeypatch)
|
||||
import tools.vision_load as vision_load_module
|
||||
|
||||
def fake_get_abs_path(*parts):
|
||||
return str(tmp_path.joinpath(*parts))
|
||||
|
||||
def fake_normalize_a0_path(path):
|
||||
return "/a0/" + str(Path(path).relative_to(tmp_path)).replace("\\", "/")
|
||||
|
||||
monkeypatch.setattr(vision_load_module.chat_media.files, "get_abs_path", fake_get_abs_path)
|
||||
monkeypatch.setattr(vision_load_module.chat_media.files, "normalize_a0_path", fake_normalize_a0_path)
|
||||
monkeypatch.setattr(vision_load_module.VisionLoad, "_config_agent", lambda self: self.agent)
|
||||
monkeypatch.setattr(
|
||||
vision_load_module,
|
||||
"get_chat_model_config",
|
||||
lambda _agent: {"vision": True, "max_embeds": 10},
|
||||
)
|
||||
monkeypatch.setattr(vision_load_module, "get_vision_model_config", lambda _agent: {})
|
||||
monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: False)
|
||||
|
||||
parent_id = "parent-vision"
|
||||
ref = vision_load_module.ephemeral_images.put_image_bytes(
|
||||
context_id=parent_id,
|
||||
mime="image/png",
|
||||
payload=b"png-data",
|
||||
name="shot.png",
|
||||
)
|
||||
context = SimpleNamespace(
|
||||
id="parallel-worker",
|
||||
get_data=lambda key: parent_id
|
||||
if key == vision_load_module.PARALLEL_WORKER_PARENT_CONTEXT_KEY
|
||||
else None,
|
||||
)
|
||||
agent = SimpleNamespace(context=context, agent_name="Agent 0")
|
||||
tool = vision_load_module.VisionLoad(
|
||||
agent=agent,
|
||||
name="vision_load",
|
||||
method=None,
|
||||
args={"paths": [ref]},
|
||||
message="",
|
||||
loop_data=None,
|
||||
)
|
||||
|
||||
await tool.execute(paths=[ref])
|
||||
|
||||
assert tool.loaded_paths == ["shot.png"]
|
||||
assert vision_load_module.ephemeral_images.get_image(ref, context_id=parent_id) is None
|
||||
stored_ref = tool.images_dict["shot.png"]
|
||||
assert stored_ref.startswith("/a0/usr/chats/parent-vision/images/vision-load/shot-")
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_independent_vision_sidecar_calls_can_run_concurrently(monkeypatch, tmp_path):
|
||||
_install_tool_stub(monkeypatch)
|
||||
import tools.vision_load as vision_load_module
|
||||
|
||||
active = 0
|
||||
max_active = 0
|
||||
call_count = 0
|
||||
|
||||
class FakeVisionModel:
|
||||
async def unified_call(self, **kwargs):
|
||||
nonlocal active, max_active, call_count
|
||||
active += 1
|
||||
call_count += 1
|
||||
max_active = max(max_active, active)
|
||||
await asyncio.sleep(0.02)
|
||||
active -= 1
|
||||
return "done", ""
|
||||
|
||||
async def direct_call(func, *args, **kwargs):
|
||||
return func(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(vision_load_module.runtime, "call_development_function", direct_call)
|
||||
monkeypatch.setattr(vision_load_module, "build_vision_model", lambda _agent: FakeVisionModel())
|
||||
monkeypatch.setattr(vision_load_module, "get_chat_model_config", lambda _agent: {"vision": False})
|
||||
monkeypatch.setattr(
|
||||
vision_load_module,
|
||||
"get_vision_model_config",
|
||||
lambda _agent: {"provider": "test", "name": "vision", "max_embeds": 10},
|
||||
)
|
||||
monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: True)
|
||||
|
||||
image_paths = [tmp_path / "one.png", tmp_path / "two.png"]
|
||||
for path in image_paths:
|
||||
path.write_bytes(b"png-data")
|
||||
|
||||
def make_tool(index):
|
||||
agent = SimpleNamespace(context=SimpleNamespace(id=""), agent_name=f"Agent {index}")
|
||||
return vision_load_module.VisionLoad(
|
||||
agent=agent,
|
||||
name="vision_load",
|
||||
method=None,
|
||||
args={"paths": [str(path) for path in image_paths]},
|
||||
message="",
|
||||
loop_data=None,
|
||||
)
|
||||
|
||||
responses = await asyncio.gather(
|
||||
*(
|
||||
make_tool(index).execute(
|
||||
paths=[str(path) for path in image_paths],
|
||||
query=f"inspection {index}",
|
||||
)
|
||||
for index in range(4)
|
||||
)
|
||||
)
|
||||
|
||||
assert call_count == 4
|
||||
assert max_active == 4
|
||||
assert all("done" in response.message for response in responses)
|
||||
|
|
|
|||
|
|
@ -1,89 +1,219 @@
|
|||
from helpers.print_style import PrintStyle
|
||||
from helpers.tool import Tool, Response
|
||||
from helpers import runtime, files, plugins, ephemeral_images, images, chat_media
|
||||
import asyncio
|
||||
import json
|
||||
from mimetypes import guess_type
|
||||
from helpers import history
|
||||
|
||||
# image token estimation for context window
|
||||
from langchain_core.messages import HumanMessage, SystemMessage
|
||||
|
||||
from helpers import chat_media, ephemeral_images, files, history, images, runtime
|
||||
from helpers.parallel_tools import PARALLEL_WORKER_PARENT_CONTEXT_KEY, coerce_bool
|
||||
from helpers.print_style import PrintStyle
|
||||
from helpers.tool import Response, Tool
|
||||
from plugins._model_config.helpers.model_config import (
|
||||
build_vision_model,
|
||||
get_chat_model_config,
|
||||
get_vision_model_config,
|
||||
use_vision_sidecar,
|
||||
)
|
||||
|
||||
TOKENS_ESTIMATE = 1500
|
||||
VISION_TIMEOUT_SECONDS = 300
|
||||
VISION_SYSTEM_PROMPT = (
|
||||
"You are a precise vision analyst. Answer only what was asked about the images. "
|
||||
"Be concise and factual. Preserve exact visible text when asked to read it."
|
||||
)
|
||||
DEFAULT_VISION_QUERY = (
|
||||
"Describe the images precisely, including the key objects, visible text, and layout."
|
||||
)
|
||||
|
||||
|
||||
class VisionLoad(Tool):
|
||||
async def execute(self, paths: list[str] = [], **kwargs) -> Response:
|
||||
|
||||
self.images_dict = {}
|
||||
async def execute(
|
||||
self,
|
||||
paths: list[str] | str | None = None,
|
||||
query: str = "",
|
||||
raw: bool = False,
|
||||
**kwargs,
|
||||
) -> Response:
|
||||
self.images_dict: dict[str, str] = {}
|
||||
self.loaded_paths: list[str] = []
|
||||
self.skipped_paths: list[str] = []
|
||||
|
||||
max_embeds = self._get_max_embeds()
|
||||
requested = [
|
||||
(str(path or "").strip(), self._display_input_path(str(path or "").strip(), idx + 1))
|
||||
for idx, path in enumerate(paths)
|
||||
]
|
||||
limited_paths = requested if max_embeds <= 0 else requested[-max_embeds:]
|
||||
self.skipped_paths = (
|
||||
[display for _, display in requested[:-max_embeds]]
|
||||
if max_embeds > 0 and len(requested) > max_embeds
|
||||
else []
|
||||
self._config_owner = self._config_agent()
|
||||
self._main_has_vision = bool(
|
||||
get_chat_model_config(self._config_owner).get("vision", False)
|
||||
)
|
||||
self._delegated = use_vision_sidecar(self._config_owner) and not (
|
||||
coerce_bool(raw, False) and self._main_has_vision
|
||||
)
|
||||
self._max_embeds = self._get_max_embeds()
|
||||
|
||||
for idx, (path, display_path) in enumerate(limited_paths):
|
||||
normalized = self._normalize_paths(paths)
|
||||
if isinstance(normalized, str):
|
||||
self._history_result = normalized
|
||||
return Response(message=normalized, break_loop=False)
|
||||
|
||||
requested = [
|
||||
(path.strip(), self._display_input_path(path.strip(), index + 1))
|
||||
for index, path in enumerate(normalized)
|
||||
]
|
||||
limited = requested if self._max_embeds <= 0 else requested[-self._max_embeds :]
|
||||
if self._max_embeds > 0 and len(requested) > self._max_embeds:
|
||||
self.skipped_paths = [display for _, display in requested[: -self._max_embeds]]
|
||||
|
||||
for index, (path, display_path) in enumerate(limited):
|
||||
if not path:
|
||||
continue
|
||||
if ephemeral_images.is_ref(path):
|
||||
image = ephemeral_images.consume_image(
|
||||
path,
|
||||
context_id=self._context_id(),
|
||||
)
|
||||
image = ephemeral_images.consume_image(path, context_id=self._context_id())
|
||||
if image is None:
|
||||
continue
|
||||
display = image.display_name or display_path
|
||||
display_path = image.display_name or display_path
|
||||
stored_ref = self._store_ephemeral_image(image)
|
||||
if stored_ref:
|
||||
self.images_dict[display] = stored_ref
|
||||
self.loaded_paths.append(display)
|
||||
continue
|
||||
if self._is_data_image_url(path):
|
||||
stored_ref = self._store_data_url(path, preferred_name=f"vision-load-{idx + 1}.png")
|
||||
if stored_ref:
|
||||
self.images_dict[display_path] = stored_ref
|
||||
self.loaded_paths.append(display_path)
|
||||
continue
|
||||
if not await runtime.call_development_function(files.exists, str(path)):
|
||||
elif self._is_data_image_url(path):
|
||||
stored_ref = self._store_data_url(
|
||||
path, preferred_name=f"vision-load-{index + 1}.png"
|
||||
)
|
||||
elif await runtime.call_development_function(files.exists, path):
|
||||
mime_type, _ = guess_type(path)
|
||||
if not mime_type or not mime_type.startswith("image/"):
|
||||
continue
|
||||
try:
|
||||
stored_ref = self._store_local_image(
|
||||
path, preferred_name=files.basename(path)
|
||||
)
|
||||
except (FileNotFoundError, OSError, ValueError):
|
||||
continue
|
||||
else:
|
||||
continue
|
||||
|
||||
if path not in self.images_dict:
|
||||
mime_type, _ = guess_type(str(path))
|
||||
if mime_type and mime_type.startswith("image/"):
|
||||
try:
|
||||
stored_ref = self._store_local_image(path, preferred_name=files.basename(path))
|
||||
self.images_dict[display_path] = stored_ref
|
||||
self.loaded_paths.append(display_path)
|
||||
except (FileNotFoundError, OSError, ValueError):
|
||||
continue
|
||||
if stored_ref:
|
||||
self.images_dict[display_path] = stored_ref
|
||||
self.loaded_paths.append(display_path)
|
||||
|
||||
return Response(message="dummy", break_loop=False)
|
||||
summary = self._summary()
|
||||
if self._delegated and self.images_dict:
|
||||
try:
|
||||
capsule = await self._call_vision_model(
|
||||
list(self.images_dict.values()), self._query(query, kwargs)
|
||||
)
|
||||
message = (
|
||||
f"Vision Model analyzed {len(self.images_dict)} image(s)"
|
||||
f"; {len(self.skipped_paths)} skipped.\n\n{capsule.strip()}"
|
||||
)
|
||||
self._history_result = message
|
||||
return Response(message=message, break_loop=False)
|
||||
except Exception as exc:
|
||||
message = f"Vision Model error: {str(exc)[:1000]}"
|
||||
self._history_result = f"{summary}\n\n{message}"
|
||||
return Response(message=message, break_loop=False)
|
||||
|
||||
if self.images_dict and not self._main_has_vision:
|
||||
summary += (
|
||||
"\n\nImages were not injected because neither Main native vision nor "
|
||||
"a usable Vision Model is active."
|
||||
)
|
||||
self._history_result = (
|
||||
summary if self.images_dict or self.skipped_paths else "No images processed"
|
||||
)
|
||||
message = (
|
||||
"No images processed"
|
||||
if not self.images_dict and not self.skipped_paths
|
||||
else f"{len(self.images_dict)} images loaded, {len(self.skipped_paths)} skipped"
|
||||
)
|
||||
return Response(message=message, break_loop=False)
|
||||
|
||||
async def after_execution(self, response: Response, **kwargs):
|
||||
log_id = str(getattr(getattr(self, "log", None), "id", "") or "")
|
||||
self.agent.hist_add_tool_result(
|
||||
self.name,
|
||||
self._history_result,
|
||||
id=log_id,
|
||||
**(response.additional or {}),
|
||||
)
|
||||
|
||||
if self.images_dict and self._main_has_vision and not self._delegated:
|
||||
content = [
|
||||
{"type": "image_url", "image_url": {"url": image_path}}
|
||||
for image_path in self.images_dict.values()
|
||||
]
|
||||
self.agent.hist_add_message(
|
||||
False,
|
||||
content=history.RawMessage(
|
||||
raw_content=content,
|
||||
preview="<Image attachments loaded by path>",
|
||||
),
|
||||
tokens=TOKENS_ESTIMATE * len(content),
|
||||
)
|
||||
|
||||
PrintStyle(
|
||||
font_color="#1B4F72", background_color="white", padding=True, bold=True
|
||||
).print(f"{self.agent.agent_name}: Response from tool '{self.name}'")
|
||||
PrintStyle(font_color="#85C1E9").print(response.message)
|
||||
if getattr(self, "log", None):
|
||||
self.log.update(result=response.message)
|
||||
|
||||
def _get_max_embeds(self) -> int:
|
||||
cfg = plugins.get_plugin_config("_model_config", agent=self.agent) or {}
|
||||
chat_cfg = cfg.get("chat_model", {})
|
||||
max_embeds = chat_cfg.get("max_embeds", 10)
|
||||
return int(max_embeds or 0)
|
||||
cfg = (
|
||||
get_vision_model_config(self._config_owner)
|
||||
if self._delegated
|
||||
else get_chat_model_config(self._config_owner)
|
||||
)
|
||||
try:
|
||||
return int(cfg.get("max_embeds", 10) or 0)
|
||||
except (TypeError, ValueError):
|
||||
return 10
|
||||
|
||||
def _context_id(self) -> str:
|
||||
return str(getattr(getattr(self.agent, "context", None), "id", "") or "").strip()
|
||||
context = getattr(self.agent, "context", None)
|
||||
if not context:
|
||||
return ""
|
||||
get_data = getattr(context, "get_data", None)
|
||||
parent_id = get_data(PARALLEL_WORKER_PARENT_CONTEXT_KEY) if get_data else ""
|
||||
return str(parent_id or getattr(context, "id", "") or "").strip()
|
||||
|
||||
def _config_agent(self):
|
||||
context = getattr(self.agent, "context", None)
|
||||
get_data = getattr(context, "get_data", None)
|
||||
parent_id = get_data(PARALLEL_WORKER_PARENT_CONTEXT_KEY) if get_data else ""
|
||||
if parent_id:
|
||||
from agent import AgentContext
|
||||
|
||||
parent = AgentContext.get(str(parent_id))
|
||||
if parent:
|
||||
return parent.agent0
|
||||
return self.agent
|
||||
|
||||
async def _call_vision_model(self, image_paths: list[str], query: str) -> str:
|
||||
model = build_vision_model(self._config_owner)
|
||||
content = [{"type": "text", "text": query or DEFAULT_VISION_QUERY}]
|
||||
content.extend(
|
||||
{"type": "image_url", "image_url": {"url": path}}
|
||||
for path in image_paths
|
||||
)
|
||||
response, _ = await asyncio.wait_for(
|
||||
model.unified_call(
|
||||
messages=[
|
||||
SystemMessage(content=VISION_SYSTEM_PROMPT),
|
||||
HumanMessage(content=content),
|
||||
],
|
||||
explicit_caching=False,
|
||||
max_tokens=2000,
|
||||
),
|
||||
timeout=VISION_TIMEOUT_SECONDS,
|
||||
)
|
||||
if not str(response or "").strip():
|
||||
raise RuntimeError("Vision Model returned an empty response.")
|
||||
return str(response)
|
||||
|
||||
def _store_ephemeral_image(self, image: ephemeral_images.EphemeralImage) -> str:
|
||||
context_id = self._context_id()
|
||||
if not context_id:
|
||||
return image.data_url
|
||||
source = chat_media.infer_source(image.ref, image.display_name)
|
||||
category = chat_media.category_for_source(source)
|
||||
saved = chat_media.save_image_base64(
|
||||
context_id=context_id,
|
||||
data=image.data,
|
||||
mime_type=image.mime,
|
||||
category=category,
|
||||
category=chat_media.category_for_source(source),
|
||||
source=source,
|
||||
preferred_name=image.display_name,
|
||||
)
|
||||
|
|
@ -94,11 +224,10 @@ class VisionLoad(Tool):
|
|||
if not context_id:
|
||||
return data_url
|
||||
source = chat_media.infer_source(data_url, preferred_name)
|
||||
category = chat_media.category_for_source(source)
|
||||
saved = chat_media.save_image_data_url(
|
||||
context_id=context_id,
|
||||
data_url=data_url,
|
||||
category=category,
|
||||
category=chat_media.category_for_source(source),
|
||||
source=source,
|
||||
preferred_name=preferred_name,
|
||||
)
|
||||
|
|
@ -115,6 +244,37 @@ class VisionLoad(Tool):
|
|||
preferred_name=preferred_name,
|
||||
)
|
||||
|
||||
def _summary(self) -> str:
|
||||
loaded = "\n".join(self.loaded_paths) if self.loaded_paths else "none"
|
||||
skipped = "\n".join(self.skipped_paths) if self.skipped_paths else "none"
|
||||
return (
|
||||
f"Loaded images ({len(self.loaded_paths)}):\n{loaded}\n\n"
|
||||
f"Skipped images ({len(self.skipped_paths)}, max {self._max_embeds}):\n{skipped}"
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _normalize_paths(paths: list[str] | str | None) -> list[str] | str:
|
||||
if isinstance(paths, str):
|
||||
try:
|
||||
decoded = json.loads(paths)
|
||||
except json.JSONDecodeError:
|
||||
decoded = paths
|
||||
paths = decoded if isinstance(decoded, list) else [paths]
|
||||
if paths is None:
|
||||
return []
|
||||
if not isinstance(paths, (list, tuple)):
|
||||
return "vision_load error: `paths` must be an array of image paths."
|
||||
return [str(path or "").strip() for path in paths]
|
||||
|
||||
@staticmethod
|
||||
def _query(query: str, kwargs: dict) -> str:
|
||||
if str(query or "").strip():
|
||||
return str(query).strip()
|
||||
for key in ("prompt", "question", "instruction", "focus", "request"):
|
||||
if str(kwargs.get(key) or "").strip():
|
||||
return str(kwargs[key]).strip()
|
||||
return DEFAULT_VISION_QUERY
|
||||
|
||||
@staticmethod
|
||||
def _is_data_image_url(value: str) -> bool:
|
||||
normalized = str(value or "").strip().lower()
|
||||
|
|
@ -125,57 +285,5 @@ class VisionLoad(Tool):
|
|||
if ephemeral_images.is_ref(value):
|
||||
return ephemeral_images.display_ref(value)
|
||||
if cls._is_data_image_url(value):
|
||||
prefix = value.split(",", 1)[0]
|
||||
return f"{prefix},<ephemeral-image-{index}>"
|
||||
return f"{value.split(',', 1)[0]},<ephemeral-image-{index}>"
|
||||
return value
|
||||
|
||||
async def after_execution(self, response: Response, **kwargs):
|
||||
|
||||
# build image data messages for LLMs, or error message
|
||||
content = []
|
||||
loaded_count = len(self.loaded_paths)
|
||||
skipped_count = len(self.skipped_paths)
|
||||
loaded_summary = "\n".join(self.loaded_paths) if self.loaded_paths else "none"
|
||||
skipped_summary = "\n".join(self.skipped_paths) if self.skipped_paths else "none"
|
||||
summary = (
|
||||
f"Loaded images: {loaded_count}\n"
|
||||
f"Loaded images:\n{loaded_summary}\n\n"
|
||||
f"Skipped images: {skipped_count}\n"
|
||||
f"Skipped images (max {self._get_max_embeds()} loaded at a time according to model configuration):\n{skipped_summary}"
|
||||
)
|
||||
if self.images_dict:
|
||||
self.agent.hist_add_tool_result(self.name, summary, id=self.log.id if self.log else "")
|
||||
for path, image_path in self.images_dict.items():
|
||||
if image_path:
|
||||
content.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": image_path},
|
||||
}
|
||||
)
|
||||
else:
|
||||
content.append(
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Error processing image " + path,
|
||||
}
|
||||
)
|
||||
# append as raw message content for LLMs with vision tokens estimate
|
||||
msg = history.RawMessage(raw_content=content, preview="<Image attachments loaded by path>")
|
||||
self.agent.hist_add_message(
|
||||
False, content=msg, tokens=TOKENS_ESTIMATE * len(content)
|
||||
)
|
||||
else:
|
||||
self.agent.hist_add_tool_result(self.name, summary if self.skipped_paths else "No images processed", id=self.log.id if self.log else "")
|
||||
|
||||
# print and log short version
|
||||
message = (
|
||||
"No images processed"
|
||||
if not self.images_dict and not self.skipped_paths
|
||||
else f"{loaded_count} images loaded, {skipped_count} skipped"
|
||||
)
|
||||
PrintStyle(
|
||||
font_color="#1B4F72", background_color="white", padding=True, bold=True
|
||||
).print(f"{self.agent.agent_name}: Response from tool '{self.name}'")
|
||||
PrintStyle(font_color="#85C1E9").print(message)
|
||||
self.log.update(result=message)
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
## Purpose
|
||||
|
||||
- Own the `vision_load.py` agent tool.
|
||||
- This module loads images into model-visible content for vision-capable models.
|
||||
- This module routes images either into Main model-visible content or through the preset's optional Vision Model.
|
||||
- Keep this file-level DOX profile synchronized with `vision_load.py` because this directory is intentionally flat.
|
||||
|
||||
## Ownership
|
||||
|
|
@ -12,13 +12,19 @@
|
|||
- `vision_load.py.dox.md` owns durable notes about responsibilities, contracts, side effects, and verification for that implementation.
|
||||
- Classes:
|
||||
- `VisionLoad` (`Tool`)
|
||||
- `async execute(self, paths: list[str]=..., **kwargs) -> Response`
|
||||
- `async execute(self, paths, query="", raw=False, **kwargs) -> Response`
|
||||
- `async after_execution(self, response: Response, **kwargs)`
|
||||
- Notable constants/configuration names: `TOKENS_ESTIMATE`.
|
||||
|
||||
## Runtime Contracts
|
||||
|
||||
- Tool modules must define `helpers.tool.Tool` subclasses and return `helpers.tool.Response` from `execute(...)`.
|
||||
- One call may contain multiple paths. The delegated route sends every selected path in one Vision Model request and returns one textual capsule.
|
||||
- Main native vision wins unless the effective preset selects the sidecar route; `raw=true` returns to Main native vision only when Main supports it.
|
||||
- Delegation completes during `execute(...)` so native Responses function output contains the real capsule before `after_execution(...)` persists it.
|
||||
- Delegated history contains the text capsule only. Native history contains the tool result followed by one raw message holding all loaded image blocks.
|
||||
- Direct parallel workers resolve ephemeral refs, model routing, and durable chat-media storage against their recorded parent context; independent vision jobs remain generic parallel jobs.
|
||||
- `max_embeds` comes from the model that actually receives the images.
|
||||
- Update this file whenever tool arguments, output shape, `break_loop` behavior, intervention handling, prompt instructions, or side effects change.
|
||||
- `VisionLoad` is a `Tool`.
|
||||
- `VisionLoad` defines `execute(...)`.
|
||||
|
|
@ -27,7 +33,7 @@
|
|||
|
||||
## Key Concepts
|
||||
|
||||
- Important called helpers/classes observed in the source: `self._get_max_embeds`, `Response`, `str.strip`, `self._context_id`, `chat_media.infer_source`, `chat_media.category_for_source`, `chat_media.save_image_base64`, `chat_media.save_image_data_url`, `chat_media.materialize_image_ref`, `str.strip.lower`, `ephemeral_images.is_ref`, `cls._is_data_image_url`, `self._is_data_image_url`, `plugins.get_plugin_config`, `images.to_data_url`, `normalized.startswith`, `ephemeral_images.display_ref`, `join`, `self.agent.hist_add_tool_result`, `history.RawMessage`.
|
||||
- Important called helpers/classes observed in the source: `build_vision_model`, `use_vision_sidecar`, `self._get_max_embeds`, `Response`, `self._context_id`, `chat_media.save_image_base64`, `chat_media.save_image_data_url`, `chat_media.materialize_image_ref`, `ephemeral_images.consume_image`, `images.to_data_url`, `self.agent.hist_add_tool_result`, `history.RawMessage`.
|
||||
- Keep request/response, tool, or helper semantics documented here at the same time as source changes.
|
||||
|
||||
## Work Guidance
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue