mirror of
https://github.com/MoonshotAI/kimi-code.git
synced 2026-07-09 17:29:12 +00:00
feat: keep image-heavy sessions within provider request-size limits (#1508)
* feat(kosong): classify HTTP 413 request-body-too-large as a dedicated error type * feat(agent-core): lower default image downscale cap to 2000px and make it configurable * feat(agent-core): strip media to text markers and retry when the compaction request is too large * feat(agent-core): cap model-initiated image reads with a configurable byte budget * feat(agent-core): resend with degraded media when the provider rejects the request body as too large * test(agent-core): add explicit timeouts to encode-heavy image budget tests * feat: add WebP decoding support with wasm integration - Introduced a new WebP decoding module using @jsquash/webp's wasm decoder. - Implemented functions to decode WebP images and check for animated WebP formats. - Updated image compression tests to include scenarios for WebP handling, including encoding and decoding. - Enhanced error handling for API request size limits to accommodate various error messages. - Updated pnpm lockfile to include new dependencies for WebP encoding and decoding. * chore(changeset): consolidate this PR's entries into one * fix(nix): update pnpmDeps hash for merged lockfile * feat(agent-core): refuse HEIC/HEIF reads with platform-matched conversion guidance
This commit is contained in:
Kai
GitHub
parent
b91099ed7a
commit
1bf2c9afee
36 changed files with 1839 additions and 215 deletions
6
.changeset/image-request-size-limits.md
Normal file
6
.changeset/image-request-size-limits.md
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
"@moonshot-ai/kimi-code": patch
|
||||
---
|
||||
|
||||
Keep image-heavy sessions within provider request-size limits: model-read images now honor a 256 KB per-image budget and a 2000px downscale cap (configurable via `[image]` in config.toml or `KIMI_IMAGE_*` env vars), oversized WebP is compressed as well, HEIC/HEIF reads are refused with a platform-matched conversion command instead of poisoning the session, and a request-too-large rejection (HTTP 413) now recovers automatically — the request and /compact both retry with older media replaced by text markers instead of failing the session.
|
||||
|
||||
|
|
@ -141,14 +141,14 @@ describe('clipboard image paste compression', () => {
|
|||
if (att?.kind !== 'image') throw new Error('expected image attachment');
|
||||
|
||||
// Stored metadata reflects the compressed size.
|
||||
expect(Math.max(att.width, att.height)).toBeLessThanOrEqual(3000);
|
||||
expect(att.placeholder).toContain('3000×1500');
|
||||
expect(Math.max(att.width, att.height)).toBeLessThanOrEqual(2000);
|
||||
expect(att.placeholder).toContain('2000×1000');
|
||||
|
||||
// The stored bytes decode to the compressed dimensions — the thumbnail and
|
||||
// the submitted image both read from these bytes, so they cannot diverge.
|
||||
const dims = parseImageMeta(att.bytes);
|
||||
expect(dims).not.toBeNull();
|
||||
expect(Math.max(dims!.width, dims!.height)).toBeLessThanOrEqual(3000);
|
||||
expect(Math.max(dims!.width, dims!.height)).toBeLessThanOrEqual(2000);
|
||||
});
|
||||
|
||||
it('records and persists the pre-compression original for an oversized paste', async () => {
|
||||
|
|
|
|||
|
|
@ -87,11 +87,12 @@ Fields in the config file fall into two categories: **top-level scalars** that d
|
|||
| `thinking` | `table` | — | Default parameters for Thinking mode → [`thinking`](#thinking) |
|
||||
| `loop_control` | `table` | — | Agent loop control parameters → [`loop_control`](#loop_control) |
|
||||
| `background` | `table` | — | Background task runtime parameters → [`background`](#background) |
|
||||
| `image` | `table` | — | Image compression parameters → [`image`](#image) |
|
||||
| `services` | `table` | — | Built-in external service configuration → [`services`](#services) |
|
||||
| `permission` | `table` | — | Initial permission rules → [`permission`](#permission) |
|
||||
| `hooks` | `array<table>` | — | Lifecycle hooks; see [Hooks](../customization/hooks.md) |
|
||||
|
||||
The following sections cover each of the nested tables in turn: `providers`, `models`, `thinking`, `loop_control`, `background`, `services`, and `permission`.
|
||||
The following sections cover each of the nested tables in turn: `providers`, `models`, `thinking`, `loop_control`, `background`, `image`, `services`, and `permission`.
|
||||
|
||||
## `providers`
|
||||
|
||||
|
|
@ -202,6 +203,17 @@ You can also switch models temporarily without touching the config file — by s
|
|||
|
||||
In print mode (`kimi -p "<prompt>"`), Kimi Code runs a single non-interactive turn and exits as soon as the main agent finishes. If you launch background tasks (for example, concurrent subagents via `Agent(run_in_background=true)`) and need them to run to completion, set `keep_alive_on_exit = true`: the process then waits for every background task to reach a terminal state before exiting, bounded by `print_wait_ceiling_s`. Without it, the single turn ending tears background tasks down with the process.
|
||||
|
||||
## `image`
|
||||
|
||||
`image` controls how images are compressed before being sent to the model, across every ingestion point (pasted images, `ReadMediaFile` reads, images in MCP tool results, and so on).
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `max_edge_px` | `integer` | `2000` | Longest-edge ceiling in pixels. Larger images are scaled down proportionally to fit; raising it preserves more detail at the cost of larger request bodies |
|
||||
| `read_byte_budget` | `integer` | `262144` (256 KB) | Per-image byte budget for images the model reads for itself (`ReadMediaFile` default reads). It bounds the accumulated request-body size when the model keeps screenshotting and reading images; fine detail stays reachable through the `region` parameter, which reads a crop back at full fidelity (`region` and `full_resolution` are not subject to this budget) |
|
||||
|
||||
`max_edge_px` can be overridden by the `KIMI_IMAGE_MAX_EDGE_PX` environment variable and `read_byte_budget` by `KIMI_IMAGE_READ_BYTE_BUDGET`; both take higher priority than `config.toml`.
|
||||
|
||||
<!--
|
||||
## `experimental`
|
||||
|
||||
|
|
|
|||
|
|
@ -122,6 +122,8 @@ Switches that control the behavior of subsystems such as telemetry, background t
|
|||
| --- | --- | --- |
|
||||
| `KIMI_DISABLE_TELEMETRY` | Disable anonymous telemetry reporting | `1`, `true`, `yes`, `y` (case-insensitive) |
|
||||
| `KIMI_CODE_BACKGROUND_KEEP_ALIVE_ON_EXIT` | Whether to keep background tasks when the session closes; takes higher priority than `config.toml`. The default is to stop them on exit | Truthy: `1`/`true`/`yes`/`on`; falsy: `0`/`false`/`no`/`off` |
|
||||
| `KIMI_IMAGE_MAX_EDGE_PX` | Longest-edge ceiling (px) for image compression; takes higher priority than `[image] max_edge_px` in `config.toml` (default `2000`) | Positive integer; invalid values are ignored |
|
||||
| `KIMI_IMAGE_READ_BYTE_BUDGET` | Per-image byte budget for model-initiated image reads (`ReadMediaFile` default reads); takes higher priority than `[image] read_byte_budget` in `config.toml` (default `262144`, i.e. 256 KB) | Positive integer; invalid values are ignored |
|
||||
| `KIMI_CODE_PLUGIN_MARKETPLACE_URL` | Override the plugin marketplace JSON loaded by `/plugins`; useful for dev loopback servers, staging CDN files, or alternate marketplace directories | `https://code.kimi.com/kimi-code/plugins/marketplace.json`; also accepts `http://`, `file://` URLs, and local paths |
|
||||
| `KIMI_CODE_AGENT_SWARM_MAX_CONCURRENCY` | Cap how many AgentSwarm subagents run concurrently during the initial ramp; leave unset for no cap | Positive integer; invalid values fail fast |
|
||||
| `KIMI_CODE_EXPERIMENTAL_FLAG` | Enable all registered experimental features for this process | `1`, `true`, `yes`, `on` |
|
||||
|
|
|
|||
|
|
@ -87,11 +87,12 @@ timeout = 5
|
|||
| `thinking` | `table` | — | Thinking 模式默认参数 → [`thinking`](#thinking) |
|
||||
| `loop_control` | `table` | — | Agent 循环控制参数 → [`loop_control`](#loop_control) |
|
||||
| `background` | `table` | — | 后台任务运行参数 → [`background`](#background) |
|
||||
| `image` | `table` | — | 图片压缩参数 → [`image`](#image) |
|
||||
| `services` | `table` | — | 内置外部服务配置 → [`services`](#services) |
|
||||
| `permission` | `table` | — | 初始权限规则 → [`permission`](#permission) |
|
||||
| `hooks` | `array<table>` | — | 生命周期 hook，详见 [Hooks](../customization/hooks.md) |
|
||||
|
||||
以下各节对 `providers`、`models`、`thinking`、`loop_control`、`background`、`services`、`permission` 等嵌套表逐一展开。
|
||||
以下各节对 `providers`、`models`、`thinking`、`loop_control`、`background`、`image`、`services`、`permission` 等嵌套表逐一展开。
|
||||
|
||||
## `providers`
|
||||
|
||||
|
|
@ -202,6 +203,17 @@ display_name = "Kimi for Coding (custom)"
|
|||
|
||||
在 print 模式（`kimi -p "<prompt>"`）下，Kimi Code 只跑一个非交互的单轮 turn，主 agent 一结束就退出。如果你启动了后台任务（例如通过 `Agent(run_in_background=true)` 并发子代理）并希望它们跑完，请设置 `keep_alive_on_exit = true`：进程会在退出前等待所有后台任务进入终态，最长不超过 `print_wait_ceiling_s`。否则，单轮 turn 结束时后台任务会随进程一起被清理。
|
||||
|
||||
## `image`
|
||||
|
||||
`image` 控制图片发送给模型前的压缩行为，对所有图片入口生效（粘贴图片、`ReadMediaFile` 读图、MCP 工具结果里的图片等）。
|
||||
|
||||
| 字段 | 类型 | 默认值 | 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| `max_edge_px` | `integer` | `2000` | 图片最长边上限（像素）。超过时按比例缩小到该值以内；调大可保留更多细节，代价是更大的请求体积 |
|
||||
| `read_byte_budget` | `integer` | `262144`（256 KB） | 模型自行读取的图片（`ReadMediaFile` 默认读取）的单图字节预算。会话中模型反复截图、读图时，累计请求体大小由它控制；细节可通过 `region` 参数按原图坐标全保真回读（`region` 与 `full_resolution` 不受此预算限制） |
|
||||
|
||||
`max_edge_px` 可被环境变量 `KIMI_IMAGE_MAX_EDGE_PX` 覆盖，`read_byte_budget` 可被 `KIMI_IMAGE_READ_BYTE_BUDGET` 覆盖，优先级均高于配置文件。
|
||||
|
||||
<!--
|
||||
## `experimental`
|
||||
|
||||
|
|
|
|||
|
|
@ -122,6 +122,8 @@ kimi
|
|||
| --- | --- | --- |
|
||||
| `KIMI_DISABLE_TELEMETRY` | 关闭匿名遥测上报 | `1`、`true`、`yes`、`y`（不区分大小写） |
|
||||
| `KIMI_CODE_BACKGROUND_KEEP_ALIVE_ON_EXIT` | 会话关闭时是否保留后台任务，优先级高于 `config.toml`。默认会在退出时停止后台任务 | 真值：`1`/`true`/`yes`/`on`；假值：`0`/`false`/`no`/`off` |
|
||||
| `KIMI_IMAGE_MAX_EDGE_PX` | 图片压缩的最长边上限（像素），优先级高于 `config.toml` 的 `[image] max_edge_px`（默认 `2000`） | 正整数；非法值被忽略 |
|
||||
| `KIMI_IMAGE_READ_BYTE_BUDGET` | 模型自行读图（`ReadMediaFile` 默认读取）的单图字节预算，优先级高于 `config.toml` 的 `[image] read_byte_budget`（默认 `262144`，即 256 KB） | 正整数；非法值被忽略 |
|
||||
| `KIMI_CODE_PLUGIN_MARKETPLACE_URL` | 覆盖 `/plugins` 加载的 plugin marketplace JSON，适合 dev loopback server、测试 CDN 文件或替换 marketplace 目录 | `https://code.kimi.com/kimi-code/plugins/marketplace.json`；也接受 `http://`、`file://` URL 和本地路径 |
|
||||
| `KIMI_CODE_AGENT_SWARM_MAX_CONCURRENCY` | 限制 AgentSwarm 初始提升并发阶段可同时运行的子 Agent 数量；不设置表示不限制 | 正整数；非法值会立即失败 |
|
||||
| `KIMI_CODE_EXPERIMENTAL_FLAG` | 在当前进程启用所有已注册的实验功能 | `1`、`true`、`yes`、`on` |
|
||||
|
|
|
|||
|
|
@ -152,7 +152,7 @@
|
|||
inherit (finalAttrs) pname version src pnpmWorkspaces;
|
||||
inherit pnpm;
|
||||
fetcherVersion = 3;
|
||||
hash = "sha256-hUn5Srn3HnEEzU5DLxgjIzFjI0ukM3iSP4QagftEXdE=";
|
||||
hash = "sha256-i3T30dxNQ7DTWOmUX47kmqpIVtDYby0buPo58+Jhhmw=";
|
||||
};
|
||||
|
||||
nativeBuildInputs = [
|
||||
|
|
|
|||
|
|
@ -59,6 +59,7 @@
|
|||
},
|
||||
"dependencies": {
|
||||
"@antfu/utils": "^9.3.0",
|
||||
"@jsquash/webp": "^1.5.0",
|
||||
"@modelcontextprotocol/sdk": "^1.29.0",
|
||||
"@moonshot-ai/kaos": "workspace:^",
|
||||
"@moonshot-ai/kimi-code-oauth": "workspace:^",
|
||||
|
|
|
|||
36
packages/agent-core/scripts/generate-webp-dec-wasm.mjs
Normal file
36
packages/agent-core/scripts/generate-webp-dec-wasm.mjs
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
/**
|
||||
* Regenerate `src/tools/support/webp-dec-wasm.ts` from the installed
|
||||
* `@jsquash/webp` package.
|
||||
*
|
||||
* The WebP decoder wasm is committed as a base64 string module because the
|
||||
* published CLI bundles every dependency into a single file with no runtime
|
||||
* node_modules — a file-path lookup for the .wasm would break there, while a
|
||||
* string constant survives every packaging (vitest on sources, tsdown
|
||||
* bundling, nix builds) unchanged. Run this after bumping @jsquash/webp:
|
||||
*
|
||||
* node scripts/generate-webp-dec-wasm.mjs
|
||||
*/
|
||||
import { createRequire } from 'node:module';
|
||||
import { readFileSync, writeFileSync } from 'node:fs';
|
||||
import { resolve } from 'node:path';
|
||||
|
||||
const packageRoot = resolve(import.meta.dirname, '..');
|
||||
const require = createRequire(resolve(packageRoot, 'package.json'));
|
||||
|
||||
const wasmPath = require.resolve('@jsquash/webp/codec/dec/webp_dec.wasm');
|
||||
const version = require('@jsquash/webp/package.json').version;
|
||||
const wasm = readFileSync(wasmPath);
|
||||
|
||||
const target = resolve(packageRoot, 'src/tools/support/webp-dec-wasm.ts');
|
||||
writeFileSync(
|
||||
target,
|
||||
`// GENERATED FILE — do not edit by hand.
|
||||
// WebP decoder wasm from @jsquash/webp@${version} (codec/dec/webp_dec.wasm),
|
||||
// base64-encoded so the bundled CLI needs no on-disk wasm asset.
|
||||
// Regenerate with: node scripts/generate-webp-dec-wasm.mjs
|
||||
|
||||
export const WEBP_DECODER_WASM_BASE64 =
|
||||
'${wasm.toString('base64')}';
|
||||
`,
|
||||
);
|
||||
console.log(`Wrote ${target} (${wasm.length} bytes of wasm)`);
|
||||
|
|
@ -8,10 +8,12 @@ import {
|
|||
APIEmptyResponseError,
|
||||
inputTotal,
|
||||
isRetryableGenerateError,
|
||||
type ContentPart,
|
||||
type GenerateResult,
|
||||
type Message,
|
||||
type TokenUsage,
|
||||
APIContextOverflowError,
|
||||
APIRequestTooLargeError,
|
||||
APIStatusError,
|
||||
createUserMessage,
|
||||
} from '@moonshot-ai/kosong';
|
||||
|
|
@ -430,6 +432,7 @@ export class FullCompaction {
|
|||
// prefix-race check and `compactedCount`.
|
||||
let historyForModel: readonly ContextMessage[] = stripDynamicToolContext(originalHistory);
|
||||
let droppedCount = 0;
|
||||
let mediaStripAttempted = false;
|
||||
let overflowShrinkCount = 0;
|
||||
let emptyOrTruncatedShrinkCount = 0;
|
||||
while (true) {
|
||||
|
|
@ -465,6 +468,24 @@ export class FullCompaction {
|
|||
summary = extractCompactionSummary(response);
|
||||
break;
|
||||
} catch (error) {
|
||||
// A request-body-size rejection (HTTP 413) is first retried with
|
||||
// media parts replaced by text markers: accumulated base64 payloads
|
||||
// are the usual culprit, and a text summary does not need them —
|
||||
// the conversation already narrates what was seen, and the
|
||||
// ReadMediaFile `<image path="...">` text wrapper survives. Only
|
||||
// the summarizer input copy is rewritten; the real history keeps
|
||||
// its media. A 413 after the strip (or with no media to strip)
|
||||
// falls through to the overflow shrink below — dropping oldest
|
||||
// messages shrinks the body too.
|
||||
if (error instanceof APIRequestTooLargeError && !mediaStripAttempted) {
|
||||
mediaStripAttempted = true;
|
||||
const stripped = replaceMediaPartsWithMarkers(historyForModel);
|
||||
if (stripped !== historyForModel) {
|
||||
historyForModel = stripped;
|
||||
retryCount = 0;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
const isContextOverflow = this.shouldRecoverFromContextOverflow(
|
||||
error,
|
||||
estimatedCompactionRequestTokens,
|
||||
|
|
@ -472,7 +493,9 @@ export class FullCompaction {
|
|||
if (isContextOverflow) {
|
||||
this.observeContextOverflow(estimatedCompactionRequestTokens);
|
||||
}
|
||||
if (isContextOverflow && historyForModel.length > 1) {
|
||||
const shouldShrinkAfterOverflow =
|
||||
isContextOverflow || error instanceof APIRequestTooLargeError;
|
||||
if (shouldShrinkAfterOverflow && historyForModel.length > 1) {
|
||||
overflowShrinkCount += 1;
|
||||
if (overflowShrinkCount > MAX_COMPACTION_OVERFLOW_SHRINK_ATTEMPTS) {
|
||||
throw error;
|
||||
|
|
@ -639,6 +662,40 @@ export class FullCompaction {
|
|||
const MAX_COMPACTION_OVERFLOW_SHRINK_ATTEMPTS = 3;
|
||||
const COMPACTION_OVERFLOW_SHRINK_RATIOS = [0.7, 0.5, 0.35] as const;
|
||||
|
||||
const MEDIA_PART_MARKERS = {
|
||||
image_url: '[image]',
|
||||
audio_url: '[audio]',
|
||||
video_url: '[video]',
|
||||
} as const;
|
||||
|
||||
function isMediaPart(part: ContentPart): part is ContentPart & { type: keyof typeof MEDIA_PART_MARKERS } {
|
||||
return part.type in MEDIA_PART_MARKERS;
|
||||
}
|
||||
|
||||
/**
|
||||
* Replace media parts (image/audio/video) with text markers in the summarizer
|
||||
* input, for the 413 strip-and-retry above. Messages without media are
|
||||
* returned by reference (keeping the per-message token-estimate cache warm),
|
||||
* and when nothing changed the input array itself is returned so the caller
|
||||
* can tell there was no media to strip.
|
||||
*/
|
||||
function replaceMediaPartsWithMarkers(
|
||||
messages: readonly ContextMessage[],
|
||||
): readonly ContextMessage[] {
|
||||
let changed = false;
|
||||
const out = messages.map((message) => {
|
||||
if (!message.content.some(isMediaPart)) return message;
|
||||
changed = true;
|
||||
return {
|
||||
...message,
|
||||
content: message.content.map((part): ContentPart =>
|
||||
isMediaPart(part) ? { type: 'text', text: MEDIA_PART_MARKERS[part.type] } : part,
|
||||
),
|
||||
};
|
||||
});
|
||||
return changed ? out : messages;
|
||||
}
|
||||
|
||||
function shrinkCompactionHistoryAfterOverflow<T extends Message>(
|
||||
messages: readonly T[],
|
||||
attempt: number,
|
||||
|
|
|
|||
|
|
@ -18,6 +18,8 @@ import {
|
|||
type CompactionResult,
|
||||
} from '../compaction';
|
||||
import {
|
||||
degradeOlderMediaParts,
|
||||
MEDIA_DEGRADE_KEEP_RECENT,
|
||||
project,
|
||||
type ProjectionAnomaly,
|
||||
type ProjectOptions,
|
||||
|
|
@ -488,6 +490,17 @@ export class ContextMemory {
|
|||
});
|
||||
}
|
||||
|
||||
// Fallback projection for the post-413 media-degraded resend: the normal
|
||||
// wire projection with all but the most recent media parts replaced by text
|
||||
// markers, so a request body bloated by accumulated base64 media fits the
|
||||
// provider's size limit. Purely read-side — the history keeps its media —
|
||||
// and only used when the provider has already rejected the normal
|
||||
// projection as too large; see the request-too-large fallback in
|
||||
// `turn-step`.
|
||||
get mediaDegradedMessages(): Message[] {
|
||||
return degradeOlderMediaParts(this.messages, MEDIA_DEGRADE_KEEP_RECENT);
|
||||
}
|
||||
|
||||
useProjectedHistoryFrom(source: ContextMemory): void {
|
||||
this.clear();
|
||||
this.pushHistory(...trimTrailingOpenToolExchange(source.project(source.history)));
|
||||
|
|
|
|||
|
|
@ -465,3 +465,58 @@ export function trimTrailingOpenToolExchange(history: readonly Message[]): Messa
|
|||
const closed = assistant.toolCalls.every((toolCall) => trailingToolCallIds.has(toolCall.id));
|
||||
return closed ? [...history] : history.slice(0, lastNonToolIndex);
|
||||
}
|
||||
|
||||
/**
|
||||
* How many of the most recent media parts survive the media-degraded
|
||||
* projection. The tail images are what the model is actively working from
|
||||
* (the screenshot it just took); everything older is replaced by a marker.
|
||||
*/
|
||||
export const MEDIA_DEGRADE_KEEP_RECENT = 2;
|
||||
|
||||
const MEDIA_DEGRADED_PLACEHOLDERS = {
|
||||
image_url:
|
||||
'[image omitted: dropped to fit the provider request size limit; re-read the file to view it]',
|
||||
audio_url:
|
||||
'[audio omitted: dropped to fit the provider request size limit; re-read the file to hear it]',
|
||||
video_url:
|
||||
'[video omitted: dropped to fit the provider request size limit; re-read the file to view it]',
|
||||
} as const;
|
||||
|
||||
function isDegradableMediaPart(
|
||||
part: ContentPart,
|
||||
): part is ContentPart & { type: keyof typeof MEDIA_DEGRADED_PLACEHOLDERS } {
|
||||
return part.type in MEDIA_DEGRADED_PLACEHOLDERS;
|
||||
}
|
||||
|
||||
/**
|
||||
* Replace all but the `keepRecent` most recent media parts with deterministic
|
||||
* text markers. This is the media-degraded projection used to resend a request
|
||||
* the provider rejected as too large (HTTP 413 on accumulated base64 media):
|
||||
* a purely read-side transform — the underlying history is left untouched —
|
||||
* that trades old pixels for bytes while the surrounding text (including
|
||||
* ReadMediaFile's `<image path="...">` wrapper) survives, so the model can
|
||||
* re-read any file it still needs. Untouched messages are returned by
|
||||
* reference, and when nothing needs degrading the input array itself is
|
||||
* returned.
|
||||
*/
|
||||
export function degradeOlderMediaParts(
|
||||
messages: readonly Message[],
|
||||
keepRecent: number,
|
||||
): Message[] {
|
||||
const mediaCount = messages.reduce(
|
||||
(count, message) => count + message.content.filter(isDegradableMediaPart).length,
|
||||
0,
|
||||
);
|
||||
let toDegrade = Math.max(0, mediaCount - keepRecent);
|
||||
if (toDegrade === 0) return messages as Message[];
|
||||
|
||||
return messages.map((message) => {
|
||||
if (toDegrade === 0 || !message.content.some(isDegradableMediaPart)) return message;
|
||||
const content = message.content.map((part): ContentPart => {
|
||||
if (toDegrade === 0 || !isDegradableMediaPart(part)) return part;
|
||||
toDegrade -= 1;
|
||||
return { type: 'text', text: MEDIA_DEGRADED_PLACEHOLDERS[part.type] };
|
||||
});
|
||||
return { ...message, content };
|
||||
});
|
||||
}
|
||||
|
|
|
|||
|
|
@ -177,8 +177,9 @@ export interface AgentRecordEvents {
|
|||
messageCount: number;
|
||||
turnStep?: string;
|
||||
attempt?: string;
|
||||
/** Set when this request is the strict wire-compliant rebuild resend. */
|
||||
projection?: 'strict';
|
||||
/** Set when this request is a fallback resend (strict rebuild or
|
||||
* media-degraded rebuild). */
|
||||
projection?: 'strict' | 'media-degraded';
|
||||
/** Compaction only: messages dropped so far by overflow/empty shrinking. */
|
||||
droppedCount?: number;
|
||||
};
|
||||
|
|
|
|||
|
|
@ -724,6 +724,7 @@ export class TurnFlow {
|
|||
llm: this.agent.llm,
|
||||
buildMessages: () => this.agent.context.messages,
|
||||
buildMessagesStrict: () => this.agent.context.strictMessages,
|
||||
buildMessagesMediaDegraded: () => this.agent.context.mediaDegradedMessages,
|
||||
dispatchEvent: this.buildDispatchEvent(turnId),
|
||||
// Re-read per step (not snapshotted per turn) so a select_tools load
|
||||
// is dispatchable on the very next step of the same turn.
|
||||
|
|
|
|||
|
|
@ -134,6 +134,25 @@ export const BackgroundConfigSchema = z.object({
|
|||
|
||||
export type BackgroundConfig = z.infer<typeof BackgroundConfigSchema>;
|
||||
|
||||
export const ImageConfigSchema = z.object({
|
||||
/**
|
||||
* Longest-edge ceiling (px) applied when compressing images for the model.
|
||||
* Overrides the built-in default; the KIMI_IMAGE_MAX_EDGE_PX env var wins
|
||||
* over this value.
|
||||
*/
|
||||
maxEdgePx: z.number().int().min(1).optional(),
|
||||
/**
|
||||
* Raw-byte budget for images the model reads for itself (ReadMediaFile's
|
||||
* default path). Overrides the built-in default; the
|
||||
* KIMI_IMAGE_READ_BYTE_BUDGET env var wins over this value. Explicit
|
||||
* region / full_resolution reads use the provider-scale per-image limit
|
||||
* instead.
|
||||
*/
|
||||
readByteBudget: z.number().int().min(1).optional(),
|
||||
});
|
||||
|
||||
export type ImageConfig = z.infer<typeof ImageConfigSchema>;
|
||||
|
||||
export const ModelCatalogConfigSchema = z.object({
|
||||
/** Interval (ms) between automatic provider-model refreshes. `0` disables. */
|
||||
refreshIntervalMs: z.number().int().min(0).optional(),
|
||||
|
|
@ -256,6 +275,7 @@ export const KimiConfigSchema = z.object({
|
|||
extraSkillDirs: z.array(z.string()).optional(),
|
||||
loopControl: LoopControlSchema.optional(),
|
||||
background: BackgroundConfigSchema.optional(),
|
||||
image: ImageConfigSchema.optional(),
|
||||
modelCatalog: ModelCatalogConfigSchema.optional(),
|
||||
experimental: ExperimentalConfigSchema.optional(),
|
||||
telemetry: z.boolean().optional(),
|
||||
|
|
@ -270,6 +290,7 @@ const ThinkingConfigPatchSchema = ThinkingConfigSchema.partial();
|
|||
const PermissionConfigPatchSchema = PermissionConfigSchema.partial();
|
||||
const LoopControlPatchSchema = LoopControlSchema.partial();
|
||||
const BackgroundConfigPatchSchema = BackgroundConfigSchema.partial();
|
||||
const ImageConfigPatchSchema = ImageConfigSchema.partial();
|
||||
const ModelCatalogConfigPatchSchema = ModelCatalogConfigSchema.partial();
|
||||
const ExperimentalConfigPatchSchema = ExperimentalConfigSchema;
|
||||
const MoonshotServiceConfigPatchSchema = MoonshotServiceConfigSchema.partial();
|
||||
|
|
@ -296,6 +317,7 @@ export const KimiConfigPatchSchema = z
|
|||
extraSkillDirs: z.array(z.string()).optional(),
|
||||
loopControl: LoopControlPatchSchema.optional(),
|
||||
background: BackgroundConfigPatchSchema.optional(),
|
||||
image: ImageConfigPatchSchema.optional(),
|
||||
modelCatalog: ModelCatalogConfigPatchSchema.optional(),
|
||||
experimental: ExperimentalConfigPatchSchema.optional(),
|
||||
telemetry: z.boolean().optional(),
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@ import {
|
|||
type BackgroundConfig,
|
||||
type ExperimentalConfig,
|
||||
type HookDefConfig,
|
||||
type ImageConfig,
|
||||
type KimiConfig,
|
||||
type LoopControl,
|
||||
type ModelAlias,
|
||||
|
|
@ -312,6 +313,8 @@ export function transformTomlData(data: Record<string, unknown>): Record<string,
|
|||
result[targetKey] = transformLoopControlData(value);
|
||||
} else if (targetKey === 'background' && isPlainObject(value)) {
|
||||
result[targetKey] = transformPlainObject(value);
|
||||
} else if (targetKey === 'image' && isPlainObject(value)) {
|
||||
result[targetKey] = transformPlainObject(value);
|
||||
} else if (targetKey === 'experimental' && isPlainObject(value)) {
|
||||
result[targetKey] = cloneRecord(value);
|
||||
} else if (!isPlainObject(value)) {
|
||||
|
|
@ -489,6 +492,7 @@ export function configToTomlData(config: KimiConfig): Record<string, unknown> {
|
|||
setSection(out, 'services', config.services, servicesToToml);
|
||||
setSection(out, 'loop_control', config.loopControl, loopControlToToml);
|
||||
setSection(out, 'background', config.background, backgroundToToml);
|
||||
setSection(out, 'image', config.image, imageToToml);
|
||||
setSection(out, 'experimental', config.experimental, experimentalToToml);
|
||||
setSection(out, 'permission', config.permission, permissionToToml);
|
||||
setHooks(out, config.hooks);
|
||||
|
|
@ -670,6 +674,14 @@ function backgroundToToml(
|
|||
return out;
|
||||
}
|
||||
|
||||
function imageToToml(image: ImageConfig, rawImage: unknown): Record<string, unknown> {
|
||||
const out = cloneRecord(rawImage);
|
||||
for (const [key, value] of Object.entries(image)) {
|
||||
setDefined(out, camelToSnake(key), value);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function experimentalToToml(
|
||||
experimental: ExperimentalConfig,
|
||||
_rawExperimental: unknown,
|
||||
|
|
|
|||
|
|
@ -33,8 +33,10 @@ export interface LLMRequestLogFields {
|
|||
readonly attempt?: string;
|
||||
/** Request purpose; absent means a regular loop step. */
|
||||
readonly kind?: 'loop' | 'compaction';
|
||||
/** Set when the messages are the strict wire-compliant rebuild resend. */
|
||||
readonly projection?: 'strict';
|
||||
/** Set when the messages are a fallback resend projection: the strict
|
||||
* wire-compliant rebuild, or the media-degraded rebuild after a
|
||||
* request-too-large rejection. */
|
||||
readonly projection?: 'strict' | 'media-degraded';
|
||||
/** Compaction only: messages dropped so far by overflow/empty shrinking. */
|
||||
readonly droppedCount?: number;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -41,6 +41,15 @@ export interface RunTurnInput {
|
|||
* a tool_use/tool_result adjacency 400 (see `executeLoopStep`).
|
||||
*/
|
||||
readonly buildMessagesStrict?: LoopMessageBuilder | undefined;
|
||||
/**
|
||||
* Optional media-degraded rebuild of the request messages: old media parts
|
||||
* replaced by text markers, the most recent kept. Used to resend once after
|
||||
* the provider rejects the request body as too large (HTTP 413 on
|
||||
* accumulated media, see `executeLoopStep`); after a successful degraded
|
||||
* resend, later steps of the same turn build from this projection directly
|
||||
* so each step does not pay a fresh rejection.
|
||||
*/
|
||||
readonly buildMessagesMediaDegraded?: LoopMessageBuilder | undefined;
|
||||
readonly dispatchEvent: LoopEventDispatcher;
|
||||
readonly tools?: readonly ExecutableTool[] | undefined;
|
||||
/**
|
||||
|
|
@ -74,6 +83,7 @@ export async function runTurn(input: RunTurnInput): Promise<TurnResult> {
|
|||
llm,
|
||||
buildMessages,
|
||||
buildMessagesStrict,
|
||||
buildMessagesMediaDegraded,
|
||||
dispatchEvent,
|
||||
tools,
|
||||
buildTools,
|
||||
|
|
@ -89,6 +99,11 @@ export async function runTurn(input: RunTurnInput): Promise<TurnResult> {
|
|||
// Normal exits overwrite this with the completed step's stop reason.
|
||||
let stopReason: LoopTurnStopReason = 'end_turn';
|
||||
let activeStep: number | undefined;
|
||||
// Once a step only succeeded via the media-degraded resend, later steps of
|
||||
// this turn build from the degraded projection directly: the full-media
|
||||
// history is deterministically over the provider's body-size limit, so
|
||||
// rebuilding it would pay a fresh rejection on every step.
|
||||
let mediaDegradedActive = false;
|
||||
const recordStepUsage = async (
|
||||
stepUsage: TokenUsage,
|
||||
): Promise<RecordStepUsageResult | void> => {
|
||||
|
|
@ -109,8 +124,12 @@ export async function runTurn(input: RunTurnInput): Promise<TurnResult> {
|
|||
const stepResult = await executeLoopStep({
|
||||
turnId,
|
||||
signal,
|
||||
buildMessages,
|
||||
buildMessages:
|
||||
mediaDegradedActive && buildMessagesMediaDegraded !== undefined
|
||||
? buildMessagesMediaDegraded
|
||||
: buildMessages,
|
||||
buildMessagesStrict,
|
||||
buildMessagesMediaDegraded,
|
||||
dispatchEvent,
|
||||
llm,
|
||||
tools,
|
||||
|
|
@ -127,6 +146,7 @@ export async function runTurn(input: RunTurnInput): Promise<TurnResult> {
|
|||
recordUsage: recordStepUsage,
|
||||
});
|
||||
activeStep = undefined;
|
||||
mediaDegradedActive = mediaDegradedActive || stepResult.mediaDegradedResendUsed === true;
|
||||
|
||||
if (stepResult.stopReason === 'tool_use') {
|
||||
continue;
|
||||
|
|
|
|||
|
|
@ -9,7 +9,11 @@
|
|||
|
||||
import { randomUUID } from 'node:crypto';
|
||||
|
||||
import { isRecoverableRequestStructureError, type TokenUsage } from '@moonshot-ai/kosong';
|
||||
import {
|
||||
APIRequestTooLargeError,
|
||||
isRecoverableRequestStructureError,
|
||||
type TokenUsage,
|
||||
} from '@moonshot-ai/kosong';
|
||||
import type { Logger } from '#/logging/types';
|
||||
|
||||
import type { LoopEventDispatcher } from './events';
|
||||
|
|
@ -35,6 +39,8 @@ export interface ExecuteLoopStepDeps {
|
|||
readonly signal: AbortSignal;
|
||||
readonly buildMessages: LoopMessageBuilder;
|
||||
readonly buildMessagesStrict?: LoopMessageBuilder | undefined;
|
||||
/** See RunTurnInput.buildMessagesMediaDegraded. */
|
||||
readonly buildMessagesMediaDegraded?: LoopMessageBuilder | undefined;
|
||||
readonly dispatchEvent: LoopEventDispatcher;
|
||||
readonly llm: LLM;
|
||||
readonly tools?: readonly ExecutableTool[] | undefined;
|
||||
|
|
@ -57,12 +63,20 @@ export interface ExecuteLoopStepDeps {
|
|||
export async function executeLoopStep(deps: ExecuteLoopStepDeps): Promise<{
|
||||
readonly usage: TokenUsage;
|
||||
readonly stopReason: LoopStepStopReason;
|
||||
/**
|
||||
* True when this step only succeeded after resending with the
|
||||
* media-degraded projection. The turn loop uses it to keep later steps on
|
||||
* that projection — re-sending the full-media history would pay a fresh
|
||||
* rejection on every step of the turn.
|
||||
*/
|
||||
readonly mediaDegradedResendUsed?: boolean;
|
||||
}> {
|
||||
const {
|
||||
turnId,
|
||||
signal,
|
||||
buildMessages,
|
||||
buildMessagesStrict,
|
||||
buildMessagesMediaDegraded,
|
||||
dispatchEvent,
|
||||
llm,
|
||||
tools,
|
||||
|
|
@ -140,49 +154,90 @@ export async function executeLoopStep(deps: ExecuteLoopStepDeps): Promise<{
|
|||
log,
|
||||
} as const;
|
||||
let response: LLMChatResponse;
|
||||
let mediaDegradedResendUsed = false;
|
||||
try {
|
||||
response = await chatWithRetry({ ...retryInput, params: chatParams });
|
||||
} catch (error) {
|
||||
// A structural request rejection (tool_use/tool_result pairing, empty or
|
||||
// whitespace-only text, non-user first message, non-alternating roles) means
|
||||
// the projected history is not wire-compliant for a strict provider — and
|
||||
// since the same history is re-sent every turn, the session would stay stuck
|
||||
// on this error forever. Resend ONCE with a strict, guaranteed-compliant
|
||||
// rebuild (every open call closed, stray results dropped, leading non-user
|
||||
// trimmed, consecutive assistants merged) as a last resort. Any other error,
|
||||
// or a host that supplied no strict builder, propagates unchanged.
|
||||
if (buildMessagesStrict === undefined || !isRecoverableRequestStructureError(error)) throw error;
|
||||
signal.throwIfAborted();
|
||||
log?.warn('provider rejected request structure; resending with strict projection', {
|
||||
turnStep: `${turnId}.${String(currentStep)}`,
|
||||
model: llm.modelName,
|
||||
});
|
||||
const strictMessages = await buildMessagesStrict();
|
||||
signal.throwIfAborted();
|
||||
try {
|
||||
response = await chatWithRetry({
|
||||
...retryInput,
|
||||
params: {
|
||||
...chatParams,
|
||||
messages: strictMessages,
|
||||
requestLogFields: { projection: 'strict' },
|
||||
},
|
||||
});
|
||||
} catch (strictError) {
|
||||
// The strictly-sanitized rebuild was still rejected — our wire-compliance
|
||||
// repair did not cover this case. Surface it loudly: the session is stuck
|
||||
// and this is the signal we need to diagnose the gap.
|
||||
log?.error('strict resend still rejected by provider; request remains wire-invalid', {
|
||||
if (buildMessagesMediaDegraded !== undefined && error instanceof APIRequestTooLargeError) {
|
||||
// The provider rejected the request BODY as too large (HTTP 413) —
|
||||
// accumulated base64 media, not tokens, so compaction's token-driven
|
||||
// recovery never fires (media is estimated at a small flat cost). The
|
||||
// same media is re-sent on every request, so without intervention the
|
||||
// session stays stuck. Resend ONCE with the media-degraded projection
|
||||
// (old media replaced by text markers, the most recent kept); a
|
||||
// rejection of that rebuild propagates unchanged.
|
||||
signal.throwIfAborted();
|
||||
log?.warn('provider rejected request as too large; resending with degraded media', {
|
||||
turnStep: `${turnId}.${String(currentStep)}`,
|
||||
model: llm.modelName,
|
||||
originalError: errorMessage(error),
|
||||
strictError: errorMessage(strictError),
|
||||
});
|
||||
throw strictError;
|
||||
const degradedMessages = await buildMessagesMediaDegraded();
|
||||
signal.throwIfAborted();
|
||||
try {
|
||||
response = await chatWithRetry({
|
||||
...retryInput,
|
||||
params: {
|
||||
...chatParams,
|
||||
messages: degradedMessages,
|
||||
requestLogFields: { projection: 'media-degraded' },
|
||||
},
|
||||
});
|
||||
} catch (degradedError) {
|
||||
log?.error('media-degraded resend still rejected by provider', {
|
||||
turnStep: `${turnId}.${String(currentStep)}`,
|
||||
model: llm.modelName,
|
||||
originalError: errorMessage(error),
|
||||
degradedError: errorMessage(degradedError),
|
||||
});
|
||||
throw degradedError;
|
||||
}
|
||||
mediaDegradedResendUsed = true;
|
||||
log?.info('recovered after media-degraded resend', {
|
||||
turnStep: `${turnId}.${String(currentStep)}`,
|
||||
});
|
||||
} else if (buildMessagesStrict !== undefined && isRecoverableRequestStructureError(error)) {
|
||||
// A structural request rejection (tool_use/tool_result pairing, empty or
|
||||
// whitespace-only text, non-user first message, non-alternating roles) means
|
||||
// the projected history is not wire-compliant for a strict provider — and
|
||||
// since the same history is re-sent every turn, the session would stay stuck
|
||||
// on this error forever. Resend ONCE with a strict, guaranteed-compliant
|
||||
// rebuild (every open call closed, stray results dropped, leading non-user
|
||||
// trimmed, consecutive assistants merged) as a last resort. Any other error,
|
||||
// or a host that supplied no strict builder, propagates unchanged.
|
||||
signal.throwIfAborted();
|
||||
log?.warn('provider rejected request structure; resending with strict projection', {
|
||||
turnStep: `${turnId}.${String(currentStep)}`,
|
||||
model: llm.modelName,
|
||||
});
|
||||
const strictMessages = await buildMessagesStrict();
|
||||
signal.throwIfAborted();
|
||||
try {
|
||||
response = await chatWithRetry({
|
||||
...retryInput,
|
||||
params: {
|
||||
...chatParams,
|
||||
messages: strictMessages,
|
||||
requestLogFields: { projection: 'strict' },
|
||||
},
|
||||
});
|
||||
} catch (strictError) {
|
||||
// The strictly-sanitized rebuild was still rejected — our wire-compliance
|
||||
// repair did not cover this case. Surface it loudly: the session is stuck
|
||||
// and this is the signal we need to diagnose the gap.
|
||||
log?.error('strict resend still rejected by provider; request remains wire-invalid', {
|
||||
turnStep: `${turnId}.${String(currentStep)}`,
|
||||
model: llm.modelName,
|
||||
originalError: errorMessage(error),
|
||||
strictError: errorMessage(strictError),
|
||||
});
|
||||
throw strictError;
|
||||
}
|
||||
log?.info('recovered after strict resend', {
|
||||
turnStep: `${turnId}.${String(currentStep)}`,
|
||||
});
|
||||
} else {
|
||||
throw error;
|
||||
}
|
||||
log?.info('recovered after strict resend', {
|
||||
turnStep: `${turnId}.${String(currentStep)}`,
|
||||
});
|
||||
}
|
||||
const usage = response.usage;
|
||||
const usageResult = await recordUsage(usage);
|
||||
|
|
@ -244,6 +299,7 @@ export async function executeLoopStep(deps: ExecuteLoopStepDeps): Promise<{
|
|||
usage,
|
||||
stopReason:
|
||||
stopTurnAfterStep && effectiveStopReason === 'tool_use' ? 'end_turn' : effectiveStopReason,
|
||||
mediaDegradedResendUsed,
|
||||
};
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import { PluginManager } from '#/plugin';
|
|||
import { LocalFetchURLProvider } from '#/tools/providers/local-fetch-url';
|
||||
import { MoonshotFetchURLProvider } from '#/tools/providers/moonshot-fetch-url';
|
||||
import { MoonshotWebSearchProvider } from '#/tools/providers/moonshot-web-search';
|
||||
import { setConfiguredMaxImageEdgePx, setConfiguredReadImageByteBudget } from '#/tools/support/image-compress';
|
||||
import type { PromisableMethods } from '#/utils/types';
|
||||
import { getCoreVersion } from '#/version';
|
||||
import { resolveThinkingEffort } from '../agent/config/thinking';
|
||||
|
|
@ -205,6 +206,8 @@ export class KimiCore implements PromisableMethods<CoreAPI> {
|
|||
FLAG_DEFINITIONS,
|
||||
this.config.experimental,
|
||||
);
|
||||
setConfiguredMaxImageEdgePx(this.config.image?.maxEdgePx);
|
||||
setConfiguredReadImageByteBudget(this.config.image?.readByteBudget);
|
||||
this.sessionStore = new SessionStore(this.homeDir);
|
||||
this.plugins = new PluginManager({ kimiHomeDir: this.homeDir });
|
||||
// Capture the error rather than swallow it: mutators and explicit /plugins
|
||||
|
|
@ -1079,6 +1082,8 @@ export class KimiCore implements PromisableMethods<CoreAPI> {
|
|||
private setRuntimeConfig(config: KimiConfig): KimiConfig {
|
||||
this.config = config;
|
||||
this.experimentalFlags.setConfigOverrides(config.experimental);
|
||||
setConfiguredMaxImageEdgePx(config.image?.maxEdgePx);
|
||||
setConfiguredReadImageByteBudget(config.image?.readByteBudget);
|
||||
return this.config;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -45,6 +45,7 @@ import {
|
|||
compressImageForModel,
|
||||
cropImageForModel,
|
||||
formatByteSize,
|
||||
resolveReadImageByteBudget,
|
||||
type ImageCompressionTelemetry,
|
||||
type ImageCropRegion,
|
||||
} from '../../support/image-compress';
|
||||
|
|
@ -213,6 +214,45 @@ function buildMediaNote(input: {
|
|||
|
||||
// ── Implementation ───────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Refusal message for HEIC/HEIF with a conversion command matching the
|
||||
* execution environment (`kaos.osEnv.osKind` — where Bash actually runs, so
|
||||
* SSH/container sessions get the right command too). macOS converts with the
|
||||
* built-in `sips`; Linux and Windows have no built-in HEIC decoder, so the
|
||||
* guidance names the common tools and how to get them.
|
||||
*/
|
||||
function buildHeicConversionGuidance(path: string, mimeType: string, osKind: string): string {
|
||||
const converted = path.replace(/\.[^./\\]+$/, '') + '.jpg';
|
||||
return (
|
||||
`"${path}" is a ${mimeType} image, which the provider does not accept. ` +
|
||||
'Convert it to JPEG first, then read the converted file. ' +
|
||||
heicConversionCommand(path, converted, osKind)
|
||||
);
|
||||
}
|
||||
|
||||
function heicConversionCommand(path: string, converted: string, osKind: string): string {
|
||||
switch (osKind) {
|
||||
case 'macOS':
|
||||
return `On macOS: sips -s format jpeg "${path}" --out "${converted}"`;
|
||||
case 'Linux':
|
||||
return (
|
||||
`On Linux: heif-convert "${path}" "${converted}" (package libheif-examples), ` +
|
||||
`or with ImageMagick: magick "${path}" "${converted}"`
|
||||
);
|
||||
case 'Windows':
|
||||
return (
|
||||
`On Windows, with ImageMagick: magick "${path}" "${converted}" ` +
|
||||
'(install it first if missing: winget install ImageMagick.ImageMagick)'
|
||||
);
|
||||
default:
|
||||
return (
|
||||
`Options: sips -s format jpeg "${path}" --out "${converted}" (macOS), ` +
|
||||
`heif-convert "${path}" "${converted}" (Linux, package libheif-examples), ` +
|
||||
`or magick "${path}" "${converted}" (ImageMagick)`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
export class ReadMediaFileTool implements BuiltinTool<ReadMediaFileInput> {
|
||||
readonly name = 'ReadMediaFile' as const;
|
||||
readonly description: string;
|
||||
|
|
@ -293,6 +333,17 @@ export class ReadMediaFileTool implements BuiltinTool<ReadMediaFileInput> {
|
|||
'Tell the user to use a model with image input capability.',
|
||||
};
|
||||
}
|
||||
// HEIC/HEIF must never reach the provider: no provider accepts them,
|
||||
// and once the image_url lands in the history every subsequent request
|
||||
// in the session is rejected. Refuse with a conversion command for the
|
||||
// execution environment instead — the model can run it through Bash
|
||||
// (under the normal permission flow) and read the converted file.
|
||||
if (fileType.mimeType === 'image/heic' || fileType.mimeType === 'image/heif') {
|
||||
return {
|
||||
isError: true,
|
||||
output: buildHeicConversionGuidance(args.path, fileType.mimeType, this.kaos.osEnv.osKind),
|
||||
};
|
||||
}
|
||||
if (fileType.kind === 'video' && !this.capabilities.video_in) {
|
||||
return {
|
||||
isError: true,
|
||||
|
|
@ -388,10 +439,14 @@ export class ReadMediaFileTool implements BuiltinTool<ReadMediaFileInput> {
|
|||
};
|
||||
} else {
|
||||
// Shrink oversized images so a large screenshot neither wastes context
|
||||
// tokens nor trips the provider's per-image byte ceiling. Best effort:
|
||||
// on any failure compressImageForModel returns the original bytes, so
|
||||
// the read still succeeds with the uncompressed image.
|
||||
// tokens nor trips the provider's per-image byte ceiling. Model-read
|
||||
// images get the much tighter read budget: they accumulate in the
|
||||
// request body on every turn, and detail stays reachable through the
|
||||
// region readback (which ignores the budget). Best effort: on any
|
||||
// failure compressImageForModel returns the original bytes, so the
|
||||
// read still succeeds with the uncompressed image.
|
||||
const compressed = await compressImageForModel(data, fileType.mimeType, {
|
||||
byteBudget: resolveReadImageByteBudget(),
|
||||
telemetry: this.compressTelemetry,
|
||||
});
|
||||
const base64 = Buffer.from(compressed.data).toString('base64');
|
||||
|
|
|
|||
|
|
@ -8,14 +8,16 @@
|
|||
* untouched — the common case is a fast, codec-free pass-through.
|
||||
*
|
||||
* Design notes:
|
||||
* - Pure JS (jimp), imported lazily so the codec is only paid for when an
|
||||
* image actually needs work; startup and the fast path stay cheap.
|
||||
* - Pure JS (jimp + a wasm WebP decoder), imported lazily so the codecs are
|
||||
* only paid for when an image actually needs work; startup and the fast
|
||||
* path stay cheap.
|
||||
* - Best effort: any decode/encode failure returns the original bytes
|
||||
* unchanged (`changed: false`), so a compression problem never blocks a
|
||||
* prompt. Callers simply send the original instead.
|
||||
* - Only PNG and JPEG are re-encoded. GIF is passed through to preserve
|
||||
* animation; WebP is passed through because the default jimp build ships no
|
||||
* WebP codec. Unknown formats are passed through.
|
||||
* - PNG, JPEG, and (non-animated) WebP are re-encoded; WebP re-encodes
|
||||
* through the PNG/JPEG ladder, so only its decoder wasm ships. GIF and
|
||||
* animated WebP are passed through to preserve animation. Unknown formats
|
||||
* are passed through.
|
||||
* - Compression must never be silent to the model: results carry the
|
||||
* original dimensions, {@link buildImageCompressionCaption} renders the
|
||||
* shared "what was compressed, where is the original" note every ingestion
|
||||
|
|
@ -31,9 +33,53 @@ import type { ContentPart } from '@moonshot-ai/kosong';
|
|||
import type { TelemetryClient } from '#/telemetry';
|
||||
|
||||
import { sniffImageDimensions } from './file-type';
|
||||
import { decodeWebp, isAnimatedWebp } from './webp-decode';
|
||||
|
||||
/** Longest-edge ceiling (px). Larger images are scaled down to fit. */
|
||||
export const MAX_IMAGE_EDGE_PX = 3000;
|
||||
/**
|
||||
* Built-in longest-edge ceiling (px). Larger images are scaled down to fit.
|
||||
* This is the default only: the effective ceiling is resolved per call by
|
||||
* {@link resolveMaxImageEdgePx} (explicit option > env > config > this).
|
||||
*/
|
||||
export const MAX_IMAGE_EDGE_PX = 2000;
|
||||
|
||||
/**
|
||||
* Env var overriding the longest-edge ceiling (px). Read live on every
|
||||
* resolution so it applies in any process without wiring; a value that is
|
||||
* not a positive integer is ignored.
|
||||
*/
|
||||
export const MAX_IMAGE_EDGE_ENV = 'KIMI_IMAGE_MAX_EDGE_PX';
|
||||
|
||||
/**
|
||||
* The `[image] max_edge_px` value from config.toml, pushed by the config
|
||||
* owner (KimiCore) on load and reload. Processes that never load config
|
||||
* (TUI paste, ACP adapter) leave this unset and get env/built-in behavior.
|
||||
*/
|
||||
let configuredMaxImageEdgePx: number | undefined;
|
||||
|
||||
/** Push (or clear, with `undefined`) the config.toml longest-edge ceiling. */
|
||||
export function setConfiguredMaxImageEdgePx(value: number | undefined): void {
|
||||
configuredMaxImageEdgePx = value !== undefined && isPositiveInt(value) ? value : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Effective default longest-edge ceiling (px), for calls that pass no
|
||||
* explicit `maxEdge`. Precedence mirrors the experimental-flag resolver:
|
||||
* env var > config.toml > built-in {@link MAX_IMAGE_EDGE_PX}.
|
||||
*/
|
||||
export function resolveMaxImageEdgePx(
|
||||
env: Readonly<Record<string, string | undefined>> = process.env,
|
||||
): number {
|
||||
const raw = env[MAX_IMAGE_EDGE_ENV]?.trim();
|
||||
if (raw !== undefined && raw.length > 0 && /^\d+$/.test(raw)) {
|
||||
const parsed = Number(raw);
|
||||
if (isPositiveInt(parsed)) return parsed;
|
||||
}
|
||||
return configuredMaxImageEdgePx ?? MAX_IMAGE_EDGE_PX;
|
||||
}
|
||||
|
||||
function isPositiveInt(value: number): boolean {
|
||||
return Number.isInteger(value) && value > 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Raw-byte budget for a single image. base64 inflates bytes by ~4/3, so a
|
||||
|
|
@ -42,16 +88,70 @@ export const MAX_IMAGE_EDGE_PX = 3000;
|
|||
*/
|
||||
export const IMAGE_BYTE_BUDGET = 3.75 * 1024 * 1024;
|
||||
|
||||
/**
|
||||
* Built-in raw-byte budget for images the model reads for itself
|
||||
* (ReadMediaFile's default path). Far below {@link IMAGE_BYTE_BUDGET}: a
|
||||
* session that keeps screenshotting and reading images accumulates every one
|
||||
* of them in the request body on every turn, so per-image size — not the
|
||||
* provider's per-image ceiling — is what keeps the total under the
|
||||
* provider's request-size limit. 256 KB keeps a clean 2000px UI screenshot
|
||||
* on the lossless fast path while capping dense content at a readable
|
||||
* q80/1000px JPEG; fine detail stays reachable through the `region`
|
||||
* readback, which deliberately ignores this budget.
|
||||
*/
|
||||
export const READ_IMAGE_BYTE_BUDGET = 256 * 1024;
|
||||
|
||||
/**
|
||||
* Env var overriding the read-image byte budget. Read live on every
|
||||
* resolution; a value that is not a positive integer is ignored.
|
||||
*/
|
||||
export const READ_IMAGE_BYTE_BUDGET_ENV = 'KIMI_IMAGE_READ_BYTE_BUDGET';
|
||||
|
||||
/** The `[image] read_byte_budget` value from config.toml; see {@link setConfiguredMaxImageEdgePx}. */
|
||||
let configuredReadImageByteBudget: number | undefined;
|
||||
|
||||
/** Push (or clear, with `undefined`) the config.toml read-image byte budget. */
|
||||
export function setConfiguredReadImageByteBudget(value: number | undefined): void {
|
||||
configuredReadImageByteBudget = value !== undefined && isPositiveInt(value) ? value : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Effective read-image byte budget. Precedence mirrors
|
||||
* {@link resolveMaxImageEdgePx}: env var > config.toml > built-in
|
||||
* {@link READ_IMAGE_BYTE_BUDGET}.
|
||||
*/
|
||||
export function resolveReadImageByteBudget(
|
||||
env: Readonly<Record<string, string | undefined>> = process.env,
|
||||
): number {
|
||||
const raw = env[READ_IMAGE_BYTE_BUDGET_ENV]?.trim();
|
||||
if (raw !== undefined && raw.length > 0 && /^\d+$/.test(raw)) {
|
||||
const parsed = Number(raw);
|
||||
if (isPositiveInt(parsed)) return parsed;
|
||||
}
|
||||
return configuredReadImageByteBudget ?? READ_IMAGE_BYTE_BUDGET;
|
||||
}
|
||||
|
||||
/** Progressively lower JPEG quality until the payload fits the byte budget. */
|
||||
const JPEG_QUALITY_STEPS = [80, 60, 40, 20] as const;
|
||||
|
||||
/**
|
||||
* Longest-edge step-downs tried when the budget cannot be met at the fitted
|
||||
* size. The 2000px step preserves the behavior of the previous 2000px cap:
|
||||
* an image whose 2000px encode fits the budget keeps that resolution
|
||||
* instead of dropping straight to the 1000px last resort.
|
||||
* size. With the built-in 2000px ceiling the first step is a no-op; it
|
||||
* matters when a larger ceiling is configured (config/env/option). The
|
||||
* sub-1000px tail exists for small (read-scale) budgets: JPEG bytes shrink
|
||||
* roughly linearly with pixel count, so stepping down to 256px lets even
|
||||
* entropy-upper-bound content (noise, photos) land within any budget of a
|
||||
* few tens of KB instead of stalling at the q20@1000px floor.
|
||||
*/
|
||||
const FALLBACK_EDGES_PX = [2000, 1000] as const;
|
||||
const FALLBACK_EDGES_PX = [2000, 1000, 768, 512, 384, 256] as const;
|
||||
|
||||
/**
|
||||
* PNG rescales stop at this edge; below it the ladder goes lossy instead.
|
||||
* For text-bearing screenshots a q80 JPEG at 1000px reads better than a
|
||||
* lossless PNG at 512px — resolution beats losslessness once both are
|
||||
* degraded — so sub-floor edges are only ever tried with the JPEG ladder.
|
||||
*/
|
||||
const PNG_RESCALE_FLOOR_PX = 1000;
|
||||
|
||||
/**
|
||||
* Pixel-count ceiling above which we skip compression entirely. A tiny-byte,
|
||||
|
|
@ -76,11 +176,13 @@ const MAX_DECODE_PIXELS = 100_000_000;
|
|||
*/
|
||||
const MAX_DECODE_BYTES = 64 * 1024 * 1024;
|
||||
|
||||
/** Formats we can both decode and re-encode with the default jimp build. */
|
||||
const RECODABLE_MIME = new Set(['image/png', 'image/jpeg']);
|
||||
/** Formats we can decode and re-encode. WebP decodes via the bundled wasm
|
||||
* codec and re-encodes through the PNG/JPEG ladder (animated WebP is gated
|
||||
* to a passthrough before decoding). */
|
||||
const RECODABLE_MIME = new Set(['image/png', 'image/jpeg', 'image/webp']);
|
||||
|
||||
export interface CompressImageOptions {
|
||||
/** Override the longest-edge ceiling (px). */
|
||||
/** Override the longest-edge ceiling (px); defaults to {@link resolveMaxImageEdgePx}. */
|
||||
readonly maxEdge?: number;
|
||||
/** Override the raw-byte budget. */
|
||||
readonly byteBudget?: number;
|
||||
|
|
@ -152,7 +254,7 @@ export async function compressImageForModel(
|
|||
options: CompressImageOptions = {},
|
||||
): Promise<CompressImageResult> {
|
||||
const startedAt = Date.now();
|
||||
const maxEdge = options.maxEdge ?? MAX_IMAGE_EDGE_PX;
|
||||
const maxEdge = options.maxEdge ?? resolveMaxImageEdgePx();
|
||||
const byteBudget = options.byteBudget ?? IMAGE_BYTE_BUDGET;
|
||||
const maxDecodeBytes = options.maxDecodeBytes ?? MAX_DECODE_BYTES;
|
||||
const normalizedMime = normalizeMime(mimeType);
|
||||
|
|
@ -183,6 +285,11 @@ export async function compressImageForModel(
|
|||
if (bytes.length === 0) return finish('passthrough_unsupported', passthrough());
|
||||
// Only re-encode formats the codec handles; everything else passes through.
|
||||
if (!RECODABLE_MIME.has(normalizedMime)) return finish('passthrough_unsupported', passthrough());
|
||||
// Animated WebP would be flattened to one frame by decoding — pass it
|
||||
// through whole, the same reason GIF is never re-encoded.
|
||||
if (normalizedMime === 'image/webp' && isAnimatedWebp(bytes)) {
|
||||
return finish('passthrough_unsupported', passthrough());
|
||||
}
|
||||
|
||||
// Fast path: already within both budgets — no codec load, no allocation.
|
||||
const longestEdge = dims ? Math.max(dims.width, dims.height) : 0;
|
||||
|
|
@ -202,9 +309,10 @@ export async function compressImageForModel(
|
|||
if (bytes.length > maxDecodeBytes) return finish('passthrough_guard', passthrough());
|
||||
|
||||
try {
|
||||
const { Jimp } = await import('jimp');
|
||||
const image = await Jimp.fromBuffer(Buffer.from(bytes));
|
||||
const sourceIsPng = normalizedMime === 'image/png';
|
||||
const image = await decodeToJimp(bytes, normalizedMime);
|
||||
// WebP joins PNG on the lossless-first ladder: both carry alpha and
|
||||
// screenshot-grade detail that the PNG rungs preserve.
|
||||
const preferLossless = normalizedMime !== 'image/jpeg';
|
||||
// The decoded bitmap is authoritative for the original size: jimp
|
||||
// applies EXIF orientation while decoding, and this is the coordinate
|
||||
// space the encoded result and any later crop region (see
|
||||
|
|
@ -218,7 +326,7 @@ export async function compressImageForModel(
|
|||
fitWithinEdge(image, maxEdge);
|
||||
|
||||
const encoded = await encodeWithinBudget(image, {
|
||||
sourceIsPng,
|
||||
preferLossless,
|
||||
byteBudget,
|
||||
fallbackEdges: FALLBACK_EDGES_PX,
|
||||
});
|
||||
|
|
@ -518,7 +626,7 @@ export async function cropImageForModel(
|
|||
options: CropImageOptions = {},
|
||||
): Promise<CropImageOutcome> {
|
||||
const startedAt = Date.now();
|
||||
const maxEdge = options.maxEdge ?? MAX_IMAGE_EDGE_PX;
|
||||
const maxEdge = options.maxEdge ?? resolveMaxImageEdgePx();
|
||||
const byteBudget = options.byteBudget ?? IMAGE_BYTE_BUDGET;
|
||||
const maxDecodeBytes = options.maxDecodeBytes ?? MAX_DECODE_BYTES;
|
||||
const normalizedMime = normalizeMime(mimeType);
|
||||
|
|
@ -538,9 +646,14 @@ export async function cropImageForModel(
|
|||
if (!RECODABLE_MIME.has(normalizedMime)) {
|
||||
return fail(
|
||||
'unsupported_format',
|
||||
`Cropping is only supported for PNG and JPEG images; got ${mimeType}.`,
|
||||
`Cropping is only supported for PNG, JPEG, and WebP images; got ${mimeType}.`,
|
||||
);
|
||||
}
|
||||
// A crop is a still image by definition; decoding an animated WebP would
|
||||
// silently crop a single frame, so refuse explicitly.
|
||||
if (normalizedMime === 'image/webp' && isAnimatedWebp(bytes)) {
|
||||
return fail('unsupported_format', 'Cropping is not supported for animated WebP images.');
|
||||
}
|
||||
// NaN slips past every </>= comparison in the bounds guard below, so gate
|
||||
// on finiteness explicitly rather than surfacing a codec-internal error.
|
||||
if (
|
||||
|
|
@ -564,8 +677,7 @@ export async function cropImageForModel(
|
|||
}
|
||||
|
||||
try {
|
||||
const { Jimp } = await import('jimp');
|
||||
const image = await Jimp.fromBuffer(Buffer.from(bytes));
|
||||
const image = await decodeToJimp(bytes, normalizedMime);
|
||||
const originalWidth = image.width;
|
||||
const originalHeight = image.height;
|
||||
|
||||
|
|
@ -582,13 +694,15 @@ export async function cropImageForModel(
|
|||
const h = Math.min(Math.floor(region.height), originalHeight - y);
|
||||
const applied: ImageCropRegion = { x, y, width: w, height: h };
|
||||
image.crop({ x, y, w, h });
|
||||
const sourceIsPng = normalizedMime === 'image/png';
|
||||
// WebP joins PNG on the lossless side: both carry alpha and
|
||||
// screenshot-grade detail that PNG output preserves.
|
||||
const preferLossless = normalizedMime !== 'image/jpeg';
|
||||
|
||||
if (options.skipResize === true) {
|
||||
// Native resolution requested: encode once, favoring fidelity (lossless
|
||||
// PNG, or high-quality JPEG), and refuse rather than degrade when the
|
||||
// result cannot fit the byte budget.
|
||||
const buffer = sourceIsPng
|
||||
const buffer = preferLossless
|
||||
? await image.getBuffer('image/png', { deflateLevel: 9 })
|
||||
: await image.getBuffer('image/jpeg', { quality: 90 });
|
||||
if (buffer.length > byteBudget) {
|
||||
|
|
@ -603,7 +717,7 @@ export async function cropImageForModel(
|
|||
return succeed({
|
||||
ok: true,
|
||||
data: new Uint8Array(buffer),
|
||||
mimeType: sourceIsPng ? 'image/png' : 'image/jpeg',
|
||||
mimeType: preferLossless ? 'image/png' : 'image/jpeg',
|
||||
width: image.width,
|
||||
height: image.height,
|
||||
originalWidth,
|
||||
|
|
@ -617,7 +731,7 @@ export async function cropImageForModel(
|
|||
|
||||
fitWithinEdge(image, maxEdge);
|
||||
const encoded = await encodeWithinBudget(image, {
|
||||
sourceIsPng,
|
||||
preferLossless,
|
||||
byteBudget,
|
||||
fallbackEdges: FALLBACK_EDGES_PX,
|
||||
});
|
||||
|
|
@ -767,28 +881,53 @@ interface EncodedImage {
|
|||
}
|
||||
|
||||
interface EncodeOptions {
|
||||
readonly sourceIsPng: boolean;
|
||||
/** Lossless-first (PNG rungs before JPEG): PNG and WebP sources, which
|
||||
* carry alpha and screenshot-grade detail. JPEG sources skip straight to
|
||||
* the quality ladder — their detail is already lossy. */
|
||||
readonly preferLossless: boolean;
|
||||
readonly byteBudget: number;
|
||||
readonly fallbackEdges: readonly number[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Decode `bytes` into a jimp image. PNG/JPEG decode through jimp itself
|
||||
* (which applies EXIF orientation); WebP decodes through the bundled wasm
|
||||
* codec and enters jimp as a raw RGBA bitmap (WebP carries no EXIF-style
|
||||
* orientation, so the decoded pixels are already display space).
|
||||
*/
|
||||
async function decodeToJimp(bytes: Uint8Array, normalizedMime: string): Promise<JimpImage> {
|
||||
const { Jimp } = await import('jimp');
|
||||
if (normalizedMime === 'image/webp') {
|
||||
const decoded = await decodeWebp(bytes);
|
||||
return Jimp.fromBitmap({
|
||||
data: Buffer.from(decoded.data.buffer, decoded.data.byteOffset, decoded.data.byteLength),
|
||||
width: decoded.width,
|
||||
height: decoded.height,
|
||||
});
|
||||
}
|
||||
return Jimp.fromBuffer(Buffer.from(bytes));
|
||||
}
|
||||
|
||||
/**
|
||||
* Encode `image` (already fitted to the edge ceiling) under the byte budget.
|
||||
*
|
||||
* Strategy — prefer the source format so a downscaled screenshot stays lossless
|
||||
* PNG (preserving text and transparency), and only fall back to lossy JPEG when
|
||||
* PNG cannot meet the byte budget:
|
||||
* - PNG source: PNG at the fitted size → smaller PNG rescales, stepping down
|
||||
* the fallback edges → JPEG ladder.
|
||||
* - PNG source: PNG at the fitted size → smaller PNG rescales down to the
|
||||
* {@link PNG_RESCALE_FLOOR_PX} floor → JPEG ladder at that size → JPEG
|
||||
* ladder again at each sub-floor edge.
|
||||
* - JPEG source: the full quality ladder at the fitted size, then again at
|
||||
* each fallback edge — a smaller rescale must not skip the high-quality
|
||||
* rungs its extra pixels just paid for.
|
||||
*
|
||||
* Always returns the smallest buffer it produced, even if no attempt met the
|
||||
* budget — the caller still gates on whether it actually helped.
|
||||
* The sub-floor edges make the ladder converge for small (read-scale)
|
||||
* budgets: any budget of a few tens of KB is met by q20 at 256px even for
|
||||
* entropy-upper-bound content. Below that, the smallest buffer produced is
|
||||
* still returned — the caller gates on whether it actually helped.
|
||||
*/
|
||||
async function encodeWithinBudget(image: JimpImage, opts: EncodeOptions): Promise<EncodedImage> {
|
||||
const { sourceIsPng, byteBudget, fallbackEdges } = opts;
|
||||
const { preferLossless, byteBudget, fallbackEdges } = opts;
|
||||
let smallest: EncodedImage | null = null;
|
||||
|
||||
const consider = (data: Buffer, mimeType: string): EncodedImage => {
|
||||
|
|
@ -799,43 +938,52 @@ async function encodeWithinBudget(image: JimpImage, opts: EncodeOptions): Promis
|
|||
return candidate;
|
||||
};
|
||||
|
||||
if (sourceIsPng) {
|
||||
const jpegLadder = async (): Promise<EncodedImage | null> => {
|
||||
for (const quality of JPEG_QUALITY_STEPS) {
|
||||
const jpeg = await image.getBuffer('image/jpeg', { quality });
|
||||
if (jpeg.length <= byteBudget) return consider(jpeg, 'image/jpeg');
|
||||
consider(jpeg, 'image/jpeg');
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
if (preferLossless) {
|
||||
// Lossless PNG first: best for screenshots/UI (sharp text) and keeps alpha.
|
||||
const png = await image.getBuffer('image/png', { deflateLevel: 9 });
|
||||
if (png.length <= byteBudget) return consider(png, 'image/png');
|
||||
consider(png, 'image/png');
|
||||
|
||||
// Over budget: progressively smaller PNGs before going lossy.
|
||||
// Over budget: progressively smaller PNGs (down to the floor) before
|
||||
// going lossy.
|
||||
for (const edge of fallbackEdges) {
|
||||
if (edge < PNG_RESCALE_FLOOR_PX) break;
|
||||
if (!fitWithinEdge(image, edge)) continue;
|
||||
const smallerPng = await image.getBuffer('image/png', { deflateLevel: 9 });
|
||||
if (smallerPng.length <= byteBudget) return consider(smallerPng, 'image/png');
|
||||
consider(smallerPng, 'image/png');
|
||||
}
|
||||
|
||||
// Last resort: lossy JPEG ladder (drops transparency) to meet the budget.
|
||||
for (const quality of JPEG_QUALITY_STEPS) {
|
||||
const jpeg = await image.getBuffer('image/jpeg', { quality });
|
||||
if (jpeg.length <= byteBudget) return consider(jpeg, 'image/jpeg');
|
||||
consider(jpeg, 'image/jpeg');
|
||||
// Lossy JPEG ladder (drops transparency) at the floored size, then at
|
||||
// each sub-floor edge until the budget is met.
|
||||
const atFloor = await jpegLadder();
|
||||
if (atFloor !== null) return atFloor;
|
||||
for (const edge of fallbackEdges) {
|
||||
if (edge >= PNG_RESCALE_FLOOR_PX) continue;
|
||||
if (!fitWithinEdge(image, edge)) continue;
|
||||
const atEdge = await jpegLadder();
|
||||
if (atEdge !== null) return atEdge;
|
||||
}
|
||||
return smallest!;
|
||||
}
|
||||
|
||||
// JPEG source: quality ladder at the fitted size, then the full ladder
|
||||
// again at each fallback rescale.
|
||||
for (const quality of JPEG_QUALITY_STEPS) {
|
||||
const jpeg = await image.getBuffer('image/jpeg', { quality });
|
||||
if (jpeg.length <= byteBudget) return consider(jpeg, 'image/jpeg');
|
||||
consider(jpeg, 'image/jpeg');
|
||||
}
|
||||
const atFitted = await jpegLadder();
|
||||
if (atFitted !== null) return atFitted;
|
||||
for (const edge of fallbackEdges) {
|
||||
if (!fitWithinEdge(image, edge)) continue;
|
||||
for (const quality of JPEG_QUALITY_STEPS) {
|
||||
const jpeg = await image.getBuffer('image/jpeg', { quality });
|
||||
if (jpeg.length <= byteBudget) return consider(jpeg, 'image/jpeg');
|
||||
consider(jpeg, 'image/jpeg');
|
||||
}
|
||||
const atEdge = await jpegLadder();
|
||||
if (atEdge !== null) return atEdge;
|
||||
}
|
||||
|
||||
return smallest!;
|
||||
|
|
|
|||
7
packages/agent-core/src/tools/support/webp-dec-wasm.ts
Normal file
7
packages/agent-core/src/tools/support/webp-dec-wasm.ts
Normal file
File diff suppressed because one or more lines are too long
86
packages/agent-core/src/tools/support/webp-decode.ts
Normal file
86
packages/agent-core/src/tools/support/webp-decode.ts
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
/**
|
||||
* WebP decoding for the image-compression pipeline.
|
||||
*
|
||||
* The default jimp build ships no WebP codec, so WebP is decoded with
|
||||
* `@jsquash/webp`'s wasm decoder instead. The decoder wasm is compiled from a
|
||||
* base64 string committed to the repo (see `webp-dec-wasm.ts`): the published
|
||||
* CLI bundles every dependency into a single file with no runtime
|
||||
* node_modules, so a file-path or fetch lookup for the .wasm (what the
|
||||
* emscripten glue would do on its own) cannot work there — the module is
|
||||
* compiled and injected manually via the codec's `init()` hook. Only the
|
||||
* decoder is bundled: re-encoding runs through the existing PNG/JPEG ladder,
|
||||
* so the (larger) WebP encoder wasm is never needed.
|
||||
*
|
||||
* The repo's tsconfig carries no DOM lib, so the global `WebAssembly` and
|
||||
* `ImageData` names are unavailable at the type level — the wasm namespace is
|
||||
* reached through a structurally-typed `globalThis` and the decoder's RGBA
|
||||
* output is described by the local {@link DecodedWebp} shape.
|
||||
*/
|
||||
|
||||
/** Decoded RGBA bitmap in the shape `Jimp.fromBitmap` accepts. */
|
||||
export interface DecodedWebp {
|
||||
readonly data: Uint8ClampedArray;
|
||||
readonly width: number;
|
||||
readonly height: number;
|
||||
}
|
||||
|
||||
type WebpDecodeFn = (bytes: Uint8Array) => Promise<DecodedWebp>;
|
||||
|
||||
interface WasmGlobal {
|
||||
readonly WebAssembly: {
|
||||
compile(bytes: Uint8Array): Promise<object>;
|
||||
};
|
||||
}
|
||||
|
||||
let decoderReady: Promise<WebpDecodeFn> | null = null;
|
||||
|
||||
async function loadDecoder(): Promise<WebpDecodeFn> {
|
||||
decoderReady ??= (async () => {
|
||||
const [decodeModule, { WEBP_DECODER_WASM_BASE64 }] = await Promise.all([
|
||||
import('@jsquash/webp/decode.js'),
|
||||
import('./webp-dec-wasm'),
|
||||
]);
|
||||
const wasm = await (globalThis as unknown as WasmGlobal).WebAssembly.compile(
|
||||
Buffer.from(WEBP_DECODER_WASM_BASE64, 'base64'),
|
||||
);
|
||||
await decodeModule.init(wasm as never);
|
||||
const decode = decodeModule.default;
|
||||
return async (bytes: Uint8Array) => {
|
||||
const copy = new Uint8Array(bytes); // detach from any shared buffer
|
||||
return (await decode(copy.buffer as ArrayBuffer)) as unknown as DecodedWebp;
|
||||
};
|
||||
})();
|
||||
return decoderReady;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decode a (non-animated) WebP payload to RGBA. Throws on undecodable input —
|
||||
* callers keep their existing best-effort catch semantics.
|
||||
*/
|
||||
export async function decodeWebp(bytes: Uint8Array): Promise<DecodedWebp> {
|
||||
const decode = await loadDecoder();
|
||||
return decode(bytes);
|
||||
}
|
||||
|
||||
/**
|
||||
* True when the payload is a WebP whose VP8X container header carries the
|
||||
* ANIM flag. Animated WebP must be passed through, not re-encoded: decoding
|
||||
* yields a single frame and would silently destroy the animation (the same
|
||||
* reason GIF is passed through).
|
||||
*/
|
||||
export function isAnimatedWebp(bytes: Uint8Array): boolean {
|
||||
if (bytes.length < 21) return false;
|
||||
return (
|
||||
hasAscii(bytes, 'RIFF', 0) &&
|
||||
hasAscii(bytes, 'WEBP', 8) &&
|
||||
hasAscii(bytes, 'VP8X', 12) &&
|
||||
(bytes[20]! & 0x02) !== 0
|
||||
);
|
||||
}
|
||||
|
||||
function hasAscii(bytes: Uint8Array, text: string, at: number): boolean {
|
||||
for (let i = 0; i < text.length; i++) {
|
||||
if (bytes[at + i] !== text.codePointAt(i)) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
|
@ -439,6 +439,33 @@ describe('compaction — probe tests (high-risk scenarios)', () => {
|
|||
});
|
||||
});
|
||||
|
||||
describe('compaction — summarizer request media handling', () => {
|
||||
// GUARD — the first summarizer attempt sends media as-is: a multimodal
|
||||
// summarizer can still read an image nobody narrated. Media is only
|
||||
// replaced with text markers when the provider rejects the request body
|
||||
// as too large (see full.test.ts "request too large" cases).
|
||||
it('keeps media parts in the summarizer request when it is not rejected', async () => {
|
||||
const ctx = testAgent();
|
||||
ctx.configure({ provider: PROVIDER, modelCapabilities: CAPS });
|
||||
ctx.agent.context.appendUserMessage(
|
||||
[
|
||||
{ type: 'text', text: '<image path="/workspace/shot.png">' },
|
||||
{ type: 'image_url', imageUrl: { url: 'data:image/png;base64,AAAA' } },
|
||||
{ type: 'text', text: '</image>' },
|
||||
],
|
||||
{ kind: 'user' },
|
||||
);
|
||||
|
||||
ctx.mockNextResponse({ type: 'text', text: 'Summary.' });
|
||||
await ctx.rpc.beginCompaction({});
|
||||
await ctx.once('compaction.completed');
|
||||
|
||||
const request = ctx.llmCalls.at(-1)!;
|
||||
const parts = request.history.flatMap((message) => message.content);
|
||||
expect(parts.some((part) => part.type === 'image_url')).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('compaction — head/tail user-message retention', () => {
|
||||
const FIRST = `FIRST ${'a'.repeat(4_000)}`; // ~1k tokens
|
||||
const MIDDLE = 'b'.repeat(88_000); // ~22k tokens, over the 20k budget on its own
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import { join } from 'pathe';
|
|||
import {
|
||||
APIConnectionError,
|
||||
APIContextOverflowError,
|
||||
APIRequestTooLargeError,
|
||||
APIStatusError,
|
||||
generate as runKosongGenerate,
|
||||
UNKNOWN_CAPABILITY,
|
||||
|
|
@ -560,6 +561,142 @@ describe('FullCompaction', () => {
|
|||
await ctx.expectResumeMatches();
|
||||
});
|
||||
|
||||
it('strips media to text markers and retries when the summarizer request is rejected as too large', async () => {
|
||||
let attempts = 0;
|
||||
const histories: Message[][] = [];
|
||||
const generate: GenerateFn = async (_provider, _system, _tools, history) => {
|
||||
attempts += 1;
|
||||
histories.push(structuredClone(history));
|
||||
if (attempts === 1) {
|
||||
throw new APIRequestTooLargeError(413, 'Request exceeds the maximum size', 'req-413');
|
||||
}
|
||||
return textResult('Compacted without media.');
|
||||
};
|
||||
const ctx = testAgent({ generate });
|
||||
ctx.configure({
|
||||
provider: CATALOGUED_PROVIDER,
|
||||
modelCapabilities: CATALOGUED_MODEL_CAPABILITIES,
|
||||
});
|
||||
// A ReadMediaFile-shaped result: path wrapper text around inline image data.
|
||||
ctx.agent.context.appendUserMessage(
|
||||
[
|
||||
{ type: 'text', text: '<image path="/workspace/shot.png">' },
|
||||
{ type: 'image_url', imageUrl: { url: 'data:image/png;base64,AAAA' } },
|
||||
{ type: 'text', text: '</image>' },
|
||||
],
|
||||
{ kind: 'user' },
|
||||
);
|
||||
ctx.agent.context.appendUserMessage(
|
||||
[
|
||||
{ type: 'video_url', videoUrl: { url: 'data:video/mp4;base64,BBBB' } },
|
||||
{ type: 'audio_url', audioUrl: { url: 'data:audio/mp3;base64,CCCC' } },
|
||||
],
|
||||
{ kind: 'user' },
|
||||
);
|
||||
const compacted = ctx.once('context.apply_compaction');
|
||||
const completed = ctx.once('compaction.completed');
|
||||
|
||||
await ctx.rpc.beginCompaction({});
|
||||
await compacted;
|
||||
await completed;
|
||||
|
||||
expect(attempts).toBe(2);
|
||||
// The first attempt goes out with the media as-is.
|
||||
const firstParts = histories[0]!.flatMap((message) => message.content);
|
||||
expect(firstParts.some((part) => part.type === 'image_url')).toBe(true);
|
||||
// The 413 retry replaces every media part with a text marker; the
|
||||
// ReadMediaFile path wrapper survives so the summary can still reference
|
||||
// the file, and no base64 payload leaks into the retried request.
|
||||
// (Projection may merge adjacent text parts, so assert on the joined text.)
|
||||
const retryParts = histories[1]!.flatMap((message) => message.content);
|
||||
expect(retryParts.some((part) => part.type !== 'text' && part.type !== 'think')).toBe(false);
|
||||
const retryText = retryParts
|
||||
.filter((part) => part.type === 'text')
|
||||
.map((part) => part.text)
|
||||
.join('\n');
|
||||
expect(retryText).toContain('[image]');
|
||||
expect(retryText).toContain('[video]');
|
||||
expect(retryText).toContain('[audio]');
|
||||
expect(retryText).toContain('<image path="/workspace/shot.png">');
|
||||
expect(JSON.stringify(histories[1])).not.toContain('base64');
|
||||
// Stripping is not an overflow shrink: the retry drops no messages.
|
||||
expect(histories[1]!.length).toBe(histories[0]!.length);
|
||||
// The real history is untouched: recent kept user messages retain media.
|
||||
const keptParts = ctx.agent.context.history.flatMap((message) => message.content);
|
||||
expect(keptParts.some((part) => part.type === 'image_url')).toBe(true);
|
||||
await ctx.expectResumeMatches();
|
||||
});
|
||||
|
||||
it('shrinks the history when the summarizer request stays too large after media stripping', async () => {
|
||||
let attempts = 0;
|
||||
const histories: Message[][] = [];
|
||||
const generate: GenerateFn = async (_provider, _system, _tools, history) => {
|
||||
attempts += 1;
|
||||
histories.push(structuredClone(history));
|
||||
if (attempts <= 2) {
|
||||
throw new APIRequestTooLargeError(413, 'Request exceeds the maximum size');
|
||||
}
|
||||
return textResult('Recovered after shrink.');
|
||||
};
|
||||
const ctx = testAgent({ generate });
|
||||
ctx.configure({
|
||||
provider: CATALOGUED_PROVIDER,
|
||||
modelCapabilities: CATALOGUED_MODEL_CAPABILITIES,
|
||||
});
|
||||
ctx.appendExchange(1, 'old user one', 'old assistant one', 20);
|
||||
ctx.appendExchange(2, 'old user two', 'old assistant two', 40);
|
||||
ctx.appendExchange(3, 'recent user three', 'recent assistant three', 120);
|
||||
ctx.agent.context.appendUserMessage(
|
||||
[{ type: 'image_url', imageUrl: { url: 'data:image/png;base64,AAAA' } }],
|
||||
{ kind: 'user' },
|
||||
);
|
||||
const completed = ctx.once('compaction.completed');
|
||||
|
||||
await ctx.rpc.beginCompaction({});
|
||||
await completed;
|
||||
|
||||
expect(attempts).toBe(3);
|
||||
// Attempt 2 is the media strip: same message count, no media parts.
|
||||
expect(histories[1]!.length).toBe(histories[0]!.length);
|
||||
expect(
|
||||
histories[1]!.flatMap((m) => m.content).some((part) => part.type === 'image_url'),
|
||||
).toBe(false);
|
||||
// Attempt 3 falls through to the overflow shrink and drops old messages.
|
||||
expect(histories[2]!.length).toBeLessThan(histories[1]!.length);
|
||||
await ctx.expectResumeMatches();
|
||||
});
|
||||
|
||||
it('shrinks immediately on a too-large rejection when the history has no media', async () => {
|
||||
let attempts = 0;
|
||||
const histories: Message[][] = [];
|
||||
const generate: GenerateFn = async (_provider, _system, _tools, history) => {
|
||||
attempts += 1;
|
||||
histories.push(structuredClone(history));
|
||||
if (attempts === 1) {
|
||||
throw new APIRequestTooLargeError(413, 'Request exceeds the maximum size');
|
||||
}
|
||||
return textResult('Recovered after shrink.');
|
||||
};
|
||||
const ctx = testAgent({ generate });
|
||||
ctx.configure({
|
||||
provider: CATALOGUED_PROVIDER,
|
||||
modelCapabilities: CATALOGUED_MODEL_CAPABILITIES,
|
||||
});
|
||||
ctx.appendExchange(1, 'old user one', 'old assistant one', 20);
|
||||
ctx.appendExchange(2, 'old user two', 'old assistant two', 40);
|
||||
ctx.appendExchange(3, 'recent user three', 'recent assistant three', 120);
|
||||
const completed = ctx.once('compaction.completed');
|
||||
|
||||
await ctx.rpc.beginCompaction({});
|
||||
await completed;
|
||||
|
||||
// With no media to strip, the too-large rejection goes straight to the
|
||||
// overflow shrink instead of wasting a retry on an identical request.
|
||||
expect(attempts).toBe(2);
|
||||
expect(histories[1]!.length).toBeLessThan(histories[0]!.length);
|
||||
await ctx.expectResumeMatches();
|
||||
});
|
||||
|
||||
it('retries compaction responses with empty summaries before applying context', async () => {
|
||||
vi.useFakeTimers();
|
||||
const firstEmptySummary = deferred<void>();
|
||||
|
|
|
|||
|
|
@ -1,7 +1,11 @@
|
|||
import type { ContentPart, Message, ToolCall } from '@moonshot-ai/kosong';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import { project, type ProjectionAnomaly } from '../../../src/agent/context/projector';
|
||||
import {
|
||||
degradeOlderMediaParts,
|
||||
project,
|
||||
type ProjectionAnomaly,
|
||||
} from '../../../src/agent/context/projector';
|
||||
import type { ContextMessage } from '../../../src/agent/context/types';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
|
@ -768,3 +772,63 @@ function shuffle<T>(items: T[], rng: Rng): void {
|
|||
[items[i], items[j]] = [items[j]!, items[i]!];
|
||||
}
|
||||
}
|
||||
|
||||
describe('degradeOlderMediaParts', () => {
|
||||
function imageMessage(name: string): Message {
|
||||
return {
|
||||
role: 'user',
|
||||
content: [
|
||||
{ type: 'text', text: `<image path="/ws/${name}.png">` },
|
||||
{ type: 'image_url', imageUrl: { url: `data:image/png;base64,${name}` } },
|
||||
{ type: 'text', text: '</image>' },
|
||||
],
|
||||
toolCalls: [],
|
||||
};
|
||||
}
|
||||
|
||||
it('replaces all but the most recent N media parts with placeholder text', () => {
|
||||
const messages: Message[] = [
|
||||
imageMessage('old'),
|
||||
{
|
||||
role: 'user',
|
||||
content: [
|
||||
{ type: 'video_url', videoUrl: { url: 'data:video/mp4;base64,VVVV' } },
|
||||
{ type: 'audio_url', audioUrl: { url: 'data:audio/mp3;base64,SSSS' } },
|
||||
],
|
||||
toolCalls: [],
|
||||
},
|
||||
imageMessage('recent'),
|
||||
];
|
||||
|
||||
const degraded = degradeOlderMediaParts(messages, 2);
|
||||
|
||||
const parts = degraded.flatMap((message) => message.content);
|
||||
// The two most recent media parts survive; older ones become text.
|
||||
expect(parts.filter((part) => part.type === 'audio_url')).toHaveLength(1);
|
||||
expect(parts.filter((part) => part.type === 'image_url')).toHaveLength(1);
|
||||
expect(parts.filter((part) => part.type === 'video_url')).toHaveLength(0);
|
||||
const texts = parts.filter((part) => part.type === 'text').map((part) => part.text);
|
||||
expect(texts.some((text) => text.startsWith('[image omitted:'))).toBe(true);
|
||||
expect(texts.some((text) => text.startsWith('[video omitted:'))).toBe(true);
|
||||
// The path wrapper of the degraded image survives for readback.
|
||||
expect(texts).toContain('<image path="/ws/old.png">');
|
||||
// The surviving image is the most recent one.
|
||||
const survivor = parts.find((part) => part.type === 'image_url');
|
||||
expect(survivor?.type === 'image_url' && survivor.imageUrl.url).toContain('recent');
|
||||
});
|
||||
|
||||
it('returns the input array by reference when nothing needs degrading', () => {
|
||||
const messages: Message[] = [imageMessage('a'), imageMessage('b')];
|
||||
expect(degradeOlderMediaParts(messages, 2)).toBe(messages);
|
||||
});
|
||||
|
||||
it('keeps untouched messages by reference and does not mutate degraded ones', () => {
|
||||
const messages: Message[] = [imageMessage('old'), imageMessage('mid'), imageMessage('new')];
|
||||
const degraded = degradeOlderMediaParts(messages, 2);
|
||||
expect(degraded).not.toBe(messages);
|
||||
expect(degraded[1]).toBe(messages[1]);
|
||||
expect(degraded[2]).toBe(messages[2]);
|
||||
// The input's first message still carries its image part.
|
||||
expect(messages[0]!.content.some((part) => part.type === 'image_url')).toBe(true);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -9,9 +9,11 @@ import { createControlledPromise } from '@antfu/utils';
|
|||
import {
|
||||
APIConnectionError,
|
||||
APIEmptyResponseError,
|
||||
APIRequestTooLargeError,
|
||||
APIStatusError,
|
||||
APITimeoutError,
|
||||
type ChatProvider,
|
||||
type Message,
|
||||
type ModelCapability,
|
||||
type ToolCall,
|
||||
} from '@moonshot-ai/kosong';
|
||||
|
|
@ -60,6 +62,76 @@ function captureLogs(): { logger: Logger; entries: CapturedLogEntry[] } {
|
|||
}
|
||||
|
||||
describe('Agent turn flow', () => {
|
||||
it('degrades older history media and retries when the provider rejects the request body as too large', async () => {
|
||||
let attempts = 0;
|
||||
const histories: Message[][] = [];
|
||||
const generate: GenerateFn = async (_provider, _system, _tools, history) => {
|
||||
attempts += 1;
|
||||
histories.push(structuredClone(history));
|
||||
if (attempts === 1) {
|
||||
throw new APIRequestTooLargeError(413, 'Request exceeds the maximum size');
|
||||
}
|
||||
return {
|
||||
id: 'mock-degraded-recovery',
|
||||
message: { role: 'assistant', content: [{ type: 'text', text: 'done' }], toolCalls: [] },
|
||||
usage: { inputOther: 1, output: 1, inputCacheRead: 0, inputCacheCreation: 0 },
|
||||
finishReason: 'completed',
|
||||
rawFinishReason: 'stop',
|
||||
};
|
||||
};
|
||||
const ctx = testAgent({ generate });
|
||||
ctx.configure({
|
||||
provider: { type: 'kimi', apiKey: 'test-key', model: 'kimi-code' },
|
||||
modelCapabilities: {
|
||||
image_in: true,
|
||||
video_in: false,
|
||||
audio_in: false,
|
||||
thinking: false,
|
||||
tool_use: true,
|
||||
max_context_tokens: 256_000,
|
||||
},
|
||||
});
|
||||
// Three ReadMediaFile-shaped image results in the history.
|
||||
for (const name of ['a', 'b', 'c']) {
|
||||
ctx.agent.context.appendUserMessage(
|
||||
[
|
||||
{ type: 'text', text: `<image path="/workspace/${name}.png">` },
|
||||
{ type: 'image_url', imageUrl: { url: `data:image/png;base64,${name}AAA` } },
|
||||
{ type: 'text', text: '</image>' },
|
||||
],
|
||||
{ kind: 'user' },
|
||||
);
|
||||
}
|
||||
|
||||
await ctx.rpc.prompt({ input: [{ type: 'text', text: 'inspect the screenshots' }] });
|
||||
await ctx.untilTurnEnd();
|
||||
|
||||
expect(attempts).toBe(2);
|
||||
// The first request carried all three images.
|
||||
const firstParts = histories[0]!.flatMap((message) => message.content);
|
||||
expect(firstParts.filter((part) => part.type === 'image_url')).toHaveLength(3);
|
||||
// The retry keeps only the two most recent images; the oldest becomes a
|
||||
// placeholder while its path wrapper survives for readback.
|
||||
const retryParts = histories[1]!.flatMap((message) => message.content);
|
||||
const retryImages = retryParts.filter((part) => part.type === 'image_url');
|
||||
expect(retryImages).toHaveLength(2);
|
||||
expect(
|
||||
retryImages.map((part) => (part.type === 'image_url' ? part.imageUrl.url : '')),
|
||||
).toEqual(['data:image/png;base64,bAAA', 'data:image/png;base64,cAAA']);
|
||||
const retryText = retryParts
|
||||
.filter((part) => part.type === 'text')
|
||||
.map((part) => part.text)
|
||||
.join('\n');
|
||||
expect(retryText).toContain('[image omitted:');
|
||||
expect(retryText).toContain('<image path="/workspace/a.png">');
|
||||
// The real history is untouched.
|
||||
expect(
|
||||
ctx.agent.context.history
|
||||
.flatMap((message) => message.content)
|
||||
.filter((part) => part.type === 'image_url'),
|
||||
).toHaveLength(3);
|
||||
});
|
||||
|
||||
it('tracks turn_started and turn_interrupted telemetry', async () => {
|
||||
const records: TelemetryRecord[] = [];
|
||||
const ctx = testAgent({ telemetry: recordingTelemetry(records) });
|
||||
|
|
|
|||
|
|
@ -104,6 +104,10 @@ keep_alive_on_exit = false
|
|||
kill_grace_period_ms = 2000
|
||||
print_wait_ceiling_s = 3600
|
||||
|
||||
[image]
|
||||
max_edge_px = 1500
|
||||
read_byte_budget = 131072
|
||||
|
||||
[[hooks]]
|
||||
event = "PreToolUse"
|
||||
matcher = "Shell"
|
||||
|
|
@ -181,6 +185,7 @@ describe('harness config TOML loader', () => {
|
|||
killGracePeriodMs: 2000,
|
||||
printWaitCeilingS: 3600,
|
||||
});
|
||||
expect(config.image).toEqual({ maxEdgePx: 1500, readByteBudget: 131072 });
|
||||
expect(config.hooks).toEqual([
|
||||
{
|
||||
event: 'PreToolUse',
|
||||
|
|
@ -201,6 +206,23 @@ describe('harness config TOML loader', () => {
|
|||
expect(config.raw?.['notifications']).toEqual({ claim_stale_after_ms: 15000 });
|
||||
});
|
||||
|
||||
it('round-trips the [image] section', async () => {
|
||||
const dir = makeTempDir();
|
||||
const configPath = join(dir, 'image-round-trip.toml');
|
||||
const toml = `
|
||||
[image]
|
||||
max_edge_px = 2500
|
||||
read_byte_budget = 524288
|
||||
`;
|
||||
const config = parseConfigString(toml, configPath);
|
||||
expect(config.image).toEqual({ maxEdgePx: 2500, readByteBudget: 524288 });
|
||||
|
||||
await writeConfigFile(configPath, config);
|
||||
const text = await readFile(configPath, 'utf-8');
|
||||
const roundTripped = parseConfigString(text, configPath);
|
||||
expect(roundTripped.image).toEqual({ maxEdgePx: 2500, readByteBudget: 524288 });
|
||||
});
|
||||
|
||||
it('round-trips a custom registry source field on a provider', async () => {
|
||||
const dir = makeTempDir();
|
||||
const configPath = join(dir, 'round-trip.toml');
|
||||
|
|
|
|||
|
|
@ -1,14 +1,22 @@
|
|||
/**
|
||||
* Post-400 strict-resend fallback.
|
||||
* Post-rejection resend fallbacks in `executeLoopStep`.
|
||||
*
|
||||
* When a strict provider rejects a step with a tool_use/tool_result adjacency
|
||||
* 400, the same history would be re-sent every turn and the session would stay
|
||||
* stuck forever. `executeLoopStep` resends ONCE with a strict, guaranteed
|
||||
* wire-compliant rebuild (`buildMessagesStrict`). Any other error propagates
|
||||
* unchanged and the strict builder is never consulted.
|
||||
* Strict resend: when a strict provider rejects a step with a
|
||||
* tool_use/tool_result adjacency 400, the same history would be re-sent every
|
||||
* turn and the session would stay stuck forever. `executeLoopStep` resends
|
||||
* ONCE with a strict, guaranteed wire-compliant rebuild
|
||||
* (`buildMessagesStrict`).
|
||||
*
|
||||
* Media-degraded resend: when the provider rejects the request BODY as too
|
||||
* large (HTTP 413, `APIRequestTooLargeError` — accumulated base64 media, not
|
||||
* tokens), the step resends ONCE with the media-degraded projection
|
||||
* (`buildMessagesMediaDegraded`), and later steps of the same turn keep using
|
||||
* it so each step does not pay a fresh 413.
|
||||
*
|
||||
* Any other error propagates unchanged and the builders are never consulted.
|
||||
*/
|
||||
|
||||
import { APIStatusError, type Message } from '@moonshot-ai/kosong';
|
||||
import { APIRequestTooLargeError, APIStatusError, type Message } from '@moonshot-ai/kosong';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import {
|
||||
|
|
@ -18,8 +26,9 @@ import {
|
|||
type RunTurnInput,
|
||||
} from '../../src/loop/index';
|
||||
import { CollectingSink } from './fixtures/collecting-sink';
|
||||
import { FakeLLM, makeEndTurnResponse } from './fixtures/fake-llm';
|
||||
import { FakeLLM, makeEndTurnResponse, makeToolCall, makeToolUseResponse } from './fixtures/fake-llm';
|
||||
import { RecordingContext } from './fixtures/recording-context';
|
||||
import { EchoTool } from './fixtures/tools';
|
||||
|
||||
const ADJACENCY_400 = new APIStatusError(
|
||||
400,
|
||||
|
|
@ -160,3 +169,154 @@ describe('executeLoopStep — tool exchange adjacency fallback', () => {
|
|||
expect(strictCount).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('executeLoopStep — request-too-large media-degraded fallback', () => {
|
||||
const REQUEST_TOO_LARGE = new APIRequestTooLargeError(413, 'Request exceeds the maximum size');
|
||||
|
||||
interface MediaHarness {
|
||||
readonly input: RunTurnInput;
|
||||
readonly llm: FakeLLM;
|
||||
readonly degradedCalls: { count: number };
|
||||
readonly degradedMessages: Message[];
|
||||
readonly strictCalls: { count: number };
|
||||
readonly normalCalls: { count: number };
|
||||
}
|
||||
|
||||
function makeMediaHarness(
|
||||
error: unknown,
|
||||
extra: Partial<Pick<RunTurnInput, 'tools'>> & { responses?: number } = {},
|
||||
): MediaHarness {
|
||||
const responseCount = extra.responses ?? 2;
|
||||
const llm = new FakeLLM({
|
||||
responses: Array.from({ length: responseCount }, (_, index) =>
|
||||
makeEndTurnResponse(index === 0 ? 'unused' : 'recovered'),
|
||||
),
|
||||
throwOnIndex: { index: 0, error },
|
||||
});
|
||||
const sink = new CollectingSink({});
|
||||
const normalCalls = { count: 0 };
|
||||
const normalMessages: Message[] = [userMessage('normal projection')];
|
||||
const context = new RecordingContext({ messages: normalMessages });
|
||||
const buildMessages: LoopMessageBuilder = () => {
|
||||
normalCalls.count += 1;
|
||||
return normalMessages;
|
||||
};
|
||||
const degradedMessages: Message[] = [userMessage('media-degraded projection')];
|
||||
const degradedCalls = { count: 0 };
|
||||
const buildMessagesMediaDegraded: LoopMessageBuilder = () => {
|
||||
degradedCalls.count += 1;
|
||||
return degradedMessages;
|
||||
};
|
||||
const strictCalls = { count: 0 };
|
||||
const buildMessagesStrict: LoopMessageBuilder = () => {
|
||||
strictCalls.count += 1;
|
||||
return [userMessage('strict projection')];
|
||||
};
|
||||
const input: RunTurnInput = {
|
||||
turnId: 'turn-1',
|
||||
signal: new AbortController().signal,
|
||||
llm,
|
||||
buildMessages,
|
||||
buildMessagesStrict,
|
||||
buildMessagesMediaDegraded,
|
||||
tools: extra.tools,
|
||||
dispatchEvent: createLoopEventDispatcher({
|
||||
appendTranscriptRecord: context.appendTranscriptRecord,
|
||||
emitLiveEvent: sink.emit,
|
||||
}),
|
||||
};
|
||||
return { input, llm, degradedCalls, degradedMessages, strictCalls, normalCalls };
|
||||
}
|
||||
|
||||
it('resends once with the media-degraded projection after a request-too-large 413 and recovers', async () => {
|
||||
const { input, llm, degradedCalls, degradedMessages, strictCalls } =
|
||||
makeMediaHarness(REQUEST_TOO_LARGE);
|
||||
|
||||
const result = await runTurn(input);
|
||||
|
||||
expect(result.stopReason).toBe('end_turn');
|
||||
// Exactly two provider calls: the rejected one and the degraded resend —
|
||||
// and the strict builder is never consulted for a body-size rejection.
|
||||
expect(llm.callCount).toBe(2);
|
||||
expect(degradedCalls.count).toBe(1);
|
||||
expect(strictCalls.count).toBe(0);
|
||||
expect(llm.calls[0]?.messages).toEqual([userMessage('normal projection')]);
|
||||
expect(llm.calls[1]?.messages).toBe(degradedMessages);
|
||||
});
|
||||
|
||||
it('does not degrade for an unclassified 413 — the error propagates', async () => {
|
||||
const { input, llm, degradedCalls } = makeMediaHarness(
|
||||
new APIStatusError(413, 'Request failed'),
|
||||
);
|
||||
|
||||
await expect(runTurn(input)).rejects.toThrow('Request failed');
|
||||
|
||||
expect(llm.callCount).toBe(1);
|
||||
expect(degradedCalls.count).toBe(0);
|
||||
});
|
||||
|
||||
it('resends only once: a degraded rebuild that is also rejected gives up (no loop)', async () => {
|
||||
const llm = new FakeLLM({ responses: [] });
|
||||
let calls = 0;
|
||||
llm.chat = async () => {
|
||||
calls += 1;
|
||||
throw REQUEST_TOO_LARGE;
|
||||
};
|
||||
const sink = new CollectingSink({});
|
||||
const context = new RecordingContext({ messages: [userMessage('normal')] });
|
||||
let degradedCount = 0;
|
||||
const input: RunTurnInput = {
|
||||
turnId: 'turn-1',
|
||||
signal: new AbortController().signal,
|
||||
llm,
|
||||
buildMessages: context.buildMessages,
|
||||
buildMessagesMediaDegraded: () => {
|
||||
degradedCount += 1;
|
||||
return [userMessage('degraded')];
|
||||
},
|
||||
dispatchEvent: createLoopEventDispatcher({
|
||||
appendTranscriptRecord: context.appendTranscriptRecord,
|
||||
emitLiveEvent: sink.emit,
|
||||
}),
|
||||
};
|
||||
|
||||
await expect(runTurn(input)).rejects.toBe(REQUEST_TOO_LARGE);
|
||||
expect(calls).toBe(2); // first attempt + one degraded resend, then give up
|
||||
expect(degradedCount).toBe(1);
|
||||
});
|
||||
|
||||
it('keeps using the degraded projection for later steps of the same turn', async () => {
|
||||
// Step 1 is rejected with a 413 and recovers via the degraded projection,
|
||||
// then issues a tool call; step 2 must build from the degraded projection
|
||||
// directly — re-sending the full-media history would deterministically
|
||||
// pay a fresh 413 on every step.
|
||||
const echo = new EchoTool();
|
||||
const llm = new FakeLLM({
|
||||
responses: [
|
||||
makeEndTurnResponse('unused'),
|
||||
makeToolUseResponse([makeToolCall('echo', { text: 'hi' }, 'tc-1')]),
|
||||
makeEndTurnResponse('done'),
|
||||
],
|
||||
throwOnIndex: { index: 0, error: REQUEST_TOO_LARGE },
|
||||
});
|
||||
const harness = makeMediaHarness(REQUEST_TOO_LARGE);
|
||||
const input: RunTurnInput = {
|
||||
...harness.input,
|
||||
llm,
|
||||
tools: [echo],
|
||||
};
|
||||
|
||||
const result = await runTurn(input);
|
||||
|
||||
expect(result.stopReason).toBe('end_turn');
|
||||
expect(llm.callCount).toBe(3);
|
||||
// Step 1: normal projection rejected, degraded resend recovers.
|
||||
expect(llm.calls[0]?.messages).toEqual([userMessage('normal projection')]);
|
||||
expect(llm.calls[1]?.messages).toBe(harness.degradedMessages);
|
||||
// Step 2: built straight from the degraded projection, not the normal one.
|
||||
expect(llm.calls[2]?.messages).toBe(harness.degradedMessages);
|
||||
expect(harness.normalCalls.count).toBe(1);
|
||||
expect(harness.degradedCalls.count).toBe(2);
|
||||
expect(echo.calls).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -10,8 +10,10 @@
|
|||
* comes back as JPEG, strictly smaller than the input
|
||||
* - alpha: a translucent PNG stays PNG when the budget allows, and only
|
||||
* drops to JPEG as a last resort to meet a tiny budget
|
||||
* - fallback: corrupt/empty bytes and non-recodable formats (GIF/WebP)
|
||||
* return the original unchanged — never throws
|
||||
* - fallback: corrupt/empty bytes and non-recodable formats (GIF,
|
||||
* animated WebP) return the original unchanged — never throws
|
||||
* - webp: still WebP decodes through the bundled wasm codec and re-encodes
|
||||
* on the lossless-first ladder; animated WebP passes through whole
|
||||
* - invariant: `changed` implies the result is strictly smaller
|
||||
* - base64 wrapper round-trips
|
||||
* - performance: the fast path is codec-free; a large image compresses
|
||||
|
|
@ -32,8 +34,10 @@
|
|||
* result is a no-op; extreme aspect ratios never collapse to zero
|
||||
*/
|
||||
|
||||
import { createRequire } from 'node:module';
|
||||
|
||||
import { Jimp, ResizeStrategy } from 'jimp';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
// eslint-disable-next-line import/no-unresolved
|
||||
import {
|
||||
|
|
@ -44,7 +48,14 @@ import {
|
|||
cropImageForModel,
|
||||
extractImageCompressionCaptions,
|
||||
IMAGE_BYTE_BUDGET,
|
||||
MAX_IMAGE_EDGE_ENV,
|
||||
MAX_IMAGE_EDGE_PX,
|
||||
READ_IMAGE_BYTE_BUDGET,
|
||||
READ_IMAGE_BYTE_BUDGET_ENV,
|
||||
resolveMaxImageEdgePx,
|
||||
resolveReadImageByteBudget,
|
||||
setConfiguredMaxImageEdgePx,
|
||||
setConfiguredReadImageByteBudget,
|
||||
} from '../../src/tools/support/image-compress';
|
||||
// eslint-disable-next-line import/no-unresolved
|
||||
import { sniffImageDimensions } from '../../src/tools/support/file-type';
|
||||
|
|
@ -184,11 +195,11 @@ describe('compressImageForModel — dimension cap', () => {
|
|||
const result = await compressImageForModel(png, 'image/png');
|
||||
expect(result.changed).toBe(true);
|
||||
expect(Math.max(result.width, result.height)).toBe(MAX_IMAGE_EDGE_PX);
|
||||
// 4500x2250 → 3000x1500 (aspect 2:1 preserved).
|
||||
expect(result.width).toBe(3000);
|
||||
expect(result.height).toBe(1500);
|
||||
// 4500x2250 → 2000x1000 (aspect 2:1 preserved).
|
||||
expect(result.width).toBe(2000);
|
||||
expect(result.height).toBe(1000);
|
||||
const dims = sniffImageDimensions(result.data);
|
||||
expect(dims).toEqual({ width: 3000, height: 1500 });
|
||||
expect(dims).toEqual({ width: 2000, height: 1000 });
|
||||
});
|
||||
|
||||
it('respects a custom maxEdge', async () => {
|
||||
|
|
@ -239,9 +250,10 @@ describe('compressImageForModel — byte budget', () => {
|
|||
});
|
||||
|
||||
it('steps down through the 2000px edge before the 1000px fallback', async () => {
|
||||
// Regression guard for the 3000px cap raise: a PNG whose fitted encode
|
||||
// is over budget but whose 2000px encode fits must come back at 2000px
|
||||
// (as it did under the old cap), not skip straight to 1000px.
|
||||
// Fallback-ladder guard, pinned with an explicit 3000px ceiling (the
|
||||
// built-in default is 2000px, where the first fallback edge is a no-op):
|
||||
// a PNG whose fitted encode is over budget but whose 2000px encode fits
|
||||
// must come back at 2000px, not skip straight to 1000px.
|
||||
// The budget is anchored to the actual 2000px encode size (probed with
|
||||
// an unlimited budget) so the test does not depend on exact deflate
|
||||
// output sizes.
|
||||
|
|
@ -258,6 +270,7 @@ describe('compressImageForModel — byte budget', () => {
|
|||
expect(probe.finalByteLength + 1024).toBeLessThan(png.length);
|
||||
|
||||
const result = await compressImageForModel(png, 'image/png', {
|
||||
maxEdge: 3000,
|
||||
byteBudget: probe.finalByteLength + 1024,
|
||||
});
|
||||
expect(result.changed).toBe(true);
|
||||
|
|
@ -299,6 +312,195 @@ describe('compressImageForModel — byte budget', () => {
|
|||
);
|
||||
});
|
||||
|
||||
// ── webp fixtures (encoder wasm loaded manually from node_modules) ──
|
||||
|
||||
/**
|
||||
* Encode RGBA pixels to WebP using the encoder wasm from node_modules —
|
||||
* test-fixture only; production never encodes WebP. The emscripten glue
|
||||
* cannot auto-locate its wasm under Node (it tries fetch on a file URL), so
|
||||
* the module is compiled and injected manually, mirroring how production
|
||||
* initializes the decoder from the bundled base64.
|
||||
*/
|
||||
async function encodeWebp(
|
||||
image: { bitmap: { data: Buffer | Uint8Array; width: number; height: number } },
|
||||
quality = 90,
|
||||
): Promise<Uint8Array> {
|
||||
const requireLocal = createRequire(import.meta.url);
|
||||
const encMod = (await import(
|
||||
requireLocal.resolve('@jsquash/webp/encode.js')
|
||||
)) as typeof import('@jsquash/webp/encode.js');
|
||||
const { readFileSync } = await import('node:fs');
|
||||
// The repo tsconfig has no DOM lib, so the global WebAssembly name is
|
||||
// reached structurally (same approach as the production decoder).
|
||||
const wasmNamespace = (
|
||||
globalThis as unknown as { WebAssembly: { compile(bytes: Uint8Array): Promise<object> } }
|
||||
).WebAssembly;
|
||||
const wasm = await wasmNamespace.compile(
|
||||
readFileSync(requireLocal.resolve('@jsquash/webp/codec/enc/webp_enc.wasm')),
|
||||
);
|
||||
await encMod.init(wasm as never);
|
||||
const { bitmap } = image;
|
||||
const encoded = await encMod.default(
|
||||
{
|
||||
data: new Uint8ClampedArray(
|
||||
bitmap.data.buffer,
|
||||
bitmap.data.byteOffset,
|
||||
bitmap.data.byteLength,
|
||||
),
|
||||
width: bitmap.width,
|
||||
height: bitmap.height,
|
||||
} as never,
|
||||
{ quality },
|
||||
);
|
||||
return new Uint8Array(encoded);
|
||||
}
|
||||
|
||||
/** Minimal VP8X container header with the ANIM flag set. */
|
||||
function animatedWebpHeader(): Uint8Array {
|
||||
const bytes = new Uint8Array(30);
|
||||
const ascii = (s: string, at: number) => {
|
||||
for (let i = 0; i < s.length; i++) bytes[at + i] = s.charCodeAt(i);
|
||||
};
|
||||
ascii('RIFF', 0);
|
||||
new DataView(bytes.buffer).setUint32(4, 22, true);
|
||||
ascii('WEBP', 8);
|
||||
ascii('VP8X', 12);
|
||||
new DataView(bytes.buffer).setUint32(16, 10, true);
|
||||
bytes[20] = 0x02; // ANIM flag
|
||||
return bytes;
|
||||
}
|
||||
|
||||
describe('compressImageForModel — webp', () => {
|
||||
it(
|
||||
'downscales an oversized WebP to the edge cap',
|
||||
async () => {
|
||||
const source = new Jimp({ width: 2600, height: 1300, color: 0x3366ccff });
|
||||
const webp = await encodeWebp(source);
|
||||
const result = await compressImageForModel(webp, 'image/webp');
|
||||
expect(result.changed).toBe(true);
|
||||
expect(Math.max(result.width, result.height)).toBe(2000);
|
||||
expect(result.originalWidth).toBe(2600);
|
||||
expect(result.originalHeight).toBe(1300);
|
||||
expect(sniffImageDimensions(result.data)).toEqual({ width: 2000, height: 1000 });
|
||||
},
|
||||
15_000,
|
||||
);
|
||||
|
||||
it(
|
||||
're-encodes an over-budget WebP within the byte budget',
|
||||
async () => {
|
||||
const budget = 128 * 1024;
|
||||
const noisy = new Jimp({ width: 1200, height: 1200, color: 0x000000ff });
|
||||
fillXorshiftNoise(noisy.bitmap.data);
|
||||
const webp = await encodeWebp(noisy, 100);
|
||||
expect(webp.length).toBeGreaterThan(budget);
|
||||
const result = await compressImageForModel(webp, 'image/webp', { byteBudget: budget });
|
||||
expect(result.changed).toBe(true);
|
||||
expect(result.finalByteLength).toBeLessThanOrEqual(budget);
|
||||
},
|
||||
15_000,
|
||||
);
|
||||
|
||||
it(
|
||||
'keeps alpha when re-encoding a translucent WebP',
|
||||
async () => {
|
||||
const translucent = new Jimp({ width: 2600, height: 1300, color: 0x33_66_cc_80 });
|
||||
const webp = await encodeWebp(translucent);
|
||||
const result = await compressImageForModel(webp, 'image/webp');
|
||||
expect(result.changed).toBe(true);
|
||||
expect(result.mimeType).toBe('image/png');
|
||||
expect(await decodeAlpha(result.data)).toBe(true);
|
||||
},
|
||||
15_000,
|
||||
);
|
||||
|
||||
it('passes an animated WebP through to preserve animation', async () => {
|
||||
const animated = animatedWebpHeader();
|
||||
const result = await compressImageForModel(animated, 'image/webp');
|
||||
expect(result.changed).toBe(false);
|
||||
expect(result.data).toBe(animated);
|
||||
});
|
||||
|
||||
it(
|
||||
'crops a region out of a WebP',
|
||||
async () => {
|
||||
const source = new Jimp({ width: 800, height: 400, color: 0x3366ccff });
|
||||
const webp = await encodeWebp(source);
|
||||
const result = await cropImageForModel(webp, 'image/webp', {
|
||||
x: 10,
|
||||
y: 20,
|
||||
width: 300,
|
||||
height: 200,
|
||||
});
|
||||
expect(result.ok).toBe(true);
|
||||
if (!result.ok) return;
|
||||
expect(result.width).toBe(300);
|
||||
expect(result.height).toBe(200);
|
||||
expect(result.originalWidth).toBe(800);
|
||||
expect(result.originalHeight).toBe(400);
|
||||
},
|
||||
15_000,
|
||||
);
|
||||
});
|
||||
|
||||
// ── small byte budgets (per-image read budget scale) ────────────────
|
||||
|
||||
// These walk many encode rungs over noise fixtures, which is slow on CI
|
||||
// runners — each carries an explicit timeout like the quality-ladder test
|
||||
// above.
|
||||
describe('compressImageForModel — small byte budgets', () => {
|
||||
it(
|
||||
'converges under a 128KB budget for high-entropy PNG content',
|
||||
async () => {
|
||||
// Statistically random noise is the entropy upper bound (photos, dense
|
||||
// charts): the old [2000, 1000] fallback floor left q20@1000px at ~200KB,
|
||||
// over a read-scale budget. The extended ladder must land within it.
|
||||
const budget = 128 * 1024;
|
||||
const png = await randomNoisePng(1200, 1200);
|
||||
expect(png.length).toBeGreaterThan(budget);
|
||||
const result = await compressImageForModel(png, 'image/png', { byteBudget: budget });
|
||||
expect(result.changed).toBe(true);
|
||||
expect(result.finalByteLength).toBeLessThanOrEqual(budget);
|
||||
expect(sniffImageDimensions(result.data)).not.toBeNull();
|
||||
},
|
||||
15_000,
|
||||
);
|
||||
|
||||
it(
|
||||
'converges under a 128KB budget for a JPEG source',
|
||||
async () => {
|
||||
const budget = 128 * 1024;
|
||||
const jpeg = await randomNoiseJpeg(1200, 1200);
|
||||
expect(jpeg.length).toBeGreaterThan(budget);
|
||||
const result = await compressImageForModel(jpeg, 'image/jpeg', { byteBudget: budget });
|
||||
expect(result.changed).toBe(true);
|
||||
expect(result.mimeType).toBe('image/jpeg');
|
||||
expect(result.finalByteLength).toBeLessThanOrEqual(budget);
|
||||
},
|
||||
15_000,
|
||||
);
|
||||
|
||||
it(
|
||||
'shrinks pixels instead of passing through an already-optimized JPEG over budget',
|
||||
async () => {
|
||||
// A JPEG already at the encoder's quality floor for its size: re-encoding
|
||||
// at the same size cannot shrink it, so without sub-size fallbacks the
|
||||
// "unhelpful" guard used to return the original — silently over budget.
|
||||
const image = new Jimp({ width: 900, height: 900, color: 0x000000ff });
|
||||
fillXorshiftNoise(image.bitmap.data);
|
||||
const optimized = new Uint8Array(await image.getBuffer('image/jpeg', { quality: 20 }));
|
||||
const budget = optimized.length - 10 * 1024;
|
||||
expect(budget).toBeGreaterThan(0);
|
||||
|
||||
const result = await compressImageForModel(optimized, 'image/jpeg', { byteBudget: budget });
|
||||
expect(result.changed).toBe(true);
|
||||
expect(result.finalByteLength).toBeLessThanOrEqual(budget);
|
||||
expect(Math.max(result.width, result.height)).toBeLessThan(900);
|
||||
},
|
||||
15_000,
|
||||
);
|
||||
});
|
||||
|
||||
// ── fallback / robustness ────────────────────────────────────────────
|
||||
|
||||
describe('compressImageForModel — fallback', () => {
|
||||
|
|
@ -325,7 +527,10 @@ describe('compressImageForModel — fallback', () => {
|
|||
expect(result.data).toBe(gif);
|
||||
});
|
||||
|
||||
it('passes WebP through (no codec in the default build)', async () => {
|
||||
it('passes undecodable WebP bytes through (never throws)', async () => {
|
||||
// A bare RIFF/WEBP container header with no image payload: the sniffer
|
||||
// reports no dimensions and the wasm decoder cannot decode it, so it
|
||||
// passes through unchanged like any other undecodable input.
|
||||
const webp = new Uint8Array([
|
||||
0x52, 0x49, 0x46, 0x46, 0, 0, 0, 0, 0x57, 0x45, 0x42, 0x50,
|
||||
]);
|
||||
|
|
@ -444,7 +649,101 @@ describe('compressImageForModel — performance', () => {
|
|||
|
||||
it('exposes a sane default budget', () => {
|
||||
expect(IMAGE_BYTE_BUDGET).toBeGreaterThan(0);
|
||||
expect(MAX_IMAGE_EDGE_PX).toBe(3000);
|
||||
expect(MAX_IMAGE_EDGE_PX).toBe(2000);
|
||||
});
|
||||
});
|
||||
|
||||
// ── default edge resolution (env + config) ──────────────────────────
|
||||
|
||||
describe('resolveMaxImageEdgePx', () => {
|
||||
afterEach(() => {
|
||||
vi.unstubAllEnvs();
|
||||
setConfiguredMaxImageEdgePx(undefined);
|
||||
});
|
||||
|
||||
it('defaults to the built-in ceiling', () => {
|
||||
expect(resolveMaxImageEdgePx()).toBe(MAX_IMAGE_EDGE_PX);
|
||||
});
|
||||
|
||||
it('uses the configured value when set, and clears with undefined', () => {
|
||||
setConfiguredMaxImageEdgePx(1200);
|
||||
expect(resolveMaxImageEdgePx()).toBe(1200);
|
||||
setConfiguredMaxImageEdgePx(undefined);
|
||||
expect(resolveMaxImageEdgePx()).toBe(MAX_IMAGE_EDGE_PX);
|
||||
});
|
||||
|
||||
it('lets the env var override the configured value', () => {
|
||||
setConfiguredMaxImageEdgePx(1200);
|
||||
vi.stubEnv(MAX_IMAGE_EDGE_ENV, '900');
|
||||
expect(resolveMaxImageEdgePx()).toBe(900);
|
||||
});
|
||||
|
||||
it.each(['abc', '-100', '0', '1.5', ' '])('ignores the invalid env value "%s"', (raw) => {
|
||||
vi.stubEnv(MAX_IMAGE_EDGE_ENV, raw);
|
||||
expect(resolveMaxImageEdgePx()).toBe(MAX_IMAGE_EDGE_PX);
|
||||
});
|
||||
|
||||
it('drives compressImageForModel when no explicit maxEdge is passed', async () => {
|
||||
setConfiguredMaxImageEdgePx(1200);
|
||||
const png = await solidPng(1600, 800);
|
||||
const result = await compressImageForModel(png, 'image/png');
|
||||
expect(result.changed).toBe(true);
|
||||
expect(result.width).toBe(1200);
|
||||
expect(result.height).toBe(600);
|
||||
});
|
||||
|
||||
it('an explicit maxEdge option still wins over env and config', async () => {
|
||||
setConfiguredMaxImageEdgePx(1200);
|
||||
vi.stubEnv(MAX_IMAGE_EDGE_ENV, '900');
|
||||
const png = await solidPng(1600, 800);
|
||||
const result = await compressImageForModel(png, 'image/png', { maxEdge: 800 });
|
||||
expect(result.changed).toBe(true);
|
||||
expect(result.width).toBe(800);
|
||||
expect(result.height).toBe(400);
|
||||
});
|
||||
|
||||
it('drives cropImageForModel region fitting', async () => {
|
||||
setConfiguredMaxImageEdgePx(400);
|
||||
const png = await solidPng(1600, 800);
|
||||
const result = await cropImageForModel(png, 'image/png', {
|
||||
x: 0,
|
||||
y: 0,
|
||||
width: 800,
|
||||
height: 800,
|
||||
});
|
||||
expect(result.ok).toBe(true);
|
||||
if (!result.ok) return;
|
||||
expect(Math.max(result.width, result.height)).toBe(400);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveReadImageByteBudget', () => {
|
||||
afterEach(() => {
|
||||
vi.unstubAllEnvs();
|
||||
setConfiguredReadImageByteBudget(undefined);
|
||||
});
|
||||
|
||||
it('defaults to the built-in read budget', () => {
|
||||
expect(READ_IMAGE_BYTE_BUDGET).toBe(256 * 1024);
|
||||
expect(resolveReadImageByteBudget()).toBe(READ_IMAGE_BYTE_BUDGET);
|
||||
});
|
||||
|
||||
it('uses the configured value when set, and clears with undefined', () => {
|
||||
setConfiguredReadImageByteBudget(512 * 1024);
|
||||
expect(resolveReadImageByteBudget()).toBe(512 * 1024);
|
||||
setConfiguredReadImageByteBudget(undefined);
|
||||
expect(resolveReadImageByteBudget()).toBe(READ_IMAGE_BYTE_BUDGET);
|
||||
});
|
||||
|
||||
it('lets the env var override the configured value', () => {
|
||||
setConfiguredReadImageByteBudget(512 * 1024);
|
||||
vi.stubEnv(READ_IMAGE_BYTE_BUDGET_ENV, '100000');
|
||||
expect(resolveReadImageByteBudget()).toBe(100000);
|
||||
});
|
||||
|
||||
it.each(['abc', '-1', '0', '1.5', ' '])('ignores the invalid env value "%s"', (raw) => {
|
||||
vi.stubEnv(READ_IMAGE_BYTE_BUDGET_ENV, raw);
|
||||
expect(resolveReadImageByteBudget()).toBe(READ_IMAGE_BYTE_BUDGET);
|
||||
});
|
||||
});
|
||||
|
||||
|
|
@ -546,7 +845,7 @@ describe('compressImageForModel — original dimensions metadata', () => {
|
|||
expect(shrunk.changed).toBe(true);
|
||||
expect(shrunk.originalWidth).toBe(4500);
|
||||
expect(shrunk.originalHeight).toBe(2250);
|
||||
expect(shrunk.width).toBe(3000);
|
||||
expect(shrunk.width).toBe(2000);
|
||||
});
|
||||
|
||||
it('reports original dimensions through the base64 wrapper', async () => {
|
||||
|
|
@ -556,8 +855,8 @@ describe('compressImageForModel — original dimensions metadata', () => {
|
|||
expect(result.changed).toBe(true);
|
||||
expect(result.originalWidth).toBe(3900);
|
||||
expect(result.originalHeight).toBe(1950);
|
||||
expect(result.width).toBe(3000);
|
||||
expect(result.height).toBe(1500);
|
||||
expect(result.width).toBe(2000);
|
||||
expect(result.height).toBe(1000);
|
||||
});
|
||||
});
|
||||
|
||||
|
|
@ -680,7 +979,7 @@ describe('cropImageForModel', () => {
|
|||
});
|
||||
expect(result.ok).toBe(false);
|
||||
if (result.ok) return;
|
||||
expect(result.error).toMatch(/PNG and JPEG/);
|
||||
expect(result.error).toMatch(/PNG, JPEG, and WebP/);
|
||||
});
|
||||
|
||||
it('rejects corrupt bytes without throwing', async () => {
|
||||
|
|
@ -1021,14 +1320,14 @@ describe('compressImageForModel — downscale quality guards', () => {
|
|||
});
|
||||
|
||||
it('keeps a degenerate aspect ratio at least 1px tall (no zero-size collapse)', async () => {
|
||||
// 9000×2 scaled to a 3000px edge would round the short side to 0.67px;
|
||||
// the resizer must clamp to 1, not produce an undecodable 3000×0 image.
|
||||
// 9000×2 scaled to a 2000px edge would round the short side to 0.44px;
|
||||
// the resizer must clamp to 1, not produce an undecodable 2000×0 image.
|
||||
const png = await solidPng(9000, 2);
|
||||
const result = await compressImageForModel(png, 'image/png');
|
||||
expect(result.changed).toBe(true);
|
||||
expect(result.width).toBe(3000);
|
||||
expect(result.width).toBe(2000);
|
||||
expect(result.height).toBe(1);
|
||||
expect(sniffImageDimensions(result.data)).toEqual({ width: 3000, height: 1 });
|
||||
expect(sniffImageDimensions(result.data)).toEqual({ width: 2000, height: 1 });
|
||||
});
|
||||
});
|
||||
|
||||
|
|
@ -1067,8 +1366,8 @@ describe('compressImageForModel — telemetry', () => {
|
|||
expect(props['final_bytes']).toBe(result.finalByteLength);
|
||||
expect(props['original_width']).toBe(4500);
|
||||
expect(props['original_height']).toBe(2250);
|
||||
expect(props['final_width']).toBe(3000);
|
||||
expect(props['final_height']).toBe(1500);
|
||||
expect(props['final_width']).toBe(2000);
|
||||
expect(props['final_height']).toBe(1000);
|
||||
expect(props['exif_transposed']).toBe(false);
|
||||
expect(typeof props['duration_ms']).toBe('number');
|
||||
});
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@
|
|||
import type { Kaos } from '@moonshot-ai/kaos';
|
||||
import type { ContentPart, ModelCapability } from '@moonshot-ai/kosong';
|
||||
import { Jimp } from 'jimp';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { ToolAccesses } from '../../src/loop';
|
||||
import type { ExecutableToolResult } from '../../src/loop';
|
||||
|
|
@ -13,9 +13,10 @@ import {
|
|||
ReadMediaFileInputSchema,
|
||||
ReadMediaFileTool,
|
||||
} from '../../src/tools/builtin/file/read-media';
|
||||
import { setConfiguredReadImageByteBudget } from '../../src/tools/support/image-compress';
|
||||
import { MEDIA_SNIFF_BYTES, sniffImageDimensions } from '../../src/tools/support/file-type';
|
||||
import type { TelemetryClient } from '../../src/telemetry';
|
||||
import { createFakeKaos, PERMISSIVE_WORKSPACE } from './fixtures/fake-kaos';
|
||||
import { createFakeKaos, FAKE_OS_ENV, PERMISSIVE_WORKSPACE } from './fixtures/fake-kaos';
|
||||
import { executeTool } from './fixtures/execute-tool';
|
||||
|
||||
const signal = new AbortController().signal;
|
||||
|
|
@ -711,7 +712,7 @@ describe('ReadMediaFileTool', () => {
|
|||
// The image actually sent to the model is downsampled to the edge cap.
|
||||
const sentBytes = Buffer.from(match![2]!, 'base64');
|
||||
const sentDims = sniffImageDimensions(sentBytes);
|
||||
expect(Math.max(sentDims!.width, sentDims!.height)).toBeLessThanOrEqual(3000);
|
||||
expect(Math.max(sentDims!.width, sentDims!.height)).toBeLessThanOrEqual(2000);
|
||||
|
||||
// The <system> note keeps the ORIGINAL size so coordinate mapping holds.
|
||||
const systemText = noteText(result);
|
||||
|
|
@ -746,7 +747,7 @@ describe('ReadMediaFileTool', () => {
|
|||
|
||||
const systemText = noteText(result);
|
||||
expect(systemText).toContain('Original dimensions: 1800x3600');
|
||||
expect(systemText).toMatch(/downsampled to 1500x3000/);
|
||||
expect(systemText).toMatch(/downsampled to 1000x2000/);
|
||||
});
|
||||
|
||||
it('reports the decoded size for a region read of an EXIF-rotated image', async () => {
|
||||
|
|
@ -893,7 +894,7 @@ describe('ReadMediaFileTool', () => {
|
|||
// Wording must not depend on serialization order: some providers keep
|
||||
// the note inline after the media, others flatten tool text and
|
||||
// re-attach the image after it — so no "above"/"below".
|
||||
expect(systemText).toMatch(/The attached image was downsampled to 3000x3000/);
|
||||
expect(systemText).toMatch(/The attached image was downsampled to 2000x2000/);
|
||||
expect(systemText).toMatch(/fine detail/i);
|
||||
expect(systemText).toContain('region');
|
||||
});
|
||||
|
|
@ -1014,4 +1015,182 @@ describe('ReadMediaFileTool', () => {
|
|||
expect(withFullRes.isError).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('provider-unsupported formats (HEIC/HEIF)', () => {
|
||||
/** Minimal ISO-BMFF header: size + 'ftyp' + the given brand. */
|
||||
function ftypHeader(brand: string): Buffer {
|
||||
const bytes = Buffer.alloc(16);
|
||||
bytes.writeUInt32BE(16, 0);
|
||||
bytes.write('ftyp', 4, 'latin1');
|
||||
bytes.write(brand, 8, 'latin1');
|
||||
return bytes;
|
||||
}
|
||||
|
||||
function heicTool(osKind: string, brand = 'heic'): ReadMediaFileTool {
|
||||
const data = ftypHeader(brand);
|
||||
const kaos = createFakeKaos({
|
||||
stat: vi.fn<Kaos['stat']>().mockResolvedValue({ ...DEFAULT_STAT, stSize: data.length }),
|
||||
readBytes: vi.fn<Kaos['readBytes']>().mockResolvedValue(data),
|
||||
osEnv: { ...FAKE_OS_ENV, osKind },
|
||||
});
|
||||
return new ReadMediaFileTool(kaos, PERMISSIVE_WORKSPACE, capabilities());
|
||||
}
|
||||
|
||||
it('refuses HEIC with sips guidance on macOS instead of sending it to the provider', async () => {
|
||||
const result = await executeTool(heicTool('macOS'), {
|
||||
turnId: 't1',
|
||||
toolCallId: 'c_heic_mac',
|
||||
args: { path: '/workspace/photo.HEIC' },
|
||||
signal,
|
||||
});
|
||||
expect(result.isError).toBe(true);
|
||||
expect(result.output).toContain('image/heic');
|
||||
expect(result.output).toContain('sips -s format jpeg');
|
||||
expect(result.output).toContain('/workspace/photo.jpg');
|
||||
});
|
||||
|
||||
it('refuses HEIC with heif-convert / ImageMagick guidance on Linux', async () => {
|
||||
const result = await executeTool(heicTool('Linux'), {
|
||||
turnId: 't1',
|
||||
toolCallId: 'c_heic_linux',
|
||||
args: { path: '/workspace/photo.HEIC' },
|
||||
signal,
|
||||
});
|
||||
expect(result.isError).toBe(true);
|
||||
expect(result.output).toContain('heif-convert');
|
||||
expect(result.output).toContain('magick');
|
||||
});
|
||||
|
||||
it('refuses HEIC with ImageMagick guidance on Windows', async () => {
|
||||
const result = await executeTool(heicTool('Windows'), {
|
||||
turnId: 't1',
|
||||
toolCallId: 'c_heic_win',
|
||||
args: { path: '/workspace/photo.HEIC' },
|
||||
signal,
|
||||
});
|
||||
expect(result.isError).toBe(true);
|
||||
expect(result.output).toContain('magick');
|
||||
expect(result.output).toContain('winget');
|
||||
});
|
||||
|
||||
it('refuses HEIF brands and region reads the same way', async () => {
|
||||
const heif = await executeTool(heicTool('macOS', 'mif1'), {
|
||||
turnId: 't1',
|
||||
toolCallId: 'c_heif',
|
||||
args: { path: '/workspace/photo.heif' },
|
||||
signal,
|
||||
});
|
||||
expect(heif.isError).toBe(true);
|
||||
expect(heif.output).toContain('image/heif');
|
||||
|
||||
const region = await executeTool(heicTool('macOS'), {
|
||||
turnId: 't1',
|
||||
toolCallId: 'c_heic_region',
|
||||
args: { path: '/workspace/photo.HEIC', region: { x: 0, y: 0, width: 10, height: 10 } },
|
||||
signal,
|
||||
});
|
||||
expect(region.isError).toBe(true);
|
||||
expect(region.output).toContain('sips');
|
||||
});
|
||||
});
|
||||
|
||||
describe('read byte budget', () => {
|
||||
afterEach(() => {
|
||||
setConfiguredReadImageByteBudget(undefined);
|
||||
});
|
||||
|
||||
/** High-entropy PNG whose bytes stay large after downscaling. */
|
||||
async function noisePng(width: number, height: number): Promise<Buffer> {
|
||||
const image = new Jimp({ width, height, color: 0x000000ff });
|
||||
const data = image.bitmap.data;
|
||||
let state = 0x9e3779b9;
|
||||
for (let i = 0; i < data.length; i += 4) {
|
||||
state ^= (state << 13) >>> 0;
|
||||
state ^= state >>> 17;
|
||||
state ^= (state << 5) >>> 0;
|
||||
state >>>= 0;
|
||||
data[i] = state & 0xff;
|
||||
data[i + 1] = (state >>> 8) & 0xff;
|
||||
data[i + 2] = (state >>> 16) & 0xff;
|
||||
data[i + 3] = 0xff;
|
||||
}
|
||||
return Buffer.from(await image.getBuffer('image/png'));
|
||||
}
|
||||
|
||||
function toolFor(data: Buffer): ReadMediaFileTool {
|
||||
return makeReadMediaTool({
|
||||
stat: vi.fn<Kaos['stat']>().mockResolvedValue({ ...DEFAULT_STAT, stSize: data.length }),
|
||||
readBytes: vi.fn<Kaos['readBytes']>().mockResolvedValue(data),
|
||||
});
|
||||
}
|
||||
|
||||
function sentBytes(result: Awaited<ReturnType<typeof executeTool>>): Buffer {
|
||||
const parts = outputParts(result);
|
||||
const url = (parts[1] as { imageUrl: { url: string } }).imageUrl.url;
|
||||
const match = /^data:image\/[a-z]+;base64,(.+)$/.exec(url);
|
||||
expect(match).not.toBeNull();
|
||||
return Buffer.from(match![1]!, 'base64');
|
||||
}
|
||||
|
||||
it(
|
||||
'compresses a default read to fit the configured read budget',
|
||||
async () => {
|
||||
const budget = 64 * 1024;
|
||||
setConfiguredReadImageByteBudget(budget);
|
||||
const data = await noisePng(1200, 1200);
|
||||
expect(data.length).toBeGreaterThan(budget);
|
||||
|
||||
const result = await executeTool(toolFor(data), {
|
||||
turnId: 't1',
|
||||
toolCallId: 'c_budget',
|
||||
args: { path: '/workspace/noisy.png' },
|
||||
signal,
|
||||
});
|
||||
|
||||
expect(result.isError).toBe(false);
|
||||
expect(sentBytes(result).length).toBeLessThanOrEqual(budget);
|
||||
expect(noteText(result)).toMatch(/downsampled/);
|
||||
},
|
||||
15_000,
|
||||
);
|
||||
|
||||
it('full_resolution ignores the read budget (per-image provider limit applies)', async () => {
|
||||
setConfiguredReadImageByteBudget(64 * 1024);
|
||||
const data = await noisePng(600, 600);
|
||||
expect(data.length).toBeGreaterThan(64 * 1024);
|
||||
|
||||
const result = await executeTool(toolFor(data), {
|
||||
turnId: 't1',
|
||||
toolCallId: 'c_fullres_budget',
|
||||
args: { path: '/workspace/noisy.png', full_resolution: true },
|
||||
signal,
|
||||
});
|
||||
|
||||
expect(result.isError).toBe(false);
|
||||
// Delivered byte-for-byte: the read budget must not shrink the
|
||||
// full-fidelity escape hatch the downsample note promises.
|
||||
expect(sentBytes(result).equals(data)).toBe(true);
|
||||
});
|
||||
|
||||
it(
|
||||
'region reads ignore the read budget so detail readback stays full-fidelity',
|
||||
async () => {
|
||||
setConfiguredReadImageByteBudget(16 * 1024);
|
||||
const data = await noisePng(800, 800);
|
||||
|
||||
const result = await executeTool(toolFor(data), {
|
||||
turnId: 't1',
|
||||
toolCallId: 'c_region_budget',
|
||||
args: { path: '/workspace/noisy.png', region: { x: 0, y: 0, width: 400, height: 400 } },
|
||||
signal,
|
||||
});
|
||||
|
||||
expect(result.isError).toBe(false);
|
||||
// A native-resolution noise crop is far larger than the read budget;
|
||||
// it must still be delivered under the provider-scale budget.
|
||||
expect(sentBytes(result).length).toBeGreaterThan(16 * 1024);
|
||||
},
|
||||
15_000,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -56,6 +56,19 @@ export class APIContextOverflowError extends APIStatusError {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* HTTP 413 that specifically means the serialized request body exceeded the
|
||||
* provider's byte ceiling (e.g. accumulated base64 images), as opposed to a
|
||||
* token-count overflow. Token overflow is recoverable by compaction; a body
|
||||
* size rejection is not — it needs media to be dropped or shrunk.
|
||||
*/
|
||||
export class APIRequestTooLargeError extends APIStatusError {
|
||||
constructor(statusCode: number, message: string, requestId?: string | null) {
|
||||
super(statusCode, message, requestId);
|
||||
this.name = 'APIRequestTooLargeError';
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* HTTP status error that specifically means the provider rate-limited the
|
||||
* request.
|
||||
|
|
@ -146,6 +159,29 @@ const PROVIDER_RATE_LIMIT_MESSAGE_PATTERNS = [
|
|||
/rate-limited/,
|
||||
] as const;
|
||||
|
||||
// Wordings that mean the serialized request BODY was too big, matched against
|
||||
// the lowercased message of a 413. Kept separate from the context-overflow
|
||||
// patterns above: those describe token counts, these describe bytes. A 413
|
||||
// whose message matches neither family stays a plain `APIStatusError` —
|
||||
// Vertex phrases prompt-too-long as a 413, so the status alone is not proof
|
||||
// of a body-size rejection.
|
||||
const REQUEST_TOO_LARGE_MESSAGE_PATTERNS = [
|
||||
// Moonshot / Kimi: "Request exceeds the maximum size".
|
||||
/request exceeds the maximum size/,
|
||||
// Reverse proxies (nginx-style HTML body): "413 Request Entity Too Large".
|
||||
/request entity too large/,
|
||||
// Anthropic: error type `request_too_large`, message "Request exceeds the
|
||||
// maximum allowed number of bytes".
|
||||
/request_too_large/,
|
||||
/exceeds? the maximum allowed number of bytes/,
|
||||
// RFC 9110 reason phrase (both the pre-2022 and current names).
|
||||
/payload too large/,
|
||||
/content too large/,
|
||||
// Plain wordings: generic gateways say "request too large"; Go's
|
||||
// http.MaxBytesReader (common in Go proxies) says "request body too large".
|
||||
/request (?:body )?too large/,
|
||||
] as const;
|
||||
|
||||
export function isContextOverflowErrorCode(code: string | null | undefined): boolean {
|
||||
return code === 'context_length_exceeded';
|
||||
}
|
||||
|
|
@ -158,9 +194,14 @@ export function normalizeAPIStatusError(
|
|||
if (statusCode === 429) {
|
||||
return new APIProviderRateLimitError(message, requestId);
|
||||
}
|
||||
// Context overflow first: Vertex returns prompt-too-long as a 413, and a
|
||||
// token overflow must keep routing to compaction even on that status.
|
||||
if (isContextOverflowStatusError(statusCode, message)) {
|
||||
return new APIContextOverflowError(statusCode, message, requestId);
|
||||
}
|
||||
if (isRequestTooLargeStatusError(statusCode, message)) {
|
||||
return new APIRequestTooLargeError(statusCode, message, requestId);
|
||||
}
|
||||
return new APIStatusError(statusCode, message, requestId);
|
||||
}
|
||||
|
||||
|
|
@ -170,6 +211,12 @@ export function isContextOverflowStatusError(statusCode: number, message: string
|
|||
return CONTEXT_OVERFLOW_MESSAGE_PATTERNS.some((pattern) => pattern.test(lowerMessage));
|
||||
}
|
||||
|
||||
export function isRequestTooLargeStatusError(statusCode: number, message: string): boolean {
|
||||
if (statusCode !== 413) return false;
|
||||
const lowerMessage = message.toLowerCase();
|
||||
return REQUEST_TOO_LARGE_MESSAGE_PATTERNS.some((pattern) => pattern.test(lowerMessage));
|
||||
}
|
||||
|
||||
// Strict providers reject a request whose assistant `tool_use`/`tool_calls` and
|
||||
// `tool_result`/`tool` blocks are not correctly paired and adjacent — a missing
|
||||
// result, a stray result with no matching call, or a result that does not
|
||||
|
|
|
|||
|
|
@ -63,12 +63,14 @@ export {
|
|||
APIContextOverflowError,
|
||||
APIEmptyResponseError,
|
||||
APIProviderRateLimitError,
|
||||
APIRequestTooLargeError,
|
||||
APIStatusError,
|
||||
APITimeoutError,
|
||||
ChatProviderError,
|
||||
isContextOverflowStatusError,
|
||||
isProviderRateLimitError,
|
||||
isRecoverableRequestStructureError,
|
||||
isRequestTooLargeStatusError,
|
||||
isRetryableGenerateError,
|
||||
isToolExchangeAdjacencyError,
|
||||
} from './errors';
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ import {
|
|||
APIContextOverflowError,
|
||||
APIEmptyResponseError,
|
||||
APIProviderRateLimitError,
|
||||
APIRequestTooLargeError,
|
||||
APIStatusError,
|
||||
APITimeoutError,
|
||||
ChatProviderError,
|
||||
|
|
@ -112,6 +113,23 @@ describe('APIProviderRateLimitError', () => {
|
|||
});
|
||||
});
|
||||
|
||||
describe('APIRequestTooLargeError', () => {
|
||||
it('extends APIStatusError and preserves HTTP details', () => {
|
||||
const err = new APIRequestTooLargeError(413, 'Request exceeds the maximum size.', 'req-large');
|
||||
expect(err).toBeInstanceOf(APIStatusError);
|
||||
expect(err).toBeInstanceOf(ChatProviderError);
|
||||
expect(err.name).toBe('APIRequestTooLargeError');
|
||||
expect(err.statusCode).toBe(413);
|
||||
expect(err.requestId).toBe('req-large');
|
||||
});
|
||||
|
||||
it('is not retryable', () => {
|
||||
expect(
|
||||
isRetryableGenerateError(new APIRequestTooLargeError(413, 'Request exceeds the maximum size.')),
|
||||
).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('isRetryableGenerateError', () => {
|
||||
it('matches transient provider errors and empty generate responses', () => {
|
||||
expect(isRetryableGenerateError(new APIConnectionError('conn'))).toBe(true);
|
||||
|
|
@ -209,6 +227,52 @@ describe('normalizeAPIStatusError', () => {
|
|||
expect(error).toBeInstanceOf(APIStatusError);
|
||||
expect(error).not.toBeInstanceOf(APIContextOverflowError);
|
||||
});
|
||||
|
||||
it.each([
|
||||
// Moonshot / Kimi 413 observed in the field when accumulated media pushed
|
||||
// the request body over the provider's byte ceiling.
|
||||
[413, 'Request exceeds the maximum size'],
|
||||
// Reverse-proxy (nginx-style) 413 with an HTML body.
|
||||
[413, '413 <html><head><title>413 Request Entity Too Large</title></head></html>'],
|
||||
// Anthropic request_too_large: body over the 32 MB API ceiling.
|
||||
[413, 'request_too_large: Request exceeds the maximum allowed number of bytes'],
|
||||
// RFC 9110 reason phrase / Node-style wording.
|
||||
[413, 'Payload Too Large'],
|
||||
[413, 'Content Too Large'],
|
||||
// Plain wordings without "entity": generic gateways say "Request too
|
||||
// large"; Go's http.MaxBytesReader says "http: request body too large".
|
||||
[413, 'Request too large'],
|
||||
[413, 'Request body too large'],
|
||||
[413, 'http: request body too large'],
|
||||
])('normalizes %i "%s" to APIRequestTooLargeError', (statusCode, message) => {
|
||||
const error = normalizeAPIStatusError(statusCode, message, 'req-large');
|
||||
expect(error).toBeInstanceOf(APIRequestTooLargeError);
|
||||
expect(error.statusCode).toBe(statusCode);
|
||||
expect(error.requestId).toBe('req-large');
|
||||
});
|
||||
|
||||
it('keeps a 413 with token-overflow wording as APIContextOverflowError', () => {
|
||||
// Vertex phrases prompt-too-long as a 413; that is a token problem
|
||||
// (recoverable by compaction), not a request-body-size problem.
|
||||
const error = normalizeAPIStatusError(413, 'prompt is too long: 210000 tokens > 200000 maximum');
|
||||
expect(error).toBeInstanceOf(APIContextOverflowError);
|
||||
expect(error).not.toBeInstanceOf(APIRequestTooLargeError);
|
||||
});
|
||||
|
||||
it.each([
|
||||
// A bare 413 with unrecognized wording stays unclassified: Vertex abuses
|
||||
// 413 for prompt-too-long, so the status alone is not proof of a
|
||||
// body-size rejection.
|
||||
[413, 'Request failed'],
|
||||
// Size wording without the 413 status is not classified either.
|
||||
[400, 'Payload too large'],
|
||||
[422, 'Request entity too large'],
|
||||
])('keeps %i "%s" as plain APIStatusError', (statusCode, message) => {
|
||||
const error = normalizeAPIStatusError(statusCode, message);
|
||||
expect(error).toBeInstanceOf(APIStatusError);
|
||||
expect(error).not.toBeInstanceOf(APIRequestTooLargeError);
|
||||
expect(error).not.toBeInstanceOf(APIContextOverflowError);
|
||||
});
|
||||
});
|
||||
|
||||
describe('isToolExchangeAdjacencyError', () => {
|
||||
|
|
|
|||
94
pnpm-lock.yaml
generated
94
pnpm-lock.yaml
generated
|
|
@ -59,7 +59,7 @@ importers:
|
|||
version: 2.13.1
|
||||
tsdown:
|
||||
specifier: 0.22.0
|
||||
version: 0.22.0(@arethetypeswrong/core@0.18.2)(publint@0.3.18)(tsx@4.21.0)(typescript@6.0.2)(unrun@0.2.34)(vue-tsc@3.2.9(typescript@6.0.2))
|
||||
version: 0.22.0(@arethetypeswrong/core@0.18.2)(publint@0.3.18)(tsx@4.21.0)(typescript@6.0.2)(unrun@0.2.34(@emnapi/core@1.10.0)(@emnapi/runtime@1.10.0))(vue-tsc@3.2.9(typescript@6.0.2))
|
||||
tsx:
|
||||
specifier: ^4.21.0
|
||||
version: 4.21.0
|
||||
|
|
@ -354,6 +354,9 @@ importers:
|
|||
'@antfu/utils':
|
||||
specifier: ^9.3.0
|
||||
version: 9.3.0
|
||||
'@jsquash/webp':
|
||||
specifier: ^1.5.0
|
||||
version: 1.5.0
|
||||
'@modelcontextprotocol/sdk':
|
||||
specifier: ^1.29.0
|
||||
version: 1.29.0(zod@4.3.6)
|
||||
|
|
@ -1791,6 +1794,9 @@ packages:
|
|||
'@jridgewell/trace-mapping@0.3.31':
|
||||
resolution: {integrity: sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==}
|
||||
|
||||
'@jsquash/webp@1.5.0':
|
||||
resolution: {integrity: sha512-KggLoj2MnRSfIqTeKe1EmbljTX2vuV7mh79k89PCL1pyqiDULcPM1L47twxXt0hkb68F70bXiL31MxsuoZtKFw==}
|
||||
|
||||
'@loaderkit/resolve@1.0.4':
|
||||
resolution: {integrity: sha512-rJzYKVcV4dxJv+vW6jlvagF8zvGxHJ2+HTr1e2qOejfmGhAApgJHl8Aog4mMszxceTRiKTTbnpgmTO1bEZHV/A==}
|
||||
|
||||
|
|
@ -7411,6 +7417,9 @@ packages:
|
|||
resolution: {integrity: sha512-o8qghlI8NZHU1lLPrpi2+Uq7abh4GGPpYANlalzWxyWteJOCsr/P+oPBA49TOLu5FTZO4d3F9MnWJfiMo4BkmA==}
|
||||
engines: {node: '>=18'}
|
||||
|
||||
wasm-feature-detect@1.8.0:
|
||||
resolution: {integrity: sha512-zksaLKM2fVlnB5jQQDqKXXwYHLQUVH9es+5TOOHwGOVJOCeRBCiPjwSg+3tN2AdTCzjgli4jijCH290kXb/zWQ==}
|
||||
|
||||
wcwidth@1.0.1:
|
||||
resolution: {integrity: sha512-XHPEwS0q6TaxcvG85+8EYkbiCux2XtWG2mkc47Ng2A77BQu9+DqIOJldST4HgPkuea7dvKSj5VgX3P1d4rW8Tg==}
|
||||
|
||||
|
|
@ -8842,6 +8851,10 @@ snapshots:
|
|||
'@jridgewell/resolve-uri': 3.1.2
|
||||
'@jridgewell/sourcemap-codec': 1.5.5
|
||||
|
||||
'@jsquash/webp@1.5.0':
|
||||
dependencies:
|
||||
wasm-feature-detect: 1.8.0
|
||||
|
||||
'@loaderkit/resolve@1.0.4':
|
||||
dependencies:
|
||||
'@braidai/lang': 1.1.2
|
||||
|
|
@ -8993,11 +9006,6 @@ snapshots:
|
|||
|
||||
'@mozilla/readability@0.6.0': {}
|
||||
|
||||
'@napi-rs/wasm-runtime@1.1.4':
|
||||
dependencies:
|
||||
'@tybys/wasm-util': 0.10.1
|
||||
optional: true
|
||||
|
||||
'@napi-rs/wasm-runtime@1.1.4(@emnapi/core@1.10.0)(@emnapi/runtime@1.10.0)':
|
||||
dependencies:
|
||||
'@emnapi/core': 1.10.0
|
||||
|
|
@ -9258,14 +9266,6 @@ snapshots:
|
|||
'@rolldown/binding-openharmony-arm64@1.0.1':
|
||||
optional: true
|
||||
|
||||
'@rolldown/binding-wasm32-wasi@1.0.0-rc.12':
|
||||
dependencies:
|
||||
'@napi-rs/wasm-runtime': 1.1.4
|
||||
transitivePeerDependencies:
|
||||
- '@emnapi/core'
|
||||
- '@emnapi/runtime'
|
||||
optional: true
|
||||
|
||||
'@rolldown/binding-wasm32-wasi@1.0.0-rc.12(@emnapi/core@1.10.0)(@emnapi/runtime@1.10.0)':
|
||||
dependencies:
|
||||
'@napi-rs/wasm-runtime': 1.1.4(@emnapi/core@1.10.0)(@emnapi/runtime@1.10.0)
|
||||
|
|
@ -9963,7 +9963,7 @@ snapshots:
|
|||
obug: 2.1.1
|
||||
std-env: 4.0.0
|
||||
tinyrainbow: 3.1.0
|
||||
vitest: 4.1.4(@types/node@22.19.17)(@vitest/coverage-v8@4.1.4)(jsdom@25.0.1)(vite@6.4.2(@types/node@22.19.17)(jiti@2.6.1)(lightningcss@1.32.0)(tsx@4.21.0)(yaml@2.8.3))
|
||||
vitest: 4.1.4(@types/node@22.19.17)(@vitest/coverage-v8@4.1.4)(jsdom@25.0.1)(vite@8.0.8(@types/node@22.19.17)(esbuild@0.27.7)(jiti@2.6.1)(tsx@4.21.0)(yaml@2.8.3))
|
||||
|
||||
'@vitest/expect@4.1.4':
|
||||
dependencies:
|
||||
|
|
@ -13777,31 +13777,6 @@ snapshots:
|
|||
transitivePeerDependencies:
|
||||
- oxc-resolver
|
||||
|
||||
rolldown@1.0.0-rc.12:
|
||||
dependencies:
|
||||
'@oxc-project/types': 0.122.0
|
||||
'@rolldown/pluginutils': 1.0.0-rc.12
|
||||
optionalDependencies:
|
||||
'@rolldown/binding-android-arm64': 1.0.0-rc.12
|
||||
'@rolldown/binding-darwin-arm64': 1.0.0-rc.12
|
||||
'@rolldown/binding-darwin-x64': 1.0.0-rc.12
|
||||
'@rolldown/binding-freebsd-x64': 1.0.0-rc.12
|
||||
'@rolldown/binding-linux-arm-gnueabihf': 1.0.0-rc.12
|
||||
'@rolldown/binding-linux-arm64-gnu': 1.0.0-rc.12
|
||||
'@rolldown/binding-linux-arm64-musl': 1.0.0-rc.12
|
||||
'@rolldown/binding-linux-ppc64-gnu': 1.0.0-rc.12
|
||||
'@rolldown/binding-linux-s390x-gnu': 1.0.0-rc.12
|
||||
'@rolldown/binding-linux-x64-gnu': 1.0.0-rc.12
|
||||
'@rolldown/binding-linux-x64-musl': 1.0.0-rc.12
|
||||
'@rolldown/binding-openharmony-arm64': 1.0.0-rc.12
|
||||
'@rolldown/binding-wasm32-wasi': 1.0.0-rc.12
|
||||
'@rolldown/binding-win32-arm64-msvc': 1.0.0-rc.12
|
||||
'@rolldown/binding-win32-x64-msvc': 1.0.0-rc.12
|
||||
transitivePeerDependencies:
|
||||
- '@emnapi/core'
|
||||
- '@emnapi/runtime'
|
||||
optional: true
|
||||
|
||||
rolldown@1.0.0-rc.12(@emnapi/core@1.10.0)(@emnapi/runtime@1.10.0):
|
||||
dependencies:
|
||||
'@oxc-project/types': 0.122.0
|
||||
|
|
@ -14553,35 +14528,6 @@ snapshots:
|
|||
- oxc-resolver
|
||||
- vue-tsc
|
||||
|
||||
tsdown@0.22.0(@arethetypeswrong/core@0.18.2)(publint@0.3.18)(tsx@4.21.0)(typescript@6.0.2)(unrun@0.2.34)(vue-tsc@3.2.9(typescript@6.0.2)):
|
||||
dependencies:
|
||||
ansis: 4.2.0
|
||||
cac: 7.0.0
|
||||
defu: 6.1.7
|
||||
empathic: 2.0.0
|
||||
hookable: 6.1.1
|
||||
import-without-cache: 0.4.0
|
||||
obug: 2.1.1
|
||||
picomatch: 4.0.4
|
||||
rolldown: 1.0.1
|
||||
rolldown-plugin-dts: 0.25.1(rolldown@1.0.1)(typescript@6.0.2)(vue-tsc@3.2.9(typescript@6.0.2))
|
||||
semver: 7.7.4
|
||||
tinyexec: 1.1.2
|
||||
tinyglobby: 0.2.16
|
||||
tree-kill: 1.2.2
|
||||
unconfig-core: 7.5.0
|
||||
optionalDependencies:
|
||||
'@arethetypeswrong/core': 0.18.2
|
||||
publint: 0.3.18
|
||||
tsx: 4.21.0
|
||||
typescript: 6.0.2
|
||||
unrun: 0.2.34
|
||||
transitivePeerDependencies:
|
||||
- '@ts-macro/tsc'
|
||||
- '@typescript/native-preview'
|
||||
- oxc-resolver
|
||||
- vue-tsc
|
||||
|
||||
tslib@2.8.1: {}
|
||||
|
||||
tsx@4.21.0:
|
||||
|
|
@ -14745,14 +14691,6 @@ snapshots:
|
|||
picomatch: 4.0.4
|
||||
webpack-virtual-modules: 0.6.2
|
||||
|
||||
unrun@0.2.34:
|
||||
dependencies:
|
||||
rolldown: 1.0.0-rc.12
|
||||
transitivePeerDependencies:
|
||||
- '@emnapi/core'
|
||||
- '@emnapi/runtime'
|
||||
optional: true
|
||||
|
||||
unrun@0.2.34(@emnapi/core@1.10.0)(@emnapi/runtime@1.10.0):
|
||||
dependencies:
|
||||
rolldown: 1.0.0-rc.12(@emnapi/core@1.10.0)(@emnapi/runtime@1.10.0)
|
||||
|
|
@ -15014,6 +14952,8 @@ snapshots:
|
|||
xml-name-validator: 5.0.0
|
||||
optional: true
|
||||
|
||||
wasm-feature-detect@1.8.0: {}
|
||||
|
||||
wcwidth@1.0.1:
|
||||
dependencies:
|
||||
defaults: 1.0.4
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue