mirror of
https://github.com/MoonshotAI/kimi-code.git
synced 2026-07-09 17:29:12 +00:00
feat(agent-core): compress oversized images before sending to the model (#1243)
* feat(agent-core): compress oversized images before sending to the model
Downsample images to a 2000px longest-edge and per-image byte budget at the
single prompt-ingestion chokepoint (the prompt/steer RPC) and on tool results
(ReadMediaFile, MCP), so every client transport — CLI, web, desktop, ACP, SDK —
is covered uniformly inside the core. PNG screenshots stay lossless and only
degrade to JPEG when the byte budget cannot otherwise be met. Best-effort: the
original image is sent unchanged if compression fails.
* fix(agent-core): serialize prompt/steer RPCs to avoid a turn-claim race
The prompt/steer RPC handlers await image compression before turn.launch()
synchronously claims the active turn, so two overlapping calls could both
compress first — letting the faster-to-compress one win the turn and strand the
other on agent_busy. Run these two RPCs through a per-agent serialization chain
so they claim in submit order; cancel and the other RPCs stay immediate.
* fix: update flake.nix pnpmDeps hash for the jimp dependency
Adding jimp to the workspace changed pnpm-lock.yaml, so the pnpmDeps
fixed-output hash was stale and the nix build failed. Update it to the value
the CI nix build reported.
* fix(agent-core): guard image compression against decompression bombs
A tiny-byte, huge-dimension image (e.g. a solid 30000x30000 PNG) would be fully
decoded into a multi-gigabyte bitmap by Jimp before any resize — an OOM vector
the byte budget never catches. Skip compression when the sniffed pixel count
exceeds MAX_DECODE_PIXELS (~100 MP), before the decode; oversized images pass
through uncompressed as they did before compression existed.
* fix(agent-core): cap decode byte size before compressing images
Compression runs before downstream size caps (e.g. the 10MB MCP per-part
limit), so a huge or invalid base64 image from an MCP tool was Buffer.from-
decoded — and handed to Jimp — just to be dropped afterward. Add a
MAX_DECODE_BYTES ceiling (64MB, overridable) checked before the base64 decode
and before Jimp, the byte-side complement to the pixel-count guard; oversized
payloads pass through uncompressed.
* refactor(agent-core): compress images at ingestion, not on the turn RPC
Move image compression off the prompt/steer RPC path and back to each ingestion
site (CLI paste, server upload resolution, ACP conversion; ReadMediaFile and MCP
already compressed at their producers). Compressing on the RPC control path put
an async step before the synchronous turn-claim, which spawned a series of
races: prompt/steer interleaving, and — with a cancel arriving mid-compression —
an ineffective abort that let a cancelled prompt launch anyway.
Treating compression as a pure input-stage transform (done while the content
part is built, before it ever enters the agent loop) removes those races
structurally: rpc.prompt/steer are plain synchronous handlers again, and the
serialization/cancel-window machinery is gone. Records stay compressed, resume
stays consistent, and coverage degrades gracefully (a new client that skips
compression just sends a larger image, as before this feature).
* fix: compress inline base64 prompts and honor ACP cancels mid-compression
Two contained ingestion-site follow-ups:
- server: resolvePromptMediaFiles now also compresses images submitted as an
inline `{ kind: 'base64' }` source, not just uploaded files, so the REST
inline-base64 path gets the same downsampling.
- acp-adapter: AcpSession tracks a pending-abort flag while prompt() awaits
image compression (before any turn exists). A session/cancel in that window
flips it, so the prompt returns `cancelled` instead of launching a turn the
client already stopped.
* fix(acp-adapter): cover all concurrent pre-turn prompts on cancel
The pending-abort marker was a single session field, so with two
`session/prompt` requests compressing large inline images at once the later
one overwrote it and a `session/cancel` could mark only one — the other
launched after the client had cancelled. Track a token per in-flight prompt in
a set and flip them all on cancel so every pre-turn prompt is covered.
* chore(node-sdk): declare jimp as a devDependency
The SDK re-exports the image compressor, whose lazy `import('jimp')` (inside
the bundled agent-core code) is inlined into the published dist. jimp was
resolved only transitively via agent-core, so declare it as an explicit build
input here — matching the CLI — to make the bundling reliable rather than
phantom. It stays a devDependency: jimp is bundled, not a runtime dependency.
This commit is contained in:
Kai
GitHub
parent
bf35f63c5d
commit
ace7901066
24 changed files with 1915 additions and 114 deletions
6
.changeset/image-compression.md
Normal file
6
.changeset/image-compression.md
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
"@moonshot-ai/kimi-code-sdk": minor
|
||||
"@moonshot-ai/kimi-code": minor
|
||||
---
|
||||
|
||||
Automatically compress oversized images before they reach the model. Whatever the source — pasted into the CLI, uploaded from the web/desktop client, sent over ACP, read via `ReadMediaFile`, or returned by an MCP tool — images are downsampled (longest edge ≤ 2000px) and re-encoded to fit a per-image byte budget, cutting vision-token cost and avoiding provider image-size errors. Screenshots stay lossless PNG and only degrade to JPEG when the byte budget cannot otherwise be met. Compression runs as an input-stage step at each ingestion point (while the content part is built), and guards against decompression bombs by skipping absurdly large pixel/byte payloads before decoding. Best-effort: if it fails for any reason the original image is sent unchanged.
|
||||
|
|
@ -94,6 +94,7 @@
|
|||
"chalk": "^5.4.1",
|
||||
"cli-highlight": "^2.1.11",
|
||||
"commander": "^13.1.0",
|
||||
"jimp": "^1.6.1",
|
||||
"pathe": "^2.0.3",
|
||||
"postject": "1.0.0-alpha.6",
|
||||
"semver": "^7.7.4",
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import type { Session } from '@moonshot-ai/kimi-code-sdk';
|
||||
import { compressImageForModel } from '@moonshot-ai/kimi-code-sdk';
|
||||
|
||||
import { ClipboardMediaError, readClipboardMedia } from '#/utils/clipboard/clipboard-image';
|
||||
import { parseImageMeta } from '#/utils/image/image-mime';
|
||||
|
|
@ -360,7 +361,19 @@ export class EditorKeyboardController {
|
|||
|
||||
const meta = parseImageMeta(media.bytes);
|
||||
if (meta === null) return false;
|
||||
const attachment = this.imageStore.addImage(media.bytes, meta.mime, meta.width, meta.height);
|
||||
// Compress at ingestion — a pure data step while building the attachment, so
|
||||
// the stored bytes, the inline thumbnail, the `[image #N (W×H)]` placeholder,
|
||||
// and the submitted image all agree, and the agent core only ever sees an
|
||||
// already-compressed image. Best effort: originals pass through on failure.
|
||||
const compressed = await compressImageForModel(media.bytes, meta.mime);
|
||||
const attachment = compressed.changed
|
||||
? this.imageStore.addImage(
|
||||
compressed.data,
|
||||
compressed.mimeType,
|
||||
compressed.width,
|
||||
compressed.height,
|
||||
)
|
||||
: this.imageStore.addImage(media.bytes, meta.mime, meta.width, meta.height);
|
||||
this.host.state.editor.insertTextAtCursor?.(`${attachment.placeholder} `);
|
||||
this.host.state.ui.requestRender();
|
||||
this.host.track('shortcut_paste', { kind: 'image' });
|
||||
|
|
|
|||
|
|
@ -0,0 +1,114 @@
|
|||
/**
|
||||
* Clipboard image paste → attachment store, with ingestion-time compression.
|
||||
*
|
||||
* Tests pin:
|
||||
* - an oversized pasted image is downsampled while building the attachment,
|
||||
* so the stored bytes, the `[image #N (W×H)]` placeholder, and the eventual
|
||||
* submitted image all agree on the compressed size
|
||||
* - a within-budget paste is stored byte-for-byte (fast path)
|
||||
*/
|
||||
|
||||
import { Jimp } from 'jimp';
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import {
|
||||
EditorKeyboardController,
|
||||
type EditorKeyboardHost,
|
||||
} from '#/tui/controllers/editor-keyboard';
|
||||
import { ImageAttachmentStore } from '#/tui/utils/image-attachment-store';
|
||||
import { parseImageMeta } from '#/utils/image/image-mime';
|
||||
|
||||
// vitest hoists vi.mock/vi.hoisted above the imports above, so the mock still
|
||||
// applies to the editor-keyboard module that pulls in readClipboardMedia.
|
||||
const { readClipboardMedia } = vi.hoisted(() => ({ readClipboardMedia: vi.fn() }));
|
||||
|
||||
vi.mock('#/utils/clipboard/clipboard-image', async (importActual) => {
|
||||
const actual = await importActual<typeof import('#/utils/clipboard/clipboard-image')>();
|
||||
return { ...actual, readClipboardMedia };
|
||||
});
|
||||
|
||||
interface PasteHarness {
|
||||
readonly store: ImageAttachmentStore;
|
||||
pasteImage(): Promise<void>;
|
||||
}
|
||||
|
||||
function createPasteHarness(): PasteHarness {
|
||||
const editor: Record<string, ((...args: never[]) => unknown) | undefined> = {};
|
||||
const store = new ImageAttachmentStore();
|
||||
const host = {
|
||||
state: {
|
||||
editor,
|
||||
activeDialog: null,
|
||||
appState: { streamingPhase: 'idle', isCompacting: false },
|
||||
footer: { setTransientHint: vi.fn() },
|
||||
ui: { requestRender: vi.fn() },
|
||||
},
|
||||
session: undefined,
|
||||
btwPanelController: { closeOrCancel: vi.fn(() => false) },
|
||||
track: vi.fn(),
|
||||
showError: vi.fn(),
|
||||
openUndoSelector: vi.fn(),
|
||||
cancelRunningShellCommand: vi.fn(),
|
||||
} as unknown as EditorKeyboardHost;
|
||||
|
||||
const controller = new EditorKeyboardController(host, store);
|
||||
controller.install();
|
||||
|
||||
return {
|
||||
store,
|
||||
async pasteImage() {
|
||||
const handler = editor['onPasteImage'];
|
||||
if (handler === undefined) throw new Error('onPasteImage handler not installed');
|
||||
await (handler as () => Promise<boolean>)();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function solidPng(width: number, height: number): Promise<Uint8Array> {
|
||||
return new Uint8Array(
|
||||
await new Jimp({ width, height, color: 0x3366ccff }).getBuffer('image/png'),
|
||||
);
|
||||
}
|
||||
|
||||
describe('clipboard image paste compression', () => {
|
||||
beforeEach(() => {
|
||||
readClipboardMedia.mockReset();
|
||||
});
|
||||
|
||||
it('downsamples an oversized pasted image before storing it', async () => {
|
||||
const big = await solidPng(2600, 2600);
|
||||
readClipboardMedia.mockResolvedValue({ kind: 'image', bytes: big, mimeType: 'image/png' });
|
||||
|
||||
const { store, pasteImage } = createPasteHarness();
|
||||
await pasteImage();
|
||||
|
||||
expect(store.size()).toBe(1);
|
||||
const att = store.get(1);
|
||||
expect(att?.kind).toBe('image');
|
||||
if (att?.kind !== 'image') throw new Error('expected image attachment');
|
||||
|
||||
// Stored metadata reflects the compressed size.
|
||||
expect(Math.max(att.width, att.height)).toBeLessThanOrEqual(2000);
|
||||
expect(att.placeholder).toContain('2000×2000');
|
||||
|
||||
// The stored bytes decode to the compressed dimensions — the thumbnail and
|
||||
// the submitted image both read from these bytes, so they cannot diverge.
|
||||
const dims = parseImageMeta(att.bytes);
|
||||
expect(dims).not.toBeNull();
|
||||
expect(Math.max(dims!.width, dims!.height)).toBeLessThanOrEqual(2000);
|
||||
});
|
||||
|
||||
it('stores a within-budget paste byte-for-byte', async () => {
|
||||
const small = await solidPng(80, 80);
|
||||
readClipboardMedia.mockResolvedValue({ kind: 'image', bytes: small, mimeType: 'image/png' });
|
||||
|
||||
const { store, pasteImage } = createPasteHarness();
|
||||
await pasteImage();
|
||||
|
||||
const att = store.get(1);
|
||||
if (att?.kind !== 'image') throw new Error('expected image attachment');
|
||||
expect(att.width).toBe(80);
|
||||
expect(att.height).toBe(80);
|
||||
expect(att.bytes).toBe(small); // identity: no re-encode on the fast path
|
||||
});
|
||||
});
|
||||
|
|
@ -152,7 +152,7 @@
|
|||
inherit (finalAttrs) pname version src pnpmWorkspaces;
|
||||
inherit pnpm;
|
||||
fetcherVersion = 3;
|
||||
hash = "sha256-oratz8x67ZEJGTiNy+s4XaKe0TtpRKh63aIqkV79vvM=";
|
||||
hash = "sha256-mqyi0VuPZwESZcdU5E8F3XUG99OH636knBfb8y6TQpw=";
|
||||
};
|
||||
|
||||
nativeBuildInputs = [
|
||||
|
|
|
|||
|
|
@ -44,5 +44,8 @@
|
|||
"@moonshot-ai/agent-core": "workspace:^",
|
||||
"@moonshot-ai/kaos": "workspace:^",
|
||||
"@moonshot-ai/kimi-code-sdk": "workspace:^"
|
||||
},
|
||||
"devDependencies": {
|
||||
"jimp": "^1.6.1"
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
import type { ContentBlock, ToolCallContent } from '@agentclientprotocol/sdk';
|
||||
import {
|
||||
log,
|
||||
compressBase64ForModel,
|
||||
type PromptPart,
|
||||
type ToolInputDisplay,
|
||||
type ToolResultEvent,
|
||||
|
|
@ -71,6 +72,41 @@ export function acpBlocksToPromptParts(
|
|||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Shrink oversized inline images in a prompt-part list — the ACP ingestion
|
||||
* point's input-stage compression, mirroring the CLI's paste-time and the
|
||||
* server's upload-time step. Best effort: a part that cannot be compressed is
|
||||
* passed through unchanged.
|
||||
*/
|
||||
export async function compressPromptImageParts(
|
||||
parts: readonly PromptPart[],
|
||||
): Promise<PromptPart[]> {
|
||||
const out: PromptPart[] = [];
|
||||
for (const part of parts) {
|
||||
if (part.type === 'image_url') {
|
||||
const parsed = parseImageDataUrl(part.imageUrl.url);
|
||||
if (parsed !== null) {
|
||||
const result = await compressBase64ForModel(parsed.base64, parsed.mimeType);
|
||||
if (result.changed) {
|
||||
out.push({
|
||||
type: 'image_url',
|
||||
imageUrl: { ...part.imageUrl, url: `data:${result.mimeType};base64,${result.base64}` },
|
||||
});
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
out.push(part);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function parseImageDataUrl(url: string): { mimeType: string; base64: string } | null {
|
||||
const match = /^data:([^;,]+);base64,(.*)$/s.exec(url);
|
||||
if (match === null) return null;
|
||||
return { mimeType: match[1]!, base64: match[2]! };
|
||||
}
|
||||
|
||||
/**
|
||||
* Minimum-viable XML-attribute escaping for prompt-embedded resource
|
||||
* wrappers. The output is consumed by an LLM, not parsed by a canonical
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@ import {
|
|||
type KimiErrorPayload,
|
||||
type KimiHarness,
|
||||
type McpServerInfo,
|
||||
type PromptPart,
|
||||
type QuestionAnswers,
|
||||
type QuestionRequest,
|
||||
type Session,
|
||||
|
|
@ -38,7 +39,7 @@ import {
|
|||
} from './builtin-commands';
|
||||
import { buildSessionConfigOptions } from './config-options';
|
||||
import { listModelsFromHarness } from './model-catalog';
|
||||
import { acpBlocksToPromptParts } from './convert';
|
||||
import { acpBlocksToPromptParts, compressPromptImageParts } from './convert';
|
||||
import {
|
||||
acpToolCallId,
|
||||
assistantDeltaToSessionUpdate,
|
||||
|
|
@ -147,6 +148,13 @@ export class AcpSession {
|
|||
*/
|
||||
private skillCommandMap: ReadonlyMap<string, string> = new Map();
|
||||
|
||||
// One token per in-flight `prompt()` that is still awaiting image compression
|
||||
// (before any turn exists). A `session/cancel` in that window has no turn to
|
||||
// abort, so it flips every token and each affected `prompt()` returns
|
||||
// `cancelled` instead of launching. A set (not a single field) so concurrent
|
||||
// prompts are all covered rather than only the most recent.
|
||||
private readonly pendingPromptAborts = new Set<{ aborted: boolean }>();
|
||||
|
||||
/**
|
||||
* The most recent command palette advertised to the ACP client. Used by
|
||||
* `/help` so the response matches the client's `available_commands_update`
|
||||
|
|
@ -268,6 +276,11 @@ export class AcpSession {
|
|||
* acceptable.
|
||||
*/
|
||||
async cancel(): Promise<void> {
|
||||
// If any prompt is mid-compression (no turn yet), mark them aborted so they
|
||||
// do not launch once compression finishes.
|
||||
for (const pending of this.pendingPromptAborts) {
|
||||
pending.aborted = true;
|
||||
}
|
||||
await this.session.cancel();
|
||||
}
|
||||
|
||||
|
|
@ -715,7 +728,20 @@ export class AcpSession {
|
|||
* sees a JSON-RPC error rather than a hung request.
|
||||
*/
|
||||
async prompt(blocks: readonly ContentBlock[]): Promise<PromptResponse> {
|
||||
const parts = acpBlocksToPromptParts(blocks);
|
||||
// Compression happens before any turn exists, so honor a `session/cancel`
|
||||
// that arrives during it: flip the flag from cancel() and bail out here
|
||||
// rather than launching a turn the client already asked to stop.
|
||||
const pending = { aborted: false };
|
||||
this.pendingPromptAborts.add(pending);
|
||||
let parts: readonly PromptPart[];
|
||||
try {
|
||||
parts = await compressPromptImageParts(acpBlocksToPromptParts(blocks));
|
||||
} finally {
|
||||
this.pendingPromptAborts.delete(pending);
|
||||
}
|
||||
if (pending.aborted) {
|
||||
return { stopReason: 'cancelled' };
|
||||
}
|
||||
const sessionId = this.id;
|
||||
const conn = this.conn;
|
||||
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ import {
|
|||
type WriteTextFileResponse,
|
||||
} from '@agentclientprotocol/sdk';
|
||||
import { log, type KimiHarness, type Session } from '@moonshot-ai/kimi-code-sdk';
|
||||
import { Jimp } from 'jimp';
|
||||
|
||||
import { AcpServer } from '../src/server';
|
||||
import { AUTHED_STATUS } from './_helpers/harness-stubs';
|
||||
|
|
@ -139,4 +140,81 @@ describe('AcpServer cancel', () => {
|
|||
expect.objectContaining({ sessionId: 'sess-erroring' }),
|
||||
);
|
||||
});
|
||||
|
||||
it('returns cancelled without launching when cancel arrives during image compression', async () => {
|
||||
let promptCalls = 0;
|
||||
const fakeSession = {
|
||||
id: 'sess-cancel-compress',
|
||||
prompt: async () => {
|
||||
promptCalls += 1;
|
||||
return undefined;
|
||||
},
|
||||
cancel: async () => undefined,
|
||||
onEvent: () => () => undefined,
|
||||
} as unknown as Session;
|
||||
const harness = {
|
||||
auth: { status: async () => AUTHED_STATUS },
|
||||
createSession: async () => fakeSession,
|
||||
} as unknown as KimiHarness;
|
||||
|
||||
const { agentStream, clientStream } = makeInMemoryStreamPair();
|
||||
new AgentSideConnection((c) => new AcpServer(harness, c), agentStream);
|
||||
const client = new ClientSideConnection((_a) => new StubClient(), clientStream);
|
||||
|
||||
const { sessionId } = await client.newSession({ cwd: '/tmp/x', mcpServers: [] });
|
||||
|
||||
// A solid 2600×2600 image is small in bytes but slow enough to compress
|
||||
// that the cancel below reliably lands mid-compression, before any turn.
|
||||
const data = Buffer.from(
|
||||
await new Jimp({ width: 2600, height: 2600, color: 0x3366ccff }).getBuffer('image/png'),
|
||||
).toString('base64');
|
||||
|
||||
const promptP = client.prompt({
|
||||
sessionId,
|
||||
prompt: [{ type: 'image', data, mimeType: 'image/png' }],
|
||||
});
|
||||
await client.cancel({ sessionId });
|
||||
const res = await promptP;
|
||||
|
||||
expect(res.stopReason).toBe('cancelled');
|
||||
expect(promptCalls).toBe(0); // the turn was never launched
|
||||
});
|
||||
|
||||
it('cancels every prompt compressing concurrently, not just the most recent', async () => {
|
||||
let promptCalls = 0;
|
||||
const fakeSession = {
|
||||
id: 'sess-cancel-concurrent',
|
||||
prompt: async () => {
|
||||
promptCalls += 1;
|
||||
return undefined;
|
||||
},
|
||||
cancel: async () => undefined,
|
||||
onEvent: () => () => undefined,
|
||||
} as unknown as Session;
|
||||
const harness = {
|
||||
auth: { status: async () => AUTHED_STATUS },
|
||||
createSession: async () => fakeSession,
|
||||
} as unknown as KimiHarness;
|
||||
|
||||
const { agentStream, clientStream } = makeInMemoryStreamPair();
|
||||
new AgentSideConnection((c) => new AcpServer(harness, c), agentStream);
|
||||
const client = new ClientSideConnection((_a) => new StubClient(), clientStream);
|
||||
|
||||
const { sessionId } = await client.newSession({ cwd: '/tmp/x', mcpServers: [] });
|
||||
|
||||
const data = Buffer.from(
|
||||
await new Jimp({ width: 2600, height: 2600, color: 0x3366ccff }).getBuffer('image/png'),
|
||||
).toString('base64');
|
||||
const imageBlock = { type: 'image' as const, data, mimeType: 'image/png' };
|
||||
|
||||
// Two prompts compressing at once; a single cancel must cover both.
|
||||
const p1 = client.prompt({ sessionId, prompt: [imageBlock] });
|
||||
const p2 = client.prompt({ sessionId, prompt: [imageBlock] });
|
||||
await client.cancel({ sessionId });
|
||||
const [r1, r2] = await Promise.all([p1, p2]);
|
||||
|
||||
expect(r1.stopReason).toBe('cancelled');
|
||||
expect(r2.stopReason).toBe('cancelled');
|
||||
expect(promptCalls).toBe(0);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -1,10 +1,15 @@
|
|||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import type { ContentBlock } from '@agentclientprotocol/sdk';
|
||||
import { Jimp } from 'jimp';
|
||||
|
||||
import { log, type ToolInputDisplay } from '@moonshot-ai/kimi-code-sdk';
|
||||
|
||||
import { acpBlocksToPromptParts, displayBlockToAcpContent } from '../src/convert';
|
||||
import {
|
||||
acpBlocksToPromptParts,
|
||||
compressPromptImageParts,
|
||||
displayBlockToAcpContent,
|
||||
} from '../src/convert';
|
||||
|
||||
const textBlock = (text: string): ContentBlock => ({ type: 'text', text });
|
||||
const imageBlock = (data: string, mimeType: string): ContentBlock => ({
|
||||
|
|
@ -320,3 +325,31 @@ describe('displayBlockToAcpContent — plan_review branch (Phase 13.2)', () => {
|
|||
expect(displayBlockToAcpContent(cmd)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('compressPromptImageParts', () => {
|
||||
async function pngBase64(width: number, height: number): Promise<string> {
|
||||
const buf = await new Jimp({ width, height, color: 0x3366ccff }).getBuffer('image/png');
|
||||
return Buffer.from(buf).toString('base64');
|
||||
}
|
||||
|
||||
it('downsamples an oversized inline image part', async () => {
|
||||
const parts = acpBlocksToPromptParts([imageBlock(await pngBase64(2600, 2600), 'image/png')]);
|
||||
const compressed = await compressPromptImageParts(parts);
|
||||
|
||||
const part = compressed[0];
|
||||
if (part?.type !== 'image_url') throw new Error('expected an image_url part');
|
||||
const match = /^data:(image\/[a-z]+);base64,(.+)$/.exec(part.imageUrl.url);
|
||||
expect(match).not.toBeNull();
|
||||
const decoded = await Jimp.fromBuffer(Buffer.from(match![2]!, 'base64'));
|
||||
expect(Math.max(decoded.width, decoded.height)).toBeLessThanOrEqual(2000);
|
||||
});
|
||||
|
||||
it('passes a within-budget image and text through unchanged', async () => {
|
||||
const parts = acpBlocksToPromptParts([
|
||||
imageBlock(await pngBase64(32, 32), 'image/png'),
|
||||
textBlock('hi'),
|
||||
]);
|
||||
const compressed = await compressPromptImageParts(parts);
|
||||
expect(compressed).toEqual(parts);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -69,6 +69,7 @@
|
|||
"ajv-formats": "^3.0.1",
|
||||
"chokidar": "^4.0.3",
|
||||
"ignore": "^5.3.2",
|
||||
"jimp": "^1.6.1",
|
||||
"js-yaml": "^4.1.1",
|
||||
"linkedom": "^0.18.12",
|
||||
"node-pty": "^1.1.0",
|
||||
|
|
|
|||
|
|
@ -44,6 +44,23 @@ export type {
|
|||
QuestionBackgroundTaskInfo,
|
||||
} from './agent/background';
|
||||
export type { ToolServices } from './tools/support/services';
|
||||
|
||||
// Image compression — the input-stage helper each ingestion site (CLI paste,
|
||||
// server upload resolution, ACP, ReadMediaFile, MCP) calls to shrink oversized
|
||||
// images while constructing the content part. Re-exported from the package root
|
||||
// so consumers (node-sdk, server) import it without a deep subpath.
|
||||
export {
|
||||
compressImageForModel,
|
||||
compressBase64ForModel,
|
||||
compressImageContentParts,
|
||||
IMAGE_BYTE_BUDGET,
|
||||
MAX_IMAGE_EDGE_PX,
|
||||
} from './tools/support/image-compress';
|
||||
export type {
|
||||
CompressImageOptions,
|
||||
CompressImageResult,
|
||||
CompressBase64Result,
|
||||
} from './tools/support/image-compress';
|
||||
export { SingleModelProvider } from './session/provider-manager';
|
||||
export type {
|
||||
BearerTokenProvider,
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@
|
|||
|
||||
import type { ContentPart } from '@moonshot-ai/kosong';
|
||||
|
||||
import { compressImageContentParts } from '../tools/support/image-compress';
|
||||
import type { MCPContentBlock, MCPToolResult } from './types';
|
||||
|
||||
// MCP servers can produce arbitrarily large outputs; cap what we feed back to
|
||||
|
|
@ -130,10 +131,10 @@ export function convertMCPContentBlock(block: MCPContentBlock): ContentPart | nu
|
|||
* `mcp__github__create_pr`) — embedded into the `<mcp_tool_result name="…">`
|
||||
* wrap when the result is media-only, so the model can attribute binary parts.
|
||||
*/
|
||||
export function mcpResultToExecutableOutput(
|
||||
export async function mcpResultToExecutableOutput(
|
||||
result: MCPToolResult,
|
||||
qualifiedToolName: string,
|
||||
): { output: string | ContentPart[]; isError: boolean; truncated?: true } {
|
||||
): Promise<{ output: string | ContentPart[]; isError: boolean; truncated?: true }> {
|
||||
const converted: ContentPart[] = [];
|
||||
for (const block of result.content) {
|
||||
const part = convertMCPContentBlock(block);
|
||||
|
|
@ -143,7 +144,11 @@ export function mcpResultToExecutableOutput(
|
|||
}
|
||||
|
||||
const wrapped = wrapMediaOnly(converted, qualifiedToolName);
|
||||
const limited = applyOutputLimits(wrapped);
|
||||
// Shrink oversized images BEFORE the per-part byte cap, so a large but
|
||||
// compressible screenshot is downsampled and kept rather than dropped to a
|
||||
// text notice. Best effort: parts that cannot be compressed pass through.
|
||||
const compressed = await compressImageContentParts(wrapped);
|
||||
const limited = applyOutputLimits(compressed);
|
||||
const output = collapseSingleText(limited.parts);
|
||||
return limited.truncated
|
||||
? { output, isError: result.isError, truncated: true }
|
||||
|
|
|
|||
|
|
@ -30,6 +30,7 @@ import type { ExecutableToolResult, ToolExecution } from '../../../loop/types';
|
|||
import { renderPrompt } from '../../../utils/render-prompt';
|
||||
import { resolvePathAccessPath } from '../../policies/path-access';
|
||||
import { MEDIA_SNIFF_BYTES, detectFileType, sniffImageDimensions } from '../../support/file-type';
|
||||
import { compressImageForModel } from '../../support/image-compress';
|
||||
import { toInputJsonSchema } from '../../support/input-schema';
|
||||
import { literalRulePattern, matchesPathRuleSubject } from '../../support/rule-match';
|
||||
import type { WorkspaceConfig } from '../../support/workspace';
|
||||
|
|
@ -222,12 +223,17 @@ export class ReadMediaFileTool implements BuiltinTool<ReadMediaFileInput> {
|
|||
}
|
||||
|
||||
const data = await this.kaos.readBytes(safePath);
|
||||
const base64 = data.toString('base64');
|
||||
let mediaPart: ContentPart;
|
||||
if (fileType.kind === 'image') {
|
||||
// Shrink oversized images so a large screenshot neither wastes context
|
||||
// tokens nor trips the provider's per-image byte ceiling. Best effort:
|
||||
// on any failure compressImageForModel returns the original bytes, so
|
||||
// the read still succeeds with the uncompressed image.
|
||||
const compressed = await compressImageForModel(data, fileType.mimeType);
|
||||
const base64 = Buffer.from(compressed.data).toString('base64');
|
||||
mediaPart = {
|
||||
type: 'image_url',
|
||||
imageUrl: { url: `data:${fileType.mimeType};base64,${base64}` },
|
||||
imageUrl: { url: `data:${compressed.mimeType};base64,${base64}` },
|
||||
};
|
||||
} else if (this.videoUploader !== undefined) {
|
||||
mediaPart = await this.videoUploader({
|
||||
|
|
@ -236,6 +242,7 @@ export class ReadMediaFileTool implements BuiltinTool<ReadMediaFileInput> {
|
|||
filename: safePath.split(/[\\/]/).at(-1),
|
||||
});
|
||||
} else {
|
||||
const base64 = data.toString('base64');
|
||||
mediaPart = {
|
||||
type: 'video_url',
|
||||
videoUrl: { url: `data:${fileType.mimeType};base64,${base64}` },
|
||||
|
|
@ -246,6 +253,10 @@ export class ReadMediaFileTool implements BuiltinTool<ReadMediaFileInput> {
|
|||
const openText = `<${tag} path="${safePath}">`;
|
||||
const closeText = `</${tag}>`;
|
||||
|
||||
// The summary always reports the ORIGINAL pixel size and byte size: the
|
||||
// model derives relative coordinates and scales them by the original
|
||||
// dimensions, so it must see the pre-compression size even when the
|
||||
// image_url above carries a downsampled copy.
|
||||
const dimensions =
|
||||
fileType.kind === 'image' ? sniffImageDimensions(data) : null;
|
||||
const systemText = buildSystemSummary({
|
||||
|
|
|
|||
371
packages/agent-core/src/tools/support/image-compress.ts
Normal file
371
packages/agent-core/src/tools/support/image-compress.ts
Normal file
|
|
@ -0,0 +1,371 @@
|
|||
/**
|
||||
* Shrink oversized images before they reach the model.
|
||||
*
|
||||
* A multimodal request carries each image as a base64 data URL; an unbounded
|
||||
* screenshot or photo wastes context tokens and can blow past the provider's
|
||||
* per-image byte ceiling. This module downsamples and re-encodes such images
|
||||
* so they fit a pixel + byte budget, while leaving already-small images
|
||||
* untouched — the common case is a fast, codec-free pass-through.
|
||||
*
|
||||
* Design notes:
|
||||
* - Pure JS (jimp), imported lazily so the codec is only paid for when an
|
||||
* image actually needs work; startup and the fast path stay cheap.
|
||||
* - Best effort: any decode/encode failure returns the original bytes
|
||||
* unchanged (`changed: false`), so a compression problem never blocks a
|
||||
* prompt. Callers simply send the original instead.
|
||||
* - Only PNG and JPEG are re-encoded. GIF is passed through to preserve
|
||||
* animation; WebP is passed through because the default jimp build ships no
|
||||
* WebP codec. Unknown formats are passed through.
|
||||
*/
|
||||
|
||||
import type { ContentPart } from '@moonshot-ai/kosong';
|
||||
|
||||
import { sniffImageDimensions } from './file-type';
|
||||
|
||||
/** Longest-edge ceiling (px). Larger images are scaled down to fit. */
|
||||
export const MAX_IMAGE_EDGE_PX = 2000;
|
||||
|
||||
/**
|
||||
* Raw-byte budget for a single image. base64 inflates bytes by ~4/3, so a
|
||||
* 3.75 MB raw payload stays under a 5 MB encoded ceiling. Tune to the active
|
||||
* provider's per-image limit.
|
||||
*/
|
||||
export const IMAGE_BYTE_BUDGET = 3.75 * 1024 * 1024;
|
||||
|
||||
/** Progressively lower JPEG quality until the payload fits the byte budget. */
|
||||
const JPEG_QUALITY_STEPS = [80, 60, 40, 20] as const;
|
||||
|
||||
/** Last-ditch longest edge when the budget cannot be met at MAX_IMAGE_EDGE_PX. */
|
||||
const FALLBACK_EDGE_PX = 1000;
|
||||
|
||||
/**
|
||||
* Pixel-count ceiling above which we skip compression entirely. A tiny-byte,
|
||||
* huge-dimension image (e.g. a solid 30000×30000 PNG) would otherwise be fully
|
||||
* decoded into a multi-gigabyte bitmap by Jimp before any resize — a
|
||||
* decompression-bomb OOM vector, since the byte budget alone never catches it.
|
||||
* The header sniff gives us the dimensions without decoding, so we gate on them
|
||||
* first. Set well above any legitimate photo/screenshot/scan (~100 MP); larger
|
||||
* images pass through uncompressed, exactly as they did before compression
|
||||
* existed.
|
||||
*/
|
||||
const MAX_DECODE_PIXELS = 100_000_000;
|
||||
|
||||
/**
|
||||
* Raw-byte ceiling above which compression is skipped rather than decoded. The
|
||||
* byte budget bounds the *output*, but the compressor still has to load the
|
||||
* *input* first: a huge base64 payload (e.g. an oversized or invalid image from
|
||||
* an MCP tool) would be `Buffer.from`-decoded — and possibly handed to Jimp —
|
||||
* before any downstream cap (like the 10 MB MCP per-part limit) can drop it.
|
||||
* This bounds that input allocation. Set well above legitimate
|
||||
* screenshots/photos; larger images pass through uncompressed.
|
||||
*/
|
||||
const MAX_DECODE_BYTES = 64 * 1024 * 1024;
|
||||
|
||||
/** Formats we can both decode and re-encode with the default jimp build. */
|
||||
const RECODABLE_MIME = new Set(['image/png', 'image/jpeg']);
|
||||
|
||||
export interface CompressImageOptions {
|
||||
/** Override the longest-edge ceiling (px). */
|
||||
readonly maxEdge?: number;
|
||||
/** Override the raw-byte budget. */
|
||||
readonly byteBudget?: number;
|
||||
/** Override the raw-byte ceiling above which compression is skipped. */
|
||||
readonly maxDecodeBytes?: number;
|
||||
}
|
||||
|
||||
export interface CompressImageResult {
|
||||
/** Bytes to send: the re-encoded image, or the original when unchanged. */
|
||||
readonly data: Uint8Array;
|
||||
/** MIME of `data`. May differ from the input (e.g. png → jpeg). */
|
||||
readonly mimeType: string;
|
||||
/** Pixel width of `data`; falls back to the input size when unknown. */
|
||||
readonly width: number;
|
||||
/** Pixel height of `data`; falls back to the input size when unknown. */
|
||||
readonly height: number;
|
||||
/** True only when `data` differs from the input bytes. */
|
||||
readonly changed: boolean;
|
||||
readonly originalByteLength: number;
|
||||
readonly finalByteLength: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Downsample/re-encode `bytes` to fit the pixel + byte budget.
|
||||
*
|
||||
* Never throws: on any failure (unsupported format, decode error, a result
|
||||
* that would be larger than the input) the original bytes are returned with
|
||||
* `changed: false`.
|
||||
*/
|
||||
export async function compressImageForModel(
|
||||
bytes: Uint8Array,
|
||||
mimeType: string,
|
||||
options: CompressImageOptions = {},
|
||||
): Promise<CompressImageResult> {
|
||||
const maxEdge = options.maxEdge ?? MAX_IMAGE_EDGE_PX;
|
||||
const byteBudget = options.byteBudget ?? IMAGE_BYTE_BUDGET;
|
||||
const maxDecodeBytes = options.maxDecodeBytes ?? MAX_DECODE_BYTES;
|
||||
const normalizedMime = normalizeMime(mimeType);
|
||||
const dims = sniffImageDimensions(bytes);
|
||||
|
||||
const passthrough = (): CompressImageResult => ({
|
||||
data: bytes,
|
||||
mimeType,
|
||||
width: dims?.width ?? 0,
|
||||
height: dims?.height ?? 0,
|
||||
changed: false,
|
||||
originalByteLength: bytes.length,
|
||||
finalByteLength: bytes.length,
|
||||
});
|
||||
|
||||
if (bytes.length === 0) return passthrough();
|
||||
// Only re-encode formats the codec handles; everything else passes through.
|
||||
if (!RECODABLE_MIME.has(normalizedMime)) return passthrough();
|
||||
|
||||
// Fast path: already within both budgets — no codec load, no allocation.
|
||||
const longestEdge = dims ? Math.max(dims.width, dims.height) : 0;
|
||||
const withinBytes = bytes.length <= byteBudget;
|
||||
const withinEdge = longestEdge > 0 && longestEdge <= maxEdge;
|
||||
if (withinBytes && (withinEdge || longestEdge === 0)) return passthrough();
|
||||
|
||||
// Decompression-bomb guard: refuse to decode absurd pixel counts. The sniff
|
||||
// above gave us the dimensions without decoding, so this costs nothing.
|
||||
if (dims && dims.width * dims.height > MAX_DECODE_PIXELS) return passthrough();
|
||||
// Refuse to decode very large byte payloads (e.g. a huge or invalid image
|
||||
// from an MCP tool) that would be loaded just to be dropped downstream.
|
||||
if (bytes.length > maxDecodeBytes) return passthrough();
|
||||
|
||||
try {
|
||||
const { Jimp } = await import('jimp');
|
||||
const image = await Jimp.fromBuffer(Buffer.from(bytes));
|
||||
const sourceIsPng = normalizedMime === 'image/png';
|
||||
|
||||
// Scale so the longest edge fits maxEdge (never enlarges).
|
||||
fitWithinEdge(image, maxEdge);
|
||||
|
||||
const encoded = await encodeWithinBudget(image, {
|
||||
sourceIsPng,
|
||||
byteBudget,
|
||||
fallbackEdge: FALLBACK_EDGE_PX,
|
||||
});
|
||||
|
||||
// Keep the result when it actually helps: fewer bytes, or fewer pixels
|
||||
// (a smaller image costs fewer vision tokens even if the byte count is
|
||||
// flat, as with near-solid graphics). Otherwise the re-encode bought us
|
||||
// nothing — send the original.
|
||||
const originalPixels = (dims?.width ?? 0) * (dims?.height ?? 0);
|
||||
const finalPixels = encoded.width * encoded.height;
|
||||
const shrankBytes = encoded.data.length < bytes.length;
|
||||
const shrankPixels = originalPixels > 0 && finalPixels < originalPixels;
|
||||
if (!shrankBytes && !shrankPixels) return passthrough();
|
||||
|
||||
return {
|
||||
data: encoded.data,
|
||||
mimeType: encoded.mimeType,
|
||||
width: encoded.width,
|
||||
height: encoded.height,
|
||||
changed: true,
|
||||
originalByteLength: bytes.length,
|
||||
finalByteLength: encoded.data.length,
|
||||
};
|
||||
} catch {
|
||||
// Decode/encode failure — keep the original bytes.
|
||||
return passthrough();
|
||||
}
|
||||
}
|
||||
|
||||
export interface CompressBase64Result {
|
||||
readonly base64: string;
|
||||
readonly mimeType: string;
|
||||
readonly changed: boolean;
|
||||
readonly originalByteLength: number;
|
||||
readonly finalByteLength: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience wrapper for call sites that already hold base64 (MCP results,
|
||||
* data URLs). Decodes, compresses, and re-encodes to base64. Best effort:
|
||||
* returns the original base64 unchanged on any failure.
|
||||
*/
|
||||
export async function compressBase64ForModel(
|
||||
base64: string,
|
||||
mimeType: string,
|
||||
options: CompressImageOptions = {},
|
||||
): Promise<CompressBase64Result> {
|
||||
// Skip very large payloads before allocating: base64 decodes to ~3/4 its
|
||||
// length, so a payload whose decoded size would exceed the cap is passed
|
||||
// through without the Buffer.from allocation (and without touching Jimp).
|
||||
const maxDecodeBytes = options.maxDecodeBytes ?? MAX_DECODE_BYTES;
|
||||
const approxBytes = Math.floor((base64.length * 3) / 4);
|
||||
if (approxBytes > maxDecodeBytes) {
|
||||
return {
|
||||
base64,
|
||||
mimeType,
|
||||
changed: false,
|
||||
originalByteLength: approxBytes,
|
||||
finalByteLength: approxBytes,
|
||||
};
|
||||
}
|
||||
let bytes: Buffer;
|
||||
try {
|
||||
bytes = Buffer.from(base64, 'base64');
|
||||
} catch {
|
||||
return {
|
||||
base64,
|
||||
mimeType,
|
||||
changed: false,
|
||||
originalByteLength: 0,
|
||||
finalByteLength: 0,
|
||||
};
|
||||
}
|
||||
const result = await compressImageForModel(bytes, mimeType, options);
|
||||
if (!result.changed) {
|
||||
return {
|
||||
base64,
|
||||
mimeType,
|
||||
changed: false,
|
||||
originalByteLength: result.originalByteLength,
|
||||
finalByteLength: result.finalByteLength,
|
||||
};
|
||||
}
|
||||
return {
|
||||
base64: Buffer.from(result.data).toString('base64'),
|
||||
mimeType: result.mimeType,
|
||||
changed: true,
|
||||
originalByteLength: result.originalByteLength,
|
||||
finalByteLength: result.finalByteLength,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Compress any inline base64 image parts in a content-part list — the single
|
||||
* helper used by the prompt-ingestion chokepoint (every client's images) and
|
||||
* the MCP tool-result path. Image parts whose URL is not a `data:` URL (e.g. a
|
||||
* remote http(s) image) are passed through, as are non-image parts. Best
|
||||
* effort: a part that fails to compress is left unchanged.
|
||||
*/
|
||||
export async function compressImageContentParts(
|
||||
parts: readonly ContentPart[],
|
||||
options: CompressImageOptions = {},
|
||||
): Promise<ContentPart[]> {
|
||||
const out: ContentPart[] = [];
|
||||
for (const part of parts) {
|
||||
if (part.type === 'image_url') {
|
||||
const parsed = parseImageDataUrl(part.imageUrl.url);
|
||||
if (parsed !== null) {
|
||||
const result = await compressBase64ForModel(parsed.base64, parsed.mimeType, options);
|
||||
if (result.changed) {
|
||||
out.push({
|
||||
type: 'image_url',
|
||||
imageUrl: { ...part.imageUrl, url: `data:${result.mimeType};base64,${result.base64}` },
|
||||
});
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
out.push(part);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function parseImageDataUrl(url: string): { mimeType: string; base64: string } | null {
|
||||
const match = /^data:([^;,]+);base64,(.*)$/s.exec(url);
|
||||
if (match === null) return null;
|
||||
return { mimeType: match[1]!, base64: match[2]! };
|
||||
}
|
||||
|
||||
// ── internals ────────────────────────────────────────────────────────
|
||||
|
||||
/** The concrete jimp image instance type, derived from the lazily-loaded module. */
|
||||
type JimpImage = Awaited<ReturnType<(typeof import('jimp'))['Jimp']['fromBuffer']>>;
|
||||
|
||||
interface EncodedImage {
|
||||
readonly data: Buffer;
|
||||
readonly mimeType: string;
|
||||
readonly width: number;
|
||||
readonly height: number;
|
||||
}
|
||||
|
||||
interface EncodeOptions {
|
||||
readonly sourceIsPng: boolean;
|
||||
readonly byteBudget: number;
|
||||
readonly fallbackEdge: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Encode `image` (already fitted to the edge ceiling) under the byte budget.
|
||||
*
|
||||
* Strategy — prefer the source format so a downscaled screenshot stays lossless
|
||||
* PNG (preserving text and transparency), and only fall back to lossy JPEG when
|
||||
* PNG cannot meet the byte budget:
|
||||
* - PNG source: PNG at the fitted size → a smaller PNG rescale → JPEG ladder.
|
||||
* - JPEG source: JPEG quality ladder → a smaller JPEG rescale.
|
||||
*
|
||||
* Always returns the smallest buffer it produced, even if no attempt met the
|
||||
* budget — the caller still gates on whether it actually helped.
|
||||
*/
|
||||
async function encodeWithinBudget(image: JimpImage, opts: EncodeOptions): Promise<EncodedImage> {
|
||||
const { sourceIsPng, byteBudget, fallbackEdge } = opts;
|
||||
let smallest: EncodedImage | null = null;
|
||||
|
||||
const consider = (data: Buffer, mimeType: string): EncodedImage => {
|
||||
const candidate: EncodedImage = { data, mimeType, width: image.width, height: image.height };
|
||||
if (smallest === null || candidate.data.length < smallest.data.length) {
|
||||
smallest = candidate;
|
||||
}
|
||||
return candidate;
|
||||
};
|
||||
|
||||
if (sourceIsPng) {
|
||||
// Lossless PNG first: best for screenshots/UI (sharp text) and keeps alpha.
|
||||
const png = await image.getBuffer('image/png', { deflateLevel: 9 });
|
||||
if (png.length <= byteBudget) return consider(png, 'image/png');
|
||||
consider(png, 'image/png');
|
||||
|
||||
// Over budget: a smaller PNG before going lossy.
|
||||
if (fitWithinEdge(image, fallbackEdge)) {
|
||||
const smallerPng = await image.getBuffer('image/png', { deflateLevel: 9 });
|
||||
if (smallerPng.length <= byteBudget) return consider(smallerPng, 'image/png');
|
||||
consider(smallerPng, 'image/png');
|
||||
}
|
||||
|
||||
// Last resort: lossy JPEG ladder (drops transparency) to meet the budget.
|
||||
for (const quality of JPEG_QUALITY_STEPS) {
|
||||
const jpeg = await image.getBuffer('image/jpeg', { quality });
|
||||
if (jpeg.length <= byteBudget) return consider(jpeg, 'image/jpeg');
|
||||
consider(jpeg, 'image/jpeg');
|
||||
}
|
||||
return smallest!;
|
||||
}
|
||||
|
||||
// JPEG source: quality ladder, then a smaller rescale at the lowest quality.
|
||||
for (const quality of JPEG_QUALITY_STEPS) {
|
||||
const jpeg = await image.getBuffer('image/jpeg', { quality });
|
||||
if (jpeg.length <= byteBudget) return consider(jpeg, 'image/jpeg');
|
||||
consider(jpeg, 'image/jpeg');
|
||||
}
|
||||
if (fitWithinEdge(image, fallbackEdge)) {
|
||||
const jpeg = await image.getBuffer('image/jpeg', { quality: JPEG_QUALITY_STEPS.at(-1) });
|
||||
consider(jpeg, 'image/jpeg');
|
||||
}
|
||||
|
||||
return smallest!;
|
||||
}
|
||||
|
||||
/**
|
||||
* Scale `image` so its longest edge is at most `edge`, preserving aspect
|
||||
* ratio. No-op (returns false) when the image already fits.
|
||||
*/
|
||||
function fitWithinEdge(image: JimpImage, edge: number): boolean {
|
||||
const longest = Math.max(image.width, image.height);
|
||||
if (longest <= edge) return false;
|
||||
const factor = edge / longest;
|
||||
image.resize({
|
||||
w: Math.max(1, Math.round(image.width * factor)),
|
||||
h: Math.max(1, Math.round(image.height * factor)),
|
||||
});
|
||||
return true;
|
||||
}
|
||||
|
||||
function normalizeMime(mimeType: string): string {
|
||||
const lower = mimeType.trim().toLowerCase();
|
||||
return lower === 'image/jpg' ? 'image/jpeg' : lower;
|
||||
}
|
||||
|
|
@ -1,9 +1,11 @@
|
|||
import { ContentBlockSchema } from '@modelcontextprotocol/sdk/types.js';
|
||||
import type { ContentPart } from '@moonshot-ai/kosong';
|
||||
import { Jimp } from 'jimp';
|
||||
import { describe, expect, test } from 'vitest';
|
||||
|
||||
import { convertMCPContentBlock, mcpResultToExecutableOutput } from '../../src/mcp/output';
|
||||
import type { MCPContentBlock, MCPToolResult } from '../../src/mcp/types';
|
||||
import { sniffImageDimensions } from '../../src/tools/support/file-type';
|
||||
|
||||
const MCP_OUTPUT_TRUNCATED_TEXT =
|
||||
'\n\n[Output truncated: exceeded 100000 character limit. ' +
|
||||
|
|
@ -205,29 +207,32 @@ describe('mcpResultToExecutableOutput', () => {
|
|||
return { content, isError };
|
||||
}
|
||||
|
||||
test('collapses a single text part into a plain string', () => {
|
||||
const out = mcpResultToExecutableOutput(result([{ type: 'text', text: 'hello' }]), 'mcp__s__t');
|
||||
test('collapses a single text part into a plain string', async () => {
|
||||
const out = await mcpResultToExecutableOutput(
|
||||
result([{ type: 'text', text: 'hello' }]),
|
||||
'mcp__s__t',
|
||||
);
|
||||
expect(out).toEqual({ output: 'hello', isError: false });
|
||||
});
|
||||
|
||||
test('propagates isError=true on the success-shape return', () => {
|
||||
const out = mcpResultToExecutableOutput(
|
||||
test('propagates isError=true on the success-shape return', async () => {
|
||||
const out = await mcpResultToExecutableOutput(
|
||||
result([{ type: 'text', text: 'oops' }], true),
|
||||
'mcp__s__t',
|
||||
);
|
||||
expect(out).toEqual({ output: 'oops', isError: true });
|
||||
});
|
||||
|
||||
test('returns an empty string when the content array is empty', () => {
|
||||
const out = mcpResultToExecutableOutput(result([]), 'mcp__s__t');
|
||||
test('returns an empty string when the content array is empty', async () => {
|
||||
const out = await mcpResultToExecutableOutput(result([]), 'mcp__s__t');
|
||||
// No parts survive; collapseSingleText has nothing to collapse so the
|
||||
// ContentPart[] branch wins. An empty array is the model-visible signal
|
||||
// that the tool returned no content.
|
||||
expect(out).toEqual({ output: [], isError: false });
|
||||
});
|
||||
|
||||
test('drops unconvertible blocks and keeps the rest', () => {
|
||||
const out = mcpResultToExecutableOutput(
|
||||
test('drops unconvertible blocks and keeps the rest', async () => {
|
||||
const out = await mcpResultToExecutableOutput(
|
||||
result([
|
||||
{ type: 'text', text: 'kept' },
|
||||
{ type: 'fancy_new_type', text: 'dropped' },
|
||||
|
|
@ -237,8 +242,8 @@ describe('mcpResultToExecutableOutput', () => {
|
|||
expect(out).toEqual({ output: 'kept', isError: false });
|
||||
});
|
||||
|
||||
test('wraps media-only output in mcp_tool_result tags using the qualified name', () => {
|
||||
const out = mcpResultToExecutableOutput(
|
||||
test('wraps media-only output in mcp_tool_result tags using the qualified name', async () => {
|
||||
const out = await mcpResultToExecutableOutput(
|
||||
result([{ type: 'image', data: 'AAA', mimeType: 'image/png' }]),
|
||||
'mcp__github__create_pr',
|
||||
);
|
||||
|
|
@ -250,8 +255,8 @@ describe('mcpResultToExecutableOutput', () => {
|
|||
]);
|
||||
});
|
||||
|
||||
test('does NOT wrap when a non-empty text part accompanies the media', () => {
|
||||
const out = mcpResultToExecutableOutput(
|
||||
test('does NOT wrap when a non-empty text part accompanies the media', async () => {
|
||||
const out = await mcpResultToExecutableOutput(
|
||||
result([
|
||||
{ type: 'text', text: 'caption' },
|
||||
{ type: 'image', data: 'AAA', mimeType: 'image/png' },
|
||||
|
|
@ -264,8 +269,8 @@ describe('mcpResultToExecutableOutput', () => {
|
|||
]);
|
||||
});
|
||||
|
||||
test('an empty-text companion still triggers the wrap', () => {
|
||||
const out = mcpResultToExecutableOutput(
|
||||
test('an empty-text companion still triggers the wrap', async () => {
|
||||
const out = await mcpResultToExecutableOutput(
|
||||
result([
|
||||
{ type: 'text', text: '' },
|
||||
{ type: 'image', data: 'AAA', mimeType: 'image/png' },
|
||||
|
|
@ -277,8 +282,8 @@ describe('mcpResultToExecutableOutput', () => {
|
|||
expect(parts.at(-1)).toEqual({ type: 'text', text: '</mcp_tool_result>' });
|
||||
});
|
||||
|
||||
test('truncates oversized text and merges the notice into the surviving text part', () => {
|
||||
const out = mcpResultToExecutableOutput(
|
||||
test('truncates oversized text and merges the notice into the surviving text part', async () => {
|
||||
const out = await mcpResultToExecutableOutput(
|
||||
result([{ type: 'text', text: 'x'.repeat(100_001) }]),
|
||||
'mcp__s__t',
|
||||
);
|
||||
|
|
@ -288,10 +293,12 @@ describe('mcpResultToExecutableOutput', () => {
|
|||
expect(out.truncated).toBe(true);
|
||||
});
|
||||
|
||||
test('drops oversized binary parts in favor of a per-part notice without touching the text budget', () => {
|
||||
// 14 MiB base64 ≈ 10.5 MiB raw — just above the 10 MiB per-part cap.
|
||||
test('drops oversized binary parts in favor of a per-part notice without touching the text budget', async () => {
|
||||
// 14 MiB base64 ≈ 10.5 MiB raw — just above the 10 MiB per-part cap. The
|
||||
// bytes are not a real image, so compression fails over and the drop path
|
||||
// still applies.
|
||||
const huge = 'x'.repeat(14 * 1024 * 1024);
|
||||
const out = mcpResultToExecutableOutput(
|
||||
const out = await mcpResultToExecutableOutput(
|
||||
result([{ type: 'image', data: huge, mimeType: 'image/png' }]),
|
||||
'mcp__s__big',
|
||||
);
|
||||
|
|
@ -308,8 +315,8 @@ describe('mcpResultToExecutableOutput', () => {
|
|||
expect(out.truncated).toBe(true);
|
||||
});
|
||||
|
||||
test('binary part within the per-part cap survives intact alongside oversized text', () => {
|
||||
const out = mcpResultToExecutableOutput(
|
||||
test('binary part within the per-part cap survives intact alongside oversized text', async () => {
|
||||
const out = await mcpResultToExecutableOutput(
|
||||
result([
|
||||
{ type: 'text', text: 'A'.repeat(100_000) },
|
||||
{ type: 'image', data: 'B'.repeat(500_000), mimeType: 'image/png' },
|
||||
|
|
@ -324,4 +331,28 @@ describe('mcpResultToExecutableOutput', () => {
|
|||
]);
|
||||
expect(out).not.toHaveProperty('truncated');
|
||||
});
|
||||
|
||||
test('downsamples an oversized real image instead of leaving it full-size', async () => {
|
||||
const big = Buffer.from(
|
||||
await new Jimp({ width: 2600, height: 2600, color: 0x3366ccff }).getBuffer('image/png'),
|
||||
).toString('base64');
|
||||
|
||||
const out = await mcpResultToExecutableOutput(
|
||||
result([{ type: 'image', data: big, mimeType: 'image/png' }]),
|
||||
'mcp__s__shot',
|
||||
);
|
||||
|
||||
const parts = out.output as ContentPart[];
|
||||
const imagePart = parts.find((p) => p.type === 'image_url');
|
||||
expect(imagePart).toBeDefined();
|
||||
const match = /^data:(image\/[a-z]+);base64,(.+)$/.exec(
|
||||
(imagePart as { imageUrl: { url: string } }).imageUrl.url,
|
||||
);
|
||||
expect(match).not.toBeNull();
|
||||
const dims = sniffImageDimensions(Buffer.from(match![2]!, 'base64'));
|
||||
expect(Math.max(dims!.width, dims!.height)).toBeLessThanOrEqual(2000);
|
||||
// The image was compressed and kept, not dropped to a notice.
|
||||
const joined = parts.map((p) => (p.type === 'text' ? p.text : '')).join('');
|
||||
expect(joined).not.toContain('image_url dropped');
|
||||
});
|
||||
});
|
||||
|
|
|
|||
357
packages/agent-core/test/tools/image-compress.test.ts
Normal file
357
packages/agent-core/test/tools/image-compress.test.ts
Normal file
|
|
@ -0,0 +1,357 @@
|
|||
/**
|
||||
* image-compress — downsample/re-encode oversized images for the model.
|
||||
*
|
||||
* Tests pin:
|
||||
* - fast path: an image within both budgets passes through untouched
|
||||
* (same byte reference, no re-encode)
|
||||
* - dimension cap: an oversized image is scaled so its longest edge is
|
||||
* exactly MAX_IMAGE_EDGE_PX, preserving aspect ratio
|
||||
* - byte budget: an over-budget image walks the JPEG quality ladder and
|
||||
* comes back as JPEG, strictly smaller than the input
|
||||
* - alpha: a translucent PNG stays PNG when the budget allows, and only
|
||||
* drops to JPEG as a last resort to meet a tiny budget
|
||||
* - fallback: corrupt/empty bytes and non-recodable formats (GIF/WebP)
|
||||
* return the original unchanged — never throws
|
||||
* - invariant: `changed` implies the result is strictly smaller
|
||||
* - base64 wrapper round-trips
|
||||
* - performance: the fast path is codec-free; a large image compresses
|
||||
* within a generous time bound
|
||||
*/
|
||||
|
||||
import { Jimp } from 'jimp';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
// eslint-disable-next-line import/no-unresolved
|
||||
import {
|
||||
compressBase64ForModel,
|
||||
compressImageContentParts,
|
||||
compressImageForModel,
|
||||
IMAGE_BYTE_BUDGET,
|
||||
MAX_IMAGE_EDGE_PX,
|
||||
} from '../../src/tools/support/image-compress';
|
||||
// eslint-disable-next-line import/no-unresolved
|
||||
import { sniffImageDimensions } from '../../src/tools/support/file-type';
|
||||
|
||||
// ── fixtures ─────────────────────────────────────────────────────────
|
||||
|
||||
async function solidPng(width: number, height: number, color = 0x3366ccff): Promise<Uint8Array> {
|
||||
const image = new Jimp({ width, height, color });
|
||||
return new Uint8Array(await image.getBuffer('image/png'));
|
||||
}
|
||||
|
||||
async function solidJpeg(width: number, height: number, color = 0x3366ccff): Promise<Uint8Array> {
|
||||
const image = new Jimp({ width, height, color });
|
||||
return new Uint8Array(await image.getBuffer('image/jpeg', { quality: 90 }));
|
||||
}
|
||||
|
||||
async function translucentPng(width: number, height: number): Promise<Uint8Array> {
|
||||
// Alpha 0x80 on every pixel → hasAlpha() is true.
|
||||
const image = new Jimp({ width, height, color: 0x33_66_cc_80 });
|
||||
return new Uint8Array(await image.getBuffer('image/png'));
|
||||
}
|
||||
|
||||
/** High-entropy image whose PNG barely compresses — used to force the ladder. */
|
||||
async function noisePng(width: number, height: number, alpha = false): Promise<Uint8Array> {
|
||||
const image = new Jimp({ width, height, color: 0x000000ff });
|
||||
const data = image.bitmap.data;
|
||||
for (let i = 0; i < data.length; i += 4) {
|
||||
// Deterministic pseudo-random bytes (no Math.random for stable fixtures).
|
||||
// Distinct multipliers per channel keep entropy high so PNG barely shrinks.
|
||||
data[i] = (i * 2_654_435_761) & 0xff;
|
||||
data[i + 1] = (i * 40_503) & 0xff;
|
||||
data[i + 2] = (i * 12_289) & 0xff;
|
||||
data[i + 3] = alpha ? (i * 7 + 17) & 0xff : 0xff;
|
||||
}
|
||||
return new Uint8Array(await image.getBuffer('image/png'));
|
||||
}
|
||||
|
||||
async function decodeAlpha(bytes: Uint8Array): Promise<boolean> {
|
||||
const image = await Jimp.fromBuffer(Buffer.from(bytes));
|
||||
return image.hasAlpha();
|
||||
}
|
||||
|
||||
// ── fast path ────────────────────────────────────────────────────────
|
||||
|
||||
describe('compressImageForModel — fast path', () => {
|
||||
it('passes a within-budget image through untouched (same reference)', async () => {
|
||||
const png = await solidPng(64, 64);
|
||||
const result = await compressImageForModel(png, 'image/png');
|
||||
expect(result.changed).toBe(false);
|
||||
expect(result.data).toBe(png); // identity: no copy, no re-encode
|
||||
expect(result.mimeType).toBe('image/png');
|
||||
expect(result.width).toBe(64);
|
||||
expect(result.height).toBe(64);
|
||||
});
|
||||
|
||||
it('treats image/jpg as image/jpeg', async () => {
|
||||
const jpeg = await solidJpeg(32, 32);
|
||||
const result = await compressImageForModel(jpeg, 'image/jpg');
|
||||
expect(result.changed).toBe(false);
|
||||
expect(result.data).toBe(jpeg);
|
||||
});
|
||||
});
|
||||
|
||||
// ── dimension cap ────────────────────────────────────────────────────
|
||||
|
||||
describe('compressImageForModel — dimension cap', () => {
|
||||
it('scales the longest edge down to MAX_IMAGE_EDGE_PX, preserving aspect', async () => {
|
||||
const png = await solidPng(3000, 1500);
|
||||
const result = await compressImageForModel(png, 'image/png');
|
||||
expect(result.changed).toBe(true);
|
||||
expect(Math.max(result.width, result.height)).toBe(MAX_IMAGE_EDGE_PX);
|
||||
// 3000x1500 → 2000x1000 (aspect 2:1 preserved).
|
||||
expect(result.width).toBe(2000);
|
||||
expect(result.height).toBe(1000);
|
||||
const dims = sniffImageDimensions(result.data);
|
||||
expect(dims).toEqual({ width: 2000, height: 1000 });
|
||||
});
|
||||
|
||||
it('respects a custom maxEdge', async () => {
|
||||
const png = await solidPng(1600, 800);
|
||||
const result = await compressImageForModel(png, 'image/png', { maxEdge: 800 });
|
||||
expect(result.changed).toBe(true);
|
||||
expect(result.width).toBe(800);
|
||||
expect(result.height).toBe(400);
|
||||
});
|
||||
|
||||
it('keeps a downscaled opaque PNG lossless (no needless JPEG conversion)', async () => {
|
||||
// A screenshot-like opaque PNG that only needs downscaling must stay PNG so
|
||||
// sharp text is not degraded by JPEG artifacts.
|
||||
const png = await solidPng(3000, 1500);
|
||||
const result = await compressImageForModel(png, 'image/png');
|
||||
expect(result.changed).toBe(true);
|
||||
expect(result.mimeType).toBe('image/png');
|
||||
expect(Math.max(result.width, result.height)).toBe(MAX_IMAGE_EDGE_PX);
|
||||
});
|
||||
});
|
||||
|
||||
// ── byte budget ──────────────────────────────────────────────────────
|
||||
|
||||
describe('compressImageForModel — byte budget', () => {
|
||||
it('walks the JPEG ladder for an over-budget non-alpha image', async () => {
|
||||
const png = await noisePng(900, 900);
|
||||
const result = await compressImageForModel(png, 'image/png', { byteBudget: 8 * 1024 });
|
||||
expect(result.changed).toBe(true);
|
||||
expect(result.mimeType).toBe('image/jpeg');
|
||||
expect(result.finalByteLength).toBeLessThan(result.originalByteLength);
|
||||
});
|
||||
|
||||
it('keeps a translucent PNG as PNG when the budget allows', async () => {
|
||||
const png = await translucentPng(2600, 2600);
|
||||
const result = await compressImageForModel(png, 'image/png');
|
||||
expect(result.changed).toBe(true);
|
||||
expect(result.mimeType).toBe('image/png');
|
||||
expect(Math.max(result.width, result.height)).toBe(MAX_IMAGE_EDGE_PX);
|
||||
expect(await decodeAlpha(result.data)).toBe(true);
|
||||
});
|
||||
|
||||
it('drops alpha to JPEG only as a last resort under a tiny budget', async () => {
|
||||
const png = await noisePng(800, 800, /* alpha */ true);
|
||||
const result = await compressImageForModel(png, 'image/png', { byteBudget: 4 * 1024 });
|
||||
expect(result.changed).toBe(true);
|
||||
expect(result.mimeType).toBe('image/jpeg');
|
||||
expect(result.finalByteLength).toBeLessThan(result.originalByteLength);
|
||||
});
|
||||
});
|
||||
|
||||
// ── fallback / robustness ────────────────────────────────────────────
|
||||
|
||||
describe('compressImageForModel — fallback', () => {
|
||||
it('returns the original on corrupt bytes (never throws)', async () => {
|
||||
// Valid PNG signature followed by garbage — decode will fail.
|
||||
const corrupt = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 1, 2, 3, 4, 5]);
|
||||
const result = await compressImageForModel(corrupt, 'image/png');
|
||||
expect(result.changed).toBe(false);
|
||||
expect(result.data).toBe(corrupt);
|
||||
});
|
||||
|
||||
it('passes empty buffers through', async () => {
|
||||
const empty = new Uint8Array(0);
|
||||
const result = await compressImageForModel(empty, 'image/png');
|
||||
expect(result.changed).toBe(false);
|
||||
expect(result.data).toBe(empty);
|
||||
});
|
||||
|
||||
it('passes GIF through (preserves animation)', async () => {
|
||||
// Minimal GIF89a header — enough for the MIME guard to skip it.
|
||||
const gif = new Uint8Array([0x47, 0x49, 0x46, 0x38, 0x39, 0x61, 1, 0, 1, 0]);
|
||||
const result = await compressImageForModel(gif, 'image/gif');
|
||||
expect(result.changed).toBe(false);
|
||||
expect(result.data).toBe(gif);
|
||||
});
|
||||
|
||||
it('passes WebP through (no codec in the default build)', async () => {
|
||||
const webp = new Uint8Array([
|
||||
0x52, 0x49, 0x46, 0x46, 0, 0, 0, 0, 0x57, 0x45, 0x42, 0x50,
|
||||
]);
|
||||
const result = await compressImageForModel(webp, 'image/webp');
|
||||
expect(result.changed).toBe(false);
|
||||
expect(result.data).toBe(webp);
|
||||
});
|
||||
|
||||
it('skips compression for absurd pixel counts without decoding (bomb guard)', async () => {
|
||||
// A PNG header advertising 30000×30000 (900 MP) with no pixel data. The
|
||||
// dimension sniff reads the IHDR; the guard must pass through before Jimp
|
||||
// is ever invoked, so this completes instantly with no multi-GB bitmap.
|
||||
const header = Buffer.alloc(24);
|
||||
Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]).copy(header, 0);
|
||||
header.writeUInt32BE(13, 8); // IHDR chunk length
|
||||
header.write('IHDR', 12, 'latin1');
|
||||
header.writeUInt32BE(30000, 16);
|
||||
header.writeUInt32BE(30000, 20);
|
||||
const bomb = new Uint8Array(header);
|
||||
|
||||
const result = await compressImageForModel(bomb, 'image/png');
|
||||
expect(result.changed).toBe(false);
|
||||
expect(result.data).toBe(bomb); // identity → Jimp was never called
|
||||
});
|
||||
|
||||
it('skips compression for payloads over the byte cap without decoding', async () => {
|
||||
// Over the edge (so not the fast path), but capped by maxDecodeBytes.
|
||||
const png = await solidPng(3000, 100);
|
||||
const result = await compressImageForModel(png, 'image/png', { maxDecodeBytes: 64 });
|
||||
expect(result.changed).toBe(false);
|
||||
expect(result.data).toBe(png); // passthrough → Jimp was never called
|
||||
});
|
||||
});
|
||||
|
||||
// ── invariants ───────────────────────────────────────────────────────
|
||||
|
||||
describe('compressImageForModel — invariants', () => {
|
||||
it('changed always yields a within-cap, decodable payload', async () => {
|
||||
const cases: Uint8Array[] = [
|
||||
await solidPng(3000, 1500),
|
||||
await noisePng(900, 900),
|
||||
await translucentPng(2600, 2600),
|
||||
];
|
||||
for (const bytes of cases) {
|
||||
const result = await compressImageForModel(bytes, 'image/png');
|
||||
expect(result.finalByteLength).toBe(result.data.length);
|
||||
if (result.changed) {
|
||||
// A change is only kept when it helped: fewer bytes or fewer pixels.
|
||||
const original = sniffImageDimensions(bytes)!;
|
||||
const shrankBytes = result.finalByteLength < result.originalByteLength;
|
||||
const shrankPixels = result.width * result.height < original.width * original.height;
|
||||
expect(shrankBytes || shrankPixels).toBe(true);
|
||||
// Dimensions never exceed the cap after a change.
|
||||
expect(Math.max(result.width, result.height)).toBeLessThanOrEqual(MAX_IMAGE_EDGE_PX);
|
||||
// The result must decode.
|
||||
expect(sniffImageDimensions(result.data)).not.toBeNull();
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ── base64 wrapper ───────────────────────────────────────────────────
|
||||
|
||||
describe('compressBase64ForModel', () => {
|
||||
it('round-trips an over-sized image', async () => {
|
||||
const png = await noisePng(700, 700);
|
||||
const base64 = Buffer.from(png).toString('base64');
|
||||
const result = await compressBase64ForModel(base64, 'image/png', { byteBudget: 8 * 1024 });
|
||||
expect(result.changed).toBe(true);
|
||||
expect(result.finalByteLength).toBeLessThan(result.originalByteLength);
|
||||
// The re-encoded base64 still decodes to a valid image.
|
||||
const dims = sniffImageDimensions(Buffer.from(result.base64, 'base64'));
|
||||
expect(dims).not.toBeNull();
|
||||
});
|
||||
|
||||
it('returns the original base64 unchanged on the fast path', async () => {
|
||||
const png = await solidPng(64, 64);
|
||||
const base64 = Buffer.from(png).toString('base64');
|
||||
const result = await compressBase64ForModel(base64, 'image/png');
|
||||
expect(result.changed).toBe(false);
|
||||
expect(result.base64).toBe(base64);
|
||||
});
|
||||
|
||||
it('skips a base64 payload over the byte cap without decoding', async () => {
|
||||
const png = await solidPng(3000, 100); // over edge, would otherwise compress
|
||||
const base64 = Buffer.from(png).toString('base64');
|
||||
const result = await compressBase64ForModel(base64, 'image/png', { maxDecodeBytes: 64 });
|
||||
expect(result.changed).toBe(false);
|
||||
expect(result.base64).toBe(base64); // unchanged → not decoded
|
||||
});
|
||||
});
|
||||
|
||||
// ── performance ──────────────────────────────────────────────────────
|
||||
|
||||
describe('compressImageForModel — performance', () => {
|
||||
it('fast path is codec-free and quick across many calls', async () => {
|
||||
const png = await solidPng(200, 200);
|
||||
const start = performance.now();
|
||||
for (let i = 0; i < 100; i += 1) {
|
||||
const result = await compressImageForModel(png, 'image/png');
|
||||
expect(result.data).toBe(png); // proves no decode/encode happened
|
||||
}
|
||||
const elapsed = performance.now() - start;
|
||||
// 100 metadata-only checks should be well under 100ms.
|
||||
expect(elapsed).toBeLessThan(100);
|
||||
});
|
||||
|
||||
it('compresses a large image within a generous time bound', async () => {
|
||||
const png = await solidPng(3000, 2000);
|
||||
const start = performance.now();
|
||||
const result = await compressImageForModel(png, 'image/png');
|
||||
const elapsed = performance.now() - start;
|
||||
expect(result.changed).toBe(true);
|
||||
expect(elapsed).toBeLessThan(5000);
|
||||
});
|
||||
|
||||
it('exposes a sane default budget', () => {
|
||||
expect(IMAGE_BYTE_BUDGET).toBeGreaterThan(0);
|
||||
expect(MAX_IMAGE_EDGE_PX).toBe(2000);
|
||||
});
|
||||
});
|
||||
|
||||
// ── content-part helper ──────────────────────────────────────────────
|
||||
|
||||
describe('compressImageContentParts', () => {
|
||||
function dataUrl(mime: string, bytes: Uint8Array): string {
|
||||
return `data:${mime};base64,${Buffer.from(bytes).toString('base64')}`;
|
||||
}
|
||||
|
||||
it('compresses an oversized inline image part, leaving other parts untouched', async () => {
|
||||
const big = await solidPng(2600, 2600);
|
||||
const parts = [
|
||||
{ type: 'text' as const, text: 'look at this' },
|
||||
{ type: 'image_url' as const, imageUrl: { url: dataUrl('image/png', big) } },
|
||||
];
|
||||
const out = await compressImageContentParts(parts);
|
||||
|
||||
expect(out[0]).toEqual({ type: 'text', text: 'look at this' });
|
||||
const imagePart = out[1];
|
||||
if (imagePart?.type !== 'image_url') throw new Error('expected image_url');
|
||||
const match = /^data:(image\/[a-z]+);base64,(.+)$/.exec(imagePart.imageUrl.url);
|
||||
expect(match).not.toBeNull();
|
||||
const dims = sniffImageDimensions(Buffer.from(match![2]!, 'base64'));
|
||||
expect(Math.max(dims!.width, dims!.height)).toBeLessThanOrEqual(MAX_IMAGE_EDGE_PX);
|
||||
});
|
||||
|
||||
it('preserves the part identity for a within-budget image (no change)', async () => {
|
||||
const small = await solidPng(48, 48);
|
||||
const url = dataUrl('image/png', small);
|
||||
const parts = [{ type: 'image_url' as const, imageUrl: { url } }];
|
||||
const out = await compressImageContentParts(parts);
|
||||
expect(out[0]).toEqual({ type: 'image_url', imageUrl: { url } });
|
||||
});
|
||||
|
||||
it('leaves remote (non-data) image URLs untouched', async () => {
|
||||
const parts = [
|
||||
{ type: 'image_url' as const, imageUrl: { url: 'https://example.com/pic.png' } },
|
||||
];
|
||||
const out = await compressImageContentParts(parts);
|
||||
expect(out[0]).toEqual({ type: 'image_url', imageUrl: { url: 'https://example.com/pic.png' } });
|
||||
});
|
||||
|
||||
it('keeps an image part id when rewriting the compressed url', async () => {
|
||||
const big = await solidPng(2600, 2600);
|
||||
const parts = [
|
||||
{ type: 'image_url' as const, imageUrl: { url: dataUrl('image/png', big), id: 'att-1' } },
|
||||
];
|
||||
const out = await compressImageContentParts(parts);
|
||||
const imagePart = out[0];
|
||||
if (imagePart?.type !== 'image_url') throw new Error('expected image_url');
|
||||
expect(imagePart.imageUrl.id).toBe('att-1');
|
||||
expect(imagePart.imageUrl.url).not.toBe(dataUrl('image/png', big));
|
||||
});
|
||||
});
|
||||
|
|
@ -4,6 +4,7 @@
|
|||
|
||||
import type { Kaos } from '@moonshot-ai/kaos';
|
||||
import type { ContentPart, ModelCapability } from '@moonshot-ai/kosong';
|
||||
import { Jimp } from 'jimp';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { ToolAccesses } from '../../src/loop';
|
||||
|
|
@ -12,7 +13,7 @@ import {
|
|||
ReadMediaFileInputSchema,
|
||||
ReadMediaFileTool,
|
||||
} from '../../src/tools/builtin/file/read-media';
|
||||
import { MEDIA_SNIFF_BYTES } from '../../src/tools/support/file-type';
|
||||
import { MEDIA_SNIFF_BYTES, sniffImageDimensions } from '../../src/tools/support/file-type';
|
||||
import { createFakeKaos, PERMISSIVE_WORKSPACE } from './fixtures/fake-kaos';
|
||||
import { executeTool } from './fixtures/execute-tool';
|
||||
|
||||
|
|
@ -651,4 +652,37 @@ describe('ReadMediaFileTool', () => {
|
|||
'"/workspace/fake.png" is not a supported image or video file. Use Read for text files, or Bash or an MCP tool for other binary formats.',
|
||||
);
|
||||
});
|
||||
|
||||
it('downsamples an oversized image but reports original dimensions', async () => {
|
||||
const big = Buffer.from(
|
||||
await new Jimp({ width: 2600, height: 2600, color: 0x3366ccff }).getBuffer('image/png'),
|
||||
);
|
||||
expect(sniffImageDimensions(big)).toEqual({ width: 2600, height: 2600 });
|
||||
|
||||
const tool = makeReadMediaTool({
|
||||
stat: vi.fn<Kaos['stat']>().mockResolvedValue({ ...DEFAULT_STAT, stSize: big.length }),
|
||||
readBytes: vi.fn<Kaos['readBytes']>().mockResolvedValue(big),
|
||||
});
|
||||
|
||||
const result = await executeTool(tool, {
|
||||
turnId: 't1',
|
||||
toolCallId: 'c_big',
|
||||
args: { path: '/workspace/big.png' },
|
||||
signal,
|
||||
});
|
||||
|
||||
const parts = outputParts(result);
|
||||
const url = (parts[2] as { imageUrl: { url: string } }).imageUrl.url;
|
||||
const match = /^data:(image\/[a-z]+);base64,(.+)$/.exec(url);
|
||||
expect(match).not.toBeNull();
|
||||
// The image actually sent to the model is downsampled to the edge cap.
|
||||
const sentBytes = Buffer.from(match![2]!, 'base64');
|
||||
const sentDims = sniffImageDimensions(sentBytes);
|
||||
expect(Math.max(sentDims!.width, sentDims!.height)).toBeLessThanOrEqual(2000);
|
||||
|
||||
// The <system> summary keeps the ORIGINAL size so coordinate mapping holds.
|
||||
const systemText = (parts[0] as { text: string }).text;
|
||||
expect(systemText).toContain('2600x2600');
|
||||
expect(systemText).toContain(`${String(big.length)} bytes`);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -66,6 +66,7 @@
|
|||
"@moonshot-ai/kaos": "workspace:^",
|
||||
"@moonshot-ai/kimi-code-oauth": "workspace:^",
|
||||
"@moonshot-ai/kosong": "workspace:^",
|
||||
"@types/yazl": "^2.4.6"
|
||||
"@types/yazl": "^2.4.6",
|
||||
"jimp": "^1.6.1"
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -67,6 +67,21 @@ export { loadRuntimeConfigSafe, resolveConfigPath } from '@moonshot-ai/agent-cor
|
|||
// outbound fetch honors HTTP_PROXY / HTTPS_PROXY / NO_PROXY.
|
||||
export { installGlobalProxyDispatcher } from '@moonshot-ai/agent-core';
|
||||
|
||||
// Image compression — ingestion sites (e.g. the CLI's clipboard paste, the ACP
|
||||
// adapter) shrink oversized images while constructing the content part, before
|
||||
// it enters a prompt. Best effort: returns the original on any failure.
|
||||
export {
|
||||
compressImageForModel,
|
||||
compressBase64ForModel,
|
||||
IMAGE_BYTE_BUDGET,
|
||||
MAX_IMAGE_EDGE_PX,
|
||||
} from '@moonshot-ai/agent-core';
|
||||
export type {
|
||||
CompressImageOptions,
|
||||
CompressImageResult,
|
||||
CompressBase64Result,
|
||||
} from '@moonshot-ai/agent-core';
|
||||
|
||||
// Experimental feature flags — types only. Resolved values come from
|
||||
// `KimiHarness.getExperimentalFeatures()` over RPC, not from a re-exported runtime value.
|
||||
export type {
|
||||
|
|
|
|||
|
|
@ -55,6 +55,7 @@
|
|||
},
|
||||
"devDependencies": {
|
||||
"@types/bcryptjs": "^2.4.6",
|
||||
"@types/ws": "^8.18.0"
|
||||
"@types/ws": "^8.18.0",
|
||||
"jimp": "^1.6.1"
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ import {
|
|||
promptSteerResultSchema,
|
||||
type PromptSubmission,
|
||||
} from '@moonshot-ai/protocol';
|
||||
import { IPromptService, AuthModelNotResolvedError, AuthProvisioningRequiredError, AuthTokenMissingError, AuthTokenUnauthorizedError, PromptAlreadyCompletedError, PromptNotFoundError, SessionBusyError, SessionNotFoundError, FileNotFoundError, IFileStore, type IInstantiationService, type GetResult } from '@moonshot-ai/agent-core';
|
||||
import { IPromptService, AuthModelNotResolvedError, AuthProvisioningRequiredError, AuthTokenMissingError, AuthTokenUnauthorizedError, PromptAlreadyCompletedError, PromptNotFoundError, SessionBusyError, SessionNotFoundError, FileNotFoundError, IFileStore, compressImageForModel, compressBase64ForModel, type IInstantiationService, type GetResult } from '@moonshot-ai/agent-core';
|
||||
import { z } from 'zod';
|
||||
|
||||
|
||||
|
|
@ -254,6 +254,22 @@ async function resolvePromptMediaFiles(
|
|||
let changed = false;
|
||||
const content: PromptSubmission['content'] = [];
|
||||
for (const part of body.content) {
|
||||
// Inline base64 image: compress the payload in place. This is the same
|
||||
// input-stage step as the file path below, for REST clients that submit an
|
||||
// image as `{ source: { kind: 'base64' } }` instead of uploading a file.
|
||||
if (part.type === 'image' && part.source.kind === 'base64') {
|
||||
const compressed = await compressBase64ForModel(part.source.data, part.source.media_type);
|
||||
if (compressed.changed) {
|
||||
content.push({
|
||||
type: 'image',
|
||||
source: { kind: 'base64', media_type: compressed.mimeType, data: compressed.base64 },
|
||||
});
|
||||
changed = true;
|
||||
} else {
|
||||
content.push(part);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if ((part.type !== 'image' && part.type !== 'video') || part.source.kind !== 'file') {
|
||||
content.push(part);
|
||||
continue;
|
||||
|
|
@ -261,10 +277,21 @@ async function resolvePromptMediaFiles(
|
|||
const file = await store.get(part.source.file_id);
|
||||
assertMediaFile(file, part.type);
|
||||
const data = await readFile(file.blobPath);
|
||||
// Compress the image while inlining it into the prompt (an input-stage data
|
||||
// step, before the prompt reaches the agent core). The stored file keeps its
|
||||
// original bytes; only the model-facing copy is shrunk. Best effort: a
|
||||
// failure leaves the original bytes. Video is never re-encoded here.
|
||||
let mediaType = file.meta.media_type;
|
||||
let bytes: Uint8Array = data;
|
||||
if (part.type === 'image') {
|
||||
const compressed = await compressImageForModel(data, mediaType);
|
||||
bytes = compressed.data;
|
||||
mediaType = compressed.mimeType;
|
||||
}
|
||||
const source = {
|
||||
kind: 'base64' as const,
|
||||
media_type: file.meta.media_type,
|
||||
data: data.toString('base64'),
|
||||
media_type: mediaType,
|
||||
data: Buffer.from(bytes).toString('base64'),
|
||||
};
|
||||
content.push(part.type === 'video' ? { type: 'video', source } : { type: 'image', source });
|
||||
changed = true;
|
||||
|
|
|
|||
|
|
@ -28,6 +28,7 @@ import { join } from 'node:path';
|
|||
import { pino } from 'pino';
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||
import { WebSocket } from 'ws';
|
||||
import { Jimp } from 'jimp';
|
||||
|
||||
import type { Event, PromptSubmission } from '@moonshot-ai/protocol';
|
||||
import { IEventService, IPromptService, PromptService } from '@moonshot-ai/agent-core';
|
||||
|
|
@ -421,6 +422,111 @@ describe('POST /api/v1/sessions/{sid}/prompts — submit validation (W7.2 / Chai
|
|||
]);
|
||||
});
|
||||
|
||||
it('compresses an oversized uploaded image when resolving the prompt, leaving the stored file intact', async () => {
|
||||
let submitted: PromptSubmission | undefined;
|
||||
const r = await bootDaemon([
|
||||
[
|
||||
IPromptService,
|
||||
createPromptServiceOverride({
|
||||
submit: async (_sid, body) => {
|
||||
submitted = body;
|
||||
return {
|
||||
prompt_id: 'prompt_from_stub',
|
||||
user_message_id: 'msg_from_stub',
|
||||
status: 'running',
|
||||
content: body.content,
|
||||
created_at: '2026-06-09T00:00:00.000Z',
|
||||
};
|
||||
},
|
||||
}),
|
||||
],
|
||||
]);
|
||||
const sid = await createSession(r);
|
||||
|
||||
const bigPng = Buffer.from(
|
||||
await new Jimp({ width: 2600, height: 2600, color: 0x3366ccff }).getBuffer('image/png'),
|
||||
);
|
||||
const upload = buildMultipart({
|
||||
file: { fieldName: 'file', filename: 'big.png', contentType: 'image/png', data: bigPng },
|
||||
});
|
||||
const uploadRes = await appOf(r).inject({
|
||||
method: 'POST',
|
||||
url: '/api/v1/files',
|
||||
payload: upload.body,
|
||||
headers: { 'content-type': upload.contentType },
|
||||
});
|
||||
const uploadEnv = envelopeOf<{ id: string; media_type: string; size: number }>(
|
||||
uploadRes.json(),
|
||||
);
|
||||
expect(uploadEnv.code).toBe(0);
|
||||
// The stored file keeps its ORIGINAL bytes — only the prompt copy is shrunk.
|
||||
expect(uploadEnv.data?.size).toBe(bigPng.length);
|
||||
|
||||
const res = await appOf(r).inject({
|
||||
method: 'POST',
|
||||
url: `/api/v1/sessions/${sid}/prompts`,
|
||||
payload: {
|
||||
content: [{ type: 'image', source: { kind: 'file', file_id: uploadEnv.data!.id } }],
|
||||
},
|
||||
});
|
||||
expect(envelopeOf(res.json()).code).toBe(0);
|
||||
|
||||
const part = submitted?.content[0];
|
||||
if (part?.type !== 'image' || part.source.kind !== 'base64') {
|
||||
throw new Error('expected a resolved base64 image part');
|
||||
}
|
||||
const sentBytes = Buffer.from(part.source.data, 'base64');
|
||||
const decoded = await Jimp.fromBuffer(sentBytes);
|
||||
// The model-facing copy is downsampled to the edge cap.
|
||||
expect(Math.max(decoded.width, decoded.height)).toBeLessThanOrEqual(2000);
|
||||
});
|
||||
|
||||
it('compresses an inline base64 image submitted directly in the prompt', async () => {
|
||||
let submitted: PromptSubmission | undefined;
|
||||
const r = await bootDaemon([
|
||||
[
|
||||
IPromptService,
|
||||
createPromptServiceOverride({
|
||||
submit: async (_sid, body) => {
|
||||
submitted = body;
|
||||
return {
|
||||
prompt_id: 'prompt_from_stub',
|
||||
user_message_id: 'msg_from_stub',
|
||||
status: 'running',
|
||||
content: body.content,
|
||||
created_at: '2026-06-09T00:00:00.000Z',
|
||||
};
|
||||
},
|
||||
}),
|
||||
],
|
||||
]);
|
||||
const sid = await createSession(r);
|
||||
|
||||
// Solid 2600×2600: over the edge cap but tiny in bytes, so it stays well
|
||||
// under Fastify's inline-JSON limit yet still benefits from downscaling.
|
||||
const base64 = Buffer.from(
|
||||
await new Jimp({ width: 2600, height: 2600, color: 0x3366ccff }).getBuffer('image/png'),
|
||||
).toString('base64');
|
||||
|
||||
const res = await appOf(r).inject({
|
||||
method: 'POST',
|
||||
url: `/api/v1/sessions/${sid}/prompts`,
|
||||
payload: {
|
||||
content: [
|
||||
{ type: 'image', source: { kind: 'base64', media_type: 'image/png', data: base64 } },
|
||||
],
|
||||
},
|
||||
});
|
||||
expect(envelopeOf(res.json()).code).toBe(0);
|
||||
|
||||
const part = submitted?.content[0];
|
||||
if (part?.type !== 'image' || part.source.kind !== 'base64') {
|
||||
throw new Error('expected a base64 image part');
|
||||
}
|
||||
const decoded = await Jimp.fromBuffer(Buffer.from(part.source.data, 'base64'));
|
||||
expect(Math.max(decoded.width, decoded.height)).toBeLessThanOrEqual(2000);
|
||||
});
|
||||
|
||||
it('returns 40407 when prompt image file_id is unknown', async () => {
|
||||
let submitted = false;
|
||||
const r = await bootDaemon([
|
||||
|
|
|
|||
668
pnpm-lock.yaml
generated
668
pnpm-lock.yaml
generated
File diff suppressed because it is too large
Load diff
Loading…
Add table
Add a link
Reference in a new issue