mirror of
https://github.com/MoonshotAI/kimi-code.git
synced 2026-07-09 17:29:12 +00:00
feat(agent-core): announce image compression and keep originals readable (#1304)
* feat(agent-core): announce image compression and keep originals readable Every image ingestion point (ReadMediaFile, MCP tool results, clipboard paste, REST upload/inline base64, ACP) now places a <system> caption next to a compressed image stating the original vs. delivered dimensions, byte size, and format, so downsampling is never silent to the model. Originals stay readable: file uploads point at the stored blob, and in-memory images are persisted into the session's media-originals dir (content-addressed, size-capped, removed with the session; temp-dir fallback when no session is known). ReadMediaFile gains region (crop in original-image pixel coordinates, delivered at full fidelity) and full_resolution (skip downscaling, with an explicit error over the per-image byte limit), so the model can zoom into fine detail instead of silently degrading on large images. * fix(agent-core): exempt compression captions from the MCP text budget The caption announcing an image's compression was inserted before the shared 100K text budget was applied, so a chatty MCP result (page text + screenshot) consumed the budget first and the caption was evicted — or sliced mid-string into an unclosed <system> fragment — while the downsampled image survived, silently reintroducing the exact degradation the caption exists to report, and orphaning the persisted original. Split the size-limit pass in two and reorder the pipeline: the text budget now runs on the tool's own text BEFORE compression inserts captions (exempt by construction), and the per-part binary cap still runs after compression so compressible screenshots are kept. * fix(agent-core): harden crop error reporting and document readback semantics - cropImageForModel rejects non-finite region coordinates with a clean message instead of surfacing the codec's internal validation dump - the full_resolution and cropped-region over-budget errors now include exact byte counts alongside the rounded sizes, so a file a hair over budget no longer reads "is 3.8 MB, over the 3.8 MB limit" - read-media.md notes that re-reading a file without region or full_resolution reproduces the same downsampled image
This commit is contained in:
Kai
GitHub
parent
77eb3a9fe4
commit
0fc0ae380b
25 changed files with 1832 additions and 74 deletions
5
.changeset/announce-image-compression-readback.md
Normal file
5
.changeset/announce-image-compression-readback.md
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
---
|
||||
"@moonshot-ai/kimi-code": minor
|
||||
---
|
||||
|
||||
Never compress images silently. Every image ingestion point (ReadMediaFile, MCP tool results, clipboard paste, REST upload/inline base64, ACP) now places a caption next to a compressed image stating the original vs. delivered dimensions, byte size, and format, and preserves the original bytes for readback: uploads point at the stored file, and in-memory images are saved into the session's `media-originals` directory (size-capped, removed with the session; a shared temp-dir cache is the fallback when no session is known). ReadMediaFile gains a `region` parameter (crop in original-image pixel coordinates, delivered at full fidelity) and a `full_resolution` flag (skip downscaling, with an explicit error when the file exceeds the per-image byte limit), so the model can zoom into fine detail instead of degrading silently.
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import type { Session } from '@moonshot-ai/kimi-code-sdk';
|
||||
import { compressImageForModel } from '@moonshot-ai/kimi-code-sdk';
|
||||
import { compressImageForModel, persistOriginalImage, sessionMediaOriginalsDir } from '@moonshot-ai/kimi-code-sdk';
|
||||
|
||||
import { ClipboardMediaError, readClipboardMedia } from '#/utils/clipboard/clipboard-image';
|
||||
import { parseImageMeta } from '#/utils/image/image-mime';
|
||||
|
|
@ -407,13 +407,29 @@ export class EditorKeyboardController {
|
|||
// the stored bytes, the inline thumbnail, the `[image #N (W×H)]` placeholder,
|
||||
// and the submitted image all agree, and the agent core only ever sees an
|
||||
// already-compressed image. Best effort: originals pass through on failure.
|
||||
// When compression changed the bytes, the original is persisted (into the
|
||||
// session's media-originals dir when known, else the temp-dir fallback)
|
||||
// and recorded on the attachment, so submit-time expansion can announce
|
||||
// the compression and point the model at the full-fidelity copy.
|
||||
const compressed = await compressImageForModel(media.bytes, meta.mime);
|
||||
const sessionDir = this.host.session?.summary?.sessionDir;
|
||||
const attachment = compressed.changed
|
||||
? this.imageStore.addImage(
|
||||
compressed.data,
|
||||
compressed.mimeType,
|
||||
compressed.width,
|
||||
compressed.height,
|
||||
{
|
||||
path: await persistOriginalImage(
|
||||
media.bytes,
|
||||
meta.mime,
|
||||
sessionDir === undefined ? {} : { dir: sessionMediaOriginalsDir(sessionDir) },
|
||||
),
|
||||
width: meta.width,
|
||||
height: meta.height,
|
||||
byteLength: media.bytes.length,
|
||||
mime: meta.mime,
|
||||
},
|
||||
)
|
||||
: this.imageStore.addImage(media.bytes, meta.mime, meta.width, meta.height);
|
||||
this.host.state.editor.insertTextAtCursor?.(`${attachment.placeholder} `);
|
||||
|
|
|
|||
|
|
@ -6,7 +6,9 @@
|
|||
* (640×480)]` / `[video #2 sample.mov]`). The placeholder is what the
|
||||
* user sees in the input field; on submit, `extractMediaAttachments`
|
||||
* walks the text and expands image placeholders to image content parts
|
||||
* and video placeholders to file-path tags for `ReadMediaFile`.
|
||||
* (preceded by a compression caption when paste-time compression shrank
|
||||
* the bytes — see `ImageAttachment.original`) and video placeholders to
|
||||
* file-path tags for `ReadMediaFile`.
|
||||
*
|
||||
* Scope is per-`KimiTUI` instance. Reloads (`/new`, `/clear`,
|
||||
* session switch) call `clear()` so ids restart from 1 and stale
|
||||
|
|
@ -15,6 +17,18 @@
|
|||
* `--resume` wouldn't know how to materialize the files anyway.
|
||||
*/
|
||||
|
||||
export interface ImageAttachmentOriginal {
|
||||
/**
|
||||
* Where the pre-compression bytes were persisted for readback
|
||||
* (ReadMediaFile + region); null when persistence failed.
|
||||
*/
|
||||
readonly path: string | null;
|
||||
readonly width: number;
|
||||
readonly height: number;
|
||||
readonly byteLength: number;
|
||||
readonly mime: string;
|
||||
}
|
||||
|
||||
export interface ImageAttachment {
|
||||
readonly id: number;
|
||||
readonly kind: 'image';
|
||||
|
|
@ -22,6 +36,12 @@ export interface ImageAttachment {
|
|||
readonly mime: string;
|
||||
readonly width: number;
|
||||
readonly height: number;
|
||||
/**
|
||||
* Pre-compression original, recorded when paste-time compression changed
|
||||
* the bytes. Drives the compression caption emitted on submit so the model
|
||||
* knows it received a downsampled copy. Absent for untouched pastes.
|
||||
*/
|
||||
readonly original?: ImageAttachmentOriginal | undefined;
|
||||
/** Rendered placeholder string, e.g. `[image #1 (640×480)]`. */
|
||||
readonly placeholder: string;
|
||||
}
|
||||
|
|
@ -43,7 +63,13 @@ export class ImageAttachmentStore {
|
|||
private nextId = 1;
|
||||
private readonly byId = new Map<number, MediaAttachment>();
|
||||
|
||||
addImage(bytes: Uint8Array, mime: string, width: number, height: number): ImageAttachment {
|
||||
addImage(
|
||||
bytes: Uint8Array,
|
||||
mime: string,
|
||||
width: number,
|
||||
height: number,
|
||||
original?: ImageAttachmentOriginal,
|
||||
): ImageAttachment {
|
||||
const id = this.nextId;
|
||||
this.nextId += 1;
|
||||
const attachment: ImageAttachment = {
|
||||
|
|
@ -53,6 +79,7 @@ export class ImageAttachmentStore {
|
|||
mime,
|
||||
width,
|
||||
height,
|
||||
original,
|
||||
placeholder: formatPlaceholder(id, width, height),
|
||||
};
|
||||
this.byId.set(id, attachment);
|
||||
|
|
|
|||
|
|
@ -16,6 +16,7 @@
|
|||
*/
|
||||
|
||||
import type { PromptPart } from '@moonshot-ai/kimi-code-sdk';
|
||||
import { buildImageCompressionCaption } from '@moonshot-ai/kimi-code-sdk';
|
||||
|
||||
import type {
|
||||
ImageAttachment,
|
||||
|
|
@ -66,6 +67,11 @@ export function extractMediaAttachments(
|
|||
pushText(parts, mediaText);
|
||||
videoAttachmentIds.push(id);
|
||||
} else {
|
||||
// Paste-time compression is announced next to the image so the model
|
||||
// knows it received a downsampled copy and where the original lives.
|
||||
if (attachment.original !== undefined) {
|
||||
pushText(parts, captionForCompressedImage(attachment));
|
||||
}
|
||||
parts.push(imagePartForAttachment(attachment));
|
||||
imageAttachmentIds.push(id);
|
||||
}
|
||||
|
|
@ -109,6 +115,26 @@ function imagePartForAttachment(att: ImageAttachment): PromptPart {
|
|||
};
|
||||
}
|
||||
|
||||
function captionForCompressedImage(att: ImageAttachment): string {
|
||||
const original = att.original;
|
||||
if (original === undefined) return '';
|
||||
return buildImageCompressionCaption({
|
||||
original: {
|
||||
width: original.width,
|
||||
height: original.height,
|
||||
byteLength: original.byteLength,
|
||||
mimeType: original.mime,
|
||||
},
|
||||
final: {
|
||||
width: att.width,
|
||||
height: att.height,
|
||||
byteLength: att.bytes.length,
|
||||
mimeType: att.mime,
|
||||
},
|
||||
originalPath: original.path,
|
||||
});
|
||||
}
|
||||
|
||||
function tagTextForVideo(att: VideoAttachment): string {
|
||||
return formatMediaTag('video', att.sourcePath);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,9 +5,17 @@
|
|||
* - an oversized pasted image is downsampled while building the attachment,
|
||||
* so the stored bytes, the `[image #N (W×H)]` placeholder, and the eventual
|
||||
* submitted image all agree on the compressed size
|
||||
* - a within-budget paste is stored byte-for-byte (fast path)
|
||||
* - the pre-compression original is persisted and recorded on the
|
||||
* attachment, so the submitted prompt can announce the compression and
|
||||
* point the model at the full-fidelity bytes
|
||||
* - a within-budget paste is stored byte-for-byte (fast path), with no
|
||||
* original recorded
|
||||
*/
|
||||
|
||||
import { mkdtemp, readFile, rm, unlink } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
|
||||
import { Jimp } from 'jimp';
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
|
|
@ -32,7 +40,7 @@ interface PasteHarness {
|
|||
pasteImage(): Promise<void>;
|
||||
}
|
||||
|
||||
function createPasteHarness(): PasteHarness {
|
||||
function createPasteHarness(options: { sessionDir?: string } = {}): PasteHarness {
|
||||
const editor: Record<string, ((...args: never[]) => unknown) | undefined> = {
|
||||
setHistoryFilter: vi.fn() as unknown as (...args: never[]) => unknown,
|
||||
};
|
||||
|
|
@ -45,7 +53,10 @@ function createPasteHarness(): PasteHarness {
|
|||
footer: { setTransientHint: vi.fn() },
|
||||
ui: { requestRender: vi.fn() },
|
||||
},
|
||||
session: undefined,
|
||||
session:
|
||||
options.sessionDir === undefined
|
||||
? undefined
|
||||
: { summary: { sessionDir: options.sessionDir } },
|
||||
btwPanelController: { closeOrCancel: vi.fn(() => false) },
|
||||
track: vi.fn(),
|
||||
showError: vi.fn(),
|
||||
|
|
@ -100,6 +111,45 @@ describe('clipboard image paste compression', () => {
|
|||
expect(Math.max(dims!.width, dims!.height)).toBeLessThanOrEqual(2000);
|
||||
});
|
||||
|
||||
it('records and persists the pre-compression original for an oversized paste', async () => {
|
||||
const big = await solidPng(2600, 2600);
|
||||
readClipboardMedia.mockResolvedValue({ kind: 'image', bytes: big, mimeType: 'image/png' });
|
||||
|
||||
const { store, pasteImage } = createPasteHarness();
|
||||
await pasteImage();
|
||||
|
||||
const att = store.get(1);
|
||||
if (att?.kind !== 'image') throw new Error('expected image attachment');
|
||||
expect(att.original).toBeDefined();
|
||||
expect(att.original?.width).toBe(2600);
|
||||
expect(att.original?.height).toBe(2600);
|
||||
expect(att.original?.byteLength).toBe(big.length);
|
||||
expect(att.original?.mime).toBe('image/png');
|
||||
|
||||
// The original bytes are readable back from the persisted path.
|
||||
expect(att.original?.path).not.toBeNull();
|
||||
const persisted = await readFile(att.original!.path!);
|
||||
expect(new Uint8Array(persisted)).toEqual(big);
|
||||
await unlink(att.original!.path!).catch(() => undefined);
|
||||
});
|
||||
|
||||
it('persists the original into the session media-originals dir when the session is known', async () => {
|
||||
const sessionDir = await mkdtemp(join(tmpdir(), 'kimi-paste-session-'));
|
||||
const big = await solidPng(2600, 2600);
|
||||
readClipboardMedia.mockResolvedValue({ kind: 'image', bytes: big, mimeType: 'image/png' });
|
||||
|
||||
const { store, pasteImage } = createPasteHarness({ sessionDir });
|
||||
await pasteImage();
|
||||
|
||||
const att = store.get(1);
|
||||
if (att?.kind !== 'image') throw new Error('expected image attachment');
|
||||
expect(att.original?.path).not.toBeNull();
|
||||
expect(att.original!.path!.startsWith(join(sessionDir, 'media-originals'))).toBe(true);
|
||||
const persisted = await readFile(att.original!.path!);
|
||||
expect(new Uint8Array(persisted)).toEqual(big);
|
||||
await rm(sessionDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('stores a within-budget paste byte-for-byte', async () => {
|
||||
const small = await solidPng(80, 80);
|
||||
readClipboardMedia.mockResolvedValue({ kind: 'image', bytes: small, mimeType: 'image/png' });
|
||||
|
|
@ -112,5 +162,6 @@ describe('clipboard image paste compression', () => {
|
|||
expect(att.width).toBe(80);
|
||||
expect(att.height).toBe(80);
|
||||
expect(att.bytes).toBe(small); // identity: no re-encode on the fast path
|
||||
expect(att.original).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -101,4 +101,52 @@ describe('extractMediaAttachments', () => {
|
|||
expect(r.videoAttachmentIds).toEqual([1]);
|
||||
expect(r.parts).toEqual([{ type: 'text', text: '<video path="/tmp/sample.mp4"></video>' }]);
|
||||
});
|
||||
|
||||
it('inserts a compression caption before an image that was compressed at paste time', () => {
|
||||
const store = new ImageAttachmentStore();
|
||||
const att = store.addImage(new Uint8Array([1, 2, 3]), 'image/png', 2000, 2000, {
|
||||
path: '/tmp/kimi-code-original-images/abc.png',
|
||||
width: 2600,
|
||||
height: 2600,
|
||||
byteLength: 123456,
|
||||
mime: 'image/png',
|
||||
});
|
||||
|
||||
const r = extractMediaAttachments(`look ${att.placeholder}`, store);
|
||||
|
||||
expect(r.parts).toHaveLength(2);
|
||||
const caption = r.parts[0];
|
||||
if (caption?.type !== 'text') throw new Error('expected leading text part');
|
||||
expect(caption.text).toContain('Image compressed');
|
||||
expect(caption.text).toContain('2600x2600');
|
||||
expect(caption.text).toContain('/tmp/kimi-code-original-images/abc.png');
|
||||
expect(r.parts[1]).toEqual({
|
||||
type: 'image_url',
|
||||
imageUrl: { url: 'data:image/png;base64,AQID' },
|
||||
});
|
||||
});
|
||||
|
||||
it('notes an unpreserved original when persistence failed at paste time', () => {
|
||||
const store = new ImageAttachmentStore();
|
||||
const att = store.addImage(new Uint8Array([1]), 'image/png', 2000, 2000, {
|
||||
path: null,
|
||||
width: 2600,
|
||||
height: 2600,
|
||||
byteLength: 123456,
|
||||
mime: 'image/png',
|
||||
});
|
||||
|
||||
const r = extractMediaAttachments(att.placeholder, store);
|
||||
|
||||
const caption = r.parts[0];
|
||||
if (caption?.type !== 'text') throw new Error('expected leading text part');
|
||||
expect(caption.text).toMatch(/not preserved/i);
|
||||
});
|
||||
|
||||
it('adds no caption for an uncompressed image attachment', () => {
|
||||
const { store, placeholder } = storeWith(new Uint8Array([0xaa]));
|
||||
const r = extractMediaAttachments(placeholder, store);
|
||||
expect(r.parts).toHaveLength(1);
|
||||
expect(r.parts[0]?.type).toBe('image_url');
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -1,7 +1,9 @@
|
|||
import type { ContentBlock, ToolCallContent } from '@agentclientprotocol/sdk';
|
||||
import {
|
||||
log,
|
||||
buildImageCompressionCaption,
|
||||
compressBase64ForModel,
|
||||
persistOriginalImage,
|
||||
type PromptPart,
|
||||
type ToolInputDisplay,
|
||||
type ToolResultEvent,
|
||||
|
|
@ -77,9 +79,16 @@ export function acpBlocksToPromptParts(
|
|||
* point's input-stage compression, mirroring the CLI's paste-time and the
|
||||
* server's upload-time step. Best effort: a part that cannot be compressed is
|
||||
* passed through unchanged.
|
||||
*
|
||||
* Compression is never silent: a re-encoded image gains a caption text part
|
||||
* immediately before it stating what the original was, and the original bytes
|
||||
* are persisted (into `originalsDir` — typically the session's
|
||||
* media-originals dir — or the shared temp-dir fallback) so the model can
|
||||
* read fine detail back via ReadMediaFile + region.
|
||||
*/
|
||||
export async function compressPromptImageParts(
|
||||
parts: readonly PromptPart[],
|
||||
options: { readonly originalsDir?: string | undefined } = {},
|
||||
): Promise<PromptPart[]> {
|
||||
const out: PromptPart[] = [];
|
||||
for (const part of parts) {
|
||||
|
|
@ -88,6 +97,29 @@ export async function compressPromptImageParts(
|
|||
if (parsed !== null) {
|
||||
const result = await compressBase64ForModel(parsed.base64, parsed.mimeType);
|
||||
if (result.changed) {
|
||||
const originalPath = await persistOriginalImage(
|
||||
Buffer.from(parsed.base64, 'base64'),
|
||||
parsed.mimeType,
|
||||
options.originalsDir === undefined ? {} : { dir: options.originalsDir },
|
||||
);
|
||||
out.push({
|
||||
type: 'text',
|
||||
text: buildImageCompressionCaption({
|
||||
original: {
|
||||
width: result.originalWidth,
|
||||
height: result.originalHeight,
|
||||
byteLength: result.originalByteLength,
|
||||
mimeType: parsed.mimeType,
|
||||
},
|
||||
final: {
|
||||
width: result.width,
|
||||
height: result.height,
|
||||
byteLength: result.finalByteLength,
|
||||
mimeType: result.mimeType,
|
||||
},
|
||||
originalPath,
|
||||
}),
|
||||
});
|
||||
out.push({
|
||||
type: 'image_url',
|
||||
imageUrl: { ...part.imageUrl, url: `data:${result.mimeType};base64,${result.base64}` },
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@ import {
|
|||
import {
|
||||
ErrorCodes,
|
||||
log,
|
||||
sessionMediaOriginalsDir,
|
||||
type ApprovalRequest,
|
||||
type ApprovalResponse,
|
||||
type BackgroundTaskInfo,
|
||||
|
|
@ -735,7 +736,11 @@ export class AcpSession {
|
|||
this.pendingPromptAborts.add(pending);
|
||||
let parts: readonly PromptPart[];
|
||||
try {
|
||||
parts = await compressPromptImageParts(acpBlocksToPromptParts(blocks));
|
||||
const sessionDir = this.session.summary?.sessionDir;
|
||||
parts = await compressPromptImageParts(acpBlocksToPromptParts(blocks), {
|
||||
originalsDir:
|
||||
sessionDir === undefined ? undefined : sessionMediaOriginalsDir(sessionDir),
|
||||
});
|
||||
} finally {
|
||||
this.pendingPromptAborts.delete(pending);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,5 +1,9 @@
|
|||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { mkdtemp, readFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
|
||||
import type { ContentBlock } from '@agentclientprotocol/sdk';
|
||||
import { Jimp } from 'jimp';
|
||||
|
||||
|
|
@ -332,16 +336,35 @@ describe('compressPromptImageParts', () => {
|
|||
return Buffer.from(buf).toString('base64');
|
||||
}
|
||||
|
||||
it('downsamples an oversized inline image part', async () => {
|
||||
const parts = acpBlocksToPromptParts([imageBlock(await pngBase64(2600, 2600), 'image/png')]);
|
||||
const compressed = await compressPromptImageParts(parts);
|
||||
it('downsamples an oversized inline image part and announces the compression', async () => {
|
||||
const originalsDir = await mkdtemp(join(tmpdir(), 'acp-originals-'));
|
||||
const originalBase64 = await pngBase64(2600, 2600);
|
||||
const parts = acpBlocksToPromptParts([imageBlock(originalBase64, 'image/png')]);
|
||||
const compressed = await compressPromptImageParts(parts, { originalsDir });
|
||||
|
||||
const part = compressed[0];
|
||||
// A caption precedes the downsampled image so the model knows it is
|
||||
// looking at a degraded copy and where the original bytes live.
|
||||
expect(compressed).toHaveLength(2);
|
||||
const caption = compressed[0];
|
||||
if (caption?.type !== 'text') throw new Error('expected a caption text part');
|
||||
expect(caption.text).toContain('Image compressed');
|
||||
expect(caption.text).toContain('2600x2600');
|
||||
|
||||
const part = compressed[1];
|
||||
if (part?.type !== 'image_url') throw new Error('expected an image_url part');
|
||||
const match = /^data:(image\/[a-z]+);base64,(.+)$/.exec(part.imageUrl.url);
|
||||
expect(match).not.toBeNull();
|
||||
const decoded = await Jimp.fromBuffer(Buffer.from(match![2]!, 'base64'));
|
||||
expect(Math.max(decoded.width, decoded.height)).toBeLessThanOrEqual(2000);
|
||||
|
||||
// The caption points at a persisted copy of the ORIGINAL bytes, placed in
|
||||
// the provided (session-scoped) originals dir.
|
||||
const pathMatch = /saved at "([^"]+)"/.exec(caption.text);
|
||||
expect(pathMatch).not.toBeNull();
|
||||
expect(pathMatch![1]!.startsWith(originalsDir)).toBe(true);
|
||||
const persisted = await readFile(pathMatch![1]!);
|
||||
expect(persisted.equals(Buffer.from(originalBase64, 'base64'))).toBe(true);
|
||||
await rm(originalsDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('passes a within-budget image and text through unchanged', async () => {
|
||||
|
|
|
|||
|
|
@ -70,6 +70,13 @@ export interface AgentOptions {
|
|||
readonly kaos: Kaos;
|
||||
readonly config?: KimiConfig;
|
||||
readonly homedir?: string;
|
||||
/**
|
||||
* Session-owned directory for pre-compression image originals
|
||||
* (`sessionMediaOriginalsDir(sessionDir)`), threaded to media-producing
|
||||
* paths (MCP tool results) so readback originals live with the session
|
||||
* rather than in the shared temp-dir fallback.
|
||||
*/
|
||||
readonly mediaOriginalsDir?: string;
|
||||
readonly rpc?: Partial<SDKAgentRPC>;
|
||||
readonly persistence?: AgentRecordPersistence;
|
||||
readonly type?: AgentType;
|
||||
|
|
@ -103,6 +110,7 @@ export class Agent {
|
|||
|
||||
readonly kimiConfig?: KimiConfig;
|
||||
readonly homedir?: string;
|
||||
readonly mediaOriginalsDir?: string;
|
||||
readonly rpc?: Partial<SDKAgentRPC>;
|
||||
readonly toolServices?: ToolServices;
|
||||
readonly pluginSessionStarts: readonly EnabledPluginSessionStart[];
|
||||
|
|
@ -146,6 +154,7 @@ export class Agent {
|
|||
this._kaos = options.kaos;
|
||||
this.kimiConfig = options.config;
|
||||
this.homedir = options.homedir;
|
||||
this.mediaOriginalsDir = options.mediaOriginalsDir;
|
||||
this.rpc = options.rpc;
|
||||
this.toolServices = options.toolServices;
|
||||
this.pluginSessionStarts = options.pluginSessionStarts ?? [];
|
||||
|
|
|
|||
|
|
@ -281,7 +281,9 @@ export class ToolManager {
|
|||
(args ?? {}) as Record<string, unknown>,
|
||||
context.signal,
|
||||
);
|
||||
return mcpResultToExecutableOutput(result, qualified);
|
||||
return mcpResultToExecutableOutput(result, qualified, {
|
||||
originalsDir: this.agent.mediaOriginalsDir,
|
||||
});
|
||||
},
|
||||
};
|
||||
},
|
||||
|
|
|
|||
|
|
@ -47,20 +47,39 @@ export type { ToolServices } from './tools/support/services';
|
|||
|
||||
// Image compression — the input-stage helper each ingestion site (CLI paste,
|
||||
// server upload resolution, ACP, ReadMediaFile, MCP) calls to shrink oversized
|
||||
// images while constructing the content part. Re-exported from the package root
|
||||
// so consumers (node-sdk, server) import it without a deep subpath.
|
||||
// images while constructing the content part. Compression is never silent:
|
||||
// buildImageCompressionCaption renders the shared "what was compressed" note,
|
||||
// persistOriginalImage keeps the pre-compression bytes readable, and
|
||||
// cropImageForModel reads a region of an original back at full fidelity.
|
||||
// Re-exported from the package root so consumers (node-sdk, server) import
|
||||
// them without a deep subpath.
|
||||
export {
|
||||
buildImageCompressionCaption,
|
||||
compressImageForModel,
|
||||
compressBase64ForModel,
|
||||
compressImageContentParts,
|
||||
cropImageForModel,
|
||||
formatByteSize,
|
||||
IMAGE_BYTE_BUDGET,
|
||||
MAX_IMAGE_EDGE_PX,
|
||||
} from './tools/support/image-compress';
|
||||
export type {
|
||||
CompressAnnotateOptions,
|
||||
CompressImageOptions,
|
||||
CompressImageResult,
|
||||
CompressBase64Result,
|
||||
CropImageOptions,
|
||||
CropImageOutcome,
|
||||
ImageCompressionCaptionInput,
|
||||
ImageCropRegion,
|
||||
ImageVariantDescription,
|
||||
} from './tools/support/image-compress';
|
||||
export {
|
||||
originalImageCacheDir,
|
||||
persistOriginalImage,
|
||||
sessionMediaOriginalsDir,
|
||||
} from './tools/support/image-originals';
|
||||
export type { PersistOriginalImageOptions } from './tools/support/image-originals';
|
||||
export { SingleModelProvider } from './session/provider-manager';
|
||||
export type {
|
||||
BearerTokenProvider,
|
||||
|
|
|
|||
|
|
@ -8,11 +8,17 @@
|
|||
* 2. Wrap media-only outputs in `<mcp_tool_result name="…">` tags so the
|
||||
* model can attribute binary output when several tools return media.
|
||||
* Mirrors the in-tree `ReadMediaFile` convention.
|
||||
* 3. Apply size limits: text/think share a 100K character budget; binary
|
||||
* parts (image/audio/video URLs) each carry an independent 10 MB cap and
|
||||
* collapse to a notice when oversize, so a single screenshot cannot
|
||||
* evict every text part.
|
||||
* 4. Collapse a single-text-part result to a plain string output; otherwise
|
||||
* 3. Apply the 100K text/think character budget to the tool's own text.
|
||||
* This runs BEFORE captions exist, so a chatty tool (page text + a
|
||||
* screenshot) can never evict or slice the compression caption — that
|
||||
* would silently reintroduce the very degradation the caption reports.
|
||||
* 4. Compress oversized inline images, announcing each compression with a
|
||||
* caption (original vs. sent size, readback path to the persisted
|
||||
* original) so downsampling is never silent.
|
||||
* 5. Apply the per-part 10 MB binary cap: oversized binary parts
|
||||
* (image/audio/video URLs) collapse to a notice, so a single
|
||||
* screenshot cannot evict every text part.
|
||||
* 6. Collapse a single-text-part result to a plain string output; otherwise
|
||||
* emit the `ContentPart[]` as-is.
|
||||
*
|
||||
* `mcpResultToExecutableOutput` is the single entry point; the per-step
|
||||
|
|
@ -22,8 +28,18 @@
|
|||
import type { ContentPart } from '@moonshot-ai/kosong';
|
||||
|
||||
import { compressImageContentParts } from '../tools/support/image-compress';
|
||||
import { persistOriginalImage } from '../tools/support/image-originals';
|
||||
import type { MCPContentBlock, MCPToolResult } from './types';
|
||||
|
||||
export interface McpOutputOptions {
|
||||
/**
|
||||
* Session-owned directory for pre-compression originals (typically
|
||||
* `sessionMediaOriginalsDir(sessionDir)` threaded down from the agent).
|
||||
* Falls back to the shared temp-dir cache when absent.
|
||||
*/
|
||||
readonly originalsDir?: string | undefined;
|
||||
}
|
||||
|
||||
// MCP servers can produce arbitrarily large outputs; cap what we feed back to
|
||||
// the model so a single chatty server does not blow up the context window. The
|
||||
// notice text is fed to the model verbatim so it can react (e.g. paginate),
|
||||
|
|
@ -134,6 +150,7 @@ export function convertMCPContentBlock(block: MCPContentBlock): ContentPart | nu
|
|||
export async function mcpResultToExecutableOutput(
|
||||
result: MCPToolResult,
|
||||
qualifiedToolName: string,
|
||||
options: McpOutputOptions = {},
|
||||
): Promise<{ output: string | ContentPart[]; isError: boolean; truncated?: true }> {
|
||||
const converted: ContentPart[] = [];
|
||||
for (const block of result.content) {
|
||||
|
|
@ -144,13 +161,32 @@ export async function mcpResultToExecutableOutput(
|
|||
}
|
||||
|
||||
const wrapped = wrapMediaOnly(converted, qualifiedToolName);
|
||||
// Text budget FIRST, on the tool's own text only: captions inserted by the
|
||||
// compression step below must never compete with a chatty tool's text for
|
||||
// the budget — an evicted or mid-string-sliced caption silently
|
||||
// reintroduces the downsampling this pipeline promises to announce.
|
||||
const budgeted = applyTextBudget(wrapped);
|
||||
// Shrink oversized images BEFORE the per-part byte cap, so a large but
|
||||
// compressible screenshot is downsampled and kept rather than dropped to a
|
||||
// text notice. Best effort: parts that cannot be compressed pass through.
|
||||
const compressed = await compressImageContentParts(wrapped);
|
||||
const limited = applyOutputLimits(compressed);
|
||||
const output = collapseSingleText(limited.parts);
|
||||
return limited.truncated
|
||||
// text notice. Compression is never silent: each re-encoded image gains a
|
||||
// caption stating what the original was, and the original bytes are
|
||||
// persisted (best effort, into the session's media-originals dir when
|
||||
// known) so the model can read detail back via ReadMediaFile + region.
|
||||
// Parts that cannot be compressed pass through.
|
||||
const compressed = await compressImageContentParts(budgeted.parts, {
|
||||
annotate: {
|
||||
persistOriginal: (bytes, mimeType) =>
|
||||
persistOriginalImage(
|
||||
bytes,
|
||||
mimeType,
|
||||
options.originalsDir === undefined ? {} : { dir: options.originalsDir },
|
||||
),
|
||||
},
|
||||
});
|
||||
const capped = applyBinaryPartCap(compressed);
|
||||
const truncated = budgeted.truncated || capped.truncated;
|
||||
const output = collapseSingleText(capped.parts);
|
||||
return truncated
|
||||
? { output, isError: result.isError, truncated: true }
|
||||
: { output, isError: result.isError };
|
||||
}
|
||||
|
|
@ -174,33 +210,33 @@ function wrapMediaOnly(parts: readonly ContentPart[], qualifiedToolName: string)
|
|||
}
|
||||
|
||||
/**
|
||||
* Apply the 100K text/think budget and the per-part 10 MB binary cap.
|
||||
* Apply the 100K text/think budget. Runs before image compression, so only
|
||||
* the tool's own text is charged — compression captions inserted afterwards
|
||||
* are exempt by construction. Binary parts pass through untouched (their
|
||||
* independent per-part cap is {@link applyBinaryPartCap}).
|
||||
*
|
||||
* When text/think parts get truncated, the truncation notice is appended to
|
||||
* the last surviving text part — this keeps the single-text-part collapse
|
||||
* working when the entire (oversized) input is a single text block.
|
||||
*/
|
||||
function applyOutputLimits(parts: readonly ContentPart[]): {
|
||||
function applyTextBudget(parts: readonly ContentPart[]): {
|
||||
readonly parts: ContentPart[];
|
||||
readonly truncated: boolean;
|
||||
} {
|
||||
let remaining = MCP_MAX_OUTPUT_CHARS;
|
||||
let truncated = false;
|
||||
let textTruncated = false;
|
||||
const out: ContentPart[] = [];
|
||||
|
||||
for (const part of parts) {
|
||||
if (part.type === 'text') {
|
||||
if (remaining <= 0) {
|
||||
truncated = true;
|
||||
textTruncated = true;
|
||||
continue;
|
||||
}
|
||||
if (part.text.length > remaining) {
|
||||
out.push({ type: 'text', text: part.text.slice(0, remaining) });
|
||||
remaining = 0;
|
||||
truncated = true;
|
||||
textTruncated = true;
|
||||
} else {
|
||||
out.push(part);
|
||||
remaining -= part.text.length;
|
||||
|
|
@ -212,14 +248,12 @@ function applyOutputLimits(parts: readonly ContentPart[]): {
|
|||
const size = part.think.length + (part.encrypted?.length ?? 0);
|
||||
if (remaining <= 0) {
|
||||
truncated = true;
|
||||
textTruncated = true;
|
||||
continue;
|
||||
}
|
||||
if (size > remaining) {
|
||||
out.push({ type: 'think', think: part.think.slice(0, remaining) });
|
||||
remaining = 0;
|
||||
truncated = true;
|
||||
textTruncated = true;
|
||||
} else {
|
||||
out.push(part);
|
||||
remaining -= size;
|
||||
|
|
@ -227,9 +261,35 @@ function applyOutputLimits(parts: readonly ContentPart[]): {
|
|||
continue;
|
||||
}
|
||||
|
||||
// image_url / audio_url / video_url: per-part byte cap, independent of the
|
||||
// text character budget. Oversized parts collapse into a per-part notice so
|
||||
// the model can pick a smaller resource instead of silently losing the blob.
|
||||
out.push(part);
|
||||
}
|
||||
|
||||
if (truncated) {
|
||||
appendTruncationNotice(out);
|
||||
}
|
||||
return { parts: out, truncated };
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply the per-part 10 MB binary cap, independent of the text character
|
||||
* budget. Oversized parts collapse into a per-part notice so the model can
|
||||
* pick a smaller resource instead of silently losing the blob. Runs after
|
||||
* image compression, so a large but compressible image has already been
|
||||
* shrunk under the cap.
|
||||
*/
|
||||
function applyBinaryPartCap(parts: readonly ContentPart[]): {
|
||||
readonly parts: ContentPart[];
|
||||
readonly truncated: boolean;
|
||||
} {
|
||||
let truncated = false;
|
||||
const out: ContentPart[] = [];
|
||||
|
||||
for (const part of parts) {
|
||||
if (part.type === 'text' || part.type === 'think') {
|
||||
out.push(part);
|
||||
continue;
|
||||
}
|
||||
|
||||
const url =
|
||||
part.type === 'image_url'
|
||||
? part.imageUrl.url
|
||||
|
|
@ -246,9 +306,6 @@ function applyOutputLimits(parts: readonly ContentPart[]): {
|
|||
out.push(part);
|
||||
}
|
||||
|
||||
if (textTruncated) {
|
||||
appendTruncationNotice(out);
|
||||
}
|
||||
return { parts: out, truncated };
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -49,6 +49,7 @@ import {
|
|||
} from '../skill';
|
||||
import { noopTelemetryClient, type TelemetryClient } from '../telemetry';
|
||||
import { SessionSubagentHost } from './subagent-host';
|
||||
import { sessionMediaOriginalsDir } from '../tools/support/image-originals';
|
||||
import type { ToolServices } from '../tools/support/services';
|
||||
import { FlagResolver, type ExperimentalFlagResolver } from '../flags';
|
||||
import { abortError } from '../utils/abort';
|
||||
|
|
@ -733,6 +734,9 @@ export class Session {
|
|||
toolServices: this.options.toolServices,
|
||||
config: this.options.config,
|
||||
homedir,
|
||||
// Session-level, shared across agents: originals persisted for
|
||||
// compression captions live with the session, not the agent.
|
||||
mediaOriginalsDir: sessionMediaOriginalsDir(this.options.homedir),
|
||||
skills: this.skills,
|
||||
rpc: proxyWithExtraPayload(this.rpc, { agentId: id }),
|
||||
modelProvider: this.options.providerManager,
|
||||
|
|
|
|||
|
|
@ -2,13 +2,13 @@ Read media content from a file.
|
|||
|
||||
**Tips:**
|
||||
- Make sure you follow the description of each tool parameter.
|
||||
- A `<system>` tag is given before the file content; it summarizes the mime type, byte size and, for images, the original pixel dimensions. When outputting coordinates, give relative coordinates first and compute absolute coordinates from the original image size. After generating or editing media via commands or scripts, read the result back before continuing.
|
||||
- A `<system>` tag is given before the file content; it summarizes the mime type, byte size and, for images, the original pixel dimensions, and states how the image was delivered (untouched, downsampled, cropped, or native resolution). When outputting coordinates, give relative coordinates first and compute absolute coordinates from the original image size. After generating or editing media via commands or scripts, read the result back before continuing.
|
||||
- Large images are downsampled by default to fit model limits, which can blur fine detail (small text, dense UI). Compute absolute coordinates from the original dimensions reported in the `<system>` block, never by measuring the displayed copy. When the `<system>` tag reports downsampling and you need that detail, call this tool again with the `region` parameter (original-image pixel coordinates) to view a crop at full fidelity, or set `full_resolution` to true when the whole file fits the per-image byte limit. Re-reading the same file without these parameters just reproduces the same downsampled image.
|
||||
- The system will notify you when there is anything wrong when reading the file.
|
||||
- This tool is a tool that you typically want to use in parallel. Always read multiple files in one response when possible.
|
||||
- This tool can only read image or video files. To read text files, use the Read tool. To list directories, use `ls` via Bash for a known directory, or Glob for pattern search.
|
||||
- If the file doesn't exist or path is invalid, an error will be returned.
|
||||
- The maximum size that can be read is {{ MAX_MEDIA_MEGABYTES }}MB. An error will be returned if the file is larger than this limit.
|
||||
- The media content will be returned in a form that you can directly view and understand.
|
||||
- Large images may be downsampled before being shown to you; the `<system>` block reports the original pixel dimensions when known — compute absolute coordinates from those, never by measuring the displayed copy.
|
||||
|
||||
**Capabilities**
|
||||
|
|
@ -7,10 +7,18 @@
|
|||
* and gates on the model's `image_in` / `video_in` capability.
|
||||
*
|
||||
* The leading `<system>` block summarizes mime type, byte size and (for
|
||||
* images) original pixel dimensions, guides the model to derive absolute
|
||||
* coordinates from that original size, and reminds it to re-read any media
|
||||
* images) original pixel dimensions, states exactly how the image was
|
||||
* delivered (untouched, downsampled, cropped, or native resolution) so
|
||||
* compression is never silent, guides the model to derive absolute
|
||||
* coordinates from the original size, and reminds it to re-read any media
|
||||
* it generates or edits.
|
||||
*
|
||||
* Images support two opt-in delivery controls: `region` cuts a rectangle
|
||||
* (original-image pixel coordinates) out of the file so fine detail survives
|
||||
* at full fidelity, and `full_resolution` skips the default downscale when
|
||||
* the payload fits the per-image byte budget (refusing explicitly when it
|
||||
* does not, instead of silently degrading).
|
||||
*
|
||||
* Path safety: goes through the shared path access resolver used by
|
||||
* Read/Write/Edit.
|
||||
*/
|
||||
|
|
@ -30,7 +38,13 @@ import type { ExecutableToolResult, ToolExecution } from '../../../loop/types';
|
|||
import { renderPrompt } from '../../../utils/render-prompt';
|
||||
import { resolvePathAccessPath } from '../../policies/path-access';
|
||||
import { MEDIA_SNIFF_BYTES, detectFileType, sniffImageDimensions } from '../../support/file-type';
|
||||
import { compressImageForModel } from '../../support/image-compress';
|
||||
import {
|
||||
IMAGE_BYTE_BUDGET,
|
||||
compressImageForModel,
|
||||
cropImageForModel,
|
||||
formatByteSize,
|
||||
type ImageCropRegion,
|
||||
} from '../../support/image-compress';
|
||||
import { toInputJsonSchema } from '../../support/input-schema';
|
||||
import { literalRulePattern, matchesPathRuleSubject } from '../../support/rule-match';
|
||||
import type { WorkspaceConfig } from '../../support/workspace';
|
||||
|
|
@ -55,6 +69,27 @@ export const ReadMediaFileInputSchema = z.object({
|
|||
'a path outside the working directory must be absolute. ' +
|
||||
'Directories and text files are not supported.',
|
||||
),
|
||||
region: z
|
||||
.object({
|
||||
x: z.number().int().min(0).describe('Left edge of the crop, in original-image pixels.'),
|
||||
y: z.number().int().min(0).describe('Top edge of the crop, in original-image pixels.'),
|
||||
width: z.number().int().min(1).describe('Crop width, in original-image pixels.'),
|
||||
height: z.number().int().min(1).describe('Crop height, in original-image pixels.'),
|
||||
})
|
||||
.optional()
|
||||
.describe(
|
||||
'Images only: view just this rectangle of the image (original-image pixel coordinates). ' +
|
||||
'Use after a downsampled full view to inspect fine detail — a region within the size ' +
|
||||
'limits is delivered at full fidelity.',
|
||||
),
|
||||
full_resolution: z
|
||||
.boolean()
|
||||
.optional()
|
||||
.describe(
|
||||
'Images only: skip the default downscaling and view at native resolution. Fails with an ' +
|
||||
'explicit error when the payload would exceed the per-image byte limit; use region for ' +
|
||||
'files that large.',
|
||||
),
|
||||
});
|
||||
|
||||
export type ReadMediaFileInput = z.Infer<typeof ReadMediaFileInputSchema>;
|
||||
|
|
@ -86,19 +121,40 @@ function buildDescription(capabilities: ModelCapability): string {
|
|||
|
||||
// ── System summary ───────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* How the image payload placed after the summary relates to the file on disk.
|
||||
* Reported verbatim so the model always knows when it is looking at a
|
||||
* degraded copy (and how to get the detail back) — silent downsampling reads
|
||||
* as "the image is just blurry" and quietly degrades the model's work.
|
||||
*/
|
||||
interface ImageDelivery {
|
||||
readonly kind: 'untouched' | 'downsampled' | 'crop' | 'full';
|
||||
/** Pixel size of the payload actually sent; 0 when unknown. */
|
||||
readonly width: number;
|
||||
readonly height: number;
|
||||
readonly byteLength: number;
|
||||
readonly mimeType: string;
|
||||
/** The crop actually applied (clamped), for kind 'crop'. */
|
||||
readonly region?: ImageCropRegion;
|
||||
/** For kind 'crop': the crop was additionally downscaled to fit budgets. */
|
||||
readonly resized?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the `<system>` summary that precedes the media content.
|
||||
*
|
||||
* Carries mime type, byte size and (for images) the original pixel
|
||||
* dimensions. When the dimensions are known it also guides the model to
|
||||
* derive absolute coordinates from that original size; it always reminds
|
||||
* the model to re-read any media it generates or edits.
|
||||
* dimensions, plus the delivery note above. When the dimensions are known it
|
||||
* also guides the model to derive absolute coordinates from that original
|
||||
* size (crops get offset-mapping guidance instead); it always reminds the
|
||||
* model to re-read any media it generates or edits.
|
||||
*/
|
||||
function buildSystemSummary(input: {
|
||||
readonly kind: 'image' | 'video';
|
||||
readonly mimeType: string;
|
||||
readonly byteSize: number;
|
||||
readonly dimensions: { readonly width: number; readonly height: number } | null;
|
||||
readonly delivery?: ImageDelivery | undefined;
|
||||
}): string {
|
||||
const parts: string[] = [
|
||||
`Read ${input.kind} file.`,
|
||||
|
|
@ -111,6 +167,34 @@ function buildSystemSummary(input: {
|
|||
if (input.kind === 'image' && input.dimensions) {
|
||||
parts.push(
|
||||
`Original dimensions: ${String(input.dimensions.width)}x${String(input.dimensions.height)} pixels.`,
|
||||
);
|
||||
}
|
||||
const delivery = input.delivery;
|
||||
if (delivery?.kind === 'downsampled') {
|
||||
parts.push(
|
||||
`The image below was downsampled to ${String(delivery.width)}x${String(delivery.height)} pixels ` +
|
||||
`(${delivery.mimeType}, ${formatByteSize(delivery.byteLength)}) to fit model limits; ` +
|
||||
'fine detail may be lost.',
|
||||
'To inspect fine detail, call ReadMediaFile again with the region parameter ' +
|
||||
'(original-image pixel coordinates) to view a crop at full fidelity.',
|
||||
);
|
||||
} else if (delivery?.kind === 'crop' && delivery.region) {
|
||||
const { x, y, width, height } = delivery.region;
|
||||
parts.push(
|
||||
`Showing region (x=${String(x)}, y=${String(y)}, width=${String(width)}, height=${String(height)}) ` +
|
||||
`of the original image${
|
||||
delivery.resized === true
|
||||
? `, downsampled to ${String(delivery.width)}x${String(delivery.height)} pixels`
|
||||
: ' at native resolution'
|
||||
}.`,
|
||||
'To output coordinates in original-image pixels, locate them within this crop and add ' +
|
||||
`the region offset (x=${String(x)}, y=${String(y)}).`,
|
||||
);
|
||||
} else if (delivery?.kind === 'full') {
|
||||
parts.push('Shown at native resolution; no downscaling applied.');
|
||||
}
|
||||
if (input.kind === 'image' && input.dimensions && delivery?.kind !== 'crop') {
|
||||
parts.push(
|
||||
'If you need to output coordinates, output relative coordinates first ' +
|
||||
'and compute absolute coordinates using the original image size.',
|
||||
);
|
||||
|
|
@ -222,19 +306,93 @@ export class ReadMediaFileTool implements BuiltinTool<ReadMediaFileInput> {
|
|||
};
|
||||
}
|
||||
|
||||
const data = await this.kaos.readBytes(safePath);
|
||||
let mediaPart: ContentPart;
|
||||
if (fileType.kind === 'image') {
|
||||
// Shrink oversized images so a large screenshot neither wastes context
|
||||
// tokens nor trips the provider's per-image byte ceiling. Best effort:
|
||||
// on any failure compressImageForModel returns the original bytes, so
|
||||
// the read still succeeds with the uncompressed image.
|
||||
const compressed = await compressImageForModel(data, fileType.mimeType);
|
||||
const base64 = Buffer.from(compressed.data).toString('base64');
|
||||
mediaPart = {
|
||||
type: 'image_url',
|
||||
imageUrl: { url: `data:${compressed.mimeType};base64,${base64}` },
|
||||
if (fileType.kind === 'video' && (args.region !== undefined || args.full_resolution === true)) {
|
||||
return {
|
||||
isError: true,
|
||||
output: 'region and full_resolution apply only to image files.',
|
||||
};
|
||||
}
|
||||
|
||||
const data = await this.kaos.readBytes(safePath);
|
||||
// The summary always reports the ORIGINAL pixel size and byte size: the
|
||||
// model derives relative coordinates and scales them by the original
|
||||
// dimensions, so it must see the pre-compression size even when the
|
||||
// image_url below carries a downsampled copy.
|
||||
let dimensions = fileType.kind === 'image' ? sniffImageDimensions(data) : null;
|
||||
let mediaPart: ContentPart;
|
||||
let delivery: ImageDelivery | undefined;
|
||||
if (fileType.kind === 'image') {
|
||||
if (args.region !== undefined) {
|
||||
// Explicit crop: read a rectangle of the original back, typically at
|
||||
// full fidelity, so a prior downsampled view can be zoomed into.
|
||||
const outcome = await cropImageForModel(data, fileType.mimeType, args.region, {
|
||||
skipResize: args.full_resolution === true,
|
||||
});
|
||||
if (!outcome.ok) {
|
||||
return { isError: true, output: `Cannot read region from "${args.path}": ${outcome.error}` };
|
||||
}
|
||||
const base64 = Buffer.from(outcome.data).toString('base64');
|
||||
mediaPart = {
|
||||
type: 'image_url',
|
||||
imageUrl: { url: `data:${outcome.mimeType};base64,${base64}` },
|
||||
};
|
||||
delivery = {
|
||||
kind: 'crop',
|
||||
width: outcome.width,
|
||||
height: outcome.height,
|
||||
byteLength: outcome.finalByteLength,
|
||||
mimeType: outcome.mimeType,
|
||||
region: outcome.region,
|
||||
resized: outcome.resized,
|
||||
};
|
||||
// The decode knows the true size even when header sniffing does not.
|
||||
dimensions ??= { width: outcome.originalWidth, height: outcome.originalHeight };
|
||||
} else if (args.full_resolution === true) {
|
||||
// Native resolution on request — but the provider's per-image byte
|
||||
// ceiling is a hard limit, so refuse explicitly rather than degrade.
|
||||
// Exact byte counts accompany the rounded sizes: a file a hair over
|
||||
// budget would otherwise read "is 3.8 MB, over the 3.8 MB limit".
|
||||
if (data.length > IMAGE_BYTE_BUDGET) {
|
||||
return {
|
||||
isError: true,
|
||||
output:
|
||||
`"${args.path}" is ${String(data.length)} bytes (${formatByteSize(data.length)}), ` +
|
||||
`over the ${String(IMAGE_BYTE_BUDGET)}-byte (${formatByteSize(IMAGE_BYTE_BUDGET)}) ` +
|
||||
'per-image limit, so full_resolution cannot be honored. ' +
|
||||
'Use region to view a crop at full fidelity instead.',
|
||||
};
|
||||
}
|
||||
const base64 = Buffer.from(data).toString('base64');
|
||||
mediaPart = {
|
||||
type: 'image_url',
|
||||
imageUrl: { url: `data:${fileType.mimeType};base64,${base64}` },
|
||||
};
|
||||
delivery = {
|
||||
kind: 'full',
|
||||
width: dimensions?.width ?? 0,
|
||||
height: dimensions?.height ?? 0,
|
||||
byteLength: data.length,
|
||||
mimeType: fileType.mimeType,
|
||||
};
|
||||
} else {
|
||||
// Shrink oversized images so a large screenshot neither wastes context
|
||||
// tokens nor trips the provider's per-image byte ceiling. Best effort:
|
||||
// on any failure compressImageForModel returns the original bytes, so
|
||||
// the read still succeeds with the uncompressed image.
|
||||
const compressed = await compressImageForModel(data, fileType.mimeType);
|
||||
const base64 = Buffer.from(compressed.data).toString('base64');
|
||||
mediaPart = {
|
||||
type: 'image_url',
|
||||
imageUrl: { url: `data:${compressed.mimeType};base64,${base64}` },
|
||||
};
|
||||
delivery = {
|
||||
kind: compressed.changed ? 'downsampled' : 'untouched',
|
||||
width: compressed.width,
|
||||
height: compressed.height,
|
||||
byteLength: compressed.finalByteLength,
|
||||
mimeType: compressed.mimeType,
|
||||
};
|
||||
}
|
||||
} else if (this.videoUploader !== undefined) {
|
||||
mediaPart = await this.videoUploader({
|
||||
data,
|
||||
|
|
@ -253,17 +411,12 @@ export class ReadMediaFileTool implements BuiltinTool<ReadMediaFileInput> {
|
|||
const openText = `<${tag} path="${safePath}">`;
|
||||
const closeText = `</${tag}>`;
|
||||
|
||||
// The summary always reports the ORIGINAL pixel size and byte size: the
|
||||
// model derives relative coordinates and scales them by the original
|
||||
// dimensions, so it must see the pre-compression size even when the
|
||||
// image_url above carries a downsampled copy.
|
||||
const dimensions =
|
||||
fileType.kind === 'image' ? sniffImageDimensions(data) : null;
|
||||
const systemText = buildSystemSummary({
|
||||
kind: fileType.kind,
|
||||
mimeType: fileType.mimeType,
|
||||
byteSize: stat.stSize,
|
||||
dimensions,
|
||||
delivery,
|
||||
});
|
||||
|
||||
const output: ContentPart[] = [
|
||||
|
|
|
|||
|
|
@ -16,6 +16,11 @@
|
|||
* - Only PNG and JPEG are re-encoded. GIF is passed through to preserve
|
||||
* animation; WebP is passed through because the default jimp build ships no
|
||||
* WebP codec. Unknown formats are passed through.
|
||||
* - Compression must never be silent to the model: results carry the
|
||||
* original dimensions, {@link buildImageCompressionCaption} renders the
|
||||
* shared "what was compressed, where is the original" note every ingestion
|
||||
* point can place next to the image, and {@link cropImageForModel} lets a
|
||||
* caller read a region of the original back at full fidelity.
|
||||
*/
|
||||
|
||||
import type { ContentPart } from '@moonshot-ai/kosong';
|
||||
|
|
@ -82,6 +87,10 @@ export interface CompressImageResult {
|
|||
readonly width: number;
|
||||
/** Pixel height of `data`; falls back to the input size when unknown. */
|
||||
readonly height: number;
|
||||
/** Pixel width of the input image; 0 when it cannot be sniffed. */
|
||||
readonly originalWidth: number;
|
||||
/** Pixel height of the input image; 0 when it cannot be sniffed. */
|
||||
readonly originalHeight: number;
|
||||
/** True only when `data` differs from the input bytes. */
|
||||
readonly changed: boolean;
|
||||
readonly originalByteLength: number;
|
||||
|
|
@ -111,6 +120,8 @@ export async function compressImageForModel(
|
|||
mimeType,
|
||||
width: dims?.width ?? 0,
|
||||
height: dims?.height ?? 0,
|
||||
originalWidth: dims?.width ?? 0,
|
||||
originalHeight: dims?.height ?? 0,
|
||||
changed: false,
|
||||
originalByteLength: bytes.length,
|
||||
finalByteLength: bytes.length,
|
||||
|
|
@ -162,6 +173,8 @@ export async function compressImageForModel(
|
|||
mimeType: encoded.mimeType,
|
||||
width: encoded.width,
|
||||
height: encoded.height,
|
||||
originalWidth: dims?.width ?? 0,
|
||||
originalHeight: dims?.height ?? 0,
|
||||
changed: true,
|
||||
originalByteLength: bytes.length,
|
||||
finalByteLength: encoded.data.length,
|
||||
|
|
@ -175,6 +188,14 @@ export async function compressImageForModel(
|
|||
export interface CompressBase64Result {
|
||||
readonly base64: string;
|
||||
readonly mimeType: string;
|
||||
/** Pixel width of the (possibly re-encoded) payload; 0 when unknown. */
|
||||
readonly width: number;
|
||||
/** Pixel height of the (possibly re-encoded) payload; 0 when unknown. */
|
||||
readonly height: number;
|
||||
/** Pixel width of the input image; 0 when it cannot be sniffed. */
|
||||
readonly originalWidth: number;
|
||||
/** Pixel height of the input image; 0 when it cannot be sniffed. */
|
||||
readonly originalHeight: number;
|
||||
readonly changed: boolean;
|
||||
readonly originalByteLength: number;
|
||||
readonly finalByteLength: number;
|
||||
|
|
@ -199,6 +220,10 @@ export async function compressBase64ForModel(
|
|||
return {
|
||||
base64,
|
||||
mimeType,
|
||||
width: 0,
|
||||
height: 0,
|
||||
originalWidth: 0,
|
||||
originalHeight: 0,
|
||||
changed: false,
|
||||
originalByteLength: approxBytes,
|
||||
finalByteLength: approxBytes,
|
||||
|
|
@ -211,6 +236,10 @@ export async function compressBase64ForModel(
|
|||
return {
|
||||
base64,
|
||||
mimeType,
|
||||
width: 0,
|
||||
height: 0,
|
||||
originalWidth: 0,
|
||||
originalHeight: 0,
|
||||
changed: false,
|
||||
originalByteLength: 0,
|
||||
finalByteLength: 0,
|
||||
|
|
@ -221,6 +250,10 @@ export async function compressBase64ForModel(
|
|||
return {
|
||||
base64,
|
||||
mimeType,
|
||||
width: result.width,
|
||||
height: result.height,
|
||||
originalWidth: result.originalWidth,
|
||||
originalHeight: result.originalHeight,
|
||||
changed: false,
|
||||
originalByteLength: result.originalByteLength,
|
||||
finalByteLength: result.finalByteLength,
|
||||
|
|
@ -229,6 +262,10 @@ export async function compressBase64ForModel(
|
|||
return {
|
||||
base64: Buffer.from(result.data).toString('base64'),
|
||||
mimeType: result.mimeType,
|
||||
width: result.width,
|
||||
height: result.height,
|
||||
originalWidth: result.originalWidth,
|
||||
originalHeight: result.originalHeight,
|
||||
changed: true,
|
||||
originalByteLength: result.originalByteLength,
|
||||
finalByteLength: result.finalByteLength,
|
||||
|
|
@ -241,18 +278,57 @@ export async function compressBase64ForModel(
|
|||
* the MCP tool-result path. Image parts whose URL is not a `data:` URL (e.g. a
|
||||
* remote http(s) image) are passed through, as are non-image parts. Best
|
||||
* effort: a part that fails to compress is left unchanged.
|
||||
*
|
||||
* With `annotate` set, every image that was actually re-encoded gains a
|
||||
* {@link buildImageCompressionCaption} text part immediately before it, so the
|
||||
* model knows it is looking at a downsampled copy. `annotate.persistOriginal`
|
||||
* additionally saves the pre-compression bytes and puts the returned path in
|
||||
* the caption so the model can read the original back; persistence failures
|
||||
* degrade to a caption without a path.
|
||||
*/
|
||||
export async function compressImageContentParts(
|
||||
parts: readonly ContentPart[],
|
||||
options: CompressImageOptions = {},
|
||||
options: CompressImageOptions & { readonly annotate?: CompressAnnotateOptions } = {},
|
||||
): Promise<ContentPart[]> {
|
||||
const { annotate, ...compressOptions } = options;
|
||||
const out: ContentPart[] = [];
|
||||
for (const part of parts) {
|
||||
if (part.type === 'image_url') {
|
||||
const parsed = parseImageDataUrl(part.imageUrl.url);
|
||||
if (parsed !== null) {
|
||||
const result = await compressBase64ForModel(parsed.base64, parsed.mimeType, options);
|
||||
const result = await compressBase64ForModel(parsed.base64, parsed.mimeType, compressOptions);
|
||||
if (result.changed) {
|
||||
if (annotate !== undefined) {
|
||||
let originalPath: string | null = null;
|
||||
if (annotate.persistOriginal !== undefined) {
|
||||
try {
|
||||
originalPath = await annotate.persistOriginal(
|
||||
Buffer.from(parsed.base64, 'base64'),
|
||||
parsed.mimeType,
|
||||
);
|
||||
} catch {
|
||||
originalPath = null;
|
||||
}
|
||||
}
|
||||
out.push({
|
||||
type: 'text',
|
||||
text: buildImageCompressionCaption({
|
||||
original: {
|
||||
width: result.originalWidth,
|
||||
height: result.originalHeight,
|
||||
byteLength: result.originalByteLength,
|
||||
mimeType: parsed.mimeType,
|
||||
},
|
||||
final: {
|
||||
width: result.width,
|
||||
height: result.height,
|
||||
byteLength: result.finalByteLength,
|
||||
mimeType: result.mimeType,
|
||||
},
|
||||
originalPath,
|
||||
}),
|
||||
});
|
||||
}
|
||||
out.push({
|
||||
type: 'image_url',
|
||||
imageUrl: { ...part.imageUrl, url: `data:${result.mimeType};base64,${result.base64}` },
|
||||
|
|
@ -266,6 +342,251 @@ export async function compressImageContentParts(
|
|||
return out;
|
||||
}
|
||||
|
||||
export interface CompressAnnotateOptions {
|
||||
/**
|
||||
* Persist the pre-compression original bytes somewhere the model can read
|
||||
* them back; return the absolute path, or null when persistence failed.
|
||||
*/
|
||||
readonly persistOriginal?: (bytes: Uint8Array, mimeType: string) => Promise<string | null>;
|
||||
}
|
||||
|
||||
// ── crop ─────────────────────────────────────────────────────────────
|
||||
|
||||
/** Crop rectangle in ORIGINAL-image pixel coordinates. */
|
||||
export interface ImageCropRegion {
|
||||
readonly x: number;
|
||||
readonly y: number;
|
||||
readonly width: number;
|
||||
readonly height: number;
|
||||
}
|
||||
|
||||
export interface CropImageOptions extends CompressImageOptions {
|
||||
/**
|
||||
* Keep the crop at native resolution (no edge-fit downscale). The byte
|
||||
* budget still applies: a crop that cannot be encoded within it fails
|
||||
* explicitly instead of being silently degraded.
|
||||
*/
|
||||
readonly skipResize?: boolean;
|
||||
}
|
||||
|
||||
export interface CropImageSuccess {
|
||||
readonly ok: true;
|
||||
readonly data: Uint8Array;
|
||||
readonly mimeType: string;
|
||||
/** Pixel size of the encoded crop actually produced. */
|
||||
readonly width: number;
|
||||
readonly height: number;
|
||||
/** Pixel size of the source image the region was cut from. */
|
||||
readonly originalWidth: number;
|
||||
readonly originalHeight: number;
|
||||
/** The region actually applied, after clamping to the image bounds. */
|
||||
readonly region: ImageCropRegion;
|
||||
/** True when the crop was downscaled to fit the pixel/byte budget. */
|
||||
readonly resized: boolean;
|
||||
readonly originalByteLength: number;
|
||||
readonly finalByteLength: number;
|
||||
}
|
||||
|
||||
export interface CropImageFailure {
|
||||
readonly ok: false;
|
||||
/** Human/model-readable reason, safe to surface as a tool error. */
|
||||
readonly error: string;
|
||||
}
|
||||
|
||||
export type CropImageOutcome = CropImageSuccess | CropImageFailure;
|
||||
|
||||
/**
|
||||
* Cut `region` out of `bytes` and encode it for the model.
|
||||
*
|
||||
* Unlike {@link compressImageForModel}, cropping is an explicit request: it
|
||||
* never falls back to the full image. Anything that prevents an accurate crop
|
||||
* (unsupported format, undecodable bytes, a region outside the image, a
|
||||
* skipResize result over the byte budget) returns `ok: false` with a reason
|
||||
* the caller can hand straight back to the model.
|
||||
*
|
||||
* The default path fits the crop to the usual pixel/byte budgets; a crop no
|
||||
* larger than the edge cap is therefore delivered at native resolution.
|
||||
*/
|
||||
export async function cropImageForModel(
|
||||
bytes: Uint8Array,
|
||||
mimeType: string,
|
||||
region: ImageCropRegion,
|
||||
options: CropImageOptions = {},
|
||||
): Promise<CropImageOutcome> {
|
||||
const maxEdge = options.maxEdge ?? MAX_IMAGE_EDGE_PX;
|
||||
const byteBudget = options.byteBudget ?? IMAGE_BYTE_BUDGET;
|
||||
const maxDecodeBytes = options.maxDecodeBytes ?? MAX_DECODE_BYTES;
|
||||
const normalizedMime = normalizeMime(mimeType);
|
||||
|
||||
if (bytes.length === 0) {
|
||||
return { ok: false, error: 'The image is empty.' };
|
||||
}
|
||||
if (!RECODABLE_MIME.has(normalizedMime)) {
|
||||
return {
|
||||
ok: false,
|
||||
error: `Cropping is only supported for PNG and JPEG images; got ${mimeType}.`,
|
||||
};
|
||||
}
|
||||
// NaN slips past every </>= comparison in the bounds guard below, so gate
|
||||
// on finiteness explicitly rather than surfacing a codec-internal error.
|
||||
if (
|
||||
![region.x, region.y, region.width, region.height].every((value) => Number.isFinite(value))
|
||||
) {
|
||||
return {
|
||||
ok: false,
|
||||
error:
|
||||
`Region coordinates must be finite numbers; got x=${String(region.x)}, ` +
|
||||
`y=${String(region.y)}, width=${String(region.width)}, height=${String(region.height)}.`,
|
||||
};
|
||||
}
|
||||
const dims = sniffImageDimensions(bytes);
|
||||
if (dims && dims.width * dims.height > MAX_DECODE_PIXELS) {
|
||||
return {
|
||||
ok: false,
|
||||
error: `The image (${String(dims.width)}x${String(dims.height)} pixels) is too large to decode for cropping.`,
|
||||
};
|
||||
}
|
||||
if (bytes.length > maxDecodeBytes) {
|
||||
return { ok: false, error: 'The image is too large to decode for cropping.' };
|
||||
}
|
||||
|
||||
try {
|
||||
const { Jimp } = await import('jimp');
|
||||
const image = await Jimp.fromBuffer(Buffer.from(bytes));
|
||||
const originalWidth = image.width;
|
||||
const originalHeight = image.height;
|
||||
|
||||
const x = Math.floor(region.x);
|
||||
const y = Math.floor(region.y);
|
||||
if (x < 0 || y < 0 || x >= originalWidth || y >= originalHeight || region.width < 1 || region.height < 1) {
|
||||
return {
|
||||
ok: false,
|
||||
error:
|
||||
`Region (x=${String(region.x)}, y=${String(region.y)}, width=${String(region.width)}, ` +
|
||||
`height=${String(region.height)}) lies outside the ${String(originalWidth)}x${String(originalHeight)} image.`,
|
||||
};
|
||||
}
|
||||
const w = Math.min(Math.floor(region.width), originalWidth - x);
|
||||
const h = Math.min(Math.floor(region.height), originalHeight - y);
|
||||
const applied: ImageCropRegion = { x, y, width: w, height: h };
|
||||
image.crop({ x, y, w, h });
|
||||
const sourceIsPng = normalizedMime === 'image/png';
|
||||
|
||||
if (options.skipResize === true) {
|
||||
// Native resolution requested: encode once, favoring fidelity (lossless
|
||||
// PNG, or high-quality JPEG), and refuse rather than degrade when the
|
||||
// result cannot fit the byte budget.
|
||||
const buffer = sourceIsPng
|
||||
? await image.getBuffer('image/png', { deflateLevel: 9 })
|
||||
: await image.getBuffer('image/jpeg', { quality: 90 });
|
||||
if (buffer.length > byteBudget) {
|
||||
return {
|
||||
ok: false,
|
||||
error:
|
||||
`The cropped region encodes to ${String(buffer.length)} bytes ` +
|
||||
`(${formatByteSize(buffer.length)}), over the ${String(byteBudget)}-byte ` +
|
||||
`(${formatByteSize(byteBudget)}) per-image limit. ` +
|
||||
'Choose a smaller region, or allow downscaling.',
|
||||
};
|
||||
}
|
||||
return {
|
||||
ok: true,
|
||||
data: new Uint8Array(buffer),
|
||||
mimeType: sourceIsPng ? 'image/png' : 'image/jpeg',
|
||||
width: image.width,
|
||||
height: image.height,
|
||||
originalWidth,
|
||||
originalHeight,
|
||||
region: applied,
|
||||
resized: false,
|
||||
originalByteLength: bytes.length,
|
||||
finalByteLength: buffer.length,
|
||||
};
|
||||
}
|
||||
|
||||
fitWithinEdge(image, maxEdge);
|
||||
const encoded = await encodeWithinBudget(image, {
|
||||
sourceIsPng,
|
||||
byteBudget,
|
||||
fallbackEdge: FALLBACK_EDGE_PX,
|
||||
});
|
||||
return {
|
||||
ok: true,
|
||||
data: new Uint8Array(encoded.data),
|
||||
mimeType: encoded.mimeType,
|
||||
width: encoded.width,
|
||||
height: encoded.height,
|
||||
originalWidth,
|
||||
originalHeight,
|
||||
region: applied,
|
||||
resized: encoded.width !== w || encoded.height !== h,
|
||||
originalByteLength: bytes.length,
|
||||
finalByteLength: encoded.data.length,
|
||||
};
|
||||
} catch (error) {
|
||||
return {
|
||||
ok: false,
|
||||
error: `Failed to decode the image for cropping: ${error instanceof Error ? error.message : String(error)}`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// ── compression caption ──────────────────────────────────────────────
|
||||
|
||||
export interface ImageVariantDescription {
|
||||
/** Pixel size; pass 0 when unknown to omit the dimensions. */
|
||||
readonly width: number;
|
||||
readonly height: number;
|
||||
readonly byteLength: number;
|
||||
readonly mimeType: string;
|
||||
}
|
||||
|
||||
export interface ImageCompressionCaptionInput {
|
||||
readonly original: ImageVariantDescription;
|
||||
readonly final: ImageVariantDescription;
|
||||
/** Absolute path where the pre-compression original can be read back. */
|
||||
readonly originalPath?: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Render the shared `<system>` note placed next to a compressed image so the
|
||||
* model knows it is looking at a downsampled copy: what the original was, what
|
||||
* was actually sent, and — when the original is on disk — where to read it
|
||||
* back (via ReadMediaFile `region`) for full-fidelity detail.
|
||||
*/
|
||||
export function buildImageCompressionCaption(input: ImageCompressionCaptionInput): string {
|
||||
const sentences = [
|
||||
`Image compressed to fit model limits: original ${describeImageVariant(input.original)} -> ` +
|
||||
`sent ${describeImageVariant(input.final)}.`,
|
||||
'Fine detail may be lost.',
|
||||
];
|
||||
if (typeof input.originalPath === 'string' && input.originalPath.length > 0) {
|
||||
sentences.push(
|
||||
`The uncompressed original is saved at "${input.originalPath}"; if you need fine detail ` +
|
||||
'(e.g. small text), call ReadMediaFile on that path with the region parameter ' +
|
||||
'(original-pixel coordinates) to view a crop at full fidelity.',
|
||||
);
|
||||
} else {
|
||||
sentences.push('The uncompressed original was not preserved.');
|
||||
}
|
||||
return `<system>${sentences.join(' ')}</system>`;
|
||||
}
|
||||
|
||||
function describeImageVariant(variant: ImageVariantDescription): string {
|
||||
const size = `${variant.mimeType} (${formatByteSize(variant.byteLength)})`;
|
||||
if (variant.width > 0 && variant.height > 0) {
|
||||
return `${String(variant.width)}x${String(variant.height)} ${size}`;
|
||||
}
|
||||
return size;
|
||||
}
|
||||
|
||||
/** Human-readable byte size: `640 B`, `128 KB`, `3.8 MB`. */
|
||||
export function formatByteSize(bytes: number): string {
|
||||
if (bytes < 1024) return `${String(bytes)} B`;
|
||||
if (bytes < 1024 * 1024) return `${String(Math.round(bytes / 1024))} KB`;
|
||||
return `${(bytes / (1024 * 1024)).toFixed(1)} MB`;
|
||||
}
|
||||
|
||||
function parseImageDataUrl(url: string): { mimeType: string; base64: string } | null {
|
||||
const match = /^data:([^;,]+);base64,(.*)$/s.exec(url);
|
||||
if (match === null) return null;
|
||||
|
|
|
|||
125
packages/agent-core/src/tools/support/image-originals.ts
Normal file
125
packages/agent-core/src/tools/support/image-originals.ts
Normal file
|
|
@ -0,0 +1,125 @@
|
|||
/**
|
||||
* Content-addressed store for pre-compression image originals.
|
||||
*
|
||||
* When an ingestion point (MCP tool result, pasted image, inline base64
|
||||
* upload) compresses an image that exists only in memory, the original bytes
|
||||
* would be gone for good — the model could never zoom into a detail the
|
||||
* downsampled copy lost. This module persists those originals so the
|
||||
* compression caption can point at a real path the model can read back with
|
||||
* `ReadMediaFile` (typically with `region`).
|
||||
*
|
||||
* Placement: callers that know their session pass
|
||||
* `{ dir: sessionMediaOriginalsDir(sessionDir) }` so originals live at
|
||||
* `<sessionDir>/media-originals/` — owned by the session, cleaned up with it,
|
||||
* and immune to OS temp reaping. The shared temp-dir cache
|
||||
* ({@link originalImageCacheDir}) is only the fallback for call sites with no
|
||||
* session context.
|
||||
*
|
||||
* Design notes:
|
||||
* - Content-addressed (sha256): duplicate pastes/results reuse one file and
|
||||
* repeated writes are idempotent.
|
||||
* - Best effort: any filesystem failure returns null; callers then emit a
|
||||
* caption without a readback path. Persistence must never block a prompt.
|
||||
* - Size-capped: after each write the store is swept oldest-first (mtime)
|
||||
* until it fits {@link DEFAULT_MAX_TOTAL_BYTES}, so long sessions cannot
|
||||
* fill the disk.
|
||||
*/
|
||||
|
||||
import { createHash } from 'node:crypto';
|
||||
import { mkdir, readdir, stat, unlink, writeFile } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
|
||||
/** Per-store ceiling; the sweep evicts oldest files beyond this. */
|
||||
const DEFAULT_MAX_TOTAL_BYTES = 1024 * 1024 * 1024; // 1 GiB
|
||||
|
||||
const MIME_EXTENSION: Readonly<Record<string, string>> = {
|
||||
'image/png': 'png',
|
||||
'image/jpeg': 'jpg',
|
||||
'image/jpg': 'jpg',
|
||||
'image/gif': 'gif',
|
||||
'image/webp': 'webp',
|
||||
'image/bmp': 'bmp',
|
||||
'image/tiff': 'tif',
|
||||
};
|
||||
|
||||
export interface PersistOriginalImageOptions {
|
||||
/**
|
||||
* Target directory — pass `sessionMediaOriginalsDir(sessionDir)` when the
|
||||
* session is known. Defaults to the shared temp-dir fallback.
|
||||
*/
|
||||
readonly dir?: string;
|
||||
/** Override the store size cap in bytes (tests). */
|
||||
readonly maxTotalBytes?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Fallback store used when a call site has no session context:
|
||||
* `<os-tmp>/kimi-code-original-images`.
|
||||
*/
|
||||
export function originalImageCacheDir(): string {
|
||||
return join(tmpdir(), 'kimi-code-original-images');
|
||||
}
|
||||
|
||||
/**
|
||||
* The session-owned originals store: `<sessionDir>/media-originals`. Sits
|
||||
* next to the session's other artifacts (`tasks/`, `cron/`, `logs/`,
|
||||
* `agents/`) and is removed with the session.
|
||||
*/
|
||||
export function sessionMediaOriginalsDir(sessionDir: string): string {
|
||||
return join(sessionDir, 'media-originals');
|
||||
}
|
||||
|
||||
/**
|
||||
* Persist `bytes` into the originals store and return the absolute file
|
||||
* path, or null on any failure. Idempotent for identical bytes.
|
||||
*/
|
||||
export async function persistOriginalImage(
|
||||
bytes: Uint8Array,
|
||||
mimeType: string,
|
||||
options: PersistOriginalImageOptions = {},
|
||||
): Promise<string | null> {
|
||||
if (bytes.length === 0) return null;
|
||||
const dir = options.dir ?? originalImageCacheDir();
|
||||
const maxTotalBytes = options.maxTotalBytes ?? DEFAULT_MAX_TOTAL_BYTES;
|
||||
try {
|
||||
const hash = createHash('sha256').update(bytes).digest('hex').slice(0, 32);
|
||||
const extension = MIME_EXTENSION[mimeType.trim().toLowerCase()] ?? 'img';
|
||||
const path = join(dir, `${hash}.${extension}`);
|
||||
await mkdir(dir, { recursive: true });
|
||||
|
||||
const existing = await stat(path).catch(() => null);
|
||||
// Content-addressed: an existing entry with the right size IS this image.
|
||||
if (existing === null || existing.size !== bytes.length) {
|
||||
await writeFile(path, bytes);
|
||||
}
|
||||
|
||||
await sweepCache(dir, maxTotalBytes);
|
||||
// The just-written file may itself have been evicted by the sweep when a
|
||||
// single original exceeds the cap; report persistence honestly.
|
||||
const persisted = await stat(path).catch(() => null);
|
||||
return persisted === null ? null : path;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** Evict oldest files (by mtime) until the store fits `maxTotalBytes`. */
|
||||
async function sweepCache(dir: string, maxTotalBytes: number): Promise<void> {
|
||||
const names = await readdir(dir);
|
||||
const entries: { path: string; size: number; mtimeMs: number }[] = [];
|
||||
for (const name of names) {
|
||||
const path = join(dir, name);
|
||||
const info = await stat(path).catch(() => null);
|
||||
if (info === null || !info.isFile()) continue;
|
||||
entries.push({ path, size: info.size, mtimeMs: info.mtimeMs });
|
||||
}
|
||||
let total = entries.reduce((sum, entry) => sum + entry.size, 0);
|
||||
if (total <= maxTotalBytes) return;
|
||||
entries.sort((a, b) => a.mtimeMs - b.mtimeMs);
|
||||
for (const entry of entries) {
|
||||
if (total <= maxTotalBytes) break;
|
||||
await unlink(entry.path).catch(() => undefined);
|
||||
total -= entry.size;
|
||||
}
|
||||
}
|
||||
|
|
@ -1,6 +1,9 @@
|
|||
import { ContentBlockSchema } from '@modelcontextprotocol/sdk/types.js';
|
||||
import type { ContentPart } from '@moonshot-ai/kosong';
|
||||
import { Jimp } from 'jimp';
|
||||
import { mkdtemp, readFile, rm, unlink } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { describe, expect, test } from 'vitest';
|
||||
|
||||
import { convertMCPContentBlock, mcpResultToExecutableOutput } from '../../src/mcp/output';
|
||||
|
|
@ -355,4 +358,137 @@ describe('mcpResultToExecutableOutput', () => {
|
|||
const joined = parts.map((p) => (p.type === 'text' ? p.text : '')).join('');
|
||||
expect(joined).not.toContain('image_url dropped');
|
||||
});
|
||||
|
||||
test('annotates a downsampled image with a caption and a readable original', async () => {
|
||||
const bigBytes = Buffer.from(
|
||||
await new Jimp({ width: 2600, height: 2600, color: 0x3366ccff }).getBuffer('image/png'),
|
||||
);
|
||||
|
||||
const out = await mcpResultToExecutableOutput(
|
||||
result([{ type: 'image', data: bigBytes.toString('base64'), mimeType: 'image/png' }]),
|
||||
'mcp__s__shot',
|
||||
);
|
||||
|
||||
const parts = out.output as ContentPart[];
|
||||
const captionIndex = parts.findIndex(
|
||||
(p) => p.type === 'text' && p.text.includes('Image compressed'),
|
||||
);
|
||||
expect(captionIndex).toBeGreaterThanOrEqual(0);
|
||||
const caption = (parts[captionIndex] as { text: string }).text;
|
||||
expect(caption).toContain('2600x2600');
|
||||
// The caption sits immediately before the image it describes.
|
||||
expect(parts[captionIndex + 1]?.type).toBe('image_url');
|
||||
|
||||
// The caption points at a persisted copy of the ORIGINAL bytes, so the
|
||||
// model can read fine detail back with ReadMediaFile + region.
|
||||
const pathMatch = /saved at "([^"]+)"/.exec(caption);
|
||||
expect(pathMatch).not.toBeNull();
|
||||
const persisted = await readFile(pathMatch![1]!);
|
||||
expect(persisted.equals(bigBytes)).toBe(true);
|
||||
await unlink(pathMatch![1]!).catch(() => undefined);
|
||||
});
|
||||
|
||||
test('adds no caption for an image that passes through unchanged', async () => {
|
||||
const small = Buffer.from(
|
||||
await new Jimp({ width: 32, height: 32, color: 0x3366ccff }).getBuffer('image/png'),
|
||||
).toString('base64');
|
||||
|
||||
const out = await mcpResultToExecutableOutput(
|
||||
result([{ type: 'image', data: small, mimeType: 'image/png' }]),
|
||||
'mcp__s__shot',
|
||||
);
|
||||
|
||||
const parts = out.output as ContentPart[];
|
||||
const joined = parts.map((p) => (p.type === 'text' ? p.text : '')).join('');
|
||||
expect(joined).not.toContain('Image compressed');
|
||||
});
|
||||
|
||||
test('persists originals into the provided session originals dir', async () => {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'mcp-originals-'));
|
||||
const bigBytes = Buffer.from(
|
||||
await new Jimp({ width: 2600, height: 2600, color: 0x3366ccff }).getBuffer('image/png'),
|
||||
);
|
||||
|
||||
const out = await mcpResultToExecutableOutput(
|
||||
result([{ type: 'image', data: bigBytes.toString('base64'), mimeType: 'image/png' }]),
|
||||
'mcp__s__shot',
|
||||
{ originalsDir: dir },
|
||||
);
|
||||
|
||||
const parts = out.output as ContentPart[];
|
||||
const caption = parts.find((p) => p.type === 'text' && p.text.includes('Image compressed'));
|
||||
if (caption?.type !== 'text') throw new Error('expected a compression caption');
|
||||
const pathMatch = /saved at "([^"]+)"/.exec(caption.text);
|
||||
expect(pathMatch).not.toBeNull();
|
||||
// The original lands inside the session-scoped dir, not the tmp cache.
|
||||
expect(pathMatch![1]!.startsWith(dir)).toBe(true);
|
||||
const persisted = await readFile(pathMatch![1]!);
|
||||
expect(persisted.equals(bigBytes)).toBe(true);
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('keeps the caption intact when the tool text exhausts the 100K budget', async () => {
|
||||
// The caption must never compete with the tool's own text for the budget:
|
||||
// a chatty tool (page text + screenshot) would otherwise silently evict
|
||||
// it, reintroducing exactly the silent downsampling the caption exists
|
||||
// to prevent — and orphaning the persisted original.
|
||||
const dir = await mkdtemp(join(tmpdir(), 'mcp-originals-'));
|
||||
const big = Buffer.from(
|
||||
await new Jimp({ width: 2600, height: 2600, color: 0x3366ccff }).getBuffer('image/png'),
|
||||
).toString('base64');
|
||||
|
||||
const out = await mcpResultToExecutableOutput(
|
||||
result([
|
||||
{ type: 'text', text: 'x'.repeat(100_001) },
|
||||
{ type: 'image', data: big, mimeType: 'image/png' },
|
||||
]),
|
||||
'mcp__s__shot',
|
||||
{ originalsDir: dir },
|
||||
);
|
||||
|
||||
const parts = out.output as ContentPart[];
|
||||
// The tool text is truncated (with the notice), the image survives…
|
||||
expect(out.truncated).toBe(true);
|
||||
expect(parts.some((p) => p.type === 'image_url')).toBe(true);
|
||||
const toolText = parts[0];
|
||||
if (toolText?.type !== 'text') throw new Error('expected the tool text part first');
|
||||
expect(toolText.text).toContain('Output truncated');
|
||||
// …and the caption survives INTACT, still pointing at the original.
|
||||
const caption = parts.find((p) => p.type === 'text' && p.text.includes('Image compressed'));
|
||||
if (caption?.type !== 'text') throw new Error('expected a compression caption');
|
||||
expect(caption.text).toMatch(/<\/system>$/);
|
||||
expect(caption.text).toContain('saved at');
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('does not slice the caption when the budget is nearly exhausted', async () => {
|
||||
// 99,900 chars of tool text fit the budget on their own; the caption
|
||||
// must not be charged the remaining 100 chars and sliced mid-string
|
||||
// into an unclosed <system> fragment.
|
||||
const dir = await mkdtemp(join(tmpdir(), 'mcp-originals-'));
|
||||
const big = Buffer.from(
|
||||
await new Jimp({ width: 2600, height: 2600, color: 0x3366ccff }).getBuffer('image/png'),
|
||||
).toString('base64');
|
||||
|
||||
const out = await mcpResultToExecutableOutput(
|
||||
result([
|
||||
{ type: 'text', text: 'y'.repeat(99_900) },
|
||||
{ type: 'image', data: big, mimeType: 'image/png' },
|
||||
]),
|
||||
'mcp__s__shot',
|
||||
{ originalsDir: dir },
|
||||
);
|
||||
|
||||
const parts = out.output as ContentPart[];
|
||||
// The tool text fits — nothing is truncated at all.
|
||||
expect(out.truncated).toBeUndefined();
|
||||
const caption = parts.find((p) => p.type === 'text' && p.text.includes('Image compressed'));
|
||||
if (caption?.type !== 'text') throw new Error('expected a compression caption');
|
||||
expect(caption.text).toMatch(/^<system>Image compressed/);
|
||||
expect(caption.text).toMatch(/<\/system>$/);
|
||||
expect(caption.text).toContain('saved at');
|
||||
const joined = parts.map((p) => (p.type === 'text' ? p.text : '')).join('');
|
||||
expect(joined).not.toContain('Output truncated');
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -16,6 +16,14 @@
|
|||
* - base64 wrapper round-trips
|
||||
* - performance: the fast path is codec-free; a large image compresses
|
||||
* within a generous time bound
|
||||
* - metadata: results always carry the original pixel dimensions
|
||||
* - crop: cropImageForModel cuts a region at native resolution, clamps
|
||||
* overflow, refuses out-of-bounds/undecodable input explicitly, and
|
||||
* honors skipResize with a hard byte-budget failure
|
||||
* - caption: buildImageCompressionCaption renders a consistent
|
||||
* `<system>` note (dims, sizes, readback path)
|
||||
* - annotate: compressImageContentParts can insert that caption next to
|
||||
* each compressed image and persist the original via a callback
|
||||
*/
|
||||
|
||||
import { Jimp } from 'jimp';
|
||||
|
|
@ -23,9 +31,11 @@ import { describe, expect, it } from 'vitest';
|
|||
|
||||
// eslint-disable-next-line import/no-unresolved
|
||||
import {
|
||||
buildImageCompressionCaption,
|
||||
compressBase64ForModel,
|
||||
compressImageContentParts,
|
||||
compressImageForModel,
|
||||
cropImageForModel,
|
||||
IMAGE_BYTE_BUDGET,
|
||||
MAX_IMAGE_EDGE_PX,
|
||||
} from '../../src/tools/support/image-compress';
|
||||
|
|
@ -355,3 +365,283 @@ describe('compressImageContentParts', () => {
|
|||
expect(imagePart.imageUrl.url).not.toBe(dataUrl('image/png', big));
|
||||
});
|
||||
});
|
||||
|
||||
// ── original-dimension metadata ──────────────────────────────────────
|
||||
|
||||
describe('compressImageForModel — original dimensions metadata', () => {
|
||||
it('reports original dimensions on passthrough and compressed results', async () => {
|
||||
const small = await solidPng(64, 64);
|
||||
const pass = await compressImageForModel(small, 'image/png');
|
||||
expect(pass.changed).toBe(false);
|
||||
expect(pass.originalWidth).toBe(64);
|
||||
expect(pass.originalHeight).toBe(64);
|
||||
|
||||
const big = await solidPng(3000, 1500);
|
||||
const shrunk = await compressImageForModel(big, 'image/png');
|
||||
expect(shrunk.changed).toBe(true);
|
||||
expect(shrunk.originalWidth).toBe(3000);
|
||||
expect(shrunk.originalHeight).toBe(1500);
|
||||
expect(shrunk.width).toBe(2000);
|
||||
});
|
||||
|
||||
it('reports original dimensions through the base64 wrapper', async () => {
|
||||
const big = await solidPng(2600, 1300);
|
||||
const base64 = Buffer.from(big).toString('base64');
|
||||
const result = await compressBase64ForModel(base64, 'image/png');
|
||||
expect(result.changed).toBe(true);
|
||||
expect(result.originalWidth).toBe(2600);
|
||||
expect(result.originalHeight).toBe(1300);
|
||||
expect(result.width).toBe(2000);
|
||||
expect(result.height).toBe(1000);
|
||||
});
|
||||
});
|
||||
|
||||
// ── crop ─────────────────────────────────────────────────────────────
|
||||
|
||||
describe('cropImageForModel', () => {
|
||||
it('crops a region out of a PNG at native resolution', async () => {
|
||||
const png = await solidPng(3000, 1500);
|
||||
const result = await cropImageForModel(png, 'image/png', {
|
||||
x: 100,
|
||||
y: 200,
|
||||
width: 500,
|
||||
height: 400,
|
||||
});
|
||||
expect(result.ok).toBe(true);
|
||||
if (!result.ok) return;
|
||||
expect(result.width).toBe(500);
|
||||
expect(result.height).toBe(400);
|
||||
expect(result.originalWidth).toBe(3000);
|
||||
expect(result.originalHeight).toBe(1500);
|
||||
expect(result.region).toEqual({ x: 100, y: 200, width: 500, height: 400 });
|
||||
expect(result.resized).toBe(false);
|
||||
expect(result.mimeType).toBe('image/png');
|
||||
expect(sniffImageDimensions(result.data)).toEqual({ width: 500, height: 400 });
|
||||
});
|
||||
|
||||
it('preserves the JPEG format when cropping a JPEG', async () => {
|
||||
const jpeg = await solidJpeg(2400, 1200);
|
||||
const result = await cropImageForModel(jpeg, 'image/jpeg', {
|
||||
x: 0,
|
||||
y: 0,
|
||||
width: 300,
|
||||
height: 300,
|
||||
});
|
||||
expect(result.ok).toBe(true);
|
||||
if (!result.ok) return;
|
||||
expect(result.mimeType).toBe('image/jpeg');
|
||||
expect(result.width).toBe(300);
|
||||
expect(result.height).toBe(300);
|
||||
});
|
||||
|
||||
it('clamps a region that overflows the image bounds', async () => {
|
||||
const png = await solidPng(3000, 1500);
|
||||
const result = await cropImageForModel(png, 'image/png', {
|
||||
x: 2500,
|
||||
y: 1000,
|
||||
width: 1000,
|
||||
height: 1000,
|
||||
});
|
||||
expect(result.ok).toBe(true);
|
||||
if (!result.ok) return;
|
||||
expect(result.region).toEqual({ x: 2500, y: 1000, width: 500, height: 500 });
|
||||
expect(result.width).toBe(500);
|
||||
expect(result.height).toBe(500);
|
||||
});
|
||||
|
||||
it('rejects a region fully outside the image, naming the original size', async () => {
|
||||
const png = await solidPng(3000, 1500);
|
||||
const result = await cropImageForModel(png, 'image/png', {
|
||||
x: 3000,
|
||||
y: 0,
|
||||
width: 100,
|
||||
height: 100,
|
||||
});
|
||||
expect(result.ok).toBe(false);
|
||||
if (result.ok) return;
|
||||
expect(result.error).toContain('3000x1500');
|
||||
});
|
||||
|
||||
it('downscales an oversized crop to the edge cap by default', async () => {
|
||||
const png = await solidPng(3000, 1500);
|
||||
const result = await cropImageForModel(png, 'image/png', {
|
||||
x: 0,
|
||||
y: 0,
|
||||
width: 2500,
|
||||
height: 1200,
|
||||
});
|
||||
expect(result.ok).toBe(true);
|
||||
if (!result.ok) return;
|
||||
expect(result.resized).toBe(true);
|
||||
expect(Math.max(result.width, result.height)).toBeLessThanOrEqual(MAX_IMAGE_EDGE_PX);
|
||||
expect(result.region).toEqual({ x: 0, y: 0, width: 2500, height: 1200 });
|
||||
});
|
||||
|
||||
it('keeps native resolution with skipResize', async () => {
|
||||
const png = await solidPng(3000, 1500);
|
||||
const result = await cropImageForModel(
|
||||
png,
|
||||
'image/png',
|
||||
{ x: 0, y: 0, width: 2500, height: 1200 },
|
||||
{ skipResize: true },
|
||||
);
|
||||
expect(result.ok).toBe(true);
|
||||
if (!result.ok) return;
|
||||
expect(result.resized).toBe(false);
|
||||
expect(result.width).toBe(2500);
|
||||
expect(result.height).toBe(1200);
|
||||
});
|
||||
|
||||
it('fails explicitly when a skipResize crop exceeds the byte budget', async () => {
|
||||
const png = await noisePng(900, 900);
|
||||
const result = await cropImageForModel(
|
||||
png,
|
||||
'image/png',
|
||||
{ x: 0, y: 0, width: 900, height: 900 },
|
||||
{ skipResize: true, byteBudget: 8 * 1024 },
|
||||
);
|
||||
expect(result.ok).toBe(false);
|
||||
if (result.ok) return;
|
||||
expect(result.error).toMatch(/smaller region/i);
|
||||
});
|
||||
|
||||
it('rejects non-recodable formats explicitly', async () => {
|
||||
const gif = new Uint8Array([0x47, 0x49, 0x46, 0x38, 0x39, 0x61, 1, 0, 1, 0]);
|
||||
const result = await cropImageForModel(gif, 'image/gif', {
|
||||
x: 0,
|
||||
y: 0,
|
||||
width: 1,
|
||||
height: 1,
|
||||
});
|
||||
expect(result.ok).toBe(false);
|
||||
if (result.ok) return;
|
||||
expect(result.error).toMatch(/PNG and JPEG/);
|
||||
});
|
||||
|
||||
it('rejects corrupt bytes without throwing', async () => {
|
||||
const corrupt = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 1, 2, 3]);
|
||||
const result = await cropImageForModel(corrupt, 'image/png', {
|
||||
x: 0,
|
||||
y: 0,
|
||||
width: 10,
|
||||
height: 10,
|
||||
});
|
||||
expect(result.ok).toBe(false);
|
||||
});
|
||||
|
||||
it('rejects non-finite region coordinates with a clean error', async () => {
|
||||
// NaN slips past every `<`/`>=` comparison, so without an explicit guard
|
||||
// it reaches jimp and surfaces as a misleading internal validation dump.
|
||||
const png = await solidPng(300, 200);
|
||||
for (const region of [
|
||||
{ x: Number.NaN, y: 0, width: 10, height: 10 },
|
||||
{ x: 0, y: Number.NaN, width: 10, height: 10 },
|
||||
{ x: 0, y: 0, width: Number.NaN, height: 10 },
|
||||
{ x: 0, y: 0, width: 10, height: Number.NaN },
|
||||
]) {
|
||||
const result = await cropImageForModel(png, 'image/png', region);
|
||||
expect(result.ok).toBe(false);
|
||||
if (result.ok) continue;
|
||||
expect(result.error).toMatch(/finite/i);
|
||||
expect(result.error).not.toMatch(/Failed to decode/);
|
||||
}
|
||||
});
|
||||
|
||||
it('refuses to decode a decompression bomb', async () => {
|
||||
const header = Buffer.alloc(24);
|
||||
Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]).copy(header, 0);
|
||||
header.writeUInt32BE(13, 8);
|
||||
header.write('IHDR', 12, 'latin1');
|
||||
header.writeUInt32BE(30000, 16);
|
||||
header.writeUInt32BE(30000, 20);
|
||||
const result = await cropImageForModel(new Uint8Array(header), 'image/png', {
|
||||
x: 0,
|
||||
y: 0,
|
||||
width: 10,
|
||||
height: 10,
|
||||
});
|
||||
expect(result.ok).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// ── compression caption ──────────────────────────────────────────────
|
||||
|
||||
describe('buildImageCompressionCaption', () => {
|
||||
it('describes the original and sent variants with a readback path', () => {
|
||||
const caption = buildImageCompressionCaption({
|
||||
original: { width: 5184, height: 3456, byteLength: 13002342, mimeType: 'image/png' },
|
||||
final: { width: 2000, height: 1333, byteLength: 1153433, mimeType: 'image/jpeg' },
|
||||
originalPath: '/tmp/originals/ab.png',
|
||||
});
|
||||
expect(caption).toMatch(/^<system>.*<\/system>$/s);
|
||||
expect(caption).toContain('5184x3456 image/png (12.4 MB)');
|
||||
expect(caption).toContain('2000x1333 image/jpeg (1.1 MB)');
|
||||
expect(caption).toContain('/tmp/originals/ab.png');
|
||||
expect(caption).toContain('region');
|
||||
});
|
||||
|
||||
it('omits dimensions when unknown and notes a missing original', () => {
|
||||
const caption = buildImageCompressionCaption({
|
||||
original: { width: 0, height: 0, byteLength: 5 * 1024 * 1024, mimeType: 'image/png' },
|
||||
final: { width: 0, height: 0, byteLength: 1024 * 1024, mimeType: 'image/jpeg' },
|
||||
});
|
||||
expect(caption).not.toContain('0x0');
|
||||
expect(caption).toContain('image/png (5.0 MB)');
|
||||
expect(caption).toContain('image/jpeg (1.0 MB)');
|
||||
expect(caption).toMatch(/not preserved/i);
|
||||
});
|
||||
});
|
||||
|
||||
// ── content-part annotation ──────────────────────────────────────────
|
||||
|
||||
describe('compressImageContentParts — annotate', () => {
|
||||
function dataUrl(mime: string, bytes: Uint8Array): string {
|
||||
return `data:${mime};base64,${Buffer.from(bytes).toString('base64')}`;
|
||||
}
|
||||
|
||||
it('inserts a caption before a compressed image and persists the original', async () => {
|
||||
const big = await solidPng(2600, 2600);
|
||||
const persisted: { bytes: Uint8Array; mimeType: string }[] = [];
|
||||
const parts = [{ type: 'image_url' as const, imageUrl: { url: dataUrl('image/png', big) } }];
|
||||
const out = await compressImageContentParts(parts, {
|
||||
annotate: {
|
||||
persistOriginal: (bytes, mimeType) => {
|
||||
persisted.push({ bytes, mimeType });
|
||||
return Promise.resolve('/tmp/originals/big.png');
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
expect(out).toHaveLength(2);
|
||||
const caption = out[0];
|
||||
if (caption?.type !== 'text') throw new Error('expected caption text part');
|
||||
expect(caption.text).toContain('2600x2600');
|
||||
expect(caption.text).toContain('/tmp/originals/big.png');
|
||||
expect(out[1]?.type).toBe('image_url');
|
||||
expect(persisted).toHaveLength(1);
|
||||
expect(persisted[0]?.mimeType).toBe('image/png');
|
||||
expect(persisted[0]?.bytes.length).toBe(big.length);
|
||||
});
|
||||
|
||||
it('adds no caption when the image passes through unchanged', async () => {
|
||||
const small = await solidPng(48, 48);
|
||||
const url = dataUrl('image/png', small);
|
||||
const out = await compressImageContentParts([{ type: 'image_url' as const, imageUrl: { url } }], {
|
||||
annotate: {},
|
||||
});
|
||||
expect(out).toHaveLength(1);
|
||||
expect(out[0]).toEqual({ type: 'image_url', imageUrl: { url } });
|
||||
});
|
||||
|
||||
it('captions without a path when persistence fails', async () => {
|
||||
const big = await solidPng(2600, 2600);
|
||||
const parts = [{ type: 'image_url' as const, imageUrl: { url: dataUrl('image/png', big) } }];
|
||||
const out = await compressImageContentParts(parts, {
|
||||
annotate: { persistOriginal: () => Promise.resolve(null) },
|
||||
});
|
||||
expect(out).toHaveLength(2);
|
||||
const caption = out[0];
|
||||
if (caption?.type !== 'text') throw new Error('expected caption text part');
|
||||
expect(caption.text).toMatch(/not preserved/i);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
114
packages/agent-core/test/tools/image-originals.test.ts
Normal file
114
packages/agent-core/test/tools/image-originals.test.ts
Normal file
|
|
@ -0,0 +1,114 @@
|
|||
/**
|
||||
* image-originals — content-addressed cache for pre-compression originals.
|
||||
*
|
||||
* Tests pin:
|
||||
* - a persisted original lands under the cache dir, named by content hash
|
||||
* with a mime-derived extension, bytes intact
|
||||
* - persistence is idempotent: same bytes → same path, no duplicate file
|
||||
* - different bytes → different paths
|
||||
* - the cache is size-capped: oldest files are evicted once the cap is
|
||||
* exceeded, newest survive
|
||||
* - best effort: an unwritable destination yields null, never a throw
|
||||
*/
|
||||
|
||||
import { mkdtemp, readdir, readFile, stat, utimes, writeFile } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
// eslint-disable-next-line import/no-unresolved
|
||||
import {
|
||||
originalImageCacheDir,
|
||||
persistOriginalImage,
|
||||
sessionMediaOriginalsDir,
|
||||
} from '../../src/tools/support/image-originals';
|
||||
|
||||
async function freshDir(): Promise<string> {
|
||||
return mkdtemp(join(tmpdir(), 'image-originals-test-'));
|
||||
}
|
||||
|
||||
describe('persistOriginalImage', () => {
|
||||
it('writes the bytes under a hash-named file with a mime extension', async () => {
|
||||
const dir = await freshDir();
|
||||
const bytes = new Uint8Array([1, 2, 3, 4, 5]);
|
||||
|
||||
const path = await persistOriginalImage(bytes, 'image/png', { dir });
|
||||
|
||||
expect(path).not.toBeNull();
|
||||
expect(path!.startsWith(dir)).toBe(true);
|
||||
expect(path!).toMatch(/\.png$/);
|
||||
expect(new Uint8Array(await readFile(path!))).toEqual(bytes);
|
||||
});
|
||||
|
||||
it('maps jpeg mime types to a .jpg extension', async () => {
|
||||
const dir = await freshDir();
|
||||
const path = await persistOriginalImage(new Uint8Array([9, 9]), 'image/jpeg', { dir });
|
||||
expect(path!).toMatch(/\.jpg$/);
|
||||
});
|
||||
|
||||
it('is idempotent for identical bytes', async () => {
|
||||
const dir = await freshDir();
|
||||
const bytes = new Uint8Array([7, 7, 7]);
|
||||
|
||||
const first = await persistOriginalImage(bytes, 'image/png', { dir });
|
||||
const second = await persistOriginalImage(bytes, 'image/png', { dir });
|
||||
|
||||
expect(second).toBe(first);
|
||||
expect(await readdir(dir)).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('gives distinct paths to distinct bytes', async () => {
|
||||
const dir = await freshDir();
|
||||
const a = await persistOriginalImage(new Uint8Array([1]), 'image/png', { dir });
|
||||
const b = await persistOriginalImage(new Uint8Array([2]), 'image/png', { dir });
|
||||
expect(a).not.toBe(b);
|
||||
expect(await readdir(dir)).toHaveLength(2);
|
||||
});
|
||||
|
||||
it('evicts the oldest files once the cache exceeds its cap', async () => {
|
||||
const dir = await freshDir();
|
||||
const old = await persistOriginalImage(new Uint8Array(64).fill(1), 'image/png', {
|
||||
dir,
|
||||
maxTotalBytes: 1024,
|
||||
});
|
||||
// Backdate the first file so eviction order is deterministic.
|
||||
const past = new Date(Date.now() - 60_000);
|
||||
await utimes(old!, past, past);
|
||||
|
||||
const fresh = await persistOriginalImage(new Uint8Array(1000).fill(2), 'image/png', {
|
||||
dir,
|
||||
maxTotalBytes: 1024,
|
||||
});
|
||||
|
||||
// old (64B) + fresh (1000B) > 1024B cap → the backdated file is evicted.
|
||||
await expect(stat(old!)).rejects.toThrow();
|
||||
await expect(stat(fresh!)).resolves.toBeDefined();
|
||||
});
|
||||
|
||||
it('returns null instead of throwing when the destination is unusable', async () => {
|
||||
const dir = await freshDir();
|
||||
const blocker = join(dir, 'not-a-dir');
|
||||
await writeFile(blocker, 'plain file');
|
||||
|
||||
const path = await persistOriginalImage(new Uint8Array([1]), 'image/png', { dir: blocker });
|
||||
|
||||
expect(path).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('originalImageCacheDir', () => {
|
||||
it('defaults to a kimi-code cache directory under the OS temp dir', () => {
|
||||
const dir = originalImageCacheDir();
|
||||
expect(dir.startsWith(tmpdir())).toBe(true);
|
||||
expect(dir).toContain('kimi-code');
|
||||
});
|
||||
});
|
||||
|
||||
describe('sessionMediaOriginalsDir', () => {
|
||||
it('nests the originals dir inside the session dir', () => {
|
||||
expect(sessionMediaOriginalsDir('/home/u/.kimi-code/sessions/ws/abc')).toBe(
|
||||
'/home/u/.kimi-code/sessions/ws/abc/media-originals',
|
||||
);
|
||||
});
|
||||
});
|
||||
|
|
@ -685,4 +685,177 @@ describe('ReadMediaFileTool', () => {
|
|||
expect(systemText).toContain('2600x2600');
|
||||
expect(systemText).toContain(`${String(big.length)} bytes`);
|
||||
});
|
||||
|
||||
describe('region and full_resolution', () => {
|
||||
async function bigPng(width: number, height: number): Promise<Buffer> {
|
||||
return Buffer.from(
|
||||
await new Jimp({ width, height, color: 0x3366ccff }).getBuffer('image/png'),
|
||||
);
|
||||
}
|
||||
|
||||
function toolFor(data: Buffer): ReadMediaFileTool {
|
||||
return makeReadMediaTool({
|
||||
stat: vi.fn<Kaos['stat']>().mockResolvedValue({ ...DEFAULT_STAT, stSize: data.length }),
|
||||
readBytes: vi.fn<Kaos['readBytes']>().mockResolvedValue(data),
|
||||
});
|
||||
}
|
||||
|
||||
it('accepts region and full_resolution in the input schema', () => {
|
||||
expect(
|
||||
ReadMediaFileInputSchema.safeParse({
|
||||
path: 'a.png',
|
||||
region: { x: 0, y: 0, width: 10, height: 10 },
|
||||
}).success,
|
||||
).toBe(true);
|
||||
expect(
|
||||
ReadMediaFileInputSchema.safeParse({ path: 'a.png', full_resolution: true }).success,
|
||||
).toBe(true);
|
||||
expect(
|
||||
ReadMediaFileInputSchema.safeParse({
|
||||
path: 'a.png',
|
||||
region: { x: -1, y: 0, width: 10, height: 10 },
|
||||
}).success,
|
||||
).toBe(false);
|
||||
expect(
|
||||
ReadMediaFileInputSchema.safeParse({
|
||||
path: 'a.png',
|
||||
region: { x: 0, y: 0, width: 0, height: 10 },
|
||||
}).success,
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it('announces a downsampled delivery and the region readback in the <system> block', async () => {
|
||||
const big = await bigPng(2600, 2600);
|
||||
const result = await executeTool(toolFor(big), {
|
||||
turnId: 't1',
|
||||
toolCallId: 'c_note',
|
||||
args: { path: '/workspace/big.png' },
|
||||
signal,
|
||||
});
|
||||
|
||||
const parts = outputParts(result);
|
||||
const systemText = (parts[0] as { text: string }).text;
|
||||
expect(systemText).toContain('2600x2600');
|
||||
expect(systemText).toMatch(/downsampled to 2000x2000/);
|
||||
expect(systemText).toMatch(/fine detail/i);
|
||||
expect(systemText).toContain('region');
|
||||
});
|
||||
|
||||
it('does not claim downsampling for an image sent untouched', async () => {
|
||||
// A real 3x4 PNG passes through unchanged — the <system> block must not
|
||||
// carry a downsample note (that would be its own kind of misreporting).
|
||||
const png = Buffer.from(
|
||||
'89504e470d0a1a0a0000000d49484452000000030000000408020000003a' +
|
||||
'63dc1c0000001949444154789c63606060f8cf80019aa0a8a020' +
|
||||
'00000000ffff03000c1d03014b0000000049454e44ae426082',
|
||||
'hex',
|
||||
);
|
||||
const result = await executeTool(toolFor(png), {
|
||||
turnId: 't1',
|
||||
toolCallId: 'c_untouched',
|
||||
args: { path: '/workspace/small.png' },
|
||||
signal,
|
||||
});
|
||||
const parts = outputParts(result);
|
||||
const systemText = (parts[0] as { text: string }).text;
|
||||
expect(systemText).not.toMatch(/downsampled/i);
|
||||
});
|
||||
|
||||
it('reads a region crop at native resolution', async () => {
|
||||
const big = await bigPng(2600, 2600);
|
||||
const result = await executeTool(toolFor(big), {
|
||||
turnId: 't1',
|
||||
toolCallId: 'c_crop',
|
||||
args: { path: '/workspace/big.png', region: { x: 100, y: 50, width: 400, height: 300 } },
|
||||
signal,
|
||||
});
|
||||
|
||||
const parts = outputParts(result);
|
||||
const url = (parts[2] as { imageUrl: { url: string } }).imageUrl.url;
|
||||
const match = /^data:(image\/[a-z]+);base64,(.+)$/.exec(url);
|
||||
const sentDims = sniffImageDimensions(Buffer.from(match![2]!, 'base64'));
|
||||
expect(sentDims).toEqual({ width: 400, height: 300 });
|
||||
|
||||
const systemText = (parts[0] as { text: string }).text;
|
||||
expect(systemText).toContain('2600x2600');
|
||||
expect(systemText).toMatch(/region \(x=100, y=50, width=400, height=300\)/);
|
||||
expect(systemText).toMatch(/native resolution/);
|
||||
expect(systemText).toContain('offset');
|
||||
});
|
||||
|
||||
it('rejects a region outside the image with the original size in the error', async () => {
|
||||
const big = await bigPng(2600, 2600);
|
||||
const result = await executeTool(toolFor(big), {
|
||||
turnId: 't1',
|
||||
toolCallId: 'c_crop_oob',
|
||||
args: { path: '/workspace/big.png', region: { x: 5000, y: 0, width: 100, height: 100 } },
|
||||
signal,
|
||||
});
|
||||
expect(result.isError).toBe(true);
|
||||
expect(result.output).toContain('2600x2600');
|
||||
});
|
||||
|
||||
it('serves full_resolution when the bytes fit the per-image budget', async () => {
|
||||
const big = await bigPng(2600, 1300); // over the edge cap, tiny in bytes
|
||||
const result = await executeTool(toolFor(big), {
|
||||
turnId: 't1',
|
||||
toolCallId: 'c_fullres',
|
||||
args: { path: '/workspace/big.png', full_resolution: true },
|
||||
signal,
|
||||
});
|
||||
|
||||
const parts = outputParts(result);
|
||||
expect((parts[2] as { imageUrl: { url: string } }).imageUrl.url).toBe(
|
||||
`data:image/png;base64,${big.toString('base64')}`,
|
||||
);
|
||||
const systemText = (parts[0] as { text: string }).text;
|
||||
expect(systemText).toMatch(/native resolution/);
|
||||
});
|
||||
|
||||
it('fails full_resolution explicitly when the file exceeds the per-image budget', async () => {
|
||||
// PNG magic followed by 4MB of filler: recognizably an image, over the
|
||||
// 3.75MB byte budget — full_resolution must refuse, not silently shrink.
|
||||
const data = Buffer.concat([PNG_HEADER, Buffer.alloc(4 * 1024 * 1024, 1)]);
|
||||
const result = await executeTool(toolFor(data), {
|
||||
turnId: 't1',
|
||||
toolCallId: 'c_fullres_big',
|
||||
args: { path: '/workspace/huge.png', full_resolution: true },
|
||||
signal,
|
||||
});
|
||||
expect(result.isError).toBe(true);
|
||||
expect(result.output).toMatch(/full_resolution/);
|
||||
expect(result.output).toMatch(/region/);
|
||||
// Exact byte counts accompany the rounded sizes: a file a hair over
|
||||
// budget would otherwise read "is 3.8 MB, over the 3.8 MB limit".
|
||||
expect(result.output).toContain(`${String(data.length)} bytes`);
|
||||
expect(result.output).toContain('3932160-byte');
|
||||
});
|
||||
|
||||
it('rejects region and full_resolution for video files', async () => {
|
||||
const tool = makeReadMediaTool({
|
||||
stat: vi.fn<Kaos['stat']>().mockResolvedValue({
|
||||
...DEFAULT_STAT,
|
||||
stSize: MP4_HEADER.length,
|
||||
}),
|
||||
readBytes: vi.fn<Kaos['readBytes']>().mockResolvedValue(MP4_HEADER),
|
||||
});
|
||||
|
||||
const withRegion = await executeTool(tool, {
|
||||
turnId: 't1',
|
||||
toolCallId: 'c_vid_region',
|
||||
args: { path: '/workspace/clip.mp4', region: { x: 0, y: 0, width: 10, height: 10 } },
|
||||
signal,
|
||||
});
|
||||
expect(withRegion.isError).toBe(true);
|
||||
expect(withRegion.output).toMatch(/image files/i);
|
||||
|
||||
const withFullRes = await executeTool(tool, {
|
||||
turnId: 't1',
|
||||
toolCallId: 'c_vid_fullres',
|
||||
args: { path: '/workspace/clip.mp4', full_resolution: true },
|
||||
signal,
|
||||
});
|
||||
expect(withFullRes.isError).toBe(true);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -70,9 +70,15 @@ export { installGlobalProxyDispatcher } from '@moonshot-ai/agent-core';
|
|||
// Image compression — ingestion sites (e.g. the CLI's clipboard paste, the ACP
|
||||
// adapter) shrink oversized images while constructing the content part, before
|
||||
// it enters a prompt. Best effort: returns the original on any failure.
|
||||
// Compression is never silent: buildImageCompressionCaption renders the note
|
||||
// placed next to a compressed image, and persistOriginalImage keeps the
|
||||
// pre-compression bytes readable (ReadMediaFile + region) for detail.
|
||||
export {
|
||||
buildImageCompressionCaption,
|
||||
compressImageForModel,
|
||||
compressBase64ForModel,
|
||||
persistOriginalImage,
|
||||
sessionMediaOriginalsDir,
|
||||
IMAGE_BYTE_BUDGET,
|
||||
MAX_IMAGE_EDGE_PX,
|
||||
} from '@moonshot-ai/agent-core';
|
||||
|
|
@ -80,6 +86,7 @@ export type {
|
|||
CompressImageOptions,
|
||||
CompressImageResult,
|
||||
CompressBase64Result,
|
||||
ImageCompressionCaptionInput,
|
||||
} from '@moonshot-ai/agent-core';
|
||||
|
||||
// Experimental feature flags — types only. Resolved values come from
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ import {
|
|||
promptSteerResultSchema,
|
||||
type PromptSubmission,
|
||||
} from '@moonshot-ai/protocol';
|
||||
import { IPromptService, AuthModelNotResolvedError, AuthProvisioningRequiredError, AuthTokenMissingError, AuthTokenUnauthorizedError, PromptAlreadyCompletedError, PromptNotFoundError, SessionBusyError, SessionNotFoundError, FileNotFoundError, IFileStore, compressImageForModel, compressBase64ForModel, type IInstantiationService, type GetResult } from '@moonshot-ai/agent-core';
|
||||
import { IPromptService, AuthModelNotResolvedError, AuthProvisioningRequiredError, AuthTokenMissingError, AuthTokenUnauthorizedError, PromptAlreadyCompletedError, PromptNotFoundError, SessionBusyError, SessionNotFoundError, FileNotFoundError, ICoreProcessService, IFileStore, buildImageCompressionCaption, compressImageForModel, compressBase64ForModel, persistOriginalImage, sessionMediaOriginalsDir, type IInstantiationService, type GetResult } from '@moonshot-ai/agent-core';
|
||||
import { z } from 'zod';
|
||||
|
||||
|
||||
|
|
@ -123,12 +123,25 @@ export function registerPromptsRoutes(
|
|||
try {
|
||||
const { session_id } = req.params;
|
||||
const body = req.body;
|
||||
const result = await ix.invokeFunction(async (a) =>
|
||||
a.get(IPromptService).submit(
|
||||
session_id,
|
||||
await resolvePromptMediaFiles(body, a.get(IFileStore)),
|
||||
),
|
||||
);
|
||||
const result = await ix.invokeFunction(async (a) => {
|
||||
// Grab service references synchronously — the accessor is only
|
||||
// valid until the first await inside invokeFunction.
|
||||
const promptService = a.get(IPromptService);
|
||||
const fileStore = a.get(IFileStore);
|
||||
const core = a.get(ICoreProcessService);
|
||||
const resolved = await resolvePromptMediaFiles(body, fileStore, {
|
||||
// Resolved lazily — only when an inline base64 image actually
|
||||
// got compressed — so image-free prompts never pay the lookup.
|
||||
resolveOriginalsDir: async () => {
|
||||
const summaries = await core.rpc.listSessions({ sessionId: session_id });
|
||||
const sessionDir = summaries[0]?.sessionDir;
|
||||
return sessionDir === undefined
|
||||
? undefined
|
||||
: sessionMediaOriginalsDir(sessionDir);
|
||||
},
|
||||
});
|
||||
return promptService.submit(session_id, resolved);
|
||||
});
|
||||
reply.send(okEnvelope(result, req.id));
|
||||
} catch (error) {
|
||||
sendMappedError(reply, req.id, error);
|
||||
|
|
@ -247,11 +260,31 @@ export function registerPromptsRoutes(
|
|||
);
|
||||
}
|
||||
|
||||
interface ResolvePromptMediaOptions {
|
||||
/**
|
||||
* Lazily resolve the session's media-originals dir for persisting the
|
||||
* pre-compression bytes of inline base64 images. Only invoked when an
|
||||
* image was actually compressed; a failure or undefined result falls back
|
||||
* to the shared temp-dir cache.
|
||||
*/
|
||||
readonly resolveOriginalsDir?: () => Promise<string | undefined>;
|
||||
}
|
||||
|
||||
async function resolvePromptMediaFiles(
|
||||
body: PromptSubmission,
|
||||
store: IFileStore,
|
||||
options: ResolvePromptMediaOptions = {},
|
||||
): Promise<PromptSubmission> {
|
||||
let changed = false;
|
||||
let originalsDir: string | undefined;
|
||||
let originalsDirResolved = false;
|
||||
const resolveOriginalsDir = async (): Promise<string | undefined> => {
|
||||
if (!originalsDirResolved) {
|
||||
originalsDirResolved = true;
|
||||
originalsDir = await options.resolveOriginalsDir?.().catch(() => undefined);
|
||||
}
|
||||
return originalsDir;
|
||||
};
|
||||
const content: PromptSubmission['content'] = [];
|
||||
for (const part of body.content) {
|
||||
// Inline base64 image: compress the payload in place. This is the same
|
||||
|
|
@ -260,6 +293,34 @@ async function resolvePromptMediaFiles(
|
|||
if (part.type === 'image' && part.source.kind === 'base64') {
|
||||
const compressed = await compressBase64ForModel(part.source.data, part.source.media_type);
|
||||
if (compressed.changed) {
|
||||
// There is no stored file to point at, so persist the original into
|
||||
// the session's media-originals dir (best effort, temp-dir fallback)
|
||||
// and announce the compression next to the image — silent
|
||||
// downsampling would leave the model unaware that detail is missing.
|
||||
const dir = await resolveOriginalsDir();
|
||||
const originalPath = await persistOriginalImage(
|
||||
Buffer.from(part.source.data, 'base64'),
|
||||
part.source.media_type,
|
||||
dir === undefined ? {} : { dir },
|
||||
);
|
||||
content.push({
|
||||
type: 'text',
|
||||
text: buildImageCompressionCaption({
|
||||
original: {
|
||||
width: compressed.originalWidth,
|
||||
height: compressed.originalHeight,
|
||||
byteLength: compressed.originalByteLength,
|
||||
mimeType: part.source.media_type,
|
||||
},
|
||||
final: {
|
||||
width: compressed.width,
|
||||
height: compressed.height,
|
||||
byteLength: compressed.finalByteLength,
|
||||
mimeType: compressed.mimeType,
|
||||
},
|
||||
originalPath,
|
||||
}),
|
||||
});
|
||||
content.push({
|
||||
type: 'image',
|
||||
source: { kind: 'base64', media_type: compressed.mimeType, data: compressed.base64 },
|
||||
|
|
@ -285,6 +346,28 @@ async function resolvePromptMediaFiles(
|
|||
let bytes: Uint8Array = data;
|
||||
if (part.type === 'image') {
|
||||
const compressed = await compressImageForModel(data, mediaType);
|
||||
if (compressed.changed) {
|
||||
// The stored file already holds the original bytes — the caption
|
||||
// points straight at it, no extra copy needed.
|
||||
content.push({
|
||||
type: 'text',
|
||||
text: buildImageCompressionCaption({
|
||||
original: {
|
||||
width: compressed.originalWidth,
|
||||
height: compressed.originalHeight,
|
||||
byteLength: compressed.originalByteLength,
|
||||
mimeType: mediaType,
|
||||
},
|
||||
final: {
|
||||
width: compressed.width,
|
||||
height: compressed.height,
|
||||
byteLength: compressed.finalByteLength,
|
||||
mimeType: compressed.mimeType,
|
||||
},
|
||||
originalPath: file.blobPath,
|
||||
}),
|
||||
});
|
||||
}
|
||||
bytes = compressed.data;
|
||||
mediaType = compressed.mimeType;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -22,6 +22,7 @@
|
|||
*/
|
||||
|
||||
import { mkdtempSync, rmSync } from 'node:fs';
|
||||
import { readFile, realpath } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
|
||||
|
|
@ -471,7 +472,7 @@ describe('POST /api/v1/sessions/{sid}/prompts — submit validation (W7.2 / Chai
|
|||
});
|
||||
expect(envelopeOf(res.json()).code).toBe(0);
|
||||
|
||||
const part = submitted?.content[0];
|
||||
const part = submitted?.content[1];
|
||||
if (part?.type !== 'image' || part.source.kind !== 'base64') {
|
||||
throw new Error('expected a resolved base64 image part');
|
||||
}
|
||||
|
|
@ -479,6 +480,19 @@ describe('POST /api/v1/sessions/{sid}/prompts — submit validation (W7.2 / Chai
|
|||
const decoded = await Jimp.fromBuffer(sentBytes);
|
||||
// The model-facing copy is downsampled to the edge cap.
|
||||
expect(Math.max(decoded.width, decoded.height)).toBeLessThanOrEqual(2000);
|
||||
|
||||
// Compression is announced next to the image, and the caption points at
|
||||
// the stored file (which keeps the original bytes) for readback.
|
||||
const caption = submitted?.content[0];
|
||||
if (caption?.type !== 'text') {
|
||||
throw new Error('expected a compression caption before the image part');
|
||||
}
|
||||
expect(caption.text).toContain('Image compressed');
|
||||
expect(caption.text).toContain('2600x2600');
|
||||
const pathMatch = /saved at "([^"]+)"/.exec(caption.text);
|
||||
expect(pathMatch).not.toBeNull();
|
||||
const persisted = await readFile(pathMatch![1]!);
|
||||
expect(persisted.equals(bigPng)).toBe(true);
|
||||
});
|
||||
|
||||
it('compresses an inline base64 image submitted directly in the prompt', async () => {
|
||||
|
|
@ -519,12 +533,30 @@ describe('POST /api/v1/sessions/{sid}/prompts — submit validation (W7.2 / Chai
|
|||
});
|
||||
expect(envelopeOf(res.json()).code).toBe(0);
|
||||
|
||||
const part = submitted?.content[0];
|
||||
const part = submitted?.content[1];
|
||||
if (part?.type !== 'image' || part.source.kind !== 'base64') {
|
||||
throw new Error('expected a base64 image part');
|
||||
}
|
||||
const decoded = await Jimp.fromBuffer(Buffer.from(part.source.data, 'base64'));
|
||||
expect(Math.max(decoded.width, decoded.height)).toBeLessThanOrEqual(2000);
|
||||
|
||||
// Inline base64 has no stored file, so the original is persisted into the
|
||||
// session's media-originals dir and the caption points there.
|
||||
const caption = submitted?.content[0];
|
||||
if (caption?.type !== 'text') {
|
||||
throw new Error('expected a compression caption before the image part');
|
||||
}
|
||||
expect(caption.text).toContain('Image compressed');
|
||||
const pathMatch = /saved at "([^"]+)"/.exec(caption.text);
|
||||
expect(pathMatch).not.toBeNull();
|
||||
// Session dirs live under the daemon home: <home>/sessions/<ws>/<id>.
|
||||
// (realpath both sides: macOS tmpdir is a /var → /private/var symlink.)
|
||||
const realHome = await realpath(bridgeHome);
|
||||
const realPersistedPath = await realpath(pathMatch![1]!);
|
||||
expect(realPersistedPath.startsWith(realHome)).toBe(true);
|
||||
expect(pathMatch![1]!).toContain('/media-originals/');
|
||||
const persisted = await readFile(pathMatch![1]!);
|
||||
expect(persisted.equals(Buffer.from(base64, 'base64'))).toBe(true);
|
||||
});
|
||||
|
||||
it('returns 40407 when prompt image file_id is unknown', async () => {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue