kimi-code/packages/agent-core/test/agent/basic.test.ts
Kai 4c763f6763
Some checks are pending
CI / lint (push) Waiting to run
CI / build (push) Waiting to run
CI / test (1) (push) Waiting to run
CI / test (2) (push) Waiting to run
CI / test (3) (push) Waiting to run
CI / test (4) (push) Waiting to run
CI / test (5) (push) Waiting to run
CI / test-pi-tui (push) Waiting to run
CI / typecheck (push) Waiting to run
Nix Build / Check flake.nix workspace sync (push) Waiting to run
Release / Native release artifact (push) Blocked by required conditions
Release / Release (push) Waiting to run
CI / test-windows (push) Waiting to run
Nix Build / nix build .#kimi-code (push) Blocked by required conditions
Release / Deploy docs (push) Blocked by required conditions
Release / Publish native release assets (push) Blocked by required conditions
feat: send prompt-attached videos directly with the prompt (#1999)
* feat: send prompt-attached videos directly with the prompt

Videos attached to a prompt (pasted in the TUI, uploaded in the web UI)
previously reached the model only after it opened the file with
ReadMediaFile — an extra tool round trip that could leave the video
unseen when the model never made that call. They are now uploaded
through the model provider's file channel and embedded directly in the
user message as the provider-issued reference; ReadMediaFile stays as
the fallback and as the model's own way to open video files.

- agent-core: new uploadVideo agent RPC and Session.uploadVideo in the
  SDK; the TUI uploads pasted videos at submit time, falling back to
  the file-tag form on failure
- kap-server: inline file-source video prompt parts at the REST edge
  and map provider file ids back to local uploads behind
  GET /files/llm/{llm_id}
- kimi-web: play provider-referenced prompt videos after reload/resume

* fix: fall back to inline video when the upload channel fails

ReadMediaFile only used the provider's video upload channel when one
was bound, and surfaced a hard tool error when the upload itself
failed — on providers without a files endpoint (or a transient upload
failure) the model was told the video could not be read at all. Now a
missing or failing upload falls back to delivering the video inline
(base64), the same shape providers without an upload channel already
get. Both engines (agent-core and agent-core-v2) are fixed.

* test: cover prompt video edge cases and failure paths

Stress coverage for the prompt video pipeline:

- agent-core uploadVideo RPC: extension/magic classification
  (.txt with video bytes accepted, extension trusted when magic is
  absent), exact 100MB boundary, directory and nonexistent paths
- TUI: mixed per-video outcomes (one inlined, one tag fallback),
  submission order behind an in-flight upload, queued video messages
  carrying final uploaded parts
- kap-server: per-video provider-id mappings, Range requests through
  the llm redirect, mapping persistence across a server restart

* fix: surface auth rejections from the video upload channel

The base64 fallback for a failing video upload must not mask auth
rejections: a 401/403 (surfaced as provider.auth_error) drives the
credential force-refresh and a clear auth error, while an inline
payload would just be rejected again by the next request. Only a
missing or broken upload channel (no files endpoint, network/server
errors) falls back to inline delivery.

* fix: keep the by-design no-hook video error from degrading to inline

Main's contract for a provider with no video upload hook is an honest
"does not support video upload" tool error — an inline payload would
be dropped on that protocol's wire anyway. The base64 fallback now only
applies to an upload channel that exists but failed at runtime. The
no-hook throw gets a stable type (VideoUploadUnsupportedError) so the
two cases are told apart without matching message text.

* fix: constrain the llm video route param to a safe alphabet

The provider file id is used as a blob-store key, and the node-fs
backend joins scope and key into a storage path. An id containing an
encoded path separator (%2F) could address a different storage path
than the intended llm-video mapping; the route now only accepts the
provider-id alphabet.

* style(agent-core-v2): move added explanations into top-of-file comment blocks

The package's comment convention keeps comments solely in the top-of-file
block; relocate the new notes (url-source id pairing, video delivery
fallback, VideoUploadUnsupportedError) from functions and schema fields
into their module headers.

* fix: reject prompt video uploads when the model lacks video input

The uploadVideo RPC only checked the provider's upload channel, so an
SDK caller on a text-only or unknown-capability model could obtain a
valid-looking video_url part the current model is not supposed to
accept. The TUI capability guard does not cover this public path, so
the agent now gates on video_in itself.

* fix: sniff uploaded video bytes before inlining them

The inline branch trusted the upload-time content type; now the bytes
are sniffed first, and anything magic-confirmed as a non-video kind
(e.g. an image mislabeled as video/mp4) falls back to the file-tag
form instead of being uploaded. Like the image gate, bytes are
authoritative where the container format allows it — an MPEG-PS
lookalike still rides the extension, matching ReadMediaFile.

* test: keep the generation stub pending until the abort lands

An immediately-answered 404 ends the turn (non-retryable) before the
test's abort call, racing the cleanup into a 409; the stub now hangs
generation until the abort cancels it.

* fix: validate prompt file references before mutating session controls

A stale file_id failed inside media resolution — after the model,
thinking and permission overrides had already been applied — so a
rejected prompt still changed the session's controls. File references
are now checked up front, keeping failed submits side-effect free.

* fix: serialize prompt submissions per session

A slow provider video upload let a later text-only request reach the
queue ahead of an earlier video one, silently reordering the
conversation for REST clients and multiple tabs. Submissions to the
same session now chain, matching the ordering the TUI already
guarantees locally.

* fix: play reloaded provider videos through the authenticated fetch path

A video recovered from an ms:// reference carried only the bare redirect
URL, which 401s under daemon auth when loaded natively. The attachment
now keeps the provider file id (llmFileId) end to end, and AuthMedia
fetches the bytes with the Bearer credential through the daemon's llm
redirect — the same blob-URL path uploaded files already use.

* fix: fence pending video submits against session and model switches

A slow paste-upload left the TUI idle, so /new, the session picker or
/model could fire mid-upload; the continuation then dispatched the old
session's provider reference into the newly selected session or a
model that cannot resolve it. The dispatch now re-checks that the
session and model are unchanged and asks for a resend instead.

* fix: preserve the caller's video path verbatim in uploadVideo

Trimming the path changed the filesystem target before validation and
upload, so a name that legitimately starts or ends with whitespace
resolved to the wrong file; the trim is now only the emptiness check.

* fix: forward provider-issued ids on image URL prompt parts too

The shared url-source schema accepts id for image and video parts, but
only the video path forwarded it, dropping provider-keyed image ids
between prompt acceptance and the model request.

* fix(web): reconcile inlined video echoes into the optimistic user message

The loose user-message matcher counted media parts and <video path>
tags but not the [video:ms://…] text shape, so a racing server echo of
an inlined upload slipped through as a duplicate user bubble.

* fix: queue bash submits behind a pending video upload

A bash-mode submit could start while a pasted video was still
uploading, recording shell context before the earlier prompt was
dispatched and reordering user actions; it now chains behind the
upload like normal submits.

* test: cast the driver through unknown for the session-switch fence test

* fix: validate prompt file references before resolving the prompt agent

A stale file_id posted to a fresh or cold session materialized the main
agent (registering it in session metadata and igniting agent-scoped
services) before the request was rejected. The file check now runs
first, so failed submits create nothing and mutate nothing.

* fix: key the playback mapping by the id embedded in the reference URL

Projections and clients read the provider id from the ms:// URL, not
the id field, so an upload that returns an id-less or mismatched
reference would inline fine but 404 on playback. The mapping now
derives its key from the URL, falling back to the explicit id.

* fix: sanitize provider video ids on the write side of the playback map

The read route got the safe-alphabet guard, but recordLlmVideoRef still
used the provider-returned id verbatim as a blob-store key; a crafted
provider response could write the mapping outside the llm-video
namespace. Out-of-alphabet ids are now dropped on both put and get.

* fix(web): keep recovered provider videos resendable through the edit path

The composer reload path only honored fileId and fell back to a
bare fetch(url) without the Bearer token, so editing a reloaded
provider-video turn dropped the chip after a 401. llmFileId is now
threaded through and the bytes are re-uploaded via the authenticated
llm redirect.

* fix: fall back to inline video for no-hook providers whose wire carries it

The by-design no-hook error is only the honest answer when the wire
would drop an inline payload anyway (the OpenAI family). Protocols
that convert video_url (kimi, anthropic, google-genai, vertex) now
take the base64 fallback instead of failing every video read; the
registrar computes the flag from the model's protocol.

* test: satisfy the full IBlobStore shape in the playback-map stub

The tsgo typecheck job rejects a structural stub missing _serviceBrand
and list even though plain tsc accepted it.

* fix: queue prompt-producing slash commands behind pending uploads

Skill activations and plugin commands started their turn immediately
while a pasted video was still uploading, so the earlier video prompt
queued behind them and user actions ran out of order. Both paths now
chain behind inputSubmitChain (and re-check session/model at dispatch)
like normal and bash submits.

* fix: validate media kinds in the prompt file-reference preflight

The preflight only proved a referenced file exists; a real upload used
with the wrong kind (e.g. a PDF submitted as video) still passed it and
mutated session controls before assertMediaFile rejected the request.
The kind assertion now runs up front with the existence check.

* fix(web): play sent and recovered videos in the file preview

The media preview returned early for every kind except image, so a
user-turn video chip's play action was a no-op even with llmFileId
threaded through. The preview now handles video: bytes come from the
authenticated file/llm fetch into a blob URL and render in a native
player.

* fix(web): preview recovered videos with the authenticated blob URL

The llm re-upload branch fetched bytes with auth but kept the protected
redirect URL as the chip preview, which 401s as a native video src; the
fetched blob now becomes the preview URL, mirroring the fileId branch.

* style(agent-core-v2): move the inline-fallback note into the module header

Same package comment convention as before: rationale lives in the
top-of-file block, not beside the registration call.

* fix: fence delayed bash submits to the originating session

The chained bash callback ran runShellCommandFromInput against whatever
session was active at dispatch time, so a command submitted in session
A could execute in session B's workspace and be recorded there after a
mid-upload switch. The originating session is now captured at submit
and re-verified at dispatch, like the prompt and skill paths.

* fix: emit prompt video telemetry from the agent scope

video_upload is an agent-level event requiring ambient agent identity,
but the uploader was built with the Core-scoped session view, leaving
prompt-upload events unattributable. The route now resolves telemetry
from the target agent for the uploader while image compression keeps
the session-scoped view.

* fix: preserve provider image ids through legacy projections and web mappers

The url-source id accepted by the prompt schema was dropped again by
the legacy message projection and the web wire mapper, so provider-
keyed image references lost their id across messages, snapshots, and
undo responses. It now flows through both directions.

* fix(web): revoke recovered video blob URLs before dropping attachments

A failed llm re-upload removed the attachment without revoking the
freshly created preview blob URL, pinning the whole video in the
browser blob store until page unload on every failed edit attempt.

* fix: serialize foreground slash commands behind pending uploads

/compact and /init started their turn while a pasted video was still
uploading, so the earlier message landed after them — and compaction
summarized the context without it. Prompt-producing builtin commands
now share the same queueBehindPendingUploads chain (with the
session/model dispatch fence) as skill, plugin, and bash submits.

* fix: defer session controls until media preparation succeeds

Media resolution now runs before any profile/model/thinking/permission/
denylist mutation, with the uploader resolved transiently from the
requested (or currently bound) model — a failed submission leaves the
session's controls untouched. A concurrent model switch during
preparation is rejected with session.busy instead of enqueueing a
reference uploaded for the previous model.

* chore: bump the new SDK video upload API as a minor release

Session.uploadVideo is new public API surface, not a patch-level tweak.

* fix: re-check busy state before draining upload-queued commands

A slash command deferred behind a video upload ran the moment the video
prompt dispatched, landing on an already-running turn: beginSessionRequest
wiped the active turn's live pane, and /init or /compact started on top
of it. The deferred callbacks for skills, plugin commands, /compact and
/init now re-run the resolver's busy check at dispatch and show the same
blocked message the user would get when typing while streaming.

* fix: resolve profile-bound models before choosing the uploader

A first prompt carrying "profile" without "model" resolved the upload
model from the still-unbound alias, so no uploader was installed and
every attached video fell back to a tool-read tag even when the
configured default model supports provider upload. The transient
resolution now mirrors AgentProfileService.bind: an explicit body model
wins, a profile bind falls back to the configured default model, and
only otherwise does the currently bound alias apply.

* refactor: resolve prompt videos at request time inside the engine

Move prompt-video delivery out of the submission edge: the TUI submits
synchronously with a local file:// part the v1 turn resolves before the
message enters history, and kap-server carries an internal kimi-file://
reference the v2 requester resolves against the effective model with an
app-scoped upload cache. History keeps the durable local file id, the
/messages projection emits structured video parts, and the web plays
videos back through the authenticated /files channel - deleting the
submit fences, per-session serialization, provider-id reverse mapping,
redirect endpoint, and the unreleased SDK upload API.

* fix: propagate abort through video upload delivery instead of degrading

A turn cancelled mid-upload used to be treated as an ordinary upload
failure: v1 fell back to an inline base64 part and appended the degraded
message to history, and the v2 resolver memoized the tag fallback for the
rest of the agent's lifetime. Both catch sites now check the delivery
signal itself - abort rejections vary in shape by provider - and re-throw
so cancellation ends the turn (v1, classified as cancelled via the abort
reason) or the request (v2, not memoized, so the next turn uploads).

* fix: keep the tag form for no-upload providers whose wire drops inline video

An OpenAI-family model configured with video_in but no provider upload
channel used to receive prompt videos as an inline base64 part - which
chat completions rejects and the Responses adapter degrades to an
omitted-video placeholder, persisting ~4/3x the file size in history for
bytes the model never sees. The prompt path now mirrors the v2 resolver's
protocol gate and degrades to the <video path> tag instead; ReadMediaFile's
own delivery is unchanged. Also merges the prompt-video changesets into a
single user-facing entry.

* fix: escape the NUL separator in the video upload cache key

The cache-key template literal contained a literal NUL byte instead of
the \0 escape, which made Git classify the whole source file as binary
- no inline diffs, unreliable text tooling. The escape produces the
byte-identical runtime string, so hashed cache keys are unchanged.

* fix: check the abort signal before the inline video fallback

The no-uploader inline path (and the post-upload-failure fall-through)
never consulted the delivery signal, so cancelling a turn while the video
bytes were being read still base64-encoded the file and appended the
degraded message to history. The inline branch now re-throws the abort
reason first, matching the upload catch.

* fix: retry transient prompt video upload failures on later steps

A generic upload failure used to memoize its tag fallback for the rest of
the agent's lifetime, freezing a transient files-endpoint error into a
permanently degraded video. The resolver now marks failure-born fallbacks
as non-memoizable: the current request keeps the lightweight tag form and
the next step retries the upload. Structural outcomes (successful uploads,
capability and sniff fallbacks, no-hook inline) stay memoized for
step-retry stability.
2026-07-22 13:46:00 +08:00

508 lines
32 KiB
TypeScript

import type { ContentPart, ToolCall } from '@moonshot-ai/kosong';
import { createServer } from 'node:http';
import type { AddressInfo } from 'node:net';
import { mkdtempSync, truncateSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { pathToFileURL } from 'node:url';
import { describe, expect, it, vi } from 'vitest';
import { FLAG_DEFINITIONS, FlagResolver } from '../../src/flags';
import { createCommandKaos, testAgent } from './harness/agent';
it('creates an independent agent with a scoped experimental flag resolver', () => {
const ctx = testAgent({
experimentalFlags: new FlagResolver({}, FLAG_DEFINITIONS),
});
// No experimental flags are currently registered, so the scoped resolver
// reports none enabled.
expect(ctx.agent.experimentalFlags.enabledIds()).toEqual([]);
});
it('runs a text-only agent turn from prompt to completion', async () => {
const ctx = testAgent();
ctx.configure();
ctx.mockNextResponse({ type: 'think', think: '<think-1>' }, { type: 'text', text: '<text-1>' });
await ctx.rpc.prompt({ input: [{ type: 'text', text: 'Hello' }] });
expect(await ctx.untilTurnEnd()).toMatchInlineSnapshot(`
[wire] turn.prompt { "input": [ { "type": "text", "text": "Hello" } ], "origin": { "kind": "user" }, "time": "<time>" }
[emit] turn.started { "turnId": 0, "origin": { "kind": "user" } }
[wire] context.append_message { "message": { "role": "user", "content": [ { "type": "text", "text": "Hello" } ], "toolCalls": [], "origin": { "kind": "user" } }, "time": "<time>" }
[wire] context.append_loop_event { "event": { "type": "step.begin", "uuid": "<uuid-1>", "turnId": "0", "step": 1 }, "time": "<time>" }
[emit] turn.step.started { "turnId": 0, "step": 1, "stepId": "<uuid-1>" }
[wire] llm.tools_snapshot { "hash": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", "tools": [], "time": "<time>" }
[wire] llm.request { "kind": "loop", "provider": "kimi", "model": "mock-model", "modelAlias": "mock-model", "thinkingEffort": "off", "maxTokens": 1000000, "toolSelect": false, "systemPromptHash": "ec9c34379c88babbc468ef2f3e0e08cd2f422c8c4a910664fb8bb394d703a575", "toolsHash": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", "messageCount": 1, "turnStep": "0.1", "time": "<time>" }
[emit] thinking.delta { "turnId": 0, "delta": "<think-1>" }
[emit] assistant.delta { "turnId": 0, "delta": "<text-1>" }
[wire] context.append_loop_event { "event": { "type": "content.part", "uuid": "<uuid-2>", "turnId": "0", "step": 1, "stepUuid": "<uuid-1>", "part": { "type": "think", "think": "<think-1>" } }, "time": "<time>" }
[wire] context.append_loop_event { "event": { "type": "content.part", "uuid": "<uuid-3>", "turnId": "0", "step": 1, "stepUuid": "<uuid-1>", "part": { "type": "text", "text": "<text-1>" } }, "time": "<time>" }
[wire] context.append_loop_event { "event": { "type": "step.end", "uuid": "<uuid-1>", "turnId": "0", "step": 1, "usage": { "inputOther": 3, "output": 8, "inputCacheRead": 0, "inputCacheCreation": 0 }, "finishReason": "end_turn", "messageId": "mock-1" }, "time": "<time>" }
[emit] turn.step.completed { "turnId": 0, "step": 1, "stepId": "<uuid-1>", "usage": { "inputOther": 3, "output": 8, "inputCacheRead": 0, "inputCacheCreation": 0 }, "finishReason": "end_turn" }
[wire] usage.record { "model": "mock-model", "usage": { "inputOther": 3, "output": 8, "inputCacheRead": 0, "inputCacheCreation": 0 }, "usageScope": "turn", "time": "<time>" }
[emit] agent.status.updated { "model": "mock-model", "contextTokens": 11, "maxContextTokens": 1000000, "contextUsage": 0.000011, "planMode": false, "swarmMode": false, "permission": "manual", "usage": { "byModel": { "mock-model": { "inputOther": 3, "output": 8, "inputCacheRead": 0, "inputCacheCreation": 0 } }, "total": { "inputOther": 3, "output": 8, "inputCacheRead": 0, "inputCacheCreation": 0 }, "currentTurn": { "inputOther": 3, "output": 8, "inputCacheRead": 0, "inputCacheCreation": 0 } } }
[emit] turn.ended { "turnId": 0, "reason": "completed" }
`);
expect(ctx.lastLlmInput()).toMatchInlineSnapshot(`
system: <system-prompt>
tools: []
messages:
user: text "Hello"
`);
await ctx.expectResumeMatches();
});
it('forwards provider finish diagnostics on filtered steps', async () => {
const ctx = testAgent();
ctx.configure();
ctx.mockNextProviderResponse({
parts: [{ type: 'text', text: 'blocked' }],
finishReason: 'filtered',
rawFinishReason: 'content_filter',
});
await ctx.rpc.prompt({ input: [{ type: 'text', text: 'Hello' }] });
await ctx.untilTurnEnd();
const wireStepEnd = ctx.allEvents.find(
(event) =>
event.type === '[wire]' &&
event.event === 'context.append_loop_event' &&
(event.args as { event?: { type?: string } }).event?.type === 'step.end',
);
const rpcStepEnd = ctx.allEvents.find(
(event) => event.type === '[rpc]' && event.event === 'turn.step.completed',
);
expect(wireStepEnd?.args).toMatchObject({
event: {
finishReason: 'filtered',
providerFinishReason: 'filtered',
rawFinishReason: 'content_filter',
},
});
expect(rpcStepEnd?.args).toMatchObject({
finishReason: 'filtered',
providerFinishReason: 'filtered',
rawFinishReason: 'content_filter',
});
await ctx.expectResumeMatches();
});
it('runs an agent turn through builtin tool approval and execution', async () => {
const bashCall: ToolCall = {
type: 'function',
id: 'call_bash',
name: 'Bash',
arguments: '{"command":"printf lookup-result","timeout":60}',
};
const ctx = testAgent({ kaos: createCommandKaos('lookup-result') });
ctx.configure({ tools: ['Bash'] });
ctx.mockNextResponse({ type: 'text', text: 'I will run that.' }, bashCall);
await ctx.rpc.prompt({
input: [{ type: 'text', text: 'Run a command that prints lookup-result' }],
});
expect(await ctx.untilApproval(true)).toMatchInlineSnapshot(`
[wire] turn.prompt { "input": [ { "type": "text", "text": "Run a command that prints lookup-result" } ], "origin": { "kind": "user" }, "time": "<time>" }
[emit] turn.started { "turnId": 0, "origin": { "kind": "user" } }
[wire] context.append_message { "message": { "role": "user", "content": [ { "type": "text", "text": "Run a command that prints lookup-result" } ], "toolCalls": [], "origin": { "kind": "user" } }, "time": "<time>" }
[wire] context.append_loop_event { "event": { "type": "step.begin", "uuid": "<uuid-1>", "turnId": "0", "step": 1 }, "time": "<time>" }
[emit] turn.step.started { "turnId": 0, "step": 1, "stepId": "<uuid-1>" }
[wire] llm.tools_snapshot { "hash": "aca3041121ee711028f726fed37e7b999f7e8885c05dbece76ef97eb43e2ec1e", "tools": [ { "name": "Bash", "description": "Execute a \`bash\` command. Use this for shell semantics — pipes, env, processes, git, package managers, build/test runners, anything genuinely interactive or multi-step.\\n\\n**Translate these to a dedicated tool instead:**\\n- \`cat\` / \`head\` / \`tail\` (known path) → \`Read\`\\n- \`sed\` / \`awk\` (in-place edit) → \`Edit\`\\n- \`echo > file\` / \`cat <<EOF\`\`Write\`\\n- \`find\` / recursive \`ls\` to locate files by name pattern → \`Glob\` (plain \`ls <known-directory>\` is fine for listing a directory)\\n- \`grep\` / \`rg\` (search file contents) → \`Grep\`\\n- \`echo\` / \`printf\` (talk to the user) → just output text directly\\n\\nThe dedicated tools render in the per-tool permission UI and keep raw stdout out of the conversation; that is why they are worth reaching for whenever one fits.\\n\\n**Output:**\\nThe stdout and stderr will be combined and returned as a string. The output may be truncated if it is too long. If the command exits non-zero, the output ends with a \`Command failed with exit code: N\` line; a command killed by its timeout or interrupted by the user ends with its own message instead.\\n\\nBackground execution is disabled for this agent. Do not set \`run_in_background=true\`.\\n\\n**Guidelines for safety and security:**\\n- Each shell tool call will be executed in a fresh shell environment. The shell variables, current working directory changes, and the shell history is not preserved between calls. To run a command in a particular directory, pass the \`cwd\` argument (or use absolute paths) rather than relying on a \`cd\` from an earlier call.\\n- The tool call will return after the command is finished. You shall not use this tool to execute an interactive command or a command that may run forever. For possibly long-running commands, set the \`timeout\` argument in seconds. The default is 60s; foreground commands allow up to 300s; a foreground command that hits its timeout is killed.\\n- Avoid using \`..\` to access files or directories outside of the working directory.\\n- Avoid modifying files outside of the working directory unless explicitly instructed to do so.\\n- Never run commands that require superuser privileges unless explicitly instructed to do so.\\n\\n**Guidelines for efficiency:**\\n- Use \`&&\` to chain commands that genuinely depend on each other, e.g. \`npm install && npm test\`. Independent read-only commands (separate \`git show\`, \`ls\`, or status checks) should be issued as separate parallel Bash calls in one response, not chained into a single call — chaining serializes their execution and mixes their output. Do not stitch outputs together with \`echo\` separators.\\n- Use \`;\` to run commands sequentially regardless of success/failure\\n- Use \`||\` for conditional execution (run second command only if first fails)\\n- Use pipe operations (\`|\`) and redirections (\`>\`, \`>>\`) to chain input and output between commands\\n- Always quote file paths containing spaces with double quotes (e.g., cd \\"/path with spaces/\\")\\n- Compose multi-step logic in a single call with \`if\` / \`case\` / \`for\` / \`while\` control flows.\\n- Do not set \`run_in_background=true\`; background task management tools are not available.\\n\\n**Commands available:**\\nThe following common command categories are usually available. Availability still depends on the host, so when in doubt run \`which <command>\` first to confirm a command exists before relying on it.\\n- Navigation and inspection: \`ls\`, \`pwd\`, \`cd\`, \`stat\`, \`file\`, \`du\`, \`df\`, \`tree\`\\n- File and directory management: \`cp\`, \`mv\`, \`rm\`, \`mkdir\`, \`touch\`, \`ln\`, \`chmod\`, \`chown\`\\n- Text and data processing: \`wc\`, \`sort\`, \`uniq\`, \`cut\`, \`tr\`, \`diff\`, \`xargs\`\\n- Archives and compression: \`tar\`, \`gzip\`, \`gunzip\`, \`zip\`, \`unzip\`\\n- Networking and transfer: \`curl\`, \`wget\`, \`ping\`, \`ssh\`, \`scp\`\\n- Version control: \`git\`; for GitHub-hosted work (PRs, issues, CI runs, API queries) prefer the \`gh\` CLI when installed — it carries the user's GitHub auth and can return structured JSON\\n- Process and system: \`ps\`, \`kill\`, \`top\`, \`env\`, \`date\`, \`uname\`, \`whoami\`\\n- Language and package toolchains: \`node\`, \`npm\`, \`pnpm\`, \`yarn\`, \`python\`, \`pip\` (use whichever the project actually relies on)\\n", "parameters": { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": { "command": { "type": "string", "minLength": 1, "description": "The command to execute." }, "cwd": { "description": "The working directory in which to run the command. When omitted, the command runs in the session's working directory.", "type": "string" }, "timeout": { "default": 60, "description": "Optional timeout in seconds for the command to execute. Foreground default 60s, max 300s. Background default 600s, max 86400s. Ignored for background commands when disable_timeout=true.", "type": "integer", "exclusiveMinimum": 0, "maximum": 9007199254740991 }, "description": { "description": "A short description for the background task. Required when run_in_background is true.", "type": "string" }, "run_in_background": { "description": "Whether to run the command as a background task.", "type": "boolean" }, "disable_timeout": { "description": "If true, do not apply a timeout to the command. Only applies when run_in_background is true.", "type": "boolean" } }, "required": [ "command" ], "additionalProperties": false } } ], "time": "<time>" }
[wire] llm.request { "kind": "loop", "provider": "kimi", "model": "mock-model", "modelAlias": "mock-model", "thinkingEffort": "off", "maxTokens": 1000000, "toolSelect": false, "systemPromptHash": "ec9c34379c88babbc468ef2f3e0e08cd2f422c8c4a910664fb8bb394d703a575", "toolsHash": "aca3041121ee711028f726fed37e7b999f7e8885c05dbece76ef97eb43e2ec1e", "messageCount": 1, "turnStep": "0.1", "time": "<time>" }
[emit] assistant.delta { "turnId": 0, "delta": "I will run that." }
[emit] tool.call.delta { "turnId": 0, "toolCallId": "call_bash", "name": "Bash", "argumentsPart": "{\\"command\\":\\"printf lookup-result\\",\\"timeout\\":60}" }
[wire] context.append_loop_event { "event": { "type": "content.part", "uuid": "<uuid-2>", "turnId": "0", "step": 1, "stepUuid": "<uuid-1>", "part": { "type": "text", "text": "I will run that." } }, "time": "<time>" }
[emit] requestApproval { "turnId": 0, "toolCallId": "call_bash", "toolName": "Bash", "action": "Running: printf lookup-result", "display": { "kind": "command", "command": "printf lookup-result", "cwd": "<cwd>", "language": "bash" } }
`);
expect(ctx.lastLlmInput()).toMatchInlineSnapshot(`
system: <system-prompt>
tools: Bash
messages:
user: text "Run a command that prints lookup-result"
`);
ctx.mockNextResponse({ type: 'text', text: 'The command printed lookup-result.' });
expect(await ctx.untilTurnEnd()).toMatchInlineSnapshot(`
[wire] permission.record_approval_result { "turnId": 0, "toolCallId": "call_bash", "toolName": "Bash", "action": "Running: printf lookup-result", "result": { "decision": "approved", "selectedLabel": "approve" }, "time": "<time>" }
[wire] context.append_loop_event { "event": { "type": "tool.call", "uuid": "call_bash", "turnId": "0", "step": 1, "stepUuid": "<uuid-1>", "toolCallId": "call_bash", "name": "Bash", "args": { "command": "printf lookup-result", "timeout": 60 }, "description": "Running: printf lookup-result", "display": { "kind": "command", "command": "printf lookup-result", "cwd": "<cwd>", "language": "bash" } }, "time": "<time>" }
[emit] tool.call.started { "turnId": 0, "toolCallId": "call_bash", "name": "Bash", "args": { "command": "printf lookup-result", "timeout": 60 }, "description": "Running: printf lookup-result", "display": { "kind": "command", "command": "printf lookup-result", "cwd": "<cwd>", "language": "bash" } }
[emit] tool.progress { "turnId": 0, "toolCallId": "call_bash", "update": { "kind": "stdout", "text": "lookup-result" } }
[wire] context.append_loop_event { "event": { "type": "tool.result", "parentUuid": "call_bash", "toolCallId": "call_bash", "result": { "output": "lookup-result" } }, "time": "<time>" }
[emit] tool.result { "turnId": 0, "toolCallId": "call_bash", "output": "lookup-result" }
[wire] context.append_loop_event { "event": { "type": "step.end", "uuid": "<uuid-1>", "turnId": "0", "step": 1, "usage": { "inputOther": 11, "output": 22, "inputCacheRead": 0, "inputCacheCreation": 0 }, "finishReason": "tool_use", "messageId": "mock-1" }, "time": "<time>" }
[emit] turn.step.completed { "turnId": 0, "step": 1, "stepId": "<uuid-1>", "usage": { "inputOther": 11, "output": 22, "inputCacheRead": 0, "inputCacheCreation": 0 }, "finishReason": "tool_use" }
[wire] usage.record { "model": "mock-model", "usage": { "inputOther": 11, "output": 22, "inputCacheRead": 0, "inputCacheCreation": 0 }, "usageScope": "turn", "time": "<time>" }
[emit] agent.status.updated { "model": "mock-model", "contextTokens": 33, "maxContextTokens": 1000000, "contextUsage": 0.000033, "planMode": false, "swarmMode": false, "permission": "manual", "usage": { "byModel": { "mock-model": { "inputOther": 11, "output": 22, "inputCacheRead": 0, "inputCacheCreation": 0 } }, "total": { "inputOther": 11, "output": 22, "inputCacheRead": 0, "inputCacheCreation": 0 }, "currentTurn": { "inputOther": 11, "output": 22, "inputCacheRead": 0, "inputCacheCreation": 0 } } }
[wire] context.append_loop_event { "event": { "type": "step.begin", "uuid": "<uuid-3>", "turnId": "0", "step": 2 }, "time": "<time>" }
[emit] turn.step.started { "turnId": 0, "step": 2, "stepId": "<uuid-3>" }
[wire] llm.request { "kind": "loop", "provider": "kimi", "model": "mock-model", "modelAlias": "mock-model", "thinkingEffort": "off", "maxTokens": 999967, "toolSelect": false, "systemPromptHash": "ec9c34379c88babbc468ef2f3e0e08cd2f422c8c4a910664fb8bb394d703a575", "toolsHash": "aca3041121ee711028f726fed37e7b999f7e8885c05dbece76ef97eb43e2ec1e", "messageCount": 3, "turnStep": "0.2", "time": "<time>" }
[emit] assistant.delta { "turnId": 0, "delta": "The command printed lookup-result." }
[wire] context.append_loop_event { "event": { "type": "content.part", "uuid": "<uuid-4>", "turnId": "0", "step": 2, "stepUuid": "<uuid-3>", "part": { "type": "text", "text": "The command printed lookup-result." } }, "time": "<time>" }
[wire] context.append_loop_event { "event": { "type": "step.end", "uuid": "<uuid-3>", "turnId": "0", "step": 2, "usage": { "inputOther": 38, "output": 12, "inputCacheRead": 0, "inputCacheCreation": 0 }, "finishReason": "end_turn", "messageId": "mock-2" }, "time": "<time>" }
[emit] turn.step.completed { "turnId": 0, "step": 2, "stepId": "<uuid-3>", "usage": { "inputOther": 38, "output": 12, "inputCacheRead": 0, "inputCacheCreation": 0 }, "finishReason": "end_turn" }
[wire] usage.record { "model": "mock-model", "usage": { "inputOther": 38, "output": 12, "inputCacheRead": 0, "inputCacheCreation": 0 }, "usageScope": "turn", "time": "<time>" }
[emit] agent.status.updated { "model": "mock-model", "contextTokens": 50, "maxContextTokens": 1000000, "contextUsage": 0.00005, "planMode": false, "swarmMode": false, "permission": "manual", "usage": { "byModel": { "mock-model": { "inputOther": 49, "output": 34, "inputCacheRead": 0, "inputCacheCreation": 0 } }, "total": { "inputOther": 49, "output": 34, "inputCacheRead": 0, "inputCacheCreation": 0 }, "currentTurn": { "inputOther": 49, "output": 34, "inputCacheRead": 0, "inputCacheCreation": 0 } } }
[emit] turn.ended { "turnId": 0, "reason": "completed" }
`);
expect(ctx.lastLlmInput()).toMatchInlineSnapshot(`
messages:
<last>
assistant: text "I will run that." calls call_bash:Bash { "command": "printf lookup-result", "timeout": 60 }
tool[call_bash]: text "lookup-result"
`);
await ctx.expectResumeMatches();
});
const VIDEO_CAPS = {
image_in: true,
video_in: true,
audio_in: false,
thinking: false,
tool_use: true,
max_context_tokens: 1000000,
} as const;
describe('prompt-attached video resolution', () => {
// Minimal ISO-BMFF header: a 24-byte ftyp box with the `isom` brand, which
// is all the media sniffer needs to classify the file as video/mp4.
const FTYP_MP4 = Buffer.from([
0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70, 0x69, 0x73, 0x6f, 0x6d, 0x00, 0x00, 0x02,
0x00, 0x69, 0x73, 0x6f, 0x6d, 0x69, 0x73, 0x6f, 0x32,
]);
function tempVideo(name = 'clip.mp4', bytes: Buffer = FTYP_MP4): string {
const dir = mkdtempSync(join(tmpdir(), 'agent-prompt-video-'));
const path = join(dir, name);
writeFileSync(path, bytes);
return path;
}
function fileUrl(path: string): string {
return pathToFileURL(path).href;
}
function firstUserContent(ctx: ReturnType<typeof testAgent>): ContentPart[] {
return ctx.agent.context.messages.find((m) => m.role === 'user')?.content ?? [];
}
interface StubFilesServer {
url: string;
requests: number;
close: () => Promise<void>;
}
// Stubs the Moonshot files endpoint. `status` drives the ladder: 200 issues a
// reference, 401 exercises the auth-rejection path, any other status is a
// non-auth upload failure.
async function stubFilesServer(status = 200): Promise<StubFilesServer> {
const state = { requests: 0 };
const server = createServer((req, res) => {
if (req.method === 'POST' && req.url === '/files') {
req.resume();
req.on('end', () => {
state.requests += 1;
if (status === 200) {
res.writeHead(200, { 'content-type': 'application/json' });
res.end(
JSON.stringify({
id: 'stub-video-file',
object: 'file',
bytes: FTYP_MP4.length,
created_at: 0,
filename: 'clip.mp4',
purpose: 'video',
}),
);
return;
}
res.writeHead(status, { 'content-type': 'application/json' });
res.end(JSON.stringify({ error: { message: 'stub upload failure' } }));
});
return;
}
res.writeHead(404);
res.end();
});
await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', resolve));
const { port } = server.address() as AddressInfo;
return {
url: `http://127.0.0.1:${String(port)}`,
get requests() {
return state.requests;
},
close: () => new Promise<void>((resolve) => server.close(() => resolve())),
};
}
function kimiProvider(baseUrl: string) {
return { type: 'kimi', apiKey: 'test-key', model: 'mock-model', baseUrl } as const;
}
it('uploads a local file:// video and sends the issued ms:// reference to the model', async () => {
const stub = await stubFilesServer();
try {
const ctx = testAgent();
ctx.configure({ provider: kimiProvider(stub.url), modelCapabilities: VIDEO_CAPS });
ctx.mockNextResponse({ type: 'text', text: 'ok' });
await ctx.rpc.prompt({ input: [{ type: 'video_url', videoUrl: { url: fileUrl(tempVideo()) } }] });
await ctx.untilTurnEnd();
expect(firstUserContent(ctx)).toContainEqual({
type: 'video_url',
videoUrl: { url: 'ms://stub-video-file', id: 'stub-video-file' },
});
expect(stub.requests).toBe(1);
} finally {
await stub.close();
}
});
it('ends the turn as cancelled when the user aborts mid-upload, appending nothing', async () => {
// A /files stub that never responds: the upload only settles when the
// turn's abort signal tears the in-flight request down, so the rejection
// reaching the resolver is whatever shape the HTTP client aborts with —
// the aborted signal alone must classify it as a cancellation.
let requestArrived!: () => void;
const arrived = new Promise<void>((resolve) => {
requestArrived = resolve;
});
const server = createServer((req) => {
req.resume();
requestArrived();
});
await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', resolve));
const { port } = server.address() as AddressInfo;
try {
const ctx = testAgent();
ctx.configure({
provider: kimiProvider(`http://127.0.0.1:${String(port)}`),
modelCapabilities: VIDEO_CAPS,
});
await ctx.rpc.prompt({ input: [{ type: 'video_url', videoUrl: { url: fileUrl(tempVideo()) } }] });
await arrived;
await ctx.rpc.cancel({ turnId: 0 });
const events = await ctx.untilTurnEnd();
expect(events).toContainEqual(
expect.objectContaining({
event: 'turn.ended',
args: expect.objectContaining({ reason: 'cancelled' }),
}),
);
// The cancelled prompt is not delivered at all: no user message enters
// history, and in particular no inline-base64 degraded copy of it.
expect(ctx.agent.context.data().history).toEqual([]);
} finally {
server.closeAllConnections();
await new Promise<void>((resolve) => server.close(() => resolve()));
}
});
it('ends the turn as cancelled when the user aborts before the inline fallback', async () => {
const ctx = testAgent();
ctx.configure({
// Anthropic: no upload channel, so resolution heads for the inline
// fallback — the abort must beat the base64 encode.
provider: { type: 'anthropic', apiKey: 'test-key', model: 'mock-model' },
modelCapabilities: VIDEO_CAPS,
});
const path = tempVideo();
const realReadBytes = ctx.agent.kaos.readBytes.bind(ctx.agent.kaos);
vi.spyOn(ctx.agent.kaos, 'readBytes').mockImplementation(async (p, length) => {
// The uncapped full-content read is the window the user cancels in.
if (length === undefined) await ctx.rpc.cancel({ turnId: 0 });
return realReadBytes(p, length);
});
await ctx.rpc.prompt({ input: [{ type: 'video_url', videoUrl: { url: fileUrl(path) } }] });
const events = await ctx.untilTurnEnd();
expect(events).toContainEqual(
expect.objectContaining({
event: 'turn.ended',
args: expect.objectContaining({ reason: 'cancelled' }),
}),
);
expect(ctx.agent.context.data().history).toEqual([]);
});
it('degrades to a <video path> tag when the model lacks video_in (no upload)', async () => {
const stub = await stubFilesServer();
try {
const ctx = testAgent();
ctx.configure({
provider: kimiProvider(stub.url),
modelCapabilities: { ...VIDEO_CAPS, video_in: false },
});
ctx.mockNextResponse({ type: 'text', text: 'ok' });
const path = tempVideo();
await ctx.rpc.prompt({ input: [{ type: 'video_url', videoUrl: { url: fileUrl(path) } }] });
await ctx.untilTurnEnd();
const text = firstUserContent(ctx)
.map((p) => (p.type === 'text' ? p.text : ''))
.join('');
expect(text).toContain(`<video path="${path}">`);
expect(firstUserContent(ctx).some((p) => p.type === 'video_url')).toBe(false);
expect(stub.requests).toBe(0);
} finally {
await stub.close();
}
});
it('falls back to an inline base64 part when the provider has no upload channel', async () => {
const ctx = testAgent();
ctx.configure({
// Anthropic: no upload channel, but the wire carries inline data: video.
provider: { type: 'anthropic', apiKey: 'test-key', model: 'mock-model' },
modelCapabilities: VIDEO_CAPS,
});
ctx.mockNextResponse({ type: 'text', text: 'ok' });
await ctx.rpc.prompt({ input: [{ type: 'video_url', videoUrl: { url: fileUrl(tempVideo()) } }] });
await ctx.untilTurnEnd();
const part = firstUserContent(ctx).find((p) => p.type === 'video_url');
expect(part?.type === 'video_url' && part.videoUrl.url).toMatch(/^data:video\/mp4;base64,/);
});
it.each(['openai', 'openai_responses'] as const)(
'degrades to a <video path> tag for a no-upload %s provider whose wire drops inline video',
async (type) => {
const ctx = testAgent();
ctx.configure({
provider: { type, apiKey: 'test-key', model: 'mock-model' },
modelCapabilities: VIDEO_CAPS,
});
ctx.mockNextResponse({ type: 'text', text: 'ok' });
const path = tempVideo();
await ctx.rpc.prompt({ input: [{ type: 'video_url', videoUrl: { url: fileUrl(path) } }] });
await ctx.untilTurnEnd();
const text = firstUserContent(ctx)
.map((p) => (p.type === 'text' ? p.text : ''))
.join('');
expect(text).toContain(`<video path="${path}">`);
expect(firstUserContent(ctx).some((p) => p.type === 'video_url')).toBe(false);
},
);
it('falls back to an inline base64 part on a non-auth upload failure', async () => {
const stub = await stubFilesServer(400);
try {
const ctx = testAgent();
ctx.configure({ provider: kimiProvider(stub.url), modelCapabilities: VIDEO_CAPS });
ctx.mockNextResponse({ type: 'text', text: 'ok' });
await ctx.rpc.prompt({ input: [{ type: 'video_url', videoUrl: { url: fileUrl(tempVideo()) } }] });
await ctx.untilTurnEnd();
const part = firstUserContent(ctx).find((p) => p.type === 'video_url');
expect(part?.type === 'video_url' && part.videoUrl.url).toMatch(/^data:video\/mp4;base64,/);
} finally {
await stub.close();
}
});
it('fails the turn on an auth (401) upload rejection without poisoning history', async () => {
const stub = await stubFilesServer(401);
try {
const ctx = testAgent();
ctx.configure({ provider: kimiProvider(stub.url), modelCapabilities: VIDEO_CAPS });
await ctx.rpc.prompt({ input: [{ type: 'video_url', videoUrl: { url: fileUrl(tempVideo()) } }] });
const events = await ctx.untilTurnEnd();
expect(events).toContainEqual(
expect.objectContaining({
event: 'turn.ended',
args: expect.objectContaining({ reason: 'failed' }),
}),
);
// The unresolved video is never appended to history.
expect(ctx.agent.context.messages.some((m) => m.role === 'user')).toBe(false);
} finally {
await stub.close();
}
});
it('degrades a non-video file to a tag (magic bytes win, no upload)', async () => {
const stub = await stubFilesServer();
try {
const ctx = testAgent();
ctx.configure({ provider: kimiProvider(stub.url), modelCapabilities: VIDEO_CAPS });
ctx.mockNextResponse({ type: 'text', text: 'ok' });
const path = tempVideo('notes.txt', Buffer.from('definitely not a video'));
await ctx.rpc.prompt({ input: [{ type: 'video_url', videoUrl: { url: fileUrl(path) } }] });
await ctx.untilTurnEnd();
expect(firstUserContent(ctx).some((p) => p.type === 'video_url')).toBe(false);
expect(stub.requests).toBe(0);
} finally {
await stub.close();
}
});
it('degrades an oversize video (>100MB) to a tag (no upload)', async () => {
const stub = await stubFilesServer();
try {
const ctx = testAgent();
ctx.configure({ provider: kimiProvider(stub.url), modelCapabilities: VIDEO_CAPS });
ctx.mockNextResponse({ type: 'text', text: 'ok' });
const path = tempVideo();
// Sparse extend: the ftyp header stays so it sniffs as video, the size
// crosses the cap.
truncateSync(path, 100 * 1024 * 1024 + 1);
await ctx.rpc.prompt({ input: [{ type: 'video_url', videoUrl: { url: fileUrl(path) } }] });
await ctx.untilTurnEnd();
expect(firstUserContent(ctx).some((p) => p.type === 'video_url')).toBe(false);
expect(stub.requests).toBe(0);
} finally {
await stub.close();
}
}, 30000);
it('resolves a video attached to a steer message', async () => {
const stub = await stubFilesServer();
try {
const ctx = testAgent();
ctx.configure({ provider: kimiProvider(stub.url), modelCapabilities: VIDEO_CAPS });
// Two model steps: the first lets the steer buffer while the turn runs,
// the second reacts to the flushed steer.
ctx.mockNextResponse({ type: 'text', text: 'first' });
ctx.mockNextResponse({ type: 'text', text: 'second' });
await ctx.rpc.prompt({ input: [{ type: 'text', text: 'start' }] });
const path = tempVideo();
ctx.agent.turn.steer([{ type: 'video_url', videoUrl: { url: fileUrl(path) } }]);
await ctx.untilTurnEnd();
// The steer flush cannot await an upload, so the local video degrades to
// an always-safe tag — no unresolved file:// reference reaches history.
const steered = ctx.agent.context.messages.filter((m) => m.role === 'user');
const hasFileUrl = steered.some((m) =>
m.content.some((p) => p.type === 'video_url' && p.videoUrl.url.startsWith('file:')),
);
expect(hasFileUrl).toBe(false);
const anyTag = steered.some((m) =>
m.content.some((p) => p.type === 'text' && p.text.includes(`<video path="${path}">`)),
);
expect(anyTag).toBe(true);
} finally {
await stub.close();
}
});
});