mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-08-28 19:13:36 +00:00
fix(channels): preserve DingTalk rich-text multi-image messages (#9922)
* docs(channels): design DingTalk multi-image delivery * docs(channels): plan DingTalk multi-image delivery * fix(channels): retain all DingTalk rich-text images * feat(channels): carry ordered prompt images * feat(channels): send all prompt images over ACP * feat(channels): persist all channel images in daemon sessions * fix(channels): clean up failed daemon image uploads * fix(channels): deliver channel images on older daemons and unknown MIME types * fix(channels): harden channel image delivery against malformed and rejected turns - Drop image entries missing data/mimeType in resolvePromptImages so one malformed attachment degrades to a prompt without the image instead of throwing a TypeError that kills the whole turn (restores the pre-diff legacy-guard behavior for out-of-contract extension adapters). - Require the image/ MIME prefix in channelImageName so non-image types whose tail spells a known extension (audio/png) are skipped with a warning instead of tripping the daemon store's name/content-type check and failing the turn. - Roll uploaded images back when the daemon definitively rejects prompt admission (DaemonHttpError on the admission request, local prompt-queue limit), matching the webui client; admitted-turn errors and transport failures keep the uploads because the daemon may already reference them. - Upload prompt images concurrently like the webui path, preserving input order and the existing rollback contract. - Log failed attachment removals during rollback via sanitizeLogText. - Point CHANNEL_IMAGE_EXTENSIONS at the daemon store's set as source of truth; document the ordered images contract in the README. * fix(channels): skip oversized channel images and pin review gaps (#9922) * fix(channels): keep channel image delivery within daemon admission limits (#9922) - Skip channel images whose base64 decodes to zero bytes before uploading: the daemon store rejects empty images with 400, which failed the whole turn instead of degrading by omission. - Bound the inline fallback for daemons without session_attachments: apply the same per-item admission gate and keep the aggregate base64 payload under the daemon's 10mb prompt-body limit, so multi-image turns against older daemons degrade by omission instead of failing with 413. - Reject oversized images from the base64 length before allocating the decoded buffer, so several oversized images no longer allocate their full decoded payloads concurrently before being discarded. - Tag the skip log lines with the session id so operators can correlate a skipped image to its channel session. * fix(channels): close channel image size-gate padding undercount (#9922) The pre-decode size estimate subtracted trailing '=' padding even when the input length does not complete a quantum; Node's lenient base64 decoder ignores such stray padding, so malformed input decoding to the 8 MiB limit plus one byte slipped past the gate and the daemon's 413 then failed the whole turn. Subtract padding only for multiple-of-4 input so the estimate is a strict upper bound. Also give the skip reason a caller-supplied label so the inline fallback for daemons without session_attachments names the inline budget instead of an attachment-store limit those daemons do not have. * fix(channels): roll back uploads on pre-admission cancel * fix(channels): roll back channel images when cancelled before prompt admission (#9922) Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(channels): preserve daemon session client compatibility --------- Co-authored-by: qwen-code-ci-bot <qwen-code-ci-bot@users.noreply.github.com> Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> Co-authored-by: qwen-code-dev-bot <qwen-code-dev-bot@users.noreply.github.com>
This commit is contained in:
BaboBen
GitHub
parent
dab4ba67ed
commit
83da7233a8
15 changed files with 2265 additions and 97 deletions
70
docs/design/dingtalk-richtext-multi-image.md
Normal file
70
docs/design/dingtalk-richtext-multi-image.md
Normal file
|
|
@ -0,0 +1,70 @@
|
|||
# DingTalk rich-text multi-image delivery
|
||||
|
||||
Issue: [#9878](https://github.com/QwenLM/qwen-code/issues/9878)
|
||||
|
||||
## Problem
|
||||
|
||||
A DingTalk message can contain several `picture` parts in one
|
||||
`content.richText` array. The DingTalk adapter extracts every part into
|
||||
`downloadCodes[]`, but the processing path calls `attachMedia` only for
|
||||
`downloadCodes[0]`. Consequently the model and daemon-backed Web Shell receive
|
||||
only the first image.
|
||||
|
||||
This design covers multiple images in one DingTalk callback. It does not change
|
||||
how separate messages received during an active turn are steered or collected.
|
||||
|
||||
## Design
|
||||
|
||||
Extend `ChannelAgentBridgePromptOptions` with an ordered `images` collection
|
||||
whose entries contain base64 data and MIME type. Keep the existing
|
||||
`imageBase64` and `imageMimeType` fields as compatibility inputs for adapters
|
||||
and bridge callers that still send one image.
|
||||
|
||||
`ChannelBase` will normalize the legacy image fields and every data-backed
|
||||
image attachment into one ordered collection. It will pass that collection to
|
||||
the bridge once per inbound turn. File-backed non-image attachments retain the
|
||||
existing prompt-path behavior.
|
||||
|
||||
The DingTalk adapter will call `attachMedia` for every download code extracted
|
||||
from a rich-text callback. Image media from the same callback will remain
|
||||
data-backed so that `ChannelBase` can pass every image as a native vision
|
||||
input, rather than degrading later images into file-path instructions.
|
||||
|
||||
`AcpBridge` will emit one ACP image content block per normalized image before
|
||||
the text block. `DaemonChannelBridge` will upload each image to the owning
|
||||
session and place every returned attachment reference before the text block.
|
||||
This lets the daemon persist all references and lets Web Shell render every
|
||||
image in the transcript.
|
||||
|
||||
## Compatibility
|
||||
|
||||
- Existing adapters using `imageBase64` and `imageMimeType` continue to work.
|
||||
- Existing one-image callers keep their current ordering and behavior.
|
||||
- An invalid partial legacy pair is ignored as it is today.
|
||||
- No new configuration or dependency is introduced.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
DingTalk media downloads remain sequential and preserve callback order. A
|
||||
failed download keeps the existing per-media warning behavior and does not
|
||||
prevent successfully downloaded images from reaching the turn. Daemon upload
|
||||
failure keeps the current prompt failure semantics; the turn is not submitted
|
||||
with a silently incomplete image set.
|
||||
|
||||
## Tests
|
||||
|
||||
1. DingTalk adapter: one `richText` callback with multiple picture parts
|
||||
downloads every code and produces ordered image attachments.
|
||||
2. ChannelBase: legacy and structured image inputs normalize into an ordered
|
||||
bridge image collection without dropping later attachments.
|
||||
3. ACP bridge: every image becomes an ACP image content block before text.
|
||||
4. Daemon bridge: every image is uploaded, referenced in the prompt, and
|
||||
therefore available to session persistence and Web Shell replay.
|
||||
5. Existing single-image tests remain green to prove compatibility.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
Sending five images together in one DingTalk message produces five attachment
|
||||
references in the daemon session turn, displays five images in Web Shell, and
|
||||
provides all five images to the selected multimodal model in the original
|
||||
order.
|
||||
313
docs/plans/dingtalk-richtext-multi-image.md
Normal file
313
docs/plans/dingtalk-richtext-multi-image.md
Normal file
|
|
@ -0,0 +1,313 @@
|
|||
# DingTalk Rich-Text Multi-Image Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Deliver every image from one DingTalk `richText` callback to the multimodal model and persist every image reference for Web Shell replay.
|
||||
|
||||
**Architecture:** Normalize legacy single-image input and structured image attachments into an ordered bridge-level image array. DingTalk downloads every picture part, while ACP emits every image block and the daemon uploads every image before submitting one prompt.
|
||||
|
||||
**Tech Stack:** TypeScript, Vitest, DingTalk Stream adapter, Agent Client Protocol, Qwen daemon session attachment API.
|
||||
|
||||
**Spec:** `docs/design/dingtalk-richtext-multi-image.md`
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Preserve `imageBase64` and `imageMimeType` compatibility for existing adapters.
|
||||
- Preserve source order from `content.richText[]` through the model prompt and Web Shell transcript.
|
||||
- Do not change collection of separate messages received during an active turn.
|
||||
- Add no dependency or configuration.
|
||||
- Run tests from each package directory.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: DingTalk downloads every picture part
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `packages/channels/dingtalk/src/DingtalkAdapter.ts:2119-2140,2343-2353`
|
||||
- Test: `packages/channels/dingtalk/src/DingtalkAdapter.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: `extractContent(data).downloadCodes: string[]` in callback order.
|
||||
- Produces: `Envelope.attachments` containing one data-backed image per successful download in the same order.
|
||||
|
||||
- [ ] **Step 1: Write the failing adapter test**
|
||||
|
||||
Add a test that sends one `richText` callback with two literal picture parts, returns distinct bytes for each `downloadCode`, and asserts that both codes were downloaded and both base64 attachments reached `handleInbound` in order.
|
||||
|
||||
```ts
|
||||
expect(downloadCodes).toEqual(['picture-1', 'picture-2']);
|
||||
expect(envelope.attachments).toEqual([
|
||||
{
|
||||
type: 'image',
|
||||
data: Buffer.from([1]).toString('base64'),
|
||||
mimeType: 'image/png',
|
||||
},
|
||||
{
|
||||
type: 'image',
|
||||
data: Buffer.from([2]).toString('base64'),
|
||||
mimeType: 'image/png',
|
||||
},
|
||||
]);
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the test and verify RED**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cd packages/channels/dingtalk && npx vitest run src/DingtalkAdapter.test.ts -t "downloads every picture in one richText callback"
|
||||
```
|
||||
|
||||
Expected: FAIL because only `picture-1` is requested and only one attachment exists.
|
||||
|
||||
- [ ] **Step 3: Implement ordered multi-download**
|
||||
|
||||
Replace the first-only call with a sequential loop:
|
||||
|
||||
```ts
|
||||
for (const downloadCode of content.downloadCodes) {
|
||||
await this.attachMedia(
|
||||
envelope,
|
||||
downloadCode,
|
||||
content.mediaType,
|
||||
content.fileName,
|
||||
content.placeholder,
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
Make every successful image download append a data-backed attachment; retain the existing temp-file path for file, audio, and video media.
|
||||
|
||||
- [ ] **Step 4: Run the targeted adapter tests and verify GREEN**
|
||||
|
||||
```bash
|
||||
cd packages/channels/dingtalk && npx vitest run src/DingtalkAdapter.test.ts
|
||||
```
|
||||
|
||||
Expected: PASS, including updated quoted-two-image expectations that both images are data-backed.
|
||||
|
||||
- [ ] **Step 5: Commit the adapter behavior**
|
||||
|
||||
```bash
|
||||
git add packages/channels/dingtalk/src/DingtalkAdapter.ts packages/channels/dingtalk/src/DingtalkAdapter.test.ts
|
||||
git commit -m "fix(channels): retain all DingTalk rich-text images"
|
||||
```
|
||||
|
||||
### Task 2: ChannelBase carries an ordered image collection
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `packages/channels/base/src/ChannelAgentBridge.ts:110-116`
|
||||
- Modify: `packages/channels/base/src/ChannelBase.ts:5251-5284,5656-5661`
|
||||
- Test: `packages/channels/base/src/ChannelBase.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Produces: `ChannelPromptImage { data: string; mimeType: string }` and `ChannelAgentBridgePromptOptions.images?: ChannelPromptImage[]`.
|
||||
- Compatibility input: `imageBase64?: string` plus `imageMimeType?: string`.
|
||||
|
||||
- [ ] **Step 1: Write failing ChannelBase tests**
|
||||
|
||||
Add one test with two data-backed `Envelope.attachments` and assert the bridge receives:
|
||||
|
||||
```ts
|
||||
images: [
|
||||
{ data: 'first', mimeType: 'image/png' },
|
||||
{ data: 'second', mimeType: 'image/jpeg' },
|
||||
];
|
||||
```
|
||||
|
||||
Keep the legacy test and change its assertion to the same one-element `images` shape.
|
||||
|
||||
- [ ] **Step 2: Run the focused tests and verify RED**
|
||||
|
||||
```bash
|
||||
cd packages/channels/base && npx vitest run src/ChannelBase.test.ts -t "image"
|
||||
```
|
||||
|
||||
Expected: FAIL because the bridge options contain only singular image fields.
|
||||
|
||||
- [ ] **Step 3: Add the bridge image type and normalization**
|
||||
|
||||
```ts
|
||||
export interface ChannelPromptImage {
|
||||
data: string;
|
||||
mimeType: string;
|
||||
}
|
||||
|
||||
export interface ChannelAgentBridgePromptOptions {
|
||||
images?: ChannelPromptImage[];
|
||||
imageBase64?: string;
|
||||
imageMimeType?: string;
|
||||
displayText?: string;
|
||||
}
|
||||
```
|
||||
|
||||
Build `images` in legacy-first, attachment-order sequence and pass it to `promptBridge.prompt`. Preserve file-path rendering for attachments without image data.
|
||||
|
||||
- [ ] **Step 4: Run ChannelBase tests and verify GREEN**
|
||||
|
||||
```bash
|
||||
cd packages/channels/base && npx vitest run src/ChannelBase.test.ts
|
||||
```
|
||||
|
||||
Expected: PASS with no change to non-image attachment behavior.
|
||||
|
||||
- [ ] **Step 5: Commit the bridge contract**
|
||||
|
||||
```bash
|
||||
git add packages/channels/base/src/ChannelAgentBridge.ts packages/channels/base/src/ChannelBase.ts packages/channels/base/src/ChannelBase.test.ts
|
||||
git commit -m "feat(channels): carry ordered prompt images"
|
||||
```
|
||||
|
||||
### Task 3: ACP sends every image as a native content block
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `packages/channels/base/src/AcpBridge.ts:285-299`
|
||||
- Test: `packages/channels/base/src/AcpBridge.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: `ChannelAgentBridgePromptOptions.images` with legacy single-image fallback.
|
||||
- Produces: ACP prompt content containing every `{ type: 'image', data, mimeType }` block before text.
|
||||
|
||||
- [ ] **Step 1: Write the failing ACP test**
|
||||
|
||||
Call `prompt` with two literal images and assert the connection receives:
|
||||
|
||||
```ts
|
||||
prompt: [
|
||||
{ type: 'image', data: 'first', mimeType: 'image/png' },
|
||||
{ type: 'image', data: 'second', mimeType: 'image/jpeg' },
|
||||
{ type: 'text', text: 'describe both' },
|
||||
];
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the ACP test and verify RED**
|
||||
|
||||
```bash
|
||||
cd packages/channels/base && npx vitest run src/AcpBridge.test.ts -t "multiple images"
|
||||
```
|
||||
|
||||
Expected: FAIL because `images` is not consumed.
|
||||
|
||||
- [ ] **Step 3: Implement image iteration**
|
||||
|
||||
Normalize `options.images` with the legacy pair as fallback, push each image block, then push the text block. Do not alter ACP metadata.
|
||||
|
||||
- [ ] **Step 4: Run ACP tests and verify GREEN**
|
||||
|
||||
```bash
|
||||
cd packages/channels/base && npx vitest run src/AcpBridge.test.ts
|
||||
```
|
||||
|
||||
Expected: PASS, including existing single-image behavior.
|
||||
|
||||
- [ ] **Step 5: Commit ACP support**
|
||||
|
||||
```bash
|
||||
git add packages/channels/base/src/AcpBridge.ts packages/channels/base/src/AcpBridge.test.ts
|
||||
git commit -m "feat(channels): send all prompt images over ACP"
|
||||
```
|
||||
|
||||
### Task 4: Daemon persists every image for Web Shell
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `packages/channels/base/src/DaemonChannelBridge.ts:43-51,125-142,428-441`
|
||||
- Test: `packages/channels/base/src/DaemonChannelBridge.test.ts:79-101,1889-1948`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: ordered `ChannelAgentBridgePromptOptions.images` with legacy fallback.
|
||||
- Produces: one `session.uploadAttachment` call and one attachment reference prompt block per image.
|
||||
|
||||
- [ ] **Step 1: Extend the existing failing daemon replay test**
|
||||
|
||||
Pass two images, return two distinct attachment references, and assert upload order plus this prompt:
|
||||
|
||||
```ts
|
||||
prompt: [
|
||||
{ type: 'image', attachmentId: 'image.png', mimeType: 'image/png', size: 12 },
|
||||
{
|
||||
type: 'image',
|
||||
attachmentId: 'image-2.jpeg',
|
||||
mimeType: 'image/jpeg',
|
||||
size: 13,
|
||||
},
|
||||
{ type: 'text', text: 'describe' },
|
||||
];
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the daemon test and verify RED**
|
||||
|
||||
```bash
|
||||
cd packages/channels/base && npx vitest run src/DaemonChannelBridge.test.ts -t "stores channel images"
|
||||
```
|
||||
|
||||
Expected: FAIL because only the singular image is uploaded.
|
||||
|
||||
- [ ] **Step 3: Implement ordered uploads with unique names**
|
||||
|
||||
Iterate all normalized images, generate deterministic names (`image.png`, `image-2.jpeg`, ...), await each `uploadAttachment`, and push every returned reference before text.
|
||||
|
||||
- [ ] **Step 4: Run daemon tests and verify GREEN**
|
||||
|
||||
```bash
|
||||
cd packages/channels/base && npx vitest run src/DaemonChannelBridge.test.ts
|
||||
```
|
||||
|
||||
Expected: PASS and the prompt contains all persisted attachment references.
|
||||
|
||||
- [ ] **Step 5: Commit daemon persistence**
|
||||
|
||||
```bash
|
||||
git add packages/channels/base/src/DaemonChannelBridge.ts packages/channels/base/src/DaemonChannelBridge.test.ts
|
||||
git commit -m "feat(channels): persist all channel images in daemon sessions"
|
||||
```
|
||||
|
||||
### Task 5: Verify the complete behavior
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify only if verification exposes a defect in the files above.
|
||||
- Record runtime evidence in the final handoff and Issue/PR text; do not commit credentials or callback payloads.
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: built channel packages and the configured local DingTalk daemon.
|
||||
- Produces: unit, build, typecheck, and user-visible Web Shell evidence.
|
||||
|
||||
- [ ] **Step 1: Run package verification**
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
npx tsc --noEmit -p packages/channels/base/tsconfig.json
|
||||
npx tsc --noEmit -p packages/channels/dingtalk/tsconfig.json
|
||||
cd packages/channels/base && npx vitest run src/ChannelBase.test.ts src/AcpBridge.test.ts src/DaemonChannelBridge.test.ts
|
||||
cd packages/channels/dingtalk && npx vitest run src/DingtalkAdapter.test.ts
|
||||
```
|
||||
|
||||
Expected: all commands exit 0.
|
||||
|
||||
- [ ] **Step 2: Reload the configured channel worker**
|
||||
|
||||
Run `npm run dev -- channel reload`, then confirm `npm run dev -- channel status` reports the DingTalk worker running.
|
||||
|
||||
- [ ] **Step 3: Perform the five-image E2E**
|
||||
|
||||
Send five hand images together as one DingTalk message and verify:
|
||||
|
||||
- one DingTalk callback is accepted;
|
||||
- five media downloads complete;
|
||||
- five daemon attachment uploads return HTTP 201;
|
||||
- the persisted user turn has five `attachmentReferences` in order;
|
||||
- Web Shell shows five image previews;
|
||||
- the model reports five hands.
|
||||
|
||||
- [ ] **Step 4: Self-audit the final diff**
|
||||
|
||||
Read `git diff HEAD^` and all untracked files without filtering for expected changes. Verify no secrets, callback payloads, unrelated lockfile changes, or separate-message buffer behavior entered the patch.
|
||||
|
|
@ -197,7 +197,11 @@ interface ChannelAgentBridge {
|
|||
prompt(
|
||||
sessionId: string,
|
||||
text: string,
|
||||
options?: { imageBase64?: string; imageMimeType?: string },
|
||||
options?: {
|
||||
images?: Array<{ data: string; mimeType: string }>; // ordered, preferred
|
||||
imageBase64?: string; // legacy fallback (first image only)
|
||||
imageMimeType?: string; // legacy fallback (first image only)
|
||||
},
|
||||
): Promise<string>;
|
||||
cancelSession(sessionId: string): Promise<void>;
|
||||
shellCommand?(
|
||||
|
|
@ -208,6 +212,8 @@ interface ChannelAgentBridge {
|
|||
}
|
||||
```
|
||||
|
||||
`prompt` carries images through `options.images` — an ordered array of `{ data, mimeType }` entries delivered to the model in array order. The legacy `imageBase64`/`imageMimeType` pair is a fallback that carries only the first image. Built-in bridges normalize MIME types before use: lowercased, parameters stripped (`image/png; charset=binary` → `image/png`), and the `image/jpg` alias mapped to `image/jpeg`. Daemon-backed bridges upload images to the daemon attachment store and restrict them to its supported subtypes (`image/bmp`, `image/gif`, `image/jpeg`, `image/png`, `image/webp`) and its per-item admission rule (non-empty and at most 8 MiB once decoded); daemons without attachment support take the same images inline in the prompt body, kept under an aggregate base64 budget. Images outside those limits are skipped with a log line instead of failing the turn, while `AcpBridge` delivers any normalized `image/*` inline.
|
||||
|
||||
### AcpBridge
|
||||
|
||||
`AcpBridge` is the current implementation used by standalone `qwen channel start`. It manages the `qwen-code --acp` child process and implements `ChannelAgentBridge`.
|
||||
|
|
@ -216,14 +222,14 @@ interface ChannelAgentBridge {
|
|||
constructor(options: { cliEntryPath: string; cwd: string; model?: string })
|
||||
```
|
||||
|
||||
| Method | Description |
|
||||
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
|
||||
| `start()` | Spawn the agent process |
|
||||
| `stop()` | Kill the agent process |
|
||||
| `newSession(cwd)` | Create a new ACP session, returns `sessionId` |
|
||||
| `loadSession(sessionId, cwd)` | Restore an existing session |
|
||||
| `prompt(sessionId, text, options?)` | Send a message to the agent, returns the full response text. Supports optional `imageBase64` and `imageMimeType`. |
|
||||
| `isConnected` | Whether the agent process is alive |
|
||||
| Method | Description |
|
||||
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `start()` | Spawn the agent process |
|
||||
| `stop()` | Kill the agent process |
|
||||
| `newSession(cwd)` | Create a new ACP session, returns `sessionId` |
|
||||
| `loadSession(sessionId, cwd)` | Restore an existing session |
|
||||
| `prompt(sessionId, text, options?)` | Send a message to the agent, returns the full response text. Supports an ordered `images` array (`{ data, mimeType }` per image, preferred), falling back to the legacy `imageBase64`/`imageMimeType` pair for a single image. |
|
||||
| `isConnected` | Whether the agent process is alive |
|
||||
|
||||
**Events** (EventEmitter):
|
||||
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@ import {
|
|||
ACP_PRIVATE_PARENT_CAPABILITY_META_KEY,
|
||||
CHANNEL_PROMPT_META_KEY,
|
||||
type ChannelLoopToolHandler,
|
||||
type ChannelPromptImage,
|
||||
} from './ChannelAgentBridge.js';
|
||||
|
||||
const child = vi.hoisted(() => {
|
||||
|
|
@ -515,6 +516,85 @@ describe('AcpBridge', () => {
|
|||
});
|
||||
});
|
||||
|
||||
it('sends multiple images before the text prompt', async () => {
|
||||
const bridge = new AcpBridge({
|
||||
cliEntryPath: '/tmp/qwen',
|
||||
cwd: '/tmp',
|
||||
}) as unknown as TestableAcpBridge;
|
||||
const prompt = vi.fn().mockResolvedValue({});
|
||||
bridge.child = { killed: false, exitCode: null };
|
||||
bridge.connection = { extMethod: vi.fn(), prompt };
|
||||
|
||||
await bridge.prompt('s-1', 'describe both', {
|
||||
images: [
|
||||
{ data: 'first', mimeType: 'image/png' },
|
||||
{ data: 'second', mimeType: 'image/jpeg' },
|
||||
],
|
||||
});
|
||||
|
||||
expect(prompt).toHaveBeenCalledWith({
|
||||
sessionId: 's-1',
|
||||
prompt: [
|
||||
{ type: 'image', data: 'first', mimeType: 'image/png' },
|
||||
{ type: 'image', data: 'second', mimeType: 'image/jpeg' },
|
||||
{ type: 'text', text: 'describe both' },
|
||||
],
|
||||
_meta: { [CHANNEL_PROMPT_META_KEY]: true },
|
||||
});
|
||||
});
|
||||
|
||||
it('sends a legacy-only image pair as one inline image block', async () => {
|
||||
const bridge = new AcpBridge({
|
||||
cliEntryPath: '/tmp/qwen',
|
||||
cwd: '/tmp',
|
||||
}) as unknown as TestableAcpBridge;
|
||||
const prompt = vi.fn().mockResolvedValue({});
|
||||
bridge.child = { killed: false, exitCode: null };
|
||||
bridge.connection = { extMethod: vi.fn(), prompt };
|
||||
|
||||
await bridge.prompt('s-1', 'describe', {
|
||||
imageBase64: 'base64-image',
|
||||
imageMimeType: 'image/png',
|
||||
});
|
||||
|
||||
expect(prompt).toHaveBeenCalledWith({
|
||||
sessionId: 's-1',
|
||||
prompt: [
|
||||
{ type: 'image', data: 'base64-image', mimeType: 'image/png' },
|
||||
{ type: 'text', text: 'describe' },
|
||||
],
|
||||
_meta: { [CHANNEL_PROMPT_META_KEY]: true },
|
||||
});
|
||||
});
|
||||
|
||||
it('drops malformed prompt image entries before the text prompt', async () => {
|
||||
const bridge = new AcpBridge({
|
||||
cliEntryPath: '/tmp/qwen',
|
||||
cwd: '/tmp',
|
||||
}) as unknown as TestableAcpBridge;
|
||||
const prompt = vi.fn().mockResolvedValue({});
|
||||
bridge.child = { killed: false, exitCode: null };
|
||||
bridge.connection = { extMethod: vi.fn(), prompt };
|
||||
|
||||
await bridge.prompt('s-1', 'describe', {
|
||||
images: [
|
||||
{ data: 'AQID', mimeType: 'image/png' },
|
||||
{ data: '', mimeType: 'image/webp' },
|
||||
{ data: 'BAUG', mimeType: '' },
|
||||
null as unknown as ChannelPromptImage,
|
||||
],
|
||||
});
|
||||
|
||||
expect(prompt).toHaveBeenCalledWith({
|
||||
sessionId: 's-1',
|
||||
prompt: [
|
||||
{ type: 'image', data: 'AQID', mimeType: 'image/png' },
|
||||
{ type: 'text', text: 'describe' },
|
||||
],
|
||||
_meta: { [CHANNEL_PROMPT_META_KEY]: true },
|
||||
});
|
||||
});
|
||||
|
||||
it('excludes nested subagent text from the final response', async () => {
|
||||
const bridge = new AcpBridge({
|
||||
cliEntryPath: '/tmp/qwen',
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@ import {
|
|||
ACP_PRIVATE_PARENT_CAPABILITY_META_KEY,
|
||||
CHANNEL_PROMPT_DISPLAY_TEXT_META_KEY,
|
||||
CHANNEL_PROMPT_META_KEY,
|
||||
resolvePromptImages,
|
||||
type AvailableCommand,
|
||||
type ChannelAgentBridge,
|
||||
type ChannelAgentBridgePromptOptions,
|
||||
|
|
@ -283,11 +284,11 @@ export class AcpBridge extends EventEmitter implements ChannelAgentBridge {
|
|||
this.on('responseBoundary', clearChunks);
|
||||
|
||||
const prompt: Array<Record<string, unknown>> = [];
|
||||
if (options?.imageBase64 && options.imageMimeType) {
|
||||
for (const image of resolvePromptImages(options)) {
|
||||
prompt.push({
|
||||
type: 'image',
|
||||
data: options.imageBase64,
|
||||
mimeType: options.imageMimeType,
|
||||
data: image.data,
|
||||
mimeType: image.mimeType,
|
||||
});
|
||||
}
|
||||
prompt.push({ type: 'text', text });
|
||||
|
|
|
|||
|
|
@ -107,7 +107,13 @@ export interface ChannelAgentBridgeSessionOptions {
|
|||
sourceId?: string;
|
||||
}
|
||||
|
||||
export interface ChannelPromptImage {
|
||||
data: string;
|
||||
mimeType: string;
|
||||
}
|
||||
|
||||
export interface ChannelAgentBridgePromptOptions {
|
||||
images?: ChannelPromptImage[];
|
||||
imageBase64?: string;
|
||||
imageMimeType?: string;
|
||||
/** User-authored text shown in transcripts when `text` includes hidden context.
|
||||
|
|
@ -115,6 +121,44 @@ export interface ChannelAgentBridgePromptOptions {
|
|||
displayText?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the ordered `images` contract, falling back to the legacy
|
||||
* single-image pair, and normalizes MIME types in one place: channel
|
||||
* adapters forward CDN `content-type` headers verbatim, so values arrive
|
||||
* with parameters and mixed case (e.g. `image/png; charset=binary`), and
|
||||
* the non-standard `image/jpg` alias rides them too. Entries missing
|
||||
* `data` or `mimeType` are dropped so one malformed attachment degrades
|
||||
* to a prompt without that image, like the legacy field guards did.
|
||||
*/
|
||||
export function resolvePromptImages(
|
||||
options?: ChannelAgentBridgePromptOptions,
|
||||
): ChannelPromptImage[] {
|
||||
const images =
|
||||
options?.images && options.images.length > 0
|
||||
? options.images
|
||||
: options?.imageBase64 && options.imageMimeType
|
||||
? [{ data: options.imageBase64, mimeType: options.imageMimeType }]
|
||||
: [];
|
||||
return images
|
||||
.filter(
|
||||
(image) =>
|
||||
!!image &&
|
||||
typeof image.data === 'string' &&
|
||||
image.data.length > 0 &&
|
||||
typeof image.mimeType === 'string' &&
|
||||
image.mimeType.length > 0,
|
||||
)
|
||||
.map((image) => {
|
||||
const cleaned =
|
||||
image.mimeType.split(';', 1)[0]?.trim().toLowerCase() ?? '';
|
||||
return {
|
||||
data: image.data,
|
||||
// Normalize the alias like the daemon attachment store's own naming.
|
||||
mimeType: cleaned === 'image/jpg' ? 'image/jpeg' : cleaned,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
export interface ChannelAgentBridge {
|
||||
readonly availableCommands: AvailableCommand[];
|
||||
getAvailableCommands?(sessionId: string): AvailableCommand[];
|
||||
|
|
|
|||
|
|
@ -10189,7 +10189,7 @@ describe('ChannelBase', () => {
|
|||
expect(pathLine).not.toContain(rlo);
|
||||
});
|
||||
|
||||
it('extracts image from attachments', async () => {
|
||||
it('forwards every image attachment in order', async () => {
|
||||
const ch = createChannel();
|
||||
await ch.handleInbound(
|
||||
envelope({
|
||||
|
|
@ -10200,11 +10200,20 @@ describe('ChannelBase', () => {
|
|||
data: 'base64data',
|
||||
mimeType: 'image/png',
|
||||
},
|
||||
{
|
||||
type: 'image',
|
||||
data: 'second-image',
|
||||
mimeType: 'image/jpeg',
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const options = (bridge.prompt as any).mock.calls[0][2];
|
||||
expect(options.images).toEqual([
|
||||
{ data: 'base64data', mimeType: 'image/png' },
|
||||
{ data: 'second-image', mimeType: 'image/jpeg' },
|
||||
]);
|
||||
expect(options.imageBase64).toBe('base64data');
|
||||
expect(options.imageMimeType).toBe('image/png');
|
||||
});
|
||||
|
|
@ -10220,7 +10229,33 @@ describe('ChannelBase', () => {
|
|||
);
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const options = (bridge.prompt as any).mock.calls[0][2];
|
||||
expect(options.imageBase64).toBe('legacydata');
|
||||
expect(options.images).toEqual([
|
||||
{ data: 'legacydata', mimeType: 'image/jpeg' },
|
||||
]);
|
||||
});
|
||||
|
||||
it('orders the legacy image before attachment images', async () => {
|
||||
const ch = createChannel();
|
||||
await ch.handleInbound(
|
||||
envelope({
|
||||
text: 'see image',
|
||||
imageBase64: 'legacydata',
|
||||
imageMimeType: 'image/jpeg',
|
||||
attachments: [
|
||||
{
|
||||
type: 'image',
|
||||
data: 'attachmentdata',
|
||||
mimeType: 'image/png',
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const options = (bridge.prompt as any).mock.calls[0][2];
|
||||
expect(options.images).toEqual([
|
||||
{ data: 'legacydata', mimeType: 'image/jpeg' },
|
||||
{ data: 'attachmentdata', mimeType: 'image/png' },
|
||||
]);
|
||||
});
|
||||
|
||||
it('prepends instructions on first message only', async () => {
|
||||
|
|
|
|||
|
|
@ -54,6 +54,7 @@ import {
|
|||
import type {
|
||||
AvailableCommand,
|
||||
ChannelAgentBridge,
|
||||
ChannelPromptImage,
|
||||
ChannelLoopToolCreateInput,
|
||||
ChannelLoopToolResult,
|
||||
PermissionRequestEvent,
|
||||
|
|
@ -5248,15 +5249,22 @@ export abstract class ChannelBase {
|
|||
promptText = `[Replying to: "${quoted}"]\n\n${promptText}`;
|
||||
}
|
||||
|
||||
// Resolve attachments: extract image for bridge, append file paths to text
|
||||
// Resolve attachments: extract images for bridge, append file paths to text
|
||||
let imageBase64 = envelope.imageBase64;
|
||||
let imageMimeType = envelope.imageMimeType;
|
||||
const images: ChannelPromptImage[] = [];
|
||||
if (imageBase64 && imageMimeType) {
|
||||
images.push({ data: imageBase64, mimeType: imageMimeType });
|
||||
}
|
||||
if (envelope.attachments?.length) {
|
||||
const filePaths: string[] = [];
|
||||
for (const att of envelope.attachments) {
|
||||
if (att.type === 'image' && att.data && !imageBase64) {
|
||||
imageBase64 = att.data;
|
||||
imageMimeType = att.mimeType;
|
||||
if (att.type === 'image' && att.data) {
|
||||
images.push({ data: att.data, mimeType: att.mimeType });
|
||||
if (!imageBase64) {
|
||||
imageBase64 = att.data;
|
||||
imageMimeType = att.mimeType;
|
||||
}
|
||||
} else if (att.filePath) {
|
||||
const label = att.type === 'file' ? 'file' : att.type;
|
||||
// The filename is attacker-supplied (e.g. DingTalk), so neutralize both
|
||||
|
|
@ -5655,6 +5663,7 @@ export abstract class ChannelBase {
|
|||
|
||||
try {
|
||||
const response = await promptBridge.prompt(sessionId, promptToSend, {
|
||||
...(images.length > 0 ? { images } : {}),
|
||||
imageBase64,
|
||||
imageMimeType,
|
||||
displayText,
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -7,6 +7,7 @@ import {
|
|||
CHANNEL_PROMPT_AUTHORIZATION_META_KEY,
|
||||
CHANNEL_PROMPT_DISPLAY_TEXT_META_KEY,
|
||||
CHANNEL_PROMPT_META_KEY,
|
||||
resolvePromptImages,
|
||||
type AvailableCommand,
|
||||
type BridgeSessionInfo,
|
||||
type ChannelAgentBridge,
|
||||
|
|
@ -16,6 +17,7 @@ import {
|
|||
type ToolCallEvent,
|
||||
} from './ChannelAgentBridge.js';
|
||||
import { readAvailableCommandAltNames } from './AcpBridge.js';
|
||||
import { sanitizeLogText } from './sanitize.js';
|
||||
import {
|
||||
ChannelLoopMcpServer,
|
||||
type JsonRpcMessage,
|
||||
|
|
@ -43,6 +45,13 @@ export interface DaemonChannelSessionClient {
|
|||
},
|
||||
signal?: AbortSignal,
|
||||
): Promise<{ stopReason?: string; [key: string]: unknown }>;
|
||||
uploadAttachment?(
|
||||
data: Blob,
|
||||
name: string,
|
||||
mimeType: string,
|
||||
signal?: AbortSignal,
|
||||
): Promise<Record<string, unknown>>;
|
||||
removeAttachment?(attachmentId: string): Promise<boolean>;
|
||||
events(opts?: {
|
||||
signal?: AbortSignal;
|
||||
lastEventId?: number;
|
||||
|
|
@ -91,6 +100,12 @@ export interface DaemonChannelBridgeOptions {
|
|||
channelLoopMcpHost?: DaemonChannelLoopMcpHost;
|
||||
deleteSessionData?: (sessionId: string) => Promise<void>;
|
||||
promptAuthorization?: string;
|
||||
/**
|
||||
* The daemon advertises the `session_attachments` capability. Daemons
|
||||
* predating the attachment upload routes receive prompt images inline
|
||||
* instead, as before the upload path existed.
|
||||
*/
|
||||
sessionAttachments?: boolean;
|
||||
}
|
||||
|
||||
export interface DaemonPermissionRequestEvent {
|
||||
|
|
@ -125,6 +140,80 @@ function getTextContent(content: unknown): string | undefined {
|
|||
return getString(content['text']);
|
||||
}
|
||||
|
||||
// Mirrors the daemon attachment store's SUPPORTED_IMAGE_MIME_TYPES
|
||||
// (packages/acp-bridge/src/sessionAttachments.ts): the store rejects uploads
|
||||
// outside that set, and channels/base keeps no acp-bridge dependency, so the
|
||||
// set is repeated here and checked before uploading.
|
||||
const CHANNEL_IMAGE_EXTENSIONS = ['bmp', 'gif', 'jpeg', 'png', 'webp'];
|
||||
|
||||
// Mirrors the store's SESSION_ATTACHMENT_MAX_ITEM_BYTES and empty-image
|
||||
// rejection (same file): checked before delivery so one inadmissible image
|
||||
// degrades by omission instead of failing the whole turn.
|
||||
const CHANNEL_IMAGE_MAX_UPLOAD_BYTES = 8 * 1024 * 1024;
|
||||
|
||||
// Daemons without `session_attachments` parse the prompt body with
|
||||
// express.json({ limit: '10mb' }), so the inline fallback keeps the
|
||||
// aggregate base64 payload below that cap with headroom for the text
|
||||
// prompt and the JSON envelope.
|
||||
const CHANNEL_IMAGE_INLINE_MAX_BASE64_BYTES = 8 * 1024 * 1024;
|
||||
|
||||
function channelImageName(mimeType: string, index = 0): string | undefined {
|
||||
if (!mimeType.startsWith('image/')) {
|
||||
return undefined;
|
||||
}
|
||||
const extension = mimeType.slice('image/'.length);
|
||||
if (!CHANNEL_IMAGE_EXTENSIONS.includes(extension)) {
|
||||
return undefined;
|
||||
}
|
||||
return index === 0 ? `image.${extension}` : `image-${index + 1}.${extension}`;
|
||||
}
|
||||
|
||||
function decodeChannelImage(
|
||||
data: string,
|
||||
oversizedReason: string,
|
||||
): { bytes: Buffer } | { skip: string } {
|
||||
// Valid base64 decodes to at most this many bytes, so an oversized image
|
||||
// is rejected on length alone instead of allocating a buffer the size
|
||||
// check would discard. Padding is subtracted only when the input length
|
||||
// completes a quantum: Node's decoder ignores a stray trailing '=' on
|
||||
// malformed input, and counting it would undercount the decoded size.
|
||||
let estimatedBytes = Math.floor((data.length * 3) / 4);
|
||||
if (data.length % 4 === 0) {
|
||||
if (data.endsWith('==')) estimatedBytes -= 2;
|
||||
else if (data.endsWith('=')) estimatedBytes -= 1;
|
||||
}
|
||||
if (estimatedBytes > CHANNEL_IMAGE_MAX_UPLOAD_BYTES) {
|
||||
return { skip: oversizedReason };
|
||||
}
|
||||
const bytes = Buffer.from(data, 'base64');
|
||||
if (bytes.byteLength === 0) {
|
||||
return { skip: 'empty once base64-decoded' };
|
||||
}
|
||||
return { bytes };
|
||||
}
|
||||
|
||||
/**
|
||||
* Structural match for the daemon SDK's definite prompt-admission
|
||||
* rejections: `DaemonHttpError` from the admission request itself, or
|
||||
* `DaemonPendingPromptLimitError` raised before any request. channels/base
|
||||
* keeps no dependency on the SDK, so match by shape; post-admission turn
|
||||
* errors carry `_daemonTurnError` and must NOT match — by then the daemon
|
||||
* may already have resolved the uploaded attachments.
|
||||
*/
|
||||
function isDefinitePromptAdmissionRejection(error: unknown): boolean {
|
||||
if (!isRecord(error)) {
|
||||
return false;
|
||||
}
|
||||
if (error['name'] === 'DaemonPendingPromptLimitError') {
|
||||
return true;
|
||||
}
|
||||
return (
|
||||
error['name'] === 'DaemonHttpError' &&
|
||||
typeof error['status'] === 'number' &&
|
||||
error['_daemonTurnError'] !== true
|
||||
);
|
||||
}
|
||||
|
||||
function getSessionUpdate(data: unknown): Record<string, unknown> | undefined {
|
||||
if (!isRecord(data) || !isRecord(data['update'])) {
|
||||
return undefined;
|
||||
|
|
@ -407,41 +496,153 @@ export class DaemonChannelBridge
|
|||
this.on('responseBoundary', clearChunks);
|
||||
this.on('sessionDied', onSessionDied);
|
||||
const turnBarrier = this.createTurnBarrier(sessionId);
|
||||
|
||||
const prompt: Array<Record<string, unknown>> = [];
|
||||
if (options?.imageBase64 && options.imageMimeType) {
|
||||
prompt.push({
|
||||
type: 'image',
|
||||
data: options.imageBase64,
|
||||
mimeType: options.imageMimeType,
|
||||
});
|
||||
}
|
||||
prompt.push({ type: 'text', text });
|
||||
// Always presented: the daemon validates it for the channel-turn
|
||||
// classification as well as the display projection, and channel
|
||||
// prompts without display text still need the classification.
|
||||
const promptAuthorization = this.options.promptAuthorization;
|
||||
const uploadedAttachmentIds: string[] = [];
|
||||
let rollbackUploadedAttachments = false;
|
||||
const uploadAttachment = session.uploadAttachment?.bind(session);
|
||||
const removeAttachment = session.removeAttachment?.bind(session);
|
||||
|
||||
try {
|
||||
const result = await session.prompt(
|
||||
{
|
||||
prompt,
|
||||
_meta: {
|
||||
[CHANNEL_PROMPT_META_KEY]: true,
|
||||
...(promptAuthorization
|
||||
? {
|
||||
[CHANNEL_PROMPT_AUTHORIZATION_META_KEY]: promptAuthorization,
|
||||
}
|
||||
: {}),
|
||||
...(options?.displayText !== undefined
|
||||
? {
|
||||
[CHANNEL_PROMPT_DISPLAY_TEXT_META_KEY]: options.displayText,
|
||||
}
|
||||
: {}),
|
||||
const prompt: Array<Record<string, unknown>> = [];
|
||||
const images = resolvePromptImages(options);
|
||||
if (
|
||||
this.options.sessionAttachments &&
|
||||
uploadAttachment &&
|
||||
removeAttachment
|
||||
) {
|
||||
try {
|
||||
// Fan the uploads out like the webui's attachment path: names are
|
||||
// index-disambiguated and prompt order comes from the array order,
|
||||
// so nothing serializes the uploads themselves.
|
||||
const uploads = await Promise.allSettled(
|
||||
images.map(async (image, index) => {
|
||||
const name = channelImageName(image.mimeType, index);
|
||||
if (!name) {
|
||||
// One unrecognized subtype must not fail the whole turn;
|
||||
// degrade by omission.
|
||||
process.stderr.write(
|
||||
`[DaemonChannelBridge] skipped channel image with unsupported MIME type ${sanitizeLogText(image.mimeType, 128)} for session ${sanitizeLogText(sessionId, 128)}\n`,
|
||||
);
|
||||
return undefined;
|
||||
}
|
||||
const decoded = decodeChannelImage(
|
||||
image.data,
|
||||
'above the daemon attachment size limit',
|
||||
);
|
||||
if ('skip' in decoded) {
|
||||
process.stderr.write(
|
||||
`[DaemonChannelBridge] skipped channel image ${decoded.skip} ${sanitizeLogText(image.mimeType, 128)} for session ${sanitizeLogText(sessionId, 128)}\n`,
|
||||
);
|
||||
return undefined;
|
||||
}
|
||||
const attachment = await uploadAttachment(
|
||||
new Blob([decoded.bytes], {
|
||||
type: image.mimeType,
|
||||
}),
|
||||
name,
|
||||
image.mimeType,
|
||||
controller.signal,
|
||||
);
|
||||
const attachmentId = getString(attachment['attachmentId']);
|
||||
if (attachmentId) uploadedAttachmentIds.push(attachmentId);
|
||||
return attachment;
|
||||
}),
|
||||
);
|
||||
const failure = uploads.find(
|
||||
(upload): upload is PromiseRejectedResult =>
|
||||
upload.status === 'rejected',
|
||||
);
|
||||
if (failure) {
|
||||
throw failure.reason;
|
||||
}
|
||||
for (const upload of uploads) {
|
||||
if (upload.status === 'fulfilled' && upload.value) {
|
||||
prompt.push(upload.value);
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
rollbackUploadedAttachments = true;
|
||||
throw error;
|
||||
}
|
||||
} else {
|
||||
// Daemons without `session_attachments` take images inline.
|
||||
let inlineBase64Bytes = 0;
|
||||
for (const image of images) {
|
||||
const decoded = decodeChannelImage(
|
||||
image.data,
|
||||
'above the inline image budget',
|
||||
);
|
||||
if ('skip' in decoded) {
|
||||
process.stderr.write(
|
||||
`[DaemonChannelBridge] skipped channel image ${decoded.skip} ${sanitizeLogText(image.mimeType, 128)} for session ${sanitizeLogText(sessionId, 128)}\n`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
if (
|
||||
inlineBase64Bytes + image.data.length >
|
||||
CHANNEL_IMAGE_INLINE_MAX_BASE64_BYTES
|
||||
) {
|
||||
process.stderr.write(
|
||||
`[DaemonChannelBridge] skipped channel image to keep the inline prompt under the daemon body limit ${sanitizeLogText(image.mimeType, 128)} for session ${sanitizeLogText(sessionId, 128)}\n`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
inlineBase64Bytes += image.data.length;
|
||||
prompt.push({
|
||||
type: 'image',
|
||||
data: image.data,
|
||||
mimeType: image.mimeType,
|
||||
});
|
||||
}
|
||||
}
|
||||
prompt.push({ type: 'text', text });
|
||||
if (controller.signal.aborted) {
|
||||
rollbackUploadedAttachments = true;
|
||||
controller.signal.throwIfAborted();
|
||||
}
|
||||
// Always presented: the daemon validates it for the channel-turn
|
||||
// classification as well as the display projection, and channel
|
||||
// prompts without display text still need the classification.
|
||||
const promptAuthorization = this.options.promptAuthorization;
|
||||
|
||||
// Aborted after the uploads settled but before admission: the SDK
|
||||
// rejects an already-aborted signal with a pre-request AbortError that
|
||||
// isDefinitePromptAdmissionRejection does not match, so the uploads
|
||||
// would leak. Non-admission is certain at this point; roll back.
|
||||
if (controller.signal.aborted) {
|
||||
rollbackUploadedAttachments = true;
|
||||
throw controller.signal.reason;
|
||||
}
|
||||
|
||||
let result: { stopReason?: string; [key: string]: unknown };
|
||||
try {
|
||||
result = await session.prompt(
|
||||
{
|
||||
prompt,
|
||||
_meta: {
|
||||
[CHANNEL_PROMPT_META_KEY]: true,
|
||||
...(promptAuthorization
|
||||
? {
|
||||
[CHANNEL_PROMPT_AUTHORIZATION_META_KEY]:
|
||||
promptAuthorization,
|
||||
}
|
||||
: {}),
|
||||
...(options?.displayText !== undefined
|
||||
? {
|
||||
[CHANNEL_PROMPT_DISPLAY_TEXT_META_KEY]: options.displayText,
|
||||
}
|
||||
: {}),
|
||||
},
|
||||
},
|
||||
},
|
||||
controller.signal,
|
||||
);
|
||||
controller.signal,
|
||||
);
|
||||
} catch (error) {
|
||||
// Roll back only when the turn was never admitted; once admitted the
|
||||
// daemon may already have resolved the uploads.
|
||||
if (isDefinitePromptAdmissionRejection(error)) {
|
||||
rollbackUploadedAttachments = true;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
// Prefer turn_complete for deterministic chunk collection (SSE path).
|
||||
// Fall back to one event-loop tick for non-SSE prompt paths (blocking
|
||||
// HTTP, non-202 responses) where turn_complete never arrives.
|
||||
|
|
@ -470,6 +671,24 @@ export class DaemonChannelBridge
|
|||
) {
|
||||
this.activePromptControllers.delete(sessionId);
|
||||
}
|
||||
if (rollbackUploadedAttachments && removeAttachment) {
|
||||
const removals = await Promise.allSettled(
|
||||
uploadedAttachmentIds.map((attachmentId) =>
|
||||
removeAttachment(attachmentId),
|
||||
),
|
||||
);
|
||||
removals.forEach((removal, index) => {
|
||||
if (removal.status === 'rejected') {
|
||||
const reason =
|
||||
removal.reason instanceof Error
|
||||
? removal.reason.message
|
||||
: String(removal.reason);
|
||||
process.stderr.write(
|
||||
`[DaemonChannelBridge] failed to remove channel image ${sanitizeLogText(uploadedAttachmentIds[index] ?? '', 128)} for session ${sanitizeLogText(sessionId, 128)} during rollback: ${sanitizeLogText(reason, 256)}\n`,
|
||||
);
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -77,6 +77,8 @@ function daemonSession(
|
|||
sessionId,
|
||||
workspaceCwd: '/tmp',
|
||||
prompt: vi.fn().mockResolvedValue({}),
|
||||
uploadAttachment: vi.fn(),
|
||||
removeAttachment: vi.fn().mockResolvedValue(true),
|
||||
events: vi.fn(async function* (options?: { signal?: AbortSignal }) {
|
||||
await new Promise<void>((resolve) => {
|
||||
if (options?.signal?.aborted) {
|
||||
|
|
|
|||
|
|
@ -3790,6 +3790,74 @@ describe('DingtalkChannel quoted media', () => {
|
|||
).onMessage(downstream);
|
||||
}
|
||||
|
||||
it('downloads every picture in one richText callback', async () => {
|
||||
const downloadCodes: string[] = [];
|
||||
vi.spyOn(globalThis, 'fetch').mockImplementation(
|
||||
(input: RequestInfo | URL, init?: RequestInit) => {
|
||||
const url = String(input);
|
||||
if (url.startsWith('https://oapi.dingtalk.com/gettoken')) {
|
||||
return Promise.resolve(
|
||||
new Response(
|
||||
JSON.stringify({ errcode: 0, access_token: 'app-token' }),
|
||||
{ status: 200 },
|
||||
),
|
||||
);
|
||||
}
|
||||
if (
|
||||
url === 'https://api.dingtalk.com/v1.0/robot/messageFiles/download'
|
||||
) {
|
||||
const request = JSON.parse(String(init?.body)) as {
|
||||
downloadCode: string;
|
||||
};
|
||||
downloadCodes.push(request.downloadCode);
|
||||
return Promise.resolve(
|
||||
new Response(
|
||||
JSON.stringify({
|
||||
downloadUrl: `https://example.com/${request.downloadCode}`,
|
||||
}),
|
||||
{ status: 200 },
|
||||
),
|
||||
);
|
||||
}
|
||||
const bytes = url.endsWith('/picture-1')
|
||||
? new Uint8Array([1])
|
||||
: new Uint8Array([2]);
|
||||
return Promise.resolve(
|
||||
new Response(bytes, {
|
||||
status: 200,
|
||||
headers: { 'content-type': 'image/png' },
|
||||
}),
|
||||
);
|
||||
},
|
||||
);
|
||||
const channel = createChannel();
|
||||
|
||||
sendDirectMedia(channel, 'richText', {
|
||||
richText: [
|
||||
{ type: 'picture', downloadCode: 'picture-1' },
|
||||
{ type: 'picture', downloadCode: 'picture-2' },
|
||||
],
|
||||
});
|
||||
|
||||
await vi.waitFor(() => {
|
||||
expect(channel.handleInbound).toHaveBeenCalledOnce();
|
||||
});
|
||||
const envelope = vi.mocked(channel.handleInbound).mock.calls[0]![0];
|
||||
expect(downloadCodes).toEqual(['picture-1', 'picture-2']);
|
||||
expect(envelope.attachments).toEqual([
|
||||
{
|
||||
type: 'image',
|
||||
data: Buffer.from([1]).toString('base64'),
|
||||
mimeType: 'image/png',
|
||||
},
|
||||
{
|
||||
type: 'image',
|
||||
data: Buffer.from([2]).toString('base64'),
|
||||
mimeType: 'image/png',
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it('downloads a replied picture and attaches it to the prompt', async () => {
|
||||
const downloadCodes = mockMediaDownload(
|
||||
'image/png',
|
||||
|
|
@ -4000,11 +4068,7 @@ describe('DingtalkChannel quoted media', () => {
|
|||
if (filePath) tempDirs.add(dirname(filePath));
|
||||
});
|
||||
|
||||
// R4-1: ChannelBase resolves a single inline image per envelope (the first
|
||||
// data-only image attachment fills imageBase64) and silently drops every
|
||||
// later data-only attachment, so the quoted image must be file-backed when
|
||||
// the message's own image already occupies the slot.
|
||||
it('file-backs a quoted image when the message already carries its own image', async () => {
|
||||
it('keeps a quoted image data-backed when the message carries its own image', async () => {
|
||||
const downloadCodes = mockMediaDownload(
|
||||
'image/png',
|
||||
new Uint8Array([1, 2, 3]),
|
||||
|
|
@ -4054,26 +4118,18 @@ describe('DingtalkChannel quoted media', () => {
|
|||
referencedText: '[image]',
|
||||
});
|
||||
expect(envelope.attachments).toHaveLength(2);
|
||||
// The own image keeps the single inline slot ChannelBase resolves.
|
||||
expect(envelope.attachments?.[0]).toEqual({
|
||||
type: 'image',
|
||||
data: Buffer.from([1, 2, 3]).toString('base64'),
|
||||
mimeType: 'image/png',
|
||||
});
|
||||
// The quoted image must not be a second data-only attachment — that shape
|
||||
// is silently dropped by ChannelBase's single-image resolution.
|
||||
const quotedAttachment = envelope.attachments?.[1];
|
||||
expect(quotedAttachment).toMatchObject({
|
||||
type: 'image',
|
||||
mimeType: 'image/png',
|
||||
});
|
||||
expect(quotedAttachment).not.toHaveProperty('data');
|
||||
const filePath = quotedAttachment?.filePath;
|
||||
if (filePath) tempDirs.add(dirname(filePath));
|
||||
expect(filePath).toBeTruthy();
|
||||
expect(existsSync(filePath!)).toBe(true);
|
||||
expect(readFileSync(filePath!)).toEqual(Buffer.from([1, 2, 3]));
|
||||
expect(quotedAttachment?.fileName).toMatch(/^dingtalk_image_\d+\.png$/);
|
||||
expect(envelope.attachments).toEqual([
|
||||
{
|
||||
type: 'image',
|
||||
data: Buffer.from([1, 2, 3]).toString('base64'),
|
||||
mimeType: 'image/png',
|
||||
},
|
||||
{
|
||||
type: 'image',
|
||||
data: Buffer.from([1, 2, 3]).toString('base64'),
|
||||
mimeType: 'image/png',
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it('cleans the generated placeholder for a direct file message', async () => {
|
||||
|
|
|
|||
|
|
@ -2116,16 +2116,7 @@ export class DingtalkChannel extends ChannelBase {
|
|||
const media = await downloadMedia(downloadCode, robotCode, token);
|
||||
if (!media) return;
|
||||
|
||||
// ChannelBase fills a single imageBase64 slot from the FIRST data-only
|
||||
// image attachment and silently drops every later one, so an image
|
||||
// arriving after the slot is taken (e.g. a quoted picture alongside the
|
||||
// message's own picture) falls through to the file-backed path — the
|
||||
// `saved to:` prompt line is what keeps it reachable for the agent.
|
||||
const inlineImageSlotFree = !(envelope.attachments || []).some(
|
||||
(attachment) => attachment.type === 'image' && attachment.data,
|
||||
);
|
||||
|
||||
if (mediaType === 'image' && inlineImageSlotFree) {
|
||||
if (mediaType === 'image') {
|
||||
const mimeType = media.mimeType.startsWith('image/')
|
||||
? media.mimeType
|
||||
: 'image/jpeg';
|
||||
|
|
@ -2341,15 +2332,17 @@ export class DingtalkChannel extends ChannelBase {
|
|||
}
|
||||
|
||||
const processMessage = async () => {
|
||||
// Download media if present (first downloadCode only for images)
|
||||
// Download media in callback order.
|
||||
if (content.downloadCodes.length > 0 && content.mediaType) {
|
||||
await this.attachMedia(
|
||||
envelope,
|
||||
content.downloadCodes[0]!,
|
||||
content.mediaType,
|
||||
content.fileName,
|
||||
content.placeholder,
|
||||
);
|
||||
for (const downloadCode of content.downloadCodes) {
|
||||
await this.attachMedia(
|
||||
envelope,
|
||||
downloadCode,
|
||||
content.mediaType,
|
||||
content.fileName,
|
||||
content.placeholder,
|
||||
);
|
||||
}
|
||||
}
|
||||
if (quoted.media) {
|
||||
await this.attachMedia(
|
||||
|
|
|
|||
|
|
@ -1319,6 +1319,43 @@ describe('runChannelDaemonWorker', () => {
|
|||
expect(bridgeFacade.shellCommand).toBeTypeOf('function');
|
||||
});
|
||||
|
||||
it('enables attachment uploads only when capabilities include session_attachments', async () => {
|
||||
const sdk = createSdk();
|
||||
sdk.client.capabilities.mockResolvedValueOnce({
|
||||
v: 1,
|
||||
mode: 'http-bridge',
|
||||
features: ['session_attachments'],
|
||||
modelServices: [],
|
||||
workspaceCwd: '/workspace',
|
||||
});
|
||||
|
||||
await runChannelDaemonWorker({
|
||||
daemonUrl: 'http://127.0.0.1:4170',
|
||||
workspace: '/workspace',
|
||||
selection: { mode: 'names', names: ['telegram'] },
|
||||
loadDaemonSdk: async () => sdk,
|
||||
});
|
||||
|
||||
expect(mockDaemonChannelBridge).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ sessionAttachments: true }),
|
||||
);
|
||||
});
|
||||
|
||||
it('keeps attachment uploads off for daemons without session_attachments', async () => {
|
||||
const sdk = createSdk();
|
||||
|
||||
await runChannelDaemonWorker({
|
||||
daemonUrl: 'http://127.0.0.1:4170',
|
||||
workspace: '/workspace',
|
||||
selection: { mode: 'names', names: ['telegram'] },
|
||||
loadDaemonSdk: async () => sdk,
|
||||
});
|
||||
|
||||
expect(mockDaemonChannelBridge).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ sessionAttachments: false }),
|
||||
);
|
||||
});
|
||||
|
||||
it('fails fast for unknown selected channel names', async () => {
|
||||
const sdk = createSdk();
|
||||
|
||||
|
|
|
|||
|
|
@ -95,6 +95,7 @@ import {
|
|||
} from './loop-runtime.js';
|
||||
|
||||
const SESSION_SHELL_COMMAND_FEATURE = 'session_shell_command';
|
||||
const SESSION_ATTACHMENTS_FEATURE = 'session_attachments';
|
||||
const MAX_ACTIVE_WEBHOOK_TASKS = 16;
|
||||
const WORKER_SHUTDOWN_DRAIN_MS = 10_000;
|
||||
|
||||
|
|
@ -492,6 +493,9 @@ export async function runChannelDaemonWorker(
|
|||
DaemonSessionClient: sdk.DaemonSessionClient,
|
||||
clientId: `qwen-channel-worker:${process.pid}`,
|
||||
}),
|
||||
sessionAttachments: capabilities.features.includes(
|
||||
SESSION_ATTACHMENTS_FEATURE,
|
||||
),
|
||||
...(opts.promptAuthorization
|
||||
? { promptAuthorization: opts.promptAuthorization }
|
||||
: {}),
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue