kimi-code/packages/kosong/test/anthropic-errors.test.ts
Petrichor cdbd33c13c
fix(kosong): fail fast on quota-exhausted 429 instead of retrying (#1857)
* fix(kosong): fail fast on quota-exhausted 429 instead of retrying

A 429 caused by an exhausted account quota or insufficient balance
(Moonshot error.type "exceeded_current_quota_error", OpenAI
"insufficient_quota") can never succeed on retry, yet it was classified
as APIProviderRateLimitError and silently retried for the whole budget
(10 attempts, ~3 minutes of backoff) with no UI feedback — the session
appeared frozen on every request.

Introduce APIProviderQuotaExhaustedError, minted in
normalizeAPIStatusError from the structured body error.type/error.code
forwarded by convertOpenAIError, with billing-anchored message patterns
as a fallback for gateways that flatten the body to text. The new class
is excluded from isRetryableGenerateError (fail fast, even when a
retry-after header is present) and from isProviderRateLimitError (no
swarm requeue/suspend). toKimiErrorPayload and translateProviderError
map it to provider.api_error (retryable: false) instead of
provider.rate_limit, and classifyApiError reports it as
quota_exhausted in telemetry. agent-core-v2 mirrors the same fix.

Transient rate-limit 429s keep the existing retry, backoff, and
Retry-After behavior (verified end-to-end against a mock provider:
quota body fails after attempt 1/10; rate-limit body still walks the
full 10-attempt ladder).

Behavior changes to note: quota-failed swarm subagents now fail
instead of suspending indefinitely as "Rate limited...", and quota
errors cross the wire as provider.api_error rather than
provider.rate_limit.

* fix(kosong): classify quota exhaustion in OpenAI Responses stream errors

Responses response.failed / error SSE events carry no HTTP status and
were minted by errorFromOpenAIResponsesEvent as either a rate-limit
error (rate_limit_exceeded / embedded status_code=429) or a base
ChatProviderError — and the base class falls into the retryable
unclassified-failure fallback, so an insufficient_quota event still
burned the whole retry budget on the openai_responses path. Route the
event code and message through the same quota-exhausted check before
the rate-limit branch, in kosong and the agent-core-v2 mirror. Covers
all three entry paths (error events, response.failed, nested gateway
frames) since they share the single converter.

* style(agent-core-v2): drop inline comments per AGENTS.md header-only rule

agent-core-v2 comments live solely in the top-of-file block, never
beside functions or statements; the kosong twins keep the full
rationale.

* refactor(kosong,agent-core-v2): move quota-429 checks to vendor hook

Per review on #1857: the knowledge of how a backend signals quota
exhaustion is vendor-specific and must not run for every
OpenAI-compatible provider from the shared conversion layer.

- Add a convertError hook: ProtocolTrait.convertError in agent-core-v2
  (single-value, last-declarer-wins, bound by composeOpenAIChatHooks /
  composeAnthropicHooks / traitConvertError) and an equivalent optional
  hook parameter on convertOpenAIError / convertAnthropicError. Bases
  consult it with the raw failure (SDK error on HTTP paths, raw event on
  the Responses in-stream path) after the abort guard, before their own
  rules.
- Declare Moonshot's quota signals (exceeded_current_quota_error,
  billing wordings) on the Kimi side: kimiOpenAITrait and
  kimiAnthropicTrait in v2, the KimiChatProvider and KimiFiles catch
  sites in kosong, all through the new classifyKimiQuotaError.
- Drop the options parameter from normalizeAPIStatusError and the
  shared quota code/pattern tables: the contract layer keeps only the
  vendor-neutral APIProviderQuotaExhaustedError type and its retry /
  rate-limit / wire-mapping semantics.
- The OpenAI bases keep recognizing only OpenAI's own documented
  insufficient_quota code (HTTP and Responses stream events) as
  protocol knowledge of that wire.

Behavior: kimi and openai provider types classify exactly as before;
an unregistered vendor speaking Moonshot billing wordings through a
plain openai transport now stays a retryable rate limit by design.

* fix(kosong,agent-core,agent-core-v2): wire kimi quota hook fully

Follow-up to the second review round on #1857, all four findings:

- Kimi-over-Anthropic (legacy engine): AnthropicOptions gains the same
  optional convertError hook as the OpenAI bases, threaded through
  AnthropicStreamedMessage and every catch site, and the provider
  manager's anthropic route now passes classifyKimiQuotaError for
  provider type kimi — a quota-exhausted 429 over this transport
  previously still burned the retry budget. classifyKimiQuotaError now
  also walks error -> .error -> .error.error for the code/type, since
  the Anthropic SDK keeps the full body on .error instead of hoisting.
- v2 telemetry: ApiErrorKind gains 'quota_exhausted' and
  classifyApiError checks APIProviderQuotaExhaustedError before the
  generic 429 branch, matching the legacy engine's reporting.
- Hook contract: converted ChatProviderErrors now pass through before
  the vendor hook is consulted in convertOpenAIError /
  convertAnthropicError (both engines), so the hook sees each raw
  failure exactly once even when a stream-minted error crosses an
  outer catch; tests assert the single consult.
- protocolTrait: the convertError member doc shrinks to the concise
  style and the consult contract moves into the file header's
  composition rules.

* test(kosong,agent-core,agent-core-v2): lock quota hook assembly paths

Third review round on #1857:

- Fix the v2 anthropic base header and AnthropicHooks doc still claiming
  withThinking is the only hook.
- Drop the two remaining non-header JSDoc blocks in protocolTrait.ts per
  the AGENTS.md header-only rule; the consult contract already lives in
  the file header.
- Update the ProtocolTrait contract test to the seventeen-hook shape
  (convertError included) and cover the traitConvertError binding.
- Add real-assembly regression probes: the v2 registry composes a
  (kimi, anthropic) provider whose mocked SDK client throws a Moonshot
  quota 429 and generate rejects with the non-retryable
  APIProviderQuotaExhaustedError (a plain anthropic composition keeps
  the same 429 retryable); the legacy ProviderManager routing test
  asserts convertError is classifyKimiQuotaError on the kimi-anthropic
  route and absent for plain anthropic; the legacy provider threads
  options.convertError to its generate catch.

* test(kosong,agent-core-v2): cover KimiFiles quota 429 and drop stale docs

Fourth review round on #1857:

- Drop the AnthropicHooks member JSDoc (its content already lives in the
  anthropic.ts and anthropicHooks.ts file headers) and fix the anthropic
  contrib header still calling the hook set single-hook.
- Add the missing KimiFiles regression in both engines: a mocked files
  client rejecting with a Moonshot quota 429 makes uploadVideo reject
  with the non-retryable APIProviderQuotaExhaustedError, locking the
  classifyKimiQuotaError argument at the upload catch sites.
2026-07-28 14:35:12 +08:00

526 lines
18 KiB
TypeScript

import {
APIConnectionError,
APIContextOverflowError,
APIProviderQuotaExhaustedError,
APIProviderRateLimitError,
APIStatusError,
APITimeoutError,
ChatProviderError,
isRetryableGenerateError,
} from '#/errors';
import { convertAnthropicError, AnthropicChatProvider } from '#/providers/anthropic';
import { classifyKimiQuotaError } from '#/providers/kimi-errors';
import {
APIConnectionError as AnthropicConnectionError,
APIConnectionTimeoutError as AnthropicTimeoutError,
APIError as AnthropicAPIError,
AnthropicError,
APIUserAbortError as AnthropicUserAbortError,
AuthenticationError as AnthropicAuthenticationError,
RateLimitError as AnthropicRateLimitError,
} from '@anthropic-ai/sdk';
import { describe, it, expect, vi } from 'vitest';
describe('convertAnthropicError', () => {
it('APIConnectionTimeoutError -> APITimeoutError (not misclassified as connection)', () => {
const err = new AnthropicTimeoutError({ message: 'timed out' });
const result = convertAnthropicError(err);
expect(result).toBeInstanceOf(APITimeoutError);
// Must NOT be a plain APIConnectionError
expect(result.constructor).toBe(APITimeoutError);
});
it('APIConnectionError -> APIConnectionError', () => {
const err = new AnthropicConnectionError({ message: 'connection refused' });
const result = convertAnthropicError(err);
expect(result).toBeInstanceOf(APIConnectionError);
});
it('APIError with status -> APIStatusError', () => {
const err = AnthropicAPIError.generate(
502,
{ type: 'error', error: { type: 'api_error', message: 'bad gateway' } },
'bad gateway',
new Headers(),
);
const result = convertAnthropicError(err);
expect(result).toBeInstanceOf(APIStatusError);
expect((result as APIStatusError).statusCode).toBe(502);
});
it('context overflow APIError -> APIContextOverflowError', () => {
const err = AnthropicAPIError.generate(
422,
{
type: 'error',
error: {
type: 'invalid_request_error',
message: 'prompt is too long: 210000 tokens exceeds the maximum',
},
},
'prompt is too long: 210000 tokens exceeds the maximum',
new Headers(),
);
const result = convertAnthropicError(err);
expect(result).toBeInstanceOf(APIContextOverflowError);
expect((result as APIContextOverflowError).statusCode).toBe(422);
});
it('AuthenticationError -> APIStatusError with 401', () => {
const err = new AnthropicAuthenticationError(
401,
{ type: 'error', error: { type: 'authentication_error', message: 'invalid key' } },
'invalid key',
new Headers(),
);
const result = convertAnthropicError(err);
expect(result).toBeInstanceOf(APIStatusError);
expect((result as APIStatusError).statusCode).toBe(401);
});
it('RateLimitError -> APIProviderRateLimitError with 429', () => {
const err = new AnthropicRateLimitError(
429,
{ type: 'error', error: { type: 'rate_limit_error', message: 'rate limited' } },
'rate limited',
new Headers(),
);
const result = convertAnthropicError(err);
expect(result).toBeInstanceOf(APIProviderRateLimitError);
expect((result as APIProviderRateLimitError).statusCode).toBe(429);
});
it('reads an integer retry-after header (seconds) onto the rate-limit error', () => {
const err = AnthropicAPIError.generate(
429,
{ type: 'error', error: { type: 'rate_limit_error', message: 'rate limited' } },
'rate limited',
new Headers({ 'retry-after': '7' }),
);
const result = convertAnthropicError(err);
expect(result).toBeInstanceOf(APIProviderRateLimitError);
expect((result as APIProviderRateLimitError).retryAfterMs).toBe(7_000);
});
it('ignores a non-integer (HTTP-date) retry-after header, leaving retryAfterMs null', () => {
const err = AnthropicAPIError.generate(
429,
{ type: 'error', error: { type: 'rate_limit_error', message: 'rate limited' } },
'rate limited',
new Headers({ 'retry-after': 'Wed, 21 Oct 2026 07:28:00 GMT' }),
);
const result = convertAnthropicError(err);
expect(result).toBeInstanceOf(APIProviderRateLimitError);
expect((result as APIProviderRateLimitError).retryAfterMs).toBeNull();
});
it('generic AnthropicError -> ChatProviderError', () => {
const err = new AnthropicError('something went wrong');
const result = convertAnthropicError(err);
expect(result).toBeInstanceOf(ChatProviderError);
expect(result.message).toContain('something went wrong');
});
it('plain Error -> ChatProviderError', () => {
const err = new Error('unexpected');
const result = convertAnthropicError(err);
expect(result).toBeInstanceOf(ChatProviderError);
expect(result.message).toContain('unexpected');
});
it('non-Error value -> ChatProviderError', () => {
const result = convertAnthropicError('string error');
expect(result).toBeInstanceOf(ChatProviderError);
expect(result.message).toContain('string error');
});
it('APIUserAbortError throws the standard abort DOMException instead of being classified', () => {
// A user cancellation must never be converted into (or returned as) a
// retryable provider error: the guard at the very front of the
// classification chain throws the standard abort shape.
const err = new AnthropicUserAbortError({ message: 'aborted by user' });
let thrown: unknown;
try {
convertAnthropicError(err);
} catch (error) {
thrown = error;
}
expect(thrown).toBeInstanceOf(DOMException);
expect((thrown as DOMException).name).toBe('AbortError');
expect(isRetryableGenerateError(thrown)).toBe(false);
});
it('bare AbortError DOMException throws the standard abort DOMException', () => {
const err = new DOMException('The operation was aborted.', 'AbortError');
let thrown: unknown;
try {
convertAnthropicError(err);
} catch (error) {
thrown = error;
}
expect(thrown).toBeInstanceOf(DOMException);
expect((thrown as DOMException).name).toBe('AbortError');
expect(isRetryableGenerateError(thrown)).toBe(false);
});
it('classifies undici TypeError("terminated") as a retryable APIConnectionError', () => {
// Node v24 + undici raises a raw `TypeError: terminated` when an SSE
// response stream is dropped mid-flight. It is NOT an Anthropic SDK error,
// so it falls into the generic Error branch — but it is a transport-layer
// connection failure and must be retryable like any dropped connection.
const err = new TypeError('terminated');
(err as { cause?: unknown }).cause = new Error('other side closed');
const result = convertAnthropicError(err);
expect(result).toBeInstanceOf(APIConnectionError);
expect(isRetryableGenerateError(result)).toBe(true);
});
it('still wraps an unrelated raw Error as a base ChatProviderError, now retryable via fallback', () => {
// An unrelated raw Error is NOT an Anthropic SDK error and carries no
// usable HTTP status, so convertAnthropicError wraps it as a base
// ChatProviderError (constructor check guards that typing). The fallback
// safety net in isRetryableGenerateError then treats such unclassified
// provider failures as transient — retry beats failing the run on the
// first blip.
const result = convertAnthropicError(new Error('something completely unrelated'));
expect(result.constructor).toBe(ChatProviderError);
expect(isRetryableGenerateError(result)).toBe(true);
});
});
describe('non-stream error propagation', () => {
function createNonStreamProvider(): AnthropicChatProvider {
return new AnthropicChatProvider({
model: 'k25',
apiKey: 'test-key',
defaultMaxTokens: 1024,
stream: false,
});
}
it('APIConnectionTimeoutError during generate is converted', async () => {
const provider = createNonStreamProvider();
const sdkError = new AnthropicTimeoutError({ message: 'stream timed out' });
(provider as any)._client.messages.create = vi.fn().mockRejectedValue(sdkError);
await expect(
provider.generate(
'',
[],
[{ role: 'user', content: [{ type: 'text', text: 'Hi' }], toolCalls: [] }],
),
).rejects.toThrow(APITimeoutError);
});
it('APIConnectionError during generate is converted', async () => {
const provider = createNonStreamProvider();
const sdkError = new AnthropicConnectionError({ message: 'connection reset' });
(provider as any)._client.messages.create = vi.fn().mockRejectedValue(sdkError);
await expect(
provider.generate(
'',
[],
[{ role: 'user', content: [{ type: 'text', text: 'Hi' }], toolCalls: [] }],
),
).rejects.toThrow(APIConnectionError);
});
it('APIError with status during generate is converted to APIStatusError', async () => {
const provider = createNonStreamProvider();
const sdkError = AnthropicAPIError.generate(
500,
{ type: 'error', error: { type: 'api_error', message: 'internal error' } },
'internal error',
new Headers(),
);
(provider as any)._client.messages.create = vi.fn().mockRejectedValue(sdkError);
await expect(
provider.generate(
'',
[],
[{ role: 'user', content: [{ type: 'text', text: 'Hi' }], toolCalls: [] }],
),
).rejects.toThrow(APIStatusError);
});
it('RateLimitError during generate is converted to APIProviderRateLimitError(429)', async () => {
const provider = createNonStreamProvider();
const sdkError = new AnthropicRateLimitError(
429,
{ type: 'error', error: { type: 'rate_limit_error', message: 'too many requests' } },
'too many requests',
new Headers(),
);
(provider as any)._client.messages.create = vi.fn().mockRejectedValue(sdkError);
try {
await provider.generate(
'',
[],
[{ role: 'user', content: [{ type: 'text', text: 'Hi' }], toolCalls: [] }],
);
expect.unreachable('Should have thrown');
} catch (error) {
expect(error).toBeInstanceOf(APIProviderRateLimitError);
expect((error as APIProviderRateLimitError).statusCode).toBe(429);
}
});
it('AuthenticationError during generate is converted to APIStatusError(401)', async () => {
const provider = createNonStreamProvider();
const sdkError = new AnthropicAuthenticationError(
401,
{ type: 'error', error: { type: 'authentication_error', message: 'invalid' } },
'invalid',
new Headers(),
);
(provider as any)._client.messages.create = vi.fn().mockRejectedValue(sdkError);
try {
await provider.generate(
'',
[],
[{ role: 'user', content: [{ type: 'text', text: 'Hi' }], toolCalls: [] }],
);
expect.unreachable('Should have thrown');
} catch (error) {
expect(error).toBeInstanceOf(APIStatusError);
expect((error as APIStatusError).statusCode).toBe(401);
}
});
});
describe('stream error propagation', () => {
function createStreamProvider(): AnthropicChatProvider {
return new AnthropicChatProvider({
model: 'k25',
apiKey: 'test-key',
defaultMaxTokens: 1024,
stream: true,
});
}
function makeErrorStream(error: Error) {
return {
async *[Symbol.asyncIterator]() {
yield {
type: 'message_start',
message: { id: 'msg_err', usage: { input_tokens: 0 } },
};
throw error;
},
};
}
it('APIConnectionTimeoutError during stream iteration is converted', async () => {
const provider = createStreamProvider();
const sdkError = new AnthropicTimeoutError({ message: 'stream timed out' });
(provider as any)._client.messages.create = vi
.fn()
.mockResolvedValue(makeErrorStream(sdkError)) as never;
const result = await provider.generate(
'',
[],
[{ role: 'user', content: [{ type: 'text', text: 'Hi' }], toolCalls: [] }],
);
const parts: unknown[] = [];
await expect(
(async () => {
for await (const part of result) {
parts.push(part);
}
})(),
).rejects.toThrow(APITimeoutError);
});
it('APIConnectionError during stream iteration is converted', async () => {
const provider = createStreamProvider();
const sdkError = new AnthropicConnectionError({ message: 'connection reset' });
(provider as any)._client.messages.create = vi
.fn()
.mockResolvedValue(makeErrorStream(sdkError)) as never;
const result = await provider.generate(
'',
[],
[{ role: 'user', content: [{ type: 'text', text: 'Hi' }], toolCalls: [] }],
);
await expect(
(async () => {
for await (const _ of result) {
void _;
}
})(),
).rejects.toThrow(APIConnectionError);
});
it('APIError with status during stream iteration is converted to APIStatusError', async () => {
const provider = createStreamProvider();
const sdkError = AnthropicAPIError.generate(
500,
{ type: 'error', error: { type: 'api_error', message: 'internal error' } },
'internal error',
new Headers(),
);
(provider as any)._client.messages.create = vi
.fn()
.mockResolvedValue(makeErrorStream(sdkError)) as never;
const result = await provider.generate(
'',
[],
[{ role: 'user', content: [{ type: 'text', text: 'Hi' }], toolCalls: [] }],
);
await expect(
(async () => {
for await (const _ of result) {
void _;
}
})(),
).rejects.toThrow(APIStatusError);
});
it('RateLimitError during stream iteration is converted to APIProviderRateLimitError(429)', async () => {
const provider = createStreamProvider();
const sdkError = new AnthropicRateLimitError(
429,
{ type: 'error', error: { type: 'rate_limit_error', message: 'too many requests' } },
'too many requests',
new Headers(),
);
(provider as any)._client.messages.create = vi
.fn()
.mockResolvedValue(makeErrorStream(sdkError)) as never;
const result = await provider.generate(
'',
[],
[{ role: 'user', content: [{ type: 'text', text: 'Hi' }], toolCalls: [] }],
);
try {
for await (const _ of result) {
void _;
}
expect.unreachable('Should have thrown');
} catch (error) {
expect(error).toBeInstanceOf(APIProviderRateLimitError);
expect((error as APIProviderRateLimitError).statusCode).toBe(429);
}
});
it('AuthenticationError during stream iteration is converted to APIStatusError(401)', async () => {
const provider = createStreamProvider();
const sdkError = new AnthropicAuthenticationError(
401,
{ type: 'error', error: { type: 'authentication_error', message: 'invalid' } },
'invalid',
new Headers(),
);
(provider as any)._client.messages.create = vi
.fn()
.mockResolvedValue(makeErrorStream(sdkError)) as never;
const result = await provider.generate(
'',
[],
[{ role: 'user', content: [{ type: 'text', text: 'Hi' }], toolCalls: [] }],
);
try {
for await (const _ of result) {
void _;
}
expect.unreachable('Should have thrown');
} catch (error) {
expect(error).toBeInstanceOf(APIStatusError);
expect((error as APIStatusError).statusCode).toBe(401);
}
});
it('undici TypeError("terminated") during stream iteration -> retryable APIConnectionError', async () => {
// The real-world failure: the SSE stream drops mid-flight and undici raises
// a raw `TypeError: terminated` from inside the for-await loop. The provider
// must surface a retryable APIConnectionError so the loop retries instead of
// failing the turn outright.
const provider = createStreamProvider();
(provider as any)._client.messages.create = vi
.fn()
.mockResolvedValue(makeErrorStream(new TypeError('terminated'))) as never;
const result = await provider.generate(
'',
[],
[{ role: 'user', content: [{ type: 'text', text: 'Hi' }], toolCalls: [] }],
);
let caught: unknown;
try {
for await (const _ of result) {
void _;
}
} catch (error) {
caught = error;
}
expect(caught).toBeInstanceOf(APIConnectionError);
expect(isRetryableGenerateError(caught)).toBe(true);
});
});
describe('convertAnthropicError: quota-exhausted 429 via the convertError hook', () => {
const QUOTA_BODY = {
type: 'error',
error: {
type: 'exceeded_current_quota_error',
message:
'Your account org-0123456789abcdef <ak-test> is suspended due to insufficient balance, please recharge your account or check your plan and billing details',
},
};
function quota429(): unknown {
return AnthropicAPIError.generate(429, QUOTA_BODY, 'Too many requests', new Headers());
}
it('keeps vendor quota signals a rate limit without the vendor hook', () => {
const result = convertAnthropicError(quota429());
expect(result).toBeInstanceOf(APIProviderRateLimitError);
expect(isRetryableGenerateError(result)).toBe(true);
});
it('classifies the Kimi quota body as quota-exhausted through the hook', () => {
const result = convertAnthropicError(quota429(), classifyKimiQuotaError);
expect(result).toBeInstanceOf(APIProviderQuotaExhaustedError);
expect(isRetryableGenerateError(result)).toBe(false);
});
it('passes already-converted errors through without re-consulting the hook', () => {
const calls: unknown[] = [];
const converted = new APIProviderQuotaExhaustedError('already classified');
const result = convertAnthropicError(converted, (error) => {
calls.push(error);
return new ChatProviderError('re-classified');
});
expect(result).toBe(converted);
expect(calls).toHaveLength(0);
});
it('the provider threads options.convertError to its generate catch', async () => {
const provider = new AnthropicChatProvider({
model: 'k25',
apiKey: 'test-key',
defaultMaxTokens: 1024,
stream: false,
convertError: classifyKimiQuotaError,
});
(provider as any)._client.messages.create = vi.fn().mockRejectedValue(quota429());
await expect(
provider.generate(
'',
[],
[{ role: 'user', content: [{ type: 'text', text: 'Hi' }], toolCalls: [] }],
),
).rejects.toThrow(APIProviderQuotaExhaustedError);
});
});