mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-05 14:30:42 +00:00
7b8055c7f8
* fix(resilience): count STREAM_EARLY_EOF as a provider failure in combo routing A STREAM_EARLY_EOF is an upstream that accepted the request (HTTP 200), opened the SSE stream, then closed it without emitting a single non-ping event. The combo path classified it together with STREAM_READINESS_TIMEOUT through isStreamReadinessFailureErrorBody(), and the readiness exemption in shouldRecordProviderBreakerFailure meant the whole-provider circuit breaker never saw it. During a provider-wide outage that makes the breaker blind. Over a 7-day window on our router we recorded 311 of these events, 302 of them on one model, 265 inside the upstream's published incident window — and the provider breaker sat at CLOSED / failure_count=0 the entire time. Every request kept being dispatched to the failing provider instead of shedding to the next combo target. The two codes are different signals. The readiness probe is a pre-flight liveness check on a connection we have not committed to, so failing it means "this connection looks stale". An early EOF means the provider took the request and then failed to serve it. The single-model path already treats it that way: shouldTripProviderBreakerForResult has no readiness exemption, so a 502 early EOF trips the breaker there. This makes the combo path consistent. isStreamReadinessFailureErrorBody keeps matching both codes, because the transient-retry and round-robin semaphore-cooldown paths in combo.ts do want identical treatment for both. Only the breaker needs to tell them apart, so the distinction is added as a narrow predicate and an optional argument rather than by changing the shared classifier. Omitting the new argument reproduces the previous behaviour exactly. Follows the additive-override pattern established by the isProxyUnreachable work, and leaves the existing exclusions for client aborts and plain 429s untouched. * test: register stream-early-eof-breaker in stryker tap.testFiles The mutation test-coverage gate (check:mutation-test-coverage --strict) detects unit tests that cover a mutated module but are missing from stryker.conf.json tap.testFiles, so their mutant kills would not count. comboPredicates.ts is one of the mutated modules, and the new stream-early-eof-breaker.test.ts covers it, so the gate correctly flagged the omission. 8376-econnrefused-breaker.test.ts -- the test this one is modeled on -- is already registered; this just brings the new file in line. No production code change. --------- Co-authored-by: Nick Sullivan <nick@technick.ai> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| _helpers | ||
| _setup | ||
| benchmarks | ||
| boundary | ||
| e2e | ||
| fixtures | ||
| golden-set | ||
| helpers | ||
| homolog | ||
| integration | ||
| live | ||
| llm-security | ||
| load | ||
| manual | ||
| security | ||
| snapshots | ||
| translator | ||
| unit | ||
| _cp_mock_hook.mts | ||
| _cp_mock_module.mts | ||
| _run_dns_guard_test.sh | ||
| e2e-tool-calls.sh | ||
| scratch_test.mjs | ||
| theoldllm-stress.test.ts | ||