mirror of
https://github.com/MoonshotAI/kimi-code.git
synced 2026-07-09 17:29:12 +00:00
42e37eb898
* feat(timing): split TTFT into api-server and client portions Time-to-first-token previously lumped in-process request building (message serialization, param assembly) together with network + server latency, making it impossible to tell whether a slow turn was the client or the API server. Add an `onRequestSent` hook to kosong's GenerateOptions, fired by every provider immediately before it dispatches the network call. The window from request start to dispatch is attributed to the client; the window from dispatch to the first streamed token is attributed to the API server. The split flows through the step.end / turn.step.completed events (and therefore wire.jsonl) and is surfaced in three places: - KIMI_CODE_DEBUG=1: `TTFT: 2.5s (api 2.4s + client 100ms)` - session log: new `llm response` line with the timing breakdown - vis: firstToken/api + firstToken/client rows and timeline label The split is omitted (total only) when a provider does not report the boundary, preserving backward compatibility. * feat(timing): split the decode window into server vs client time Time-to-first-token now reports a client/server split, but the slow part of a long turn is the decode window (inter-token streaming), which was still a single opaque number. Profiling long sessions showed decode throughput halving over a session's lifetime independent of context size, which the synchronous per-chunk stream pipeline can cause: kosong awaits the host callback for every streamed part, so a loaded main thread throttles how fast tokens are pulled off the wire. Account for this directly in the stream loop: the time awaiting the next part (server + network) versus the time spent processing each part in-process (deep copy, host callback, part merge). The split is reported through onStreamEnd and flows through the step.end / turn.step.completed events (and wire.jsonl) into the same three surfaces as the TTFT split: - KIMI_CODE_DEBUG=1: `TPS: 40.0 tok/s (200 tokens in 5.0s; server 4.6s + client 400ms)` - session log: serverDecodeMs / clientConsumeMs on the `llm response` line - vis: streamDuration/server + streamDuration/client rows and timeline label A large, growing client share confirms host-side throttling; a dominant server share points at the server/connection. The per-chunk accounting is wrapped in try/finally so it stays correct across `continue` and aborts, and is omitted when the stream reports nothing.
471 B
471 B
| @moonshot-ai/kimi-code |
|---|
| patch |
Split LLM streaming timing in the session log and KIMI_CODE_DEBUG=1 output into client vs. API-server portions, so slow turns can be attributed without parsing the wire log. Time-to-first-token splits into the API-server portion (network + server) and the client portion (in-process request building); the decode window splits into time awaiting tokens from the server and time the client spends processing each streamed chunk.