mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-07-09 17:19:02 +00:00
e3a247f99e
estimateTextTokens scanned every string char-by-char via charCodeAt to classify ASCII vs non-ASCII code units. For pure-ASCII text (code, English prose - the common case) a single regex scan using V8's optimized string search replaces the JS loop, and the mixed-text path now counts only non-ASCII units, deriving the ASCII count from the length. The token formula is unchanged, so results are byte-identical for every input; verified exhaustively over all 65536 single code units plus 20k randomized mixed strings against the previous implementation. Median of 6 solo benchmark runs over a deterministic mixed fixture set: 51.9ms -> 32.2ms (-38%, 1.61x). |
||
|---|---|---|
| .. | ||
| acp-bridge | ||
| audio-capture | ||
| channels | ||
| chrome-extension | ||
| cli | ||
| core | ||
| cua-driver | ||
| desktop | ||
| mobile-mcp | ||
| sdk-java | ||
| sdk-python | ||
| sdk-typescript | ||
| vscode-ide-companion | ||
| web-shell | ||
| web-templates | ||
| webui | ||
| zed-extension | ||