mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-08-25 08:33:55 +00:00
43d46be912
* refactor(core): shrink content generator interface * refactor(core): remove orphaned request-tokenizer estimator cluster Removing countTokens from both providers deleted the last production consumers of RequestTokenEstimator. Delete the orphaned cluster: requestTokenizer.ts (330), imageTokenizer.ts (534), types.ts (36), the directory barrel (11), and both test files (608 lines). Also drop the inert vi.mock of requestTokenizer.js left in client.test.ts and the stale dimension-extractor cross-reference in review/lib/assets.ts. textTokenizer.ts and supportedImageFormats.ts stay: converter.ts, pdf.ts, and fileUtils.ts still consume them and the core barrel re-exports them. * docs(design): sync lazy-google-genai-loading record with shrunk interface countTokens and useSummarizedThinking no longer exist on ContentGenerator, so the design record for the lazy-wrapper architecture must not keep advertising them: list the three remaining shared async operations, drop the useSummarizedThinking sentence and the summarized-thinking item from the consumer audit and Verification section, and add a dated note recording the interface shrink from PR #9676. * ci: record cd-cua-driver.yml size growth in .size-baseline Same latent main-side violation as fixed in #9682: #9587 grew the workflow without a baseline update; record the new size as the check message directs (precedent #9747). * docs: finish scrubbing tokenizer references after estimator-cluster removal Follow-up to 0ee17632c7/1871bb5b81 (review round 2): - supportedImageFormats.ts header and getSupportedImageFormatsString doc no longer describe a tokenizer decode/metadata-extraction stage; the list is now documented as the vision-input acceptance list, with token accounting noted as the flat DEFAULT_IMAGE_TOKEN_ESTIMATE. - web-shell-image-drag-and-drop.md's BMP rationale no longer claims ImageTokenizer parses BMP dimensions; dated sync note added stating BMP support rests on SUPPORTED_IMAGE_MIME_TYPES plus converter passthrough since PR #9676. * docs: drop tokenizer from the BMP test-plan line Follow-up to 18f08c0924: the test plan still required converter/tokenizer focused tests for image paths; the image-tokenizer estimator cluster was removed in PR #9676 (text tokenizer is unaffected and out of scope here). |
||
|---|---|---|
| .. | ||
| src | ||
| .gitignore | ||
| index.ts | ||
| package.json | ||
| test-setup.ts | ||
| tsconfig.json | ||
| vitest.config.ts | ||