mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-08-16 04:05:15 +00:00
cf03af5d31
* feat(web-shell): stream-highlight code blocks and fix fence-language aliases - Highlight assistant code fences live as they stream in via @shikijs/stream, replacing the per-token whole-block re-tokenize that flickered between plain grey and colored and only settled at the end of the turn. - Fix fence-language aliases (`ts`/`js`/`py`/... and uppercase like `SQL`) that previously rendered unhighlighted because the supported-language gate only matched canonical Shiki ids. - Consolidate the static and streaming paths onto a single lazily-created JavaScript-regex-engine highlighter (no WASM, on-demand language loading) and hand off without a grey flash; bump shiki ^1 -> ^4 and add @shikijs/stream. * fix(web-shell): address review feedback on streaming code highlight - Stabilize the `code` renderer via an IsStreamingContext so the CodeBlock/StreamingCodeBlock instances are reused (not remounted) across the streaming→settled transition, keeping the no-flash hold effective; drop the now-redundant COMPONENTS_STREAMING split. - Log a warning when the streaming highlighter fails to load, instead of failing silently to plain text. - Export `enqueueSuffix` and add unit tests for its no-op/extend/diverge/throw branches, plus a streaming→settled hand-off render test. * fix(web-shell): harden code highlighter (review round 2) - Guard highlightToHtmlSync with try/catch so a synchronous tokenization error falls back to the async path instead of crashing the render tree. - Dedupe concurrent loadLanguage calls via a pending-languages map (two callers could previously both pass the `has` check and double-load). - Add unit tests for codeHighlighter: highlightToHtml output, the sync cold→warm transition, the 30K-char sync bail-out, and concurrent-load dedup. * fix(web-shell): guard oversized streaming fences + review cleanups - Fall back to the plain renderer while streaming once the block or its trailing line exceeds a threshold (`exceedsStreamingLimit`), so a large single-line / minified fence can't pin the UI via @shikijs/stream re-tokenizing the unstable current line on every chunk. - Drop `shell`/`zsh` from SUPPORTED_LANGUAGES — they were dead entries shadowed by the alias map (→ bash). - Tests: add `__resetForTesting` to drop cross-test ordering coupling; add highlighter retry / pending-language cleanup tests, `exceedsStreamingLimit` unit tests, an oversized-fence fallback test, and a highlighter-load-failure fallback test for StreamingCodeBlock. * refactor(web-shell): stream-highlight code blocks via synchronous per-chunk re-render (drop @shikijs/stream) --------- Co-authored-by: 易良 <1204183885@qq.com>
76 lines
2.6 KiB
TypeScript
76 lines
2.6 KiB
TypeScript
import { beforeEach, describe, expect, it } from 'vitest';
|
|
import {
|
|
__resetForTesting,
|
|
getCachedHtml,
|
|
getCodeHighlighter,
|
|
highlightToHtmlSync,
|
|
isTooLargeToHighlight,
|
|
MAX_HIGHLIGHT_LINE_CHARS,
|
|
MAX_HIGHLIGHT_TOTAL_CHARS,
|
|
} from './codeHighlighter';
|
|
|
|
const THEME = 'github-dark-default';
|
|
|
|
// Reset the module-level highlighter singleton so each test is order-independent
|
|
// (loadedLanguages would otherwise accumulate across tests).
|
|
beforeEach(() => {
|
|
__resetForTesting();
|
|
});
|
|
|
|
describe('isTooLargeToHighlight', () => {
|
|
it('allows normal multi-line code', () => {
|
|
expect(isTooLargeToHighlight('const x = 1;\nconst y = 2;\n')).toBe(false);
|
|
});
|
|
|
|
it('bails on any single line over the per-line limit (anywhere in the block)', () => {
|
|
expect(
|
|
isTooLargeToHighlight('x'.repeat(MAX_HIGHLIGHT_LINE_CHARS + 1)),
|
|
).toBe(true);
|
|
// A long line earlier in the block (not just the trailing one) also bails.
|
|
expect(
|
|
isTooLargeToHighlight(
|
|
'x'.repeat(MAX_HIGHLIGHT_LINE_CHARS + 1) + '\nshort',
|
|
),
|
|
).toBe(true);
|
|
});
|
|
|
|
it('bails when the whole block exceeds the total limit (many short lines)', () => {
|
|
const line = 'a'.repeat(80) + '\n';
|
|
const block = line.repeat(
|
|
Math.ceil(MAX_HIGHLIGHT_TOTAL_CHARS / line.length) + 5,
|
|
);
|
|
expect(block.length).toBeGreaterThan(MAX_HIGHLIGHT_TOTAL_CHARS);
|
|
expect(isTooLargeToHighlight(block)).toBe(true);
|
|
});
|
|
});
|
|
|
|
describe('codeHighlighter', () => {
|
|
it('highlightToHtmlSync is null until the language is warm, then returns HTML', async () => {
|
|
// Cold: the language has not been loaded yet.
|
|
expect(highlightToHtmlSync('SELECT 1', 'sql', THEME)).toBeNull();
|
|
await getCodeHighlighter('sql');
|
|
expect(highlightToHtmlSync('SELECT 1', 'sql', THEME)).toContain('shiki');
|
|
});
|
|
|
|
it('does not persist streaming intermediates when persist=false', async () => {
|
|
await getCodeHighlighter('sql');
|
|
// persist=false highlights but doesn't write the cache...
|
|
expect(highlightToHtmlSync('SELECT 2', 'sql', THEME, false)).toContain(
|
|
'shiki',
|
|
);
|
|
expect(getCachedHtml('SELECT 2', 'sql', THEME)).toBeNull();
|
|
// ...persist=true (default) does.
|
|
highlightToHtmlSync('SELECT 3', 'sql', THEME);
|
|
expect(getCachedHtml('SELECT 3', 'sql', THEME)).toContain('shiki');
|
|
});
|
|
|
|
it('dedupes concurrent loads of the same language without throwing', async () => {
|
|
const results = await Promise.all([
|
|
getCodeHighlighter('python'),
|
|
getCodeHighlighter('python'),
|
|
getCodeHighlighter('python'),
|
|
]);
|
|
expect(results).toHaveLength(3);
|
|
expect(highlightToHtmlSync('x = 1', 'python', THEME)).toContain('shiki');
|
|
});
|
|
});
|