qwen-code/esbuild.config.js
ChiGao 9d20536343
perf(cli): code-split lowlight to cut startup V8 parse cost (#4070)
* perf(cli): code-split lowlight to cut startup V8 parse cost

Move the syntax-highlight engine out of the synchronously-parsed cli.js
entry into a separately-emitted chunk and load it via dynamic import on
the first code-block render. Until the chunk arrives, code blocks render
as plain text; the next React commit of the surrounding subtree picks up
the highlighted version, so users never see incorrect highlighting –
just an imperceptibly later transition for the very first code block.

Mechanics:
- esbuild config: switch entry to outdir + splitting:true so that
  `await import('lowlight')` produces an actual on-disk chunk that's
  only parsed by V8 when first needed.
- esbuild-shims: rename injected __dirname/__filename to qwen-prefixed
  symbols + use `define` to redirect free references. Previous inject
  collided with vendored libraries (yargs) that ship their own
  `var __dirname` ESM-compat polyfill once splitting flattens chunks.
- prepare-package: include the new chunks/ directory in the published
  package's files list.
- CodeColorizer: keep the public colorize{Code,Line} signatures and HAST
  rendering identical; on first call when the chunk hasn't loaded it
  returns the plain line and fires the dynamic import via a tiny
  standalone loader module.
- lowlightLoader (new): isolates the lazy-load surface to a module with
  zero transitive imports (no themeManager, settings, or core). This
  lets test-setup prime the cache without dragging the whole UI module
  graph into every test file, which was observed to perturb theme and
  settings test outcomes when CodeColorizer was imported directly.
- test-setup: await loadLowlight() once via the standalone loader so
  synchronous snapshot tests see the highlighted output deterministically.

Measurements (real $HOME, n=15 interleaved A/B vs main HEAD, macOS):

| Metric             | Before (mean±sd ms) | After (mean±sd ms) | Δ        | t      | p        |
| ------------------ | ------------------- | ------------------ | -------- | ------ | -------- |
| firstByte (wall)   | 1633.5 ± 88.7       | 1475.8 ± 73.3      | -157.7   | 5.31   | 1.33e-5  |
| idle (wall)        | 2048.7 ± 93.6       | 1902.3 ± 80.2      | -146.3   | 4.60   | 8.71e-5  |
| cli.js size        | 25 MB               | 6.9 MB             | -18.1 MB | —      | —        |

Both metrics clear the +50ms-or-10% Welch's t-test bar by an order of
magnitude. cli.js drops 72%; total payload (cli.js + chunks/) is
similar but only cli.js is parsed at module-eval time, which is the
phase that dominates the user-visible startup gap.

How to validate:
  npm run bundle
  ls dist/                         # cli.js + chunks/lowlight-*.js
  node dist/cli.js -y              # interactive UI still renders

Generated with AI

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* fix(cli): resolve chunk-relative sibling paths under esbuild splitting

With `splitting: true`, esbuild hoists modules with shared dependencies
into `dist/chunks/`. Three modules derived runtime paths from
`import.meta.url` assuming they were co-located with `cli.js`; once
hoisted, `path.dirname(fileURLToPath(import.meta.url))` resolved to
`dist/chunks/` and sibling-asset lookups silently missed:

- `skill-manager.ts`: bundledSkillsDir → `dist/chunks/bundled` (actual
  `dist/bundled/`). The `existsSync` guard swallowed the miss, dropping
  all four bundled skills (`/review`, `/qc-helper`, `/batch`, `/loop`)
  with no user-visible signal.
- `ripgrepUtils.ts`: `getBuiltinRipgrep()` → `dist/chunks/vendor/...`.
  Falls back to system rg if installed, otherwise null on minimal
  hosts — degrading grep to the slow internal scanner.
- `i18n/index.ts`: `getBuiltinLocalesDir()` → `dist/chunks/locales`.
  User-visible behavior survives via the static glob import in
  `tryImportBundledTranslations`, but the loose-on-disk override path
  is dead.

Each module now strips a trailing `chunks` segment when present, so
the lookup resolves under `dist/`. In source / transpiled modes the
basename is never `chunks`, so the fallback is a no-op.

Also:
- Add `chunks` to `DIST_REQUIRED_PATHS` in `create-standalone-package.js`
  so a regressed bundle that produces only `cli.js` fails the
  pre-packaging check instead of shipping a broken archive.
- Expand `esbuild-shims.js` header so future contributors understand
  that `__qwen_filename` / `__qwen_dirname` always resolve to the
  shim's chunk file (dist/chunks/) and that sibling-asset lookups
  must strip the `chunks` segment.

Reported by claude-opus-4-7 via Qwen Code /qreview on #4070.

Generated with AI

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* perf(cli): prefetch lowlight from AppContainer + harden loader

Three follow-ups to the lowlight code-split:

- AppContainer fires `loadLowlight()` from a mount effect so the dynamic
  import is already in flight before any code block needs colorizing.
  Without this, code blocks committed to ink's append-only `<Static>`
  region before the import resolves stay plain text for the rest of
  the session — Static can only be re-rendered via `refreshStatic`,
  which is not wired to lowlight load completion. Common reachable
  paths: short `--prompt -p` runs that finalize quickly, Ctrl+C-
  cancelled first turns, and the first-paint history replay on
  `--resume`. The startup parse-cost win is preserved (V8 still
  parses off the critical path).

- `lowlightLoader.ts` latches the first import failure so subsequent
  calls short-circuit to a rejected promise instead of re-attempting
  `import('lowlight')` on every keystroke. The colorizer already falls
  back to plain text on miss; recovery requires a fresh process anyway.

- `test-setup.ts` wraps the top-level `await loadLowlight()` in
  try/catch. A transient import failure no longer crashes the entire
  vitest run — tests that hit a code block render the plain-text
  fallback and surface a warning.

- `CodeColorizer.tsx` header comment updated to point at the
  AppContainer prefetch instead of claiming first-paint always sees
  a loaded instance.

Reported by DeepSeek/deepseek-v4-pro and claude-opus-4-7 via Qwen Code
/review and /qreview on #4070.

Generated with AI

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* refactor(bundle): extract resolveBundleDir helper, apply to extensions/new

Centralises the `chunks/` strip pattern that three sites
(`i18n/index.ts`, `skills/skill-manager.ts`, `utils/ripgrepUtils.ts`)
each duplicated after the round-3 fix in d581da04d. The implicit
coupling to `esbuild.config.js`'s `chunkNames: 'chunks/[name]-[hash]'`
now lives in a single helper (`packages/core/src/utils/bundlePaths.ts`),
so a future rename only needs updating in one place.

Also applies the same anchor to `commands/extensions/new.ts:EXAMPLES_PATH`.
That module is currently bundled into `cli.js` (so the strip is a no-op
today), but `qwen extensions new --help` always reads the examples
directory in its yargs `builder` — confirmed against the built bundle
that the lookup hits `dist/examples/` (sibling of `cli.js`). Using the
helper future-proofs against esbuild later hoisting the module into a
shared chunk, where the bare `__dirname`/`import.meta.url` lookup would
silently break the command for every end user.

While here, surface lowlight-load failures from `AppContainer`'s
prefetch effect to the debug channel (`debugLogger.warn`) instead of
swallowing them silently. The loader already latches failures
permanently, so this fires at most once per session; `CodeColorizer`
continues to fall back to plain text on miss, so user-visible behaviour
is unchanged.

Generated with AI

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* fix(bundle): restore __filename shadow in ripgrepUtils; harden lowlight loader

Round-4 review (wenshao 2026-05-13 13:12) flagged five issues in the
recent code-split work. This commit addresses all of them.

CRITICAL — `packages/core/src/utils/ripgrepUtils.ts`: the round-3
`resolveBundleDir` refactor removed the local `__filename` declaration
but `getBuiltinRipgrep` still references bare `__filename` to decide
how many `..` segments to walk. In `npm run dev` (tsx, ESM) `__filename`
is undefined so the function throws `ReferenceError`. In the bundle
esbuild's `define` rewrites it to `__qwen_filename` (the shim chunk
path), which is the wrong string but happens to short-circuit to
`levelsUp = 0` — accidentally correct only because the chunk-path
string never contains `path.join('src', 'utils')`. Reproduced via tsx:
`__filename is not defined`; fixed by re-introducing the explicit
local shadow plus a comment explaining why centralising both helpers
into `resolveBundleDir` cannot replace the per-file shadow.

`packages/cli/src/ui/utils/lowlightLoader.ts`: the previous permanent
`lowlightFailed` latch left syntax highlighting dead for the entire
process lifetime on transient errors (EMFILE, antivirus locks,
slow-disk-after-wake). Replaced with a 30-second cooldown — within the
window subsequent calls return the cached rejection synchronously
(keeps the per-render short-circuit that protects against
permanently-broken installs); after the cooldown the next call retries
the dynamic import. Exposes `isLowlightCoolingDown()` so render-hot
callers can also skip duplicate failure logging.

`packages/cli/src/ui/utils/CodeColorizer.tsx`: hoisted
`loadLowlight()` + log out of the per-line render loop into a single
`ensureLowlightLoading()` call at the top of `colorizeCode`. In the
failure case this collapses hundreds of duplicate debug entries (one
per line) to one per block. The instance is now passed down to
`highlightAndRenderLine` as a parameter.

`packages/core/src/utils/bundlePaths.ts` + `esbuild.config.js`:
exposed `BUNDLE_CHUNK_DIR = 'chunks'` as a named constant and updated
`esbuild.config.js` to interpolate the same name into `chunkNames`
(plus an explicit "MUST stay in sync" comment). Renaming on one side
without the other now stands out at review time. Also expanded the
`define` comment with a contributor-facing warning describing exactly
why bare `__dirname` / `__filename` in source files becomes the shim
chunk path, and pointing future contributors at the
`fileURLToPath(import.meta.url)` shadow pattern (and
`resolveBundleDir` for sibling-asset lookups).

Verified:
- typecheck (all 4 workspaces): clean
- packages/core tests: 7747 passing (no regressions)
- packages/cli tests: only the pre-existing `useAtCompletion.test.ts`
  filesystem-order failures remain (confirmed against `git stash`)
- `npm run bundle` succeeds; `node dist/cli.js --version` returns
  `0.15.10`; `node dist/cli.js --help` renders normally
- `npx tsx <call getBuiltinRipgrep>` now returns the vendored path
  instead of throwing `ReferenceError`

Generated with AI

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* fix(bundle): validate lowlight API shape; sync doc-comment drift; add tests

- lowlightLoader: validate runtime shape of createLowlight() before the
  `as Lowlight` cast so an upstream API rename routes through the cooldown
  latch instead of silently degrading every code block to plain text.
- bundlePaths: correct doc comment — esbuild.config.js maintains its own
  `BUNDLE_CHUNK_DIR` constant rather than importing this one (it runs
  before any TS compile step).
- AppContainer: update prefetch-failure comment to reference the cooldown
  symbols (`LOWLIGHT_RETRY_COOLDOWN_MS` / `lowlightLastFailureAt`) that
  replaced the removed `lowlightFailed` latch.
- New unit tests covering the lowlightLoader state machine (success,
  in-flight dedup, shape mismatch, cooldown skip, post-cooldown retry)
  and `resolveBundleDir`'s strip-only-on-exact-match contract.

* test(bundlePaths): use path.resolve for Windows-compatible absolute paths

CI failure on Windows: the new `resolveBundleDir` tests built expected
values with `path.join(path.sep, ...)` (e.g. `\tmp\dist`), but
`pathToFileURL` resolves drive-less paths against the current drive
on Windows. The URL -> `fileURLToPath` round-trip returned `D:\tmp\dist`,
while the expectation stayed `\tmp\dist`, tripping all three new
assertions.

Switched both the URL source and the expected value to a single
`path.resolve(path.sep, ...)` anchor per test so both sides absorb
whatever the platform considers absolute. POSIX behaviour is unchanged
(`/tmp/dist` -> `/tmp/dist`).

---------

Co-authored-by: 秦奇 <gary.gq@alibaba-inc.com>
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
2026-05-15 17:26:18 +08:00

155 lines
5.4 KiB
JavaScript

/**
* @license
* Copyright 2025 Google LLC
* SPDX-License-Identifier: Apache-2.0
*/
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { createRequire } from 'node:module';
import { writeFileSync, rmSync, readFileSync } from 'node:fs';
import { wasmLoader } from 'esbuild-plugin-wasm';
let esbuild;
try {
esbuild = (await import('esbuild')).default;
} catch (_error) {
console.warn('esbuild not available, skipping bundle step');
process.exit(0);
}
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const require = createRequire(import.meta.url);
const pkg = require(path.resolve(__dirname, 'package.json'));
// Clean dist directory (cross-platform)
rmSync(path.resolve(__dirname, 'dist'), { recursive: true, force: true });
/**
* Resolve `import X from '*.wasm?binary'` imports to an inline Uint8Array.
*
* The `?binary` suffix is a build-time hint: at bundle time (esbuild) the WASM
* bytes are embedded as base64 and exported as a default Uint8Array, so no
* external vendor files are needed at runtime. In source / transpiled mode
* the dynamic import throws and the caller falls back to reading from
* node_modules via `require.resolve`.
*/
const wasmBinaryPlugin = {
name: 'wasm-binary',
setup(build) {
build.onResolve({ filter: /\.wasm\?binary$/ }, (args) => {
const specifier = args.path.replace(/\?binary$/, '');
const localRequire = createRequire(
path.resolve(args.resolveDir || __dirname, '_dummy_.js'),
);
return {
path: localRequire.resolve(specifier),
namespace: 'wasm-binary',
};
});
build.onLoad({ filter: /.*/, namespace: 'wasm-binary' }, (args) => {
const contents = readFileSync(args.path);
return { contents, loader: 'binary' };
});
},
};
const external = [
'@lydell/node-pty',
'node-pty',
'@lydell/node-pty-darwin-arm64',
'@lydell/node-pty-darwin-x64',
'@lydell/node-pty-linux-x64',
'@lydell/node-pty-win32-arm64',
'@lydell/node-pty-win32-x64',
'@teddyzhu/clipboard',
'@teddyzhu/clipboard-darwin-arm64',
'@teddyzhu/clipboard-darwin-x64',
'@teddyzhu/clipboard-linux-x64-gnu',
'@teddyzhu/clipboard-linux-arm64-gnu',
'@teddyzhu/clipboard-win32-x64-msvc',
'@teddyzhu/clipboard-win32-arm64-msvc',
];
// Name of the directory under `dist/` that esbuild emits shared chunks into.
// MUST stay in sync with `BUNDLE_CHUNK_DIR` in
// `packages/core/src/utils/bundlePaths.ts`, whose `resolveBundleDir` helper
// strips this exact segment when modules look up sibling assets at runtime.
// Renaming here without renaming there silently breaks bundled-binary lookup
// in skill-manager / ripgrepUtils / i18n / extensions/new.
const BUNDLE_CHUNK_DIR = 'chunks';
esbuild
.build({
entryPoints: { cli: 'packages/cli/index.ts' },
bundle: true,
outdir: 'dist',
entryNames: '[name]',
chunkNames: `${BUNDLE_CHUNK_DIR}/[name]-[hash]`,
splitting: true,
platform: 'node',
format: 'esm',
target: 'node22',
external,
packages: 'bundle',
inject: [path.resolve(__dirname, 'scripts/esbuild-shims.js')],
banner: {
js: `// Force strict mode and setup for ESM
"use strict";`,
},
alias: {
'is-in-ci': path.resolve(
__dirname,
'packages/cli/src/patches/is-in-ci.ts',
),
'@qwen-code/web-templates': path.resolve(
__dirname,
'packages/web-templates/src/index.ts',
),
// Resolve to userland punycode instead of deprecated node:punycode built-in
punycode: require.resolve('punycode/'),
},
define: {
'process.env.CLI_VERSION': JSON.stringify(pkg.version),
// Make global available for compatibility
global: 'globalThis',
// Redirect free __dirname/__filename references to the shim so that
// vendored libraries that emit their own `var __dirname` locals don't
// collide with our injected bindings when code-splitting is enabled.
//
// CONTRIBUTOR WARNING: this rewrite applies to *all* source files, so
// any bare `__dirname` / `__filename` in our own code resolves to the
// shim chunk's on-disk location (i.e. `dist/chunks/`), NOT the source
// file's own directory. To get a per-file path, declare a local shadow
// at the top of the module:
//
// import { fileURLToPath } from 'node:url';
// const __filename = fileURLToPath(import.meta.url);
// const __dirname = path.dirname(__filename);
//
// esbuild leaves the local binding alone (it's a declared identifier,
// not a free reference). For sibling-asset lookups in modules that may
// be hoisted into a shared chunk, prefer
// `resolveBundleDir(import.meta.url)` from
// `packages/core/src/utils/bundlePaths.ts` — it both produces a
// per-file path and strips the chunk segment when the module ends up
// under `dist/chunks/`.
__dirname: '__qwen_dirname',
__filename: '__qwen_filename',
},
loader: { '.node': 'file' },
plugins: [wasmBinaryPlugin, wasmLoader({ mode: 'embedded' })],
metafile: true,
write: true,
keepNames: true,
})
.then(({ metafile }) => {
if (process.env.DEV === 'true') {
writeFileSync('./dist/esbuild.json', JSON.stringify(metafile, null, 2));
}
})
.catch((error) => {
console.error('esbuild build failed:', error);
process.exitCode = 1;
});