mirror of
https://github.com/MoonshotAI/kimi-code.git
synced 2026-08-04 14:02:22 +00:00
119a33f7f1
Some checks are pending
CI / test (2) (push) Waiting to run
CI / test (3) (push) Waiting to run
CI / test (4) (push) Waiting to run
CI / test (5) (push) Waiting to run
CI / test-pi-tui (push) Waiting to run
CI / test-windows (push) Waiting to run
CI / lint (push) Waiting to run
CI / build (push) Waiting to run
CI / test (1) (push) Waiting to run
CI / typecheck (push) Waiting to run
Nix Build / Check flake.nix workspace sync (push) Waiting to run
Nix Build / nix build .#kimi-code (push) Blocked by required conditions
Release / Release (push) Waiting to run
Release / Deploy docs (push) Blocked by required conditions
Release / Native release artifact (push) Blocked by required conditions
Release / Publish native release assets (push) Blocked by required conditions
* perf(minidb): skip idle everysec fsyncs and add lifecycle stats - everysec WAL now fsyncs on the timer only while dirty (tracked by a write/sync generation watermark); close() keeps its unconditional final sync, and background sync failures surface via walFsyncErrors plus a sticky lastWalFsyncError instead of being silently swallowed - add WAL queue/group-commit counters (walQueuedBytes, walMaxQueuedBytes, walGroupCommits, walGroupCommitFrames) and lifecycle phase stats: recovery bytes/frames/duration, index/text rebuild durations, compaction total/snapshot/rotation/postings durations, rotation pause, and query candidates/decoded/sorted rows; add a syncIntervalMs open option threaded through compaction WAL rotation - rewrite the bench on fixed-seed synthetic data with a stable machine-readable JSON report (cold open 10k/50k/100k, word/ngram search, idle-fsync acceptance, 100k compaction, event-loop delay, peak heap/RSS per scenario) and pin the schema in test/bench-json.test.ts; add app-side baselines with loose complexity budgets in sessionIndex and searchService tests - fix ClusterDb lock-pool closeAll() leaking in-flight shard opens and drain the query store's async close on server shutdown, eliminating the ENOTEMPTY directory-teardown race * perf(minidb): bound startup rebuild and steady-state hot paths - rebuild all derived indexes in one shared store walk: a single decode per record fans out to staged builders, dt rebuild reads record metadata only, and index-less opens no longer decode at all - rank full-text results with a bounded min-heap plus a stable key tie-break instead of sorting every candidate - remove/overwrite text docs via a docID -> delta-terms reverse map instead of scanning the whole delta vocabulary - validate unique batches incrementally against touched postings instead of copying the full per-index owner map - reap due TTL entries from the expiry heap on the write path instead of a full-store sweep per write * feat(session-index): add minidb read model with keyset pagination - add ISessionIndex read-model lifecycle (prepare/status, ready/degraded states) behind the persistence_minidb_readmodel experimental flag - add ISessionIndexMirror write side recording fresh summaries into a bounded, coalescing queue after the authoritative document is durable - replace the offset cursor with before/after keyset pagination; rename list/countActive to listRecent/count - extend IQueryStore with ordered columns and pageByColumn, plus getMany/listKeys/dropCollection - wire the read model through kap-server routes and start, and update the klient sessions contract - index every session for global search instead of the 500 most recent * feat(kap-server): bound search sync lifecycle, pagination, and query budgets - split search requests from sync work: searchIndex() no longer awaits runSync/reopen/reindex; a single-flight sync coordinator with debounce and backpressure runs in the background, stale generations keep serving with explicit stale/degraded state, and refresh/sync/reindex failures surface via lastRefreshError instead of being swallowed - scope file-meta keys by session id (\0meta\file\<sessionId>\<hash>) with lazy + one-shot background migration from the legacy hash-only keys, so one session sync only touches its own meta rows - make authoritative scans incremental (mtime/ino/size rescan conditions, unchanged files no longer rewrite meta) and read wire deltas in 1 MiB chunks instead of whole-file buffer + split - replace offset pagination with versioned v2 keyset page tokens (fingerprint + index generation + sort boundary); generation changes fail old tokens with invalid_page_token, legacy v1 offset tokens are served once and upgraded, and pages collect via bounded top-K instead of full sort + offset skip - add query budgets enforced at the postings/score stage: max query terms, literal length cap, postings visit budget (minidb searchBounded/maxVisits with prefix decoding that never fabricates hits and skips the postings LRU), candidate caps, deadline and text budget; truncation is reported via incomplete reasons candidate_cap/postings_budget/deadline - reopen read-only dbs by opening the next handle before closing the previous one so a failed refresh keeps the old generation serving; failed opens now self-heal through search traffic 100k-message bench: first page p95 < 300ms and page-100 cost on par with page 1; event-loop delay during queries stays sub-millisecond. * fix(minidb): poison, roll back, and recover the WAL on write failures - give WAL writes a commit point: a failed flushBatch poisons the WAL (WAL_POISONED, tracked separately as walWriteErrors vs walFsyncErrors), rejects queued frames in reverse enqueue order, and stops scheduling further batches; everysec background sync failures stay non-rejecting per stage-1 semantics - recover in place to a known-safe point: a serialized recovery chain truncates the WAL back to the first un-acked frame, rebuilds size/nextOffset, and clears the poison; writes queue behind the recovery gate (zero-cost when idle), a failed truncate flips the instance into an explicit writeDisabled state, and a stale truncate offset (WAL file replaced by a rotation) skips the truncate - roll failed flush groups back as a unit: frames are stamped with their batchId, MiniDb keeps per-group earliest pre-state, and the first rejection restores every key of the group (rejected writes no longer reappear after reopen, and in-memory state matches reopen for any failure interleaving); the per-op seq guard remains for cross-group and rotation-retry races - wrap applyOp and the following in-memory mutations so a contract violation poisons the WAL and rolls the group back instead of escaping as a half-commit; frames never enqueued (seal race) roll back per-op without poisoning - tag errors past the commit point with ambiguous: true so callers can distinguish "definitely not applied" from "maybe applied but revoked" - close() waits for the recovery chain to go idle and backup() fences behind in-flight recovery before copying files Controlled A/B bench (22 alternating iterations, 100k concurrent sets): write-path throughput regression is within the 2% budget. * fix(minidb): turn the file lock into an instance-owned serialized lease - distinguish lock ownership by instance instead of pid: every acquire mints a pid:uuid token carried by lock/bid/watch files, inspect().mine compares tokens, liveness still follows pid, tokenless legacy files keep the old stale-takeover path, and hasLiveForeignWatch excludes self by token so same-process contenders see each other (closing the double-win takeover and the cross-instance release); a live same-pid lock is still respected, and re-acquiring a held lock is idempotent - serialize acquire/renew/release through a per-instance promise-chain mutex: renew re-checks held inside the chain and release waits for an in-flight renew, eliminating the renew/rename-after-unlink ghost lock - make MiniDb.close() a state machine (open/closing/closed) with a shared closePromise: cleanup runs per-resource try/catch in dependency order (text indexes, store, valueReader, WAL, lock), aggregates every cleanup error into an AggregateError, stays in 'closing' on failure so a retry finishes the cleanup, and no longer leaks the lock when the WAL close fails; a rejected in-flight compaction no longer escapes the cleanup pass * fix(minidb): keep readers on one consistent file generation - add an internal persistent-files module as the single source of truth for the persisted file set (snapshot, WAL, sidecars, postings pattern, fingerprint subset); lock-pool fingerprints, persistentFiles, open stale-tmp cleanup, and backup/restore filtering all derive from it, and fingerprints upgrade to dev:ino:size:mtimeMs so compound sidecar changes can no longer hide from cluster readers - pair snapshot and WAL generations during recovery (transitional stat-pairing until stage-5 manifests): each pass anchors the fds it scans, re-stats afterwards, tolerates append-only WAL growth, retries bounded times on generation churn with a clean store reset, and throws RECOVERY_GENERATION_CHURN when churn exceeds the budget; the disk-mode ValueReader attach re-validates inodes so stale offsets never read a replaced file - make the rotation directory fsyncs strict: failures abort the rotation through the existing rollback path instead of being swallowed, while platforms without directory fsync degrade once with a warn and stats.dirFsyncUnsupported * fix(minidb): serialize index-definition sidecar mutations, persist before publish - extract the promise-chain mutex into a shared createSerializer() and give each sidecar family (secondary/compound/text) its own chain: create/drop run uninterruptibly (memory change + rebuild + persist), different families stay independent, and the data write path never shares these chains - reverse the publication order to staged -> persist -> publish: a create stages the definition, rebuilds via the staged builder, persists the sidecar including the new definition, then publishes atomically; any failure discards the staged state leaving live and sidecar untouched (no phantom indexes, retry-safe); a drop persists the sidecar without the definition before removing it live; text index create/drop adopt the same pattern, replacing the hand-rolled unwind, and a dropping marker keeps compaction postings rebuilds out of the persist window - feed staged indexes from the incremental write path (add/remove/ checkUnique/checkUniqueBatch visit live+staged) so writes landing in the persist window are not lost at publish; queries still see live only - harden writeFileAtomic: instance-unique tmp names (.tmp-pid-seq), a strict fsyncDir after rename so a successful persist is crash durable, and whitelist-based stale-tmp cleanup that never touches lock tmp files * fix(minidb): validate writes before any side effect, canonicalize values once - canonical value at the write boundary: the json codec re-parses the encoded bytes once and every downstream consumer (unique checks, secondary/compound/text indexes, dt extraction) sees exactly the persisted representation, so getter/toJSON/Proxy documents can no longer diverge between the index view and the storage view - reorder the set/batch pipeline so every fallible check happens before any visible side effect: prepare (key/ttl checks, encoding, canonical decode, index field extraction, tokenization) -> unique checks -> ensureMemoryFor eviction -> commit; a constraint failure now leaves the database untouched (no more evicted victims on rejected inserts), and applyOp is structurally pure against pre-validated data - tokenize at the prepare boundary: TextIndex gains prepareAdd/ addPrepared and the buildQueue carries validated key+tokens mutations instead of raw docs, so a throwing custom tokenizer can no longer poison the live view or the queue, and custom-tokenizer output is rejected per token over 0xffff bytes before it can permanently break postings rebuilds; prepared tokens are keyed by index instance so a same-name drop+create mid-write re-tokenizes instead of crossing tokenizers - strict batch structure validation: scanBatchOpRefs/decodeBatchOps reject unknown op types, out-of-bounds lengths, and trailing bytes (offset must equal body length), so a valid-CRC but malformed batch is skipped as a unit and counted via RecoveryInfo.corruptBatches instead of being partially applied Bench vs the stage-1 baseline: json write throughput regression is within the 5% budget (median ~2-4% depending on the measurement). * feat(minidb): add OpTracker drain primitive and atomic backup, harden tests - introduce the internal OpTracker (close gate + in-flight counter with enter/leave/close/whenIdle and reference-counted pause/resume) and drive every shutdown/drain path from it: WAL background syncs are tracked so close() waits out an in-flight sync before closing the fd, cluster lock-pool closeAll() closes the gates and drains busy callbacks before closing handles, and MiniDb writes pass a write gate - make backup() atomic with a defined linearization point: pause the write gate, drain in-flight writes (every acknowledged write is now included), copy to a sibling temp dir with per-file fsyncs, write the manifest last as the commit marker, and rename into place; failures clean up and leave no partial backup, and concurrent writes are rejected with BACKUP_IN_PROGRESS - reap emptied compound-index groups on remove (the groups map no longer grows monotonically), move the open-time mkdir behind the readOnly check so a read-only open of a missing directory fails with ENOENT instead of creating it, and never run a destructive rebuild for a read-only open failure (explicit or onLockFail fallback) - consolidate every review fault-injection repro into the formal suite behind deterministic barrier helpers (programmable writev/sync/ rename/tokenize hooks) and convert the six timing-based tests to barrier/tick-driven assertions; the .tmp repro scripts are removed The converted timing tests and the full suite pass 50 repeat runs (including under CPU load injection) with zero flakes. * feat(minidb): persist derived indexes as atomic generations, open from WAL delta - checkpoint the store, dt/secondary/compound indexes, and text dictionary/postings/docs into immutable generations under generations/g-NNNNNN published atomically (tmp build, per-file checksums and fsyncs, dir rename, CURRENT swap, strict dir fsyncs); the manifest records the format version, WAL/snapshot checkpoint anchors, per-index definition hashes, and codec/value-mode compatibility - open now loads the published generation and replays only the WAL delta after its checkpoint: no full value decode, corpus tokenization, or postings rewrite on a normal reopen (warm opens are 3.5-13.8x faster at 100k/1M records); a definition change rebuilds only the affected index, and corrupt generation files fall back to the previous generation or the legacy full recovery without ever touching the authoritative snapshot/WAL - build generations transactionally with compaction (rotation plus derived state publish as one unit, replacing the synchronous rebuildTextPostings tail), capture concurrent writes through a sealed op queue with byte/op caps, hard-link clean postings and the snapshot into the new generation, and repoint every live text base into the CURRENT generation after publish - cluster/read-only refresh watches CURRENT and the WAL watermark: pure generation publishes keep readers on incremental catch-up while rotations reopen onto the new generation; writers building the next generation never disturb readers of the current one - legacy databases open through the old path unchanged and gain their first generation in the background; OpenOptions.indexGenerations: false fully restores the pre-generation behavior * feat(minidb): workerize text-index builds and split MiniDb into facets - split the monolithic src/index.ts into facet modules (mini-db, types, value-codec, memory-guard, backup, query-engine, text-registry, wal-group, generation-builder/loader, write-path, read-path, index-admin, lifecycle, stats) and move text-index.ts to text-index/ - run corpus-scale text-index builds off the main thread via the bounded worker engine (src/worker/), exported through the new worker-runtime subpath, with inline fallback for small corpora and rollback switches - defer the open-time fallback text rebuild into a maintenance task; searches on a not-yet-committed base raise TextIndexBuildingError - add the unified maintenance scheduler, bounded async read surface, and a maintenance bench - kap-server search: switch to searchBoundedAsync and serve the building page while the index base rebuilds after fallback recovery - kimi-code: install the SEA-bundled minidb text-build worker at startup, bundle it via the native asset scripts, and add the startup-trace util plus the KIMI_TUI_INPUT_LATENCY debug probe * fix(minidb): treat win32 EPERM as unsupported directory fsync - extract isUnsupportedDirectoryFsyncError and cover win32 EPERM - drop the one-shot console.warn; stats.dirFsyncUnsupported carries the degraded state * fix(kap-server): harden search-index dispose and drain lifecycle - dispose() now closes an OpTracker gate and drains in-flight sync/refresh passes before closing the db, so no background write can hit a closed handle; the deleteSessionDocs loop and trailing stats write skip once the gate closes (review #20) - drainGlobalSearchDisposals loops to a fixpoint so disposals registered while a drain is in flight are also awaited (review #21) - pin the post-open failure semantics with a regression test: a failed text-index setup closes the handle and the next open reacquires the writer lock instead of self-locking read-only (review #19) - export OpTracker from the minidb root for the search service's drain * chore: fix oxlint type-aware lint errors |
||
|---|---|---|
| .. | ||
| acp-adapter | ||
| acp-server | ||
| agent-core | ||
| agent-core-v2 | ||
| kaos | ||
| kap-server | ||
| klient | ||
| kosong | ||
| migration-legacy | ||
| minidb | ||
| node-sdk | ||
| oauth | ||
| pi-tui | ||
| protocol | ||
| telemetry | ||
| transcript | ||
| tree-sitter-bash | ||