mirror of
https://github.com/MoonshotAI/kimi-code.git
synced 2026-08-03 05:24:54 +00:00
44f3341919
* feat(minidb): add ClusterDb sharding and harden engine under stress Cluster layer: - add ClusterDb: hash-routed keys over N MiniDb shard directories with a per-shard lock pool (lease renewal, lockHoldMs yield, takeover on dead PID), merged ordered scans, cross-shard index registry, live cross-process read visibility, and crash-recovery handoff - add cluster bench suite and multi-process test suites Engine hardening (stress-driven fixes): - compaction: pre-copy now gives up when the tail copy is not converging and rotation seals the old WAL (retryable), so compaction always terminates under sustained write storms and no committed write slips through rotation - recovery: re-sync WAL size bookkeeping after torn-tail truncation; read-only opens create/modify no files and never compact under a live writer - lockfile: stale-lock takeover via atomic bid-rename + settle; release only unlinks its own pid - query: streaming candidates with skip/limit applied before materialization, plus Store.rawKeys for value-free key scans - eviction: O(1) LRU victim picking via insertion-ordered access set - TTL: adaptive expire budget drains simultaneous-expiry storms in seconds - store: O(1) size fast path when no TTL is set; has() no longer materializes disk-backed values - openOrRebuild preserves data when only a sidecar definition file is corrupt; sidecar definitions written atomically; stale compaction temps cleaned on open - RESP server serializes replies per connection; over-64KiB tokens can no longer poison the full-text index * feat(minidb): incremental WAL catch-up for cluster readers - cluster readers: track a per-shard WAL watermark (dev/ino/offset) and catch up from appended frames instead of fully reopening; fall back to a full reopen on rotation, truncation, or index-definition changes - MiniDb.catchUpFromWal applies WAL tail frames to a live instance (store plus secondary/dt/compound/text indexes), sharing recover()'s frame interpretation; RecoveryInfo exposes walScanEnd/dev/ino as the safe anchor - lock-pool: incremental refresh with stats (incrementalCatchups, catchupFramesApplied); bench/reader-catchup shows p50 read latency drop from 130ms to 0.4ms at 10k keys and from 532ms to 0.4ms at 50k keys while a neighbor process writes - compaction: keep the db writable on rotation failure (fresh WAL swap, remap once); count stats.compactions only on full success including the onCompacted hook - restoreKey: seq guard so a failed op never wipes a concurrently committed value; eviction DELs retry on WAL seal - text index: atomic build (stage then swap), createTextIndex registers only after a successful build, open-time cleanup of db.text-*.postings.tmp - RESP server: swallow per-connection socket errors, reset the parser buffer after an oversized request, isolate per-command errors while preserving reply order * chore(minidb): add changesets for reader catch-up and review hardening * fix(minidb): support Windows in WAL rotation and lock takeover - compaction rotation: on Windows, renaming over an open destination is EPERM, so replace renames with a retrying renameReplace helper and let go of the db's own ValueReader handles for the renames (reopened right after the pointer remap) - value reader: hold snapshot/WAL handles only in valueMode 'disk'; in memory mode the handles were idle and, on Windows, blocked rotation - lock takeover: retry the takeover bid's rename on Windows, but re-check the corpse before every attempt — a blind retry loop could land our bid late and overwrite an already-verified winner, double-holding the lock - lock takeover: raise the co-bidder settle window to 50ms so loaded CI machines with tens-of-milliseconds descheduling still elect one winner - test hardening for shared CI runners: a 30s minidb-wide timeout floor, batched prefills instead of sequential setup loops, explicit timeouts for process-spawning and heavy e2e tests, and Vitest 4 test() signature normalization Verified green across macOS (arm64), Windows Server 2022 (x64, 2-core), and Ubuntu 22.04 (launchpad aarch64): 321 passed, 1 skipped in each. * fix(minidb): make stale-lock takeover exactly-one under CI load - takeover now registers a liveness watch file before touching the lock, so every contender is visible to every other for the whole attempt; the settle/verify loop abstains while any live foreign watch exists. Settle-only heuristics could not survive a bidder descheduled before its bid write on shard-parallel CI runners (observed double-holds on ubuntu-latest and on a 2-core Windows box). - adaptive settle scales with the attempt's own wall clock (floored, capped), replacing the fixed window. - keep the Windows EPERM tolerance in the bid rename, re-inspecting the corpse before every attempt on ALL platforms, not just win32. - lint: fix restrict-template-expressions in an e2e RESP helper. - test: widen the cluster wait-read budget for slow CI spawns. * fix(minidb): clean remaining CI lint and consumer-test failures - wrap the compaction-storm error interpolation as String() so the type-aware restrict-template-expressions lint passes - agent-core-v2's minidb query-store corruption test now expects the intended semantics: a corrupt index-definition sidecar is dropped while the data survives, and the definition can be re-registered * fix(minidb): harden ClusterDb index administration across processes Address three multi-process administration races in the cluster layer: - findRange now merges candidates from all shards first and only then applies reverse/offset/count globally, instead of clipping per shard and discarding reverse - cluster.indexes.json mutations go through a compare-and-swap loop (reload, re-apply idempotently, publish, verify) with an in-process mutex, so concurrent create/drop from two processes loses neither registry entries nor shard sidecars - a failed createIndex/createTextIndex fan-out now rolls back exactly the shards it already created on, so the registry and every shard agree whether an index exists Also make LockFile sidecars (tmp/bid/watch) unique per acquire attempt: two lock users in the same process (independent shard pools) must never share a path, or one user's cleanup would delete the other's in-flight file. |
||
|---|---|---|
| .. | ||
| baseline.ts | ||
| bench.ts | ||
| cluster-worker.ts | ||
| cluster.ts | ||
| import-kimi-code.ts | ||
| measure-session-memory.ts | ||
| message-composed.ts | ||
| message-range.ts | ||
| query.ts | ||
| reader-catchup.ts | ||
| reader-worker.ts | ||
| search-baseline.ts | ||
| search-kimi-code.ts | ||
| session-children.ts | ||
| session-store-demo.ts | ||
| session-store.ts | ||