mirror of
https://github.com/bal-spec/sillytavern-character-memory.git
synced 2026-08-18 04:53:45 +00:00
b38655dcee
Two defects found by pointing the live tier at hosted non-reasoning models via NanoGPT, both of which had been masked by the tier being red for unrelated reasons. 1. The test never exercised map-reduce. It packed against a 24000-char budget in the stated belief that ~90 blocks "comfortably exceeds" it. They don't — the synthetic set is ~6.7k chars, so it packed into a single chunk and the multi-chunk fan-out this test exists to cover never ran. The budget is now derived from the fixture's actual size (total/3), and an assertion on the orchestrator's own chunk count guards the premise, so this can't silently regress to one chunk again while every other assertion still passes. 2. The prompt was under-specified. It asked for "bulleted contents", but parseMemories only recognises lines starting with "- ", so a model that reaches for "*" produced well-formed <memory> tags wrapping zero usable bullets and every chunk was dropped as "no parseable blocks" — observed with llama-3.3-70b, which failed all five chunks. The real consolidation prompts in index.js spell the format out; this now mirrors them, and llama's map phase passes. Also raises maxRetries to 3. Now that the test genuinely splits into several chunks it makes 6+ sequential calls to a provider, and a single transient 503 aborts the run. Retrying is the orchestrator's documented behaviour, so this exercises that path rather than hiding anything. Diagnostics extended: on a zero-block parse the test now dumps the result, which is what identified defect 2. The test remains flaky by nature — reliable in isolation, ~50% as part of the full tier where ~10 calls land in quick succession. Every failure captured has been provider-side (NanoGPT 503 all_fallbacks_failed, or a reduce pass exceeding the timeout on a slow 70B), not an assertion. Documented in CLAUDE.md along with per-model observations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| __snapshots__ | ||
| live | ||
| live.test.js | ||
| llm-client.js | ||
| snapshot.test.js | ||