Anton Razzhigaev 293bb259dd benchmarks: derive the truncation vocabulary from the runtime, not beside the check ... CRITICAL (advisory, v6.81.0). The disclosure field added this release to stop false capability claims was making one. RUNTIME_TRUNCATION_REASON_CODES was hand-copied next to the check and listed four codes the runtime has never emitted (max_rounds_exceeded, task_timeout, context_exhausted, rate_limited) while omitting the two it actually uses for the round cap and the loop-local deadline (round_limit at loop.py:3128 via _handle_round_limit, deadline_local at loop.py:3220). A round-capped or deadline-stopped task therefore published an affirmative `truncated: false` and run_tb.py filed it under genuine_failure_count -- "the agent got a fair shot and got it wrong" about a trial cut off mid-attempt. The comment above the set even named ouroboros.outcomes.BEST_EFFORT_REASON_CODES as the SSOT while contradicting it. Fixed as the CLASS: the set is now DERIVED from BEST_EFFORT_REASON_CODES. All six of its codes are also "an auditor must not read this as a capability result" -- forced finalization means the attempt was cut short by a rail rather than ended by the agent -- so it is taken whole, with no subtraction. One explicit additive delta, llm_api_error (loop_llm_call.py:630): not a best-effort code, but the same class for an auditor as provider_unavailable, and adapters without a separate infra channel would otherwise publish `truncated: false` for a transport death. Every kept code is grepped to an emitting line in ouroboros/; nothing that cannot be located there survives. Three copies of one vocabulary become one derivation plus one pinned mirror: * harbor_installed_agent.py now INTERPOLATES the set into the container runner template (it is generated source, so the literal cannot go stale); * the CL-Bench operator patch genuinely cannot import it -- that module lives in an external clone reached only through a call-time sys.path insert -- so its mirror stays, but a test now parses the vocabulary out of the .patch file and fails on divergence. The patch is regenerated in place with recomputed hunk counts and verified to apply cleanly to a pristine adapter checkout (@549998d). Drift guard (modelled on test_credential_groups_cover_every_routable_provider): every literal reason_code in ouroboros/ must have a recorded decision in _TRUNCATION_DECISIONS, with the emitting line and the reasoning, and the truncating subset must equal the published set. A code added to the runtime tomorrow fails the suite instead of silently defaulting to an affirmative `truncated: false`. Prose the code contradicted, now true and verified: run_tb.py's trial comment and the 'cost_truncated' bucket docstring, the OSWorld METHODOLOGY disclosure paragraph, and this test module's FIX B header. run_tb's _provider_reasons also shed rate_limited and provider_error -- inert, never emitted, same hand-written-vocabulary defect. MAJOR (advisory). continual_learning/METHODOLOGY.md still documented `extra.runtime_attested: false` and attributed the patch probe to the execution clone; a5bdf5e renamed the field to runtime_attestation_available and corrected the target to the --runner-path ADAPTER checkout, but only in section 6. A test asserts the old field's absence, so the document pointed at a key the manifest provably no longer carries. Swept the tree for runtime_attested and "execution clone": no other stale occurrence. Verification: ruff --select F clean; node --test web/tests clean; audit_all_launchers() and version_carrier_desyncs() empty; serial CI lane green. The parallel CI lane is red on exactly one unrelated, pre-existing cross-test-pollution failure (test_deep_self_review.py::TestIsReviewAvailable::test_openai) reproduced identically on a5bdf5e and passing in isolation. VERSION stays 6.81.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>		2026-07-26 05:26:48 +00:00
..
__init__.py	feat(devtools-benchmarks): add official benchmark harnesses and workspace executor	2026-06-06 12:03:30 +03:00
launcher_audit.py	feat(v6.81.0): benchmark provenance becomes a gate, and the artefacts stop lying	2026-07-26 03:40:44 +00:00
manifests.py	v6.81.0: benchmark artefacts must not carry or claim what did not happen	2026-07-26 05:04:14 +00:00
model_slots.py	devtools(swe-bench-pro): adapter hardening — crash-resilience + musl install-in-image + gold-history strip	2026-06-25 18:40:52 +03:00
official_commands.py	feat: v6.55.0 bench devtools alignment — scaffold defaults, ProgramBench e2e, CLB launcher, OSWorld 2.0	2026-07-03 22:49:01 +03:00
result_index.py	benchmarks: derive the truncation vocabulary from the runtime, not beside the check	2026-07-26 05:26:48 +00:00
run_roots.py	feat(v6.81.0): benchmark provenance becomes a gate, and the artefacts stop lying	2026-07-26 03:40:44 +00:00
secrets.py	v6.81.0: benchmark artefacts must not carry or claim what did not happen	2026-07-26 05:04:14 +00:00
server_runner.py	v6.81.0: benchmark artefacts must not carry or claim what did not happen	2026-07-26 05:04:14 +00:00
subprocesses.py	feat(devtools-benchmarks): add official benchmark harnesses and workspace executor	2026-06-06 12:03:30 +03:00