vrr/openclaw
2
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-10-03 01:29:56 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 29
openclaw/docs/help/testing/writing-tests.md
Peter Steinberger 760f8b7715
test(ci): bound hosted tooling growth inventory (#145759) 
The hosted job-cap regression rebuilt the growing repository's full compact
plan twelve times, resetting the module graph for every inventory variation.
Its discovery cache avoided repeat enumeration but left import, ownership,
and packing work unbounded; the case timed out at 120 seconds in hosted CI.

Use fixed timing inputs and a bounded synthetic inventory through the real
planner. Fifty-six full-budget anchors leave twenty-four tooling jobs at the
80-job cap, including a stronger-runner tail. Retain all twelve variants and
every original coverage, ownership, runner, and execution-budget assertion.
Document the fixture contract and assert exact baseline capacity/membership.

Measured case runtime fell from 8.858 seconds to 0.575 seconds locally. The
complete owner file passes all 61 tests; the unmocked repository planner still
produces 78 jobs within the cap. No runtime or workflow policy changes.
2026-09-12 00:42:16 -07:00

4.1 KiB
Raw Blame History

summary title read_when
Temp-directory rules, agent reliability eval gaps, and how to add a regression Writing and adding tests
You are writing a new test
You are adding a regression for a provider bug

Test Temp Directories

Use the shared helpers in test/helpers/temp-dir.ts for test-owned temporary directories so ownership is explicit and cleanup stays in the test lifecycle:

import { afterEach } from "vitest";
import { useAutoCleanupTempDirTracker } from "../helpers/temp-dir.js";

const tempDirs = useAutoCleanupTempDirTracker(afterEach);

it("uses a temp workspace", () => {
  const workspace = tempDirs.make("openclaw-example-");
  // use workspace
});

useAutoCleanupTempDirTracker(afterEach) intentionally exposes no manual cleanup method - Vitest owns cleanup after each test. Older lower-level helpers (makeTempDir, cleanupTempDirs, createTempDirTracker) still exist for tests that have not migrated; avoid new usage of them and avoid new bare fs.mkdtemp* calls unless a test is explicitly verifying raw temp-dir behavior. When a bare temp dir is genuinely needed, add an auditable allow comment with a reason:

// openclaw-temp-dir: allow verifies raw fs cleanup behavior
const workspace = fs.mkdtempSync(prefix);

node scripts/report-test-temp-creations.mjs reports new bare temp-dir creation and new manual shared-helper usage in added diff lines, without blocking existing cleanup styles. It follows the same test-path classification as scripts/changed-lanes.mjs and skips the shared helper implementation itself. check:changed runs this report for changed test paths as a warning-only CI signal (GitHub warning annotations, not failures).

Agent reliability evals (skills)

We already have a few CI-safe tests that behave like "agent reliability evals":

  • Agent admission, run-ID responses, and abort requests through the real Gateway with a mock OpenAI provider (src/gateway/gateway.test.ts).
  • End-to-end wizard flows that validate session wiring and config effects (src/gateway/gateway.test.ts).

What's still missing for skills (see Skills):

  • Decisioning: when skills are listed in the prompt, does the agent pick the right skill (or avoid irrelevant ones)?
  • Compliance: does the agent read SKILL.md before use and follow required steps/args?
  • Workflow contracts: multi-turn scenarios that assert tool order, session history carryover, and sandbox boundaries.

Future evals should stay deterministic first:

  • A scenario runner using mock providers to assert tool calls + order, skill file reads, and session wiring.
  • A small suite of skill-focused scenarios (use vs avoid, gating, prompt injection).
  • Optional live evals (opt-in, env-gated) only after the CI-safe suite is in place.

Adding regressions (guidance)

For inventory-growth and capacity regressions, pass bounded synthetic inventories and fixed timing data through the real planner. Put the fixture at the capacity boundary and preserve coverage, ownership, and execution-budget assertions. Keep real-checkout inventory coverage in its integration tests instead of rebuilding the growing repository plan for every synthetic variation.

When you fix a provider/model issue discovered in live:

  • Add a CI-safe regression if possible (mock/stub provider, or capture the exact request-shape transformation)
  • If it's inherently live-only (rate limits, auth policies), keep the live test narrow and opt-in via env vars
  • Prefer targeting the smallest layer that catches the bug:
    • provider request conversion/replay bug -> direct models test
    • gateway session/history/tool pipeline bug -> gateway live smoke or CI-safe gateway mock test
  • SecretRef traversal guardrail:
    • src/secrets/exec-secret-ref-id-parity.test.ts derives one sampled target per SecretRef class from registry metadata (listSecretTargetRegistryEntries()), then asserts traversal-segment exec ids are rejected.
    • If you add a new includeInPlan SecretRef target family in src/secrets/target-registry-data.ts, update classifyTargetClass in that test. The test intentionally fails on unclassified target ids so new classes cannot be skipped silently.