mirror of
https://github.com/razzant/ouroboros.git
synced 2026-10-02 19:58:46 +00:00
Add scripts/run_tests.py, the one fast local test battery
The documented local command (`make test`, CONTRIBUTING §4, README) was `pytest tests/ -q --tb=short`: the whole default suite (23k tests) in a single process, behind a stale "smoke tests" comment. An Ouroboros task followed it, hit its 1200 s timeout and withheld its PR. The correct recipe — a parallel `not serial` xdist pass, then the serial pass — lived only inside the hermetic commit gate. scripts/run_tests.py is the single entry point for every actor. A bare call always runs the FULL battery: the node lane (a missing or too-old node is a red NOT_RUN, never a silent skip), then every default-lane test in ONE xdist run under `--dist loadgroup --serial-shards=N`. tests/conftest.py gains that option: each serial FILE is pinned to one of N file-sharded groups (one per four workers, at most four; four cores keep a single group), so serial tests overlap the parallel ones while a serial file still never splits across workers or runs beside another file of its shard; other tests keep the scope loadscope uses. The option defaults to 0 and is inert for CI and the commit gate, which keep the two-pass split and stay the authority. `--sequential` keeps the gate's marker split and xdist flags in two passes; extra arguments forward a focused run. Marker exclusions and xdist flags are imported from ouroboros.preflight_runner, never restated. Measured on an 18-core Mac shared with other agents: the two-pass battery took 649 s (parallel 203 s + serial 446 s); the one-run battery took 217–287 s with identical verdicts (22 950 passed). Also: `make test` now calls the script; CONTRIBUTING, README, the Build & CI chapter, the architecture map and one sentence of prompts/SYSTEM.md point at it and say which opt-in lanes it leaves out and what to record when a full run could not finish; tests/test_plan_author_presentation.py resolves node through ouroboros.preflight_node like the rest of the suite instead of a bare PATH `node`; tests/test_run_tests_script.py pins the recipe parity, the bounded shard count, bare call = full battery, node NOT_RUN = red, and the grouping semantics. Not changed: the commit gate and .github/workflows/ci.yml; the three gate self-tests that wait out real 30/20 s timeouts (their slack exists for nested-xdist startup on slow runners); no version carrier. Co-authored-by: Ouroboros <311266734+ouroboros-agent@users.noreply.github.com>
This commit is contained in:
Anton Razzhigaev
parent
472f8c5cc7
commit
915f93604d
10 changed files with 349 additions and 9 deletions
|
|
@ -107,12 +107,26 @@ base-to-head range, never uncommitted edits.
|
|||
## 4. Verify the Change
|
||||
|
||||
Use [`README.md` → Run from Source](README.md#run-from-source) for setup. Run
|
||||
focused tests while developing, then the default local suite when practical:
|
||||
focused tests while developing — you choose the narrowest targets that exercise
|
||||
the change — then the full local battery when practical:
|
||||
|
||||
```bash
|
||||
make test
|
||||
python scripts/run_tests.py # or: make test
|
||||
```
|
||||
|
||||
That is the one fast recipe (node lane, then every default-lane test in a single
|
||||
xdist run); a bare `pytest tests/` runs the same tests in one process and takes
|
||||
many times longer. Like CI's default lanes it leaves out the opt-in marker
|
||||
lanes (`size_ratchet`, `browser`, `ui_browser`, `portable_detail`,
|
||||
`skill_smoke`, `integration`); a change that touches repository size or a
|
||||
reference-book chapter also runs `python -m pytest tests/ -m size_ratchet`. `python scripts/run_tests.py tests/test_x.py` forwards a
|
||||
focused run. Inside Ouroboros a reviewed commit (`commit_reviewed` /
|
||||
`vcs_commit_reviewed`) runs the complete battery in its hermetic gate unless
|
||||
`skip_tests` is set, so running it by hand first is optional rather than a
|
||||
second requirement; for a pull request, CI is that gate. If the full battery could
|
||||
not finish on your machine or within your budget, say so — `NOT_RUN` with the
|
||||
reason — and deliver the focused evidence; never report it as green.
|
||||
|
||||
Record exact commands, outcomes, and producer exit codes. If a check could not
|
||||
run, record `NOT_RUN` and the reason instead of claiming it passed.
|
||||
|
||||
|
|
|
|||
7
Makefile
7
Makefile
|
|
@ -3,11 +3,12 @@
|
|||
|
||||
.PHONY: test lint health clean
|
||||
|
||||
# Run smoke tests (fast, no external deps needed at runtime)
|
||||
# Full local battery: node lane + every default-lane test in one xdist run
|
||||
# (modes and focused runs: scripts/run_tests.py)
|
||||
test:
|
||||
uv run --locked python -m pytest tests/ -q --tb=short
|
||||
uv run --locked python scripts/run_tests.py
|
||||
|
||||
# Run smoke tests with verbose output
|
||||
# Single-process verbose run (slow: the whole default suite in one process)
|
||||
test-v:
|
||||
uv run --locked python -m pytest tests/ -v --tb=long
|
||||
|
||||
|
|
|
|||
|
|
@ -321,7 +321,8 @@ The server binds to `127.0.0.1:8765` by default. Read [`docs/DEPLOYMENT.md`](doc
|
|||
#### Run Tests
|
||||
|
||||
```bash
|
||||
make test
|
||||
python scripts/run_tests.py # full local battery (same as `make test`)
|
||||
python scripts/run_tests.py tests/test_x.py # focused run
|
||||
```
|
||||
|
||||
`pyproject.toml` is the direct-dependency authority and `uv.lock` is the
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ Tool-schema compatibility has two layers. Fork-safe PR tests build the complete
|
|||
|
||||
`claudexor-platform-gate.yml` proves the managed Claudexor runtime on three OSes: a fixture lane (fake harness, offline, $0) always, and a live lane only on explicit API keys — subscription auth stays deliberately out of CI, because an interactive machine-bound token must not enter CI secrets. `dependency-graph.yml` reads `ouroboros/claudexor_runtime_pin.json` and submits that direct runtime relationship to GitHub's dependency graph (runs only on pin/workflow changes on `main`/`ouroboros`, plus manual dispatch; `contents: write` only) without presenting Claudexor as a Python or Node package dependency. The Scorecard workflow (`.github/workflows/scorecard.yml`) runs on `main` pushes and weekly, pins every action by full commit SHA, defaults permissions to read-only, and adds only `security-events: write` + `id-token: write` for SARIF upload and OpenSSF publication. `CODE_OF_CONDUCT.md` owns community rules; `CITATION.cff` owns the software and preferred technical-report citations; `site/paper/index.html` owns the canonical paper landing page; `docs/benchmarks/evidence.json` holds the release-bound public benchmark projection; README remains the claim SSOT (guarded by `tests/test_trust_metadata.py` and `tests/test_public_site_metadata.py`).
|
||||
|
||||
Dev-facing release/review scripts: `scripts/run_external_review.py` (dual-lane external review; `--contributor` binding; `READY_FOR_INTEGRATION` is evidence, never merge authority), `scripts/contributor_review_evidence.py` (route-neutral contributor-packet binding), `scripts/run_plan_review.py` (the same engine as `plan_task`; review-exempt dev tool), `scripts/validate_scope_receipt.py`, `scripts/claudexor_platform_smoke.py`, `scripts/fetch_claudexor_runtime.py` (pin SSOT: `claudexor_runtime_pin.json`), `scripts/cleanup_test_pollution.py` (dry-run-first). `site/` is the Vite source of the public pages (`site/scripts/sync-assets.mjs` syncs assets); `skills/telegram/` and `skills/unix_computer_use/` are the bundled skills; `packaging/cli/` holds the CLI wrappers and installers; `packaging/appimage/` the AppRun dispatch + desktop metadata; `packaging/systemd/` the opt-in user unit (§1 Runtime topology owns its no-restart rationale).
|
||||
Dev-facing release/review scripts: `scripts/run_tests.py` (test battery; DEVELOPMENT Build & CI), `scripts/run_external_review.py` (dual-lane external review; `--contributor` binding; `READY_FOR_INTEGRATION` is evidence, never merge authority), `scripts/contributor_review_evidence.py` (contributor-packet binding), `scripts/run_plan_review.py` (the `plan_task` engine; review-exempt), `scripts/validate_scope_receipt.py`, `scripts/claudexor_platform_smoke.py`, `scripts/fetch_claudexor_runtime.py` (pin SSOT: `claudexor_runtime_pin.json`), `scripts/cleanup_test_pollution.py` (dry-run-first). `site/` is the Vite source of the public pages (`site/scripts/sync-assets.mjs`); `skills/telegram/` and `skills/unix_computer_use/` are the bundled skills; `packaging/cli/` holds the CLI wrappers and installers; `packaging/appimage/` the AppRun dispatch + desktop metadata; `packaging/systemd/` the opt-in user unit (no-restart rationale: §1 Runtime topology).
|
||||
|
||||
### System E2E suite (`tests/system_e2e/`)
|
||||
|
||||
|
|
|
|||
|
|
@ -24,6 +24,10 @@ CI runs the default suite in parallel — `python -m pytest tests/` with `-m "no
|
|||
- Mark real-process / real-port tests, and tests that mutate process-global state WITHOUT reliable fixture isolation, `@pytest.mark.serial` (or add the file to `_SERIAL_TEST_FILES` in `tests/conftest.py`): under `-n` such a test flakes on kill/reap or port-reclaim timing or crashes its worker, and with `--max-worker-restart=0` a dead worker fails its WHOLE co-located batch as spurious failures in unrelated files.
|
||||
- Keep every other test parallel-safe so it stays in the fast pass: `tmp_path` (never a fixed path), `monkeypatch.setenv`/`delenv`/`setattr` for environment and attribute changes, no execution-order assumptions. The autouse `tests/conftest.py::_os_environ_isolation` snapshot restores `os.environ` at every test boundary, but monkeypatch stays the rule because it reverses exactly the named change inside the test. A module-global mutation reliably snapshot-and-restored by a fixture may stay in the parallel pass (pattern: `tests/conftest.py::_isolate_workspace_executor_globals`).
|
||||
|
||||
### The local battery entry point
|
||||
|
||||
`python scripts/run_tests.py` (also `make test`) is the documented local run. A bare call always means the FULL battery — the node lane (a missing node is a red `NOT_RUN`), then every default-lane test — and never skips a test on the strength of an earlier run; extra arguments are forwarded to pytest as a focused run. It imports `LANE_EXCLUSION_EXPR` and `PARALLEL_PASS_FLAGS` from the gate rather than restating them. Its default mode is ONE xdist run under `--dist loadgroup --serial-shards=N` (`tests/conftest.py::_pin_lane_groups`): every serial FILE is pinned to one of N file-sharded groups, so a serial file never splits across workers and never runs beside another file of its own shard, while every other test is grouped by file; N is one per four workers, at most four, so a four-core machine keeps a single serial group. Serial tests thus also get the parallel flags' 300 s per-test timeout, which the gate's serial pass lacks. `--sequential` keeps the gate's marker split and xdist flags in two passes. The option defaults to 0 and is inert for CI and the commit gate, which keep the two-pass split and stay the authority; a failure seen only in overlapped mode is re-checked with `--sequential` before it is believed. Enforcement: `tests/test_run_tests_script.py`.
|
||||
|
||||
### The commit gate mirrors the CI split
|
||||
|
||||
`ouroboros/preflight_runner.py::run_hermetic_pytest` mirrors CI in one disposable checkout and scrubbed temporary data root: the node test lane (`cd web && node --test tests/*.test.js`, content-keyed — a candidate without web tests never requires node, while an active web suite cannot silently disappear when node is missing), then the same two logical pytest passes (parallel `not serial`, then flag-free `serial`). The browser no-undef check has two layers: the dependency-free acorn walker in that suite (`web/tests/no_undef.test.js`) is the hermetic gate's, and both CI jobs additionally run ESLint's `no-undef` (`web/eslint.config.js`, exact-pinned, installed with `npm ci`) as an independent second opinion — CI-only, never part of the gate. `LANE_EXCLUSION_EXPR` and `PARALLEL_PASS_FLAGS` are executable SSOTs pinned against both CI jobs; the candidate is captured as one hardened worktree-vs-`HEAD` binary diff, and a capture or apply failure is the typed `PREFLIGHT_CANDIDATE_ASSEMBLY` hard block, never a test failure. The `pyproject.toml` `addopts` line is the single home of the per-test timing report (`--durations=25 --durations-min=1.0`), prepended to every argv, so the same slowest-test evidence appears locally, in both CI jobs and in both gate passes. Contributor rules:
|
||||
|
|
|
|||
|
|
@ -211,7 +211,10 @@ need it.
|
|||
- When a shared contract, format, prompt, route, setting, or lifecycle changes,
|
||||
I read every reader and writer, update docs, prompts, and tests in the same
|
||||
diff, keep `docs/ARCHITECTURE.md` rationale in sync for non-obvious
|
||||
decisions, and run focused tests before review.
|
||||
decisions, and run focused tests before review. `python scripts/run_tests.py`
|
||||
is the fast full battery (arguments forward a focused run); a bare
|
||||
`pytest tests/` is one slow process, and the reviewed commit runs the full
|
||||
battery itself unless tests are explicitly skipped.
|
||||
- I preserve my own work: never delete or overwrite a viable result, candidate,
|
||||
or unique input without a recoverable copy; save a working deliverable as
|
||||
soon as I have one, then improve copies.
|
||||
|
|
|
|||
143
scripts/run_tests.py
Normal file
143
scripts/run_tests.py
Normal file
|
|
@ -0,0 +1,143 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Run the local test battery the fast, correct way.
|
||||
|
||||
python scripts/run_tests.py full battery (node lane + every default-lane test)
|
||||
python scripts/run_tests.py --sequential full battery exactly as the commit gate and CI split it
|
||||
python scripts/run_tests.py tests/test_x.py focused run; every argument is forwarded to pytest
|
||||
|
||||
A bare call always means the FULL battery — it never skips a test on the
|
||||
strength of an earlier run. The default mode is one xdist run in which the
|
||||
`serial` tests are pinned to a few file-sharded groups (`--serial-shards`,
|
||||
tests/conftest.py) and therefore overlap the parallel tests instead of waiting
|
||||
for them; `--sequential` keeps the marker split and xdist flags of
|
||||
`ouroboros.preflight_runner` and `.github/workflows/ci.yml` in two passes.
|
||||
The reviewed-commit gate and CI stay the authority: this script is feedback.
|
||||
|
||||
The marker exclusions and xdist flags are imported from the gate's SSOT
|
||||
(`LANE_EXCLUSION_EXPR`, `PARALLEL_PASS_FLAGS`), never restated here. Data-root
|
||||
isolation is `tests/conftest.py`'s job and is not repeated either.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import pathlib
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
REPO = pathlib.Path(__file__).resolve().parents[1]
|
||||
if str(REPO) not in sys.path:
|
||||
sys.path.insert(0, str(REPO))
|
||||
|
||||
_MAX_SERIAL_SHARDS = 4
|
||||
_PYTEST_EXIT_NO_TESTS = 5
|
||||
|
||||
|
||||
def _workers() -> int:
|
||||
"""What `-n auto` will resolve to: the operator override, else xdist's own count."""
|
||||
raw = os.environ.get("OUROBOROS_PREFLIGHT_TEST_WORKERS", "").strip()
|
||||
if raw.isdigit():
|
||||
return max(2, int(raw))
|
||||
try: # xdist prefers physical cores whenever psutil is importable
|
||||
import psutil
|
||||
physical = psutil.cpu_count(logical=False)
|
||||
except Exception:
|
||||
physical = None
|
||||
return max(2, physical or os.cpu_count() or 2)
|
||||
|
||||
|
||||
def serial_shards(workers: int) -> int:
|
||||
"""How many serial groups may run at once: one per four workers, at most four.
|
||||
|
||||
Four cores or fewer keep ONE serial group, i.e. serial tests still never
|
||||
overlap each other there — they only stop waiting for the parallel tests.
|
||||
"""
|
||||
return max(1, min(_MAX_SERIAL_SHARDS, workers // 4))
|
||||
|
||||
|
||||
def _with(flags: list[str], option: str, value: str) -> list[str]:
|
||||
out = list(flags)
|
||||
out[out.index(option) + 1] = value
|
||||
return out
|
||||
|
||||
|
||||
def battery_commands(sequential: bool) -> list[tuple[str, list[str]]]:
|
||||
from ouroboros.preflight_runner import LANE_EXCLUSION_EXPR, PARALLEL_PASS_FLAGS
|
||||
|
||||
base = [sys.executable, "-m", "pytest", "tests/"]
|
||||
if sequential:
|
||||
return [
|
||||
("parallel", [*base, "-m", f"not serial and {LANE_EXCLUSION_EXPR}", *PARALLEL_PASS_FLAGS]),
|
||||
("serial", [*base, "-m", f"serial and {LANE_EXCLUSION_EXPR}"]),
|
||||
]
|
||||
overlapped = _with(PARALLEL_PASS_FLAGS, "--dist", "loadgroup")
|
||||
return [(
|
||||
"overlapped",
|
||||
[*base, "-m", LANE_EXCLUSION_EXPR, *overlapped, f"--serial-shards={serial_shards(_workers())}"],
|
||||
)]
|
||||
|
||||
|
||||
def _run(label: str, argv: list[str], cwd: pathlib.Path) -> int:
|
||||
print(f"\n=== {label}: {shlex.join(argv[2:] if argv[0] == sys.executable else argv)}", flush=True)
|
||||
started = time.monotonic()
|
||||
env = dict(os.environ)
|
||||
override = env.get("OUROBOROS_PREFLIGHT_TEST_WORKERS", "").strip()
|
||||
if override.isdigit(): # the gate steers `-n auto` the same way (preflight_runner._preflight_env)
|
||||
env["PYTEST_XDIST_AUTO_NUM_WORKERS"] = str(max(2, int(override)))
|
||||
code = subprocess.call(argv, cwd=str(cwd), env=env)
|
||||
print(f"=== {label}: exit {code} in {time.monotonic() - started:.0f}s", flush=True)
|
||||
return code
|
||||
|
||||
|
||||
def _node_lane() -> int:
|
||||
from ouroboros.preflight_node import (
|
||||
NODE_MIN_VERSION, WEB_DIR, _version_tuple, candidate_node_tests, probe_node_version, resolve_node,
|
||||
)
|
||||
|
||||
files = candidate_node_tests(REPO)
|
||||
if not files:
|
||||
return 0
|
||||
node = resolve_node()
|
||||
version = probe_node_version(node) if node else ""
|
||||
floor = ".".join(str(part) for part in NODE_MIN_VERSION)
|
||||
if not version or _version_tuple(version) < NODE_MIN_VERSION:
|
||||
found = f"{node} is v{version}" if version else (f"{node} did not answer --version" if node else "no node found")
|
||||
print(f"\n=== node: NOT_RUN — {found}; the web lane needs node >= {floor}, so this is not the full "
|
||||
"battery: install node or pass focused targets", flush=True)
|
||||
return 1
|
||||
return _run("node", [node, "--test", *files], REPO / WEB_DIR)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("--sequential", action="store_true", help="two passes, exactly as the commit gate and CI")
|
||||
args, pytest_args = parser.parse_known_args(argv)
|
||||
if not (REPO / "tests" / "conftest.py").is_file():
|
||||
print(f"{REPO} is not an Ouroboros checkout (no tests/conftest.py)", file=sys.stderr)
|
||||
return 2
|
||||
if pytest_args:
|
||||
if args.sequential:
|
||||
parser.error("--sequential selects the full two-pass battery; drop it or drop the pytest arguments")
|
||||
return _run("focused", [sys.executable, "-m", "pytest", *pytest_args], REPO)
|
||||
started = time.monotonic()
|
||||
code = _node_lane()
|
||||
ran = 0
|
||||
for label, command in battery_commands(args.sequential) if code == 0 else []:
|
||||
code = _run(label, command, REPO)
|
||||
if code == _PYTEST_EXIT_NO_TESTS:
|
||||
code = 0
|
||||
continue
|
||||
ran += 1
|
||||
if code != 0:
|
||||
break
|
||||
if code == 0 and ran == 0:
|
||||
print("no tests were collected in any lane", file=sys.stderr)
|
||||
code = 1
|
||||
print(f"\n=== battery: {'GREEN' if code == 0 else 'RED'} in {time.monotonic() - started:.0f}s", flush=True)
|
||||
return code
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -13,6 +13,7 @@ import sys
|
|||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import zlib
|
||||
|
||||
import pytest
|
||||
pytest.register_assert_rewrite("tests.ui_media_delivery_smoke")
|
||||
|
|
@ -305,6 +306,35 @@ def pytest_collection_modifyitems(config, items): # noqa: ARG001
|
|||
for item in items:
|
||||
if pathlib.Path(str(item.fspath)).name in _SERIAL_TEST_FILES:
|
||||
item.add_marker(pytest.mark.serial)
|
||||
_pin_lane_groups(items, config.getoption("--serial-shards"))
|
||||
|
||||
|
||||
def pytest_addoption(parser):
|
||||
parser.addoption(
|
||||
"--serial-shards", type=int, default=0,
|
||||
help="With `--dist loadgroup`: pin the serial tests to this many file-sharded xdist groups "
|
||||
"so ONE run covers both lanes (scripts/run_tests.py). 0 leaves scheduling untouched.",
|
||||
)
|
||||
|
||||
|
||||
def _pin_lane_groups(items, shards: int) -> None:
|
||||
"""One-run lane scheduling for `scripts/run_tests.py`; inert (shards == 0) for CI and the gate.
|
||||
|
||||
Under `--dist loadgroup` a serial FILE never splits across workers and never runs beside
|
||||
another file of its own shard, so the shard count bounds how many serial files run at once.
|
||||
Every other test keeps the scope `--dist loadscope` gives the two-pass recipe (file, or class
|
||||
when there is one). Serial tests sort first so their long groups are handed out before the queue drains.
|
||||
It runs inside the tryfirst hook because xdist reads `xdist_group` in the same hook.
|
||||
"""
|
||||
if shards <= 0:
|
||||
return
|
||||
for item in items:
|
||||
path = item.nodeid.split("::", 1)[0]
|
||||
serial = item.get_closest_marker("serial") is not None
|
||||
# Serial: by FILE (the isolation unit). Others: the scope `--dist loadscope` uses (class when present).
|
||||
group = f"serial{zlib.crc32(path.encode('utf-8')) % shards}" if serial else item.nodeid.rsplit("::", 1)[0]
|
||||
item.add_marker(pytest.mark.xdist_group(group))
|
||||
items.sort(key=lambda item: item.get_closest_marker("serial") is None)
|
||||
|
||||
|
||||
def pytest_sessionstart(session): # noqa: ARG001
|
||||
|
|
|
|||
|
|
@ -8,12 +8,15 @@ import pytest
|
|||
|
||||
from ouroboros.outcomes import public_task_result
|
||||
from ouroboros.task_results import load_plan_review_state, load_task_result
|
||||
from ouroboros.preflight_node import resolve_node
|
||||
from tests.test_plan_review_engine import DECK_SPEC, _call, _finding, harness as _harness
|
||||
from tests.test_preflight_node import requires_node
|
||||
|
||||
harness = _harness
|
||||
pytestmark = pytest.mark.serial
|
||||
|
||||
|
||||
@requires_node
|
||||
@pytest.mark.parametrize("action", ["finish", "stop"])
|
||||
@pytest.mark.parametrize("cap", ["1", "2"])
|
||||
def test_current_plan_author_is_not_an_extra_reviewer(harness, monkeypatch, action, cap):
|
||||
|
|
@ -70,6 +73,6 @@ def test_current_plan_author_is_not_an_extra_reviewer(harness, monkeypatch, acti
|
|||
assert.equal(pending.state, 'running');
|
||||
assert.equal(pending.authorDecisionText, group.authorDecisionText);
|
||||
"""
|
||||
completed = subprocess.run(["node", "--input-type=module", "-e", script],
|
||||
completed = subprocess.run([resolve_node(), "--input-type=module", "-e", script],
|
||||
input=json.dumps(detail), capture_output=True, text=True, timeout=30)
|
||||
assert completed.returncode == 0, completed.stdout + completed.stderr
|
||||
|
|
|
|||
141
tests/test_run_tests_script.py
Normal file
141
tests/test_run_tests_script.py
Normal file
|
|
@ -0,0 +1,141 @@
|
|||
"""The local battery entry point keeps the gate's recipe and a bounded serial overlap."""
|
||||
|
||||
import importlib.util
|
||||
import pathlib
|
||||
import types
|
||||
|
||||
import pytest
|
||||
|
||||
from ouroboros.preflight_runner import LANE_EXCLUSION_EXPR, PARALLEL_PASS_FLAGS
|
||||
from tests import conftest
|
||||
|
||||
REPO = pathlib.Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def run_tests():
|
||||
spec = importlib.util.spec_from_file_location("run_tests_script", REPO / "scripts" / "run_tests.py")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def _item(nodeid, *, serial):
|
||||
markers = [types.SimpleNamespace(name="serial")] if serial else []
|
||||
item = types.SimpleNamespace(nodeid=nodeid, added=[])
|
||||
item.get_closest_marker = lambda name: next((m for m in markers if m.name == name), None)
|
||||
item.add_marker = item.added.append
|
||||
return item
|
||||
|
||||
|
||||
def _group(item):
|
||||
(mark,) = item.added
|
||||
assert mark.name == "xdist_group"
|
||||
return mark.args[0]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("workers, shards", [(2, 1), (4, 1), (7, 1), (8, 2), (12, 3), (16, 4), (64, 4)])
|
||||
def test_serial_overlap_is_bounded_by_the_worker_count(run_tests, workers, shards):
|
||||
assert run_tests.serial_shards(workers) == shards
|
||||
|
||||
|
||||
def test_sequential_mode_keeps_the_gate_marker_split_and_xdist_flags(run_tests, monkeypatch):
|
||||
monkeypatch.setenv("OUROBOROS_PREFLIGHT_TEST_WORKERS", "6")
|
||||
(parallel_label, parallel), (serial_label, serial) = run_tests.battery_commands(sequential=True)
|
||||
assert (parallel_label, serial_label) == ("parallel", "serial")
|
||||
assert parallel[3:] == ["tests/", "-m", f"not serial and {LANE_EXCLUSION_EXPR}", *PARALLEL_PASS_FLAGS]
|
||||
assert serial[3:] == ["tests/", "-m", f"serial and {LANE_EXCLUSION_EXPR}"]
|
||||
|
||||
|
||||
def test_default_mode_is_one_run_over_both_lanes(run_tests, monkeypatch):
|
||||
monkeypatch.setenv("OUROBOROS_PREFLIGHT_TEST_WORKERS", "8")
|
||||
((label, command),) = run_tests.battery_commands(sequential=False)
|
||||
assert label == "overlapped"
|
||||
assert command[:3] == [run_tests.sys.executable, "-m", "pytest"]
|
||||
command = command[3:]
|
||||
assert command[command.index("-m") + 1] == LANE_EXCLUSION_EXPR
|
||||
assert command[command.index("--dist") + 1] == "loadgroup"
|
||||
assert command[command.index("-n") + 1] == "auto"
|
||||
assert command[-1] == "--serial-shards=2"
|
||||
untouched = [flag for flag in PARALLEL_PASS_FLAGS if flag.startswith("--") and flag != "--dist"]
|
||||
assert all(flag in command for flag in untouched)
|
||||
|
||||
|
||||
def test_the_worker_override_steers_xdist_like_the_gate(run_tests, monkeypatch):
|
||||
seen = {}
|
||||
monkeypatch.setenv("OUROBOROS_PREFLIGHT_TEST_WORKERS", "3")
|
||||
monkeypatch.setattr(run_tests.subprocess, "call", lambda argv, cwd, env: seen.update(env=env) or 0)
|
||||
assert run_tests._run("x", [run_tests.sys.executable, "-m", "pytest"], run_tests.REPO) == 0
|
||||
assert seen["env"]["PYTEST_XDIST_AUTO_NUM_WORKERS"] == "3"
|
||||
|
||||
|
||||
def test_sequential_with_pytest_arguments_is_refused(run_tests, monkeypatch):
|
||||
monkeypatch.setattr(run_tests, "_run", lambda label, argv, cwd: pytest.fail("nothing must run"))
|
||||
with pytest.raises(SystemExit) as exit_info:
|
||||
run_tests.main(["--sequential", "tests/test_x.py"])
|
||||
assert exit_info.value.code == 2
|
||||
|
||||
|
||||
def test_a_bare_call_never_means_a_subset(run_tests, monkeypatch):
|
||||
calls = []
|
||||
monkeypatch.setattr(run_tests, "_node_lane", lambda: 0)
|
||||
monkeypatch.setattr(run_tests, "_run", lambda label, argv, cwd: calls.append((label, argv)) or 0)
|
||||
assert run_tests.main([]) == 0
|
||||
assert [label for label, _ in calls] == ["overlapped"]
|
||||
calls.clear()
|
||||
assert run_tests.main(["tests/test_x.py", "-k", "one"]) == 0
|
||||
assert calls == [("focused", [run_tests.sys.executable, "-m", "pytest", "tests/test_x.py", "-k", "one"])]
|
||||
|
||||
|
||||
def test_a_missing_node_is_a_red_not_run_never_a_silent_skip(run_tests, monkeypatch):
|
||||
monkeypatch.setattr(run_tests, "resolve_node", lambda: None, raising=False)
|
||||
monkeypatch.setattr(run_tests, "_run", lambda label, argv, cwd: pytest.fail("pytest must not start"))
|
||||
from ouroboros import preflight_node
|
||||
monkeypatch.setattr(preflight_node, "resolve_node", lambda: None)
|
||||
monkeypatch.setattr(preflight_node, "candidate_node_tests", lambda repo: ["tests/x.test.js"])
|
||||
assert run_tests.main([]) == 1
|
||||
|
||||
|
||||
def test_a_red_lane_stops_the_battery_and_an_empty_battery_is_red(run_tests, monkeypatch):
|
||||
monkeypatch.setattr(run_tests, "_node_lane", lambda: 0)
|
||||
codes = iter([1])
|
||||
seen = []
|
||||
monkeypatch.setattr(run_tests, "_run", lambda label, argv, cwd: seen.append(label) or next(codes))
|
||||
assert run_tests.main(["--sequential"]) == 1
|
||||
assert seen == ["parallel"]
|
||||
monkeypatch.setattr(run_tests, "_run", lambda label, argv, cwd: 5)
|
||||
assert run_tests.main(["--sequential"]) == 1
|
||||
|
||||
|
||||
def test_lane_groups_pin_serial_files_and_keep_other_files_whole():
|
||||
items = [
|
||||
_item("tests/test_a.py::test_one", serial=False),
|
||||
_item("tests/test_s.py::test_one", serial=True),
|
||||
_item("tests/test_a.py::test_two", serial=False),
|
||||
_item("tests/test_s.py::test_two[x]", serial=True),
|
||||
_item("tests/test_t.py::test_one", serial=True),
|
||||
_item("tests/test_c.py::TestBox::test_one", serial=False),
|
||||
_item("tests/test_s.py::TestSer::test_three", serial=True),
|
||||
]
|
||||
conftest._pin_lane_groups(items, 2)
|
||||
assert [item.nodeid.split("::")[0] for item in items[:4]] == ["tests/test_s.py", "tests/test_s.py", "tests/test_t.py", "tests/test_s.py"]
|
||||
groups = {item.nodeid: _group(item) for item in items}
|
||||
assert groups["tests/test_a.py::test_one"] == groups["tests/test_a.py::test_two"] == "tests/test_a.py"
|
||||
assert groups["tests/test_c.py::TestBox::test_one"] == "tests/test_c.py::TestBox"
|
||||
assert groups["tests/test_s.py::TestSer::test_three"] == groups["tests/test_s.py::test_one"]
|
||||
assert groups["tests/test_s.py::test_one"] == groups["tests/test_s.py::test_two[x]"]
|
||||
assert {groups["tests/test_s.py::test_one"], groups["tests/test_t.py::test_one"]} <= {"serial0", "serial1"}
|
||||
|
||||
|
||||
def test_lane_groups_are_inert_without_the_option():
|
||||
items = [_item("tests/test_s.py::test_one", serial=True), _item("tests/test_a.py::test_one", serial=False)]
|
||||
conftest._pin_lane_groups(items, 0)
|
||||
assert [item.added for item in items] == [[], []]
|
||||
assert [item.nodeid for item in items] == ["tests/test_s.py::test_one", "tests/test_a.py::test_one"]
|
||||
|
||||
|
||||
def test_the_documented_command_is_the_script():
|
||||
makefile = (REPO / "Makefile").read_text(encoding="utf-8")
|
||||
assert "test:\n\tuv run --locked python scripts/run_tests.py\n" in makefile
|
||||
for doc in ("CONTRIBUTING.md", "README.md", "docs/development/14-build-and-ci.md", "prompts/SYSTEM.md"):
|
||||
assert "scripts/run_tests.py" in (REPO / doc).read_text(encoding="utf-8"), doc
|
||||
Loading…
Add table
Add a link
Reference in a new issue