vrr/unsloth
2
0
Fork 0
mirror of https://github.com/unslothai/unsloth.git synced 2026-08-25 08:42:25 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 29
unsloth/.github/workflows/studio-backend-ci.yml
Daniel Han 86d891253f
Assert the event-loop property directly in the two load_freeze latency tests (#9532) 
* Assert the event-loop property directly in the two load_freeze latency tests

Both drove a live uvicorn server and compared real elapsed time against a
threshold: /health under 250 ms while a blocking probe ran, and a 104-request
burst under 350 ms. Neither threshold is the contract. The contract is that the
route hands its blocking call to a thread and leaves the event loop free, and a
descheduled runner thread falsifies the threshold with the code correct.

They now hold the blocking call open on an event, require /health to be answered
while it is held, and only then release it, so the ordering is the evidence and
no duration is compared. On the pre-#5642 shape those two waits are mutually
dependent and the run deadlocks rather than running late, so a faster or slower
machine cannot turn the regression green either.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: danielhanchen <unslothshared@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-08-22 17:10:38 -07:00

538 lines
30 KiB
YAML
Raw Permalink Blame History

# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
# Runs the existing studio/backend/tests/ suite (~860 tests, all CPU-friendly)
# on every PR that touches the backend or unsloth library. Until this lands,
# none of those tests run automatically. Verified locally on Python 3.13 with
# the surgical exclusions below: 861 pass, 4 skipped.
#
# Exclusions:
# - tests/test_studio_api.py: end-to-end against a live model + GGUF download,
# too heavy for free runners. Run separately when GPU CI is available.
# - -k 'not llama_cpp_load_progress_live': spawns a real llama.cpp process,
# not appropriate for CPU-only runners.
#
# Two jobs:
# - pytest on 3.13 over studio/backend/tests, with the floor checked statically
# - repo-cpu-tests: auto-discovered tests/ + state-isolated spoof files
#
# Whole-repo Python lint (syntax + ruff + debugger-leftover scan)
# moved to the dedicated `Lint CI` workflow (.github/workflows/lint-ci.yml)
# so it fires on every PR rather than only on studio/unsloth/tests
# path changes.
name: Backend CI
on:
pull_request:
paths:
- 'studio/**'
- 'unsloth/**'
- 'unsloth_cli/**'
- 'tests/**'
# The root installers: tests/sh/*.sh and tests/studio/install/* assert
# against these two files, so a change here must run the suite that
# covers it. Without them an install-only edit (the shape most AMD/ROCm
# routing fixes take) skipped Backend CI entirely.
- 'install.sh'
- 'install.ps1'
- 'scripts/**'
- 'pyproject.toml'
- '.github/workflows/studio-backend-ci.yml'
push:
branches: [main]
concurrency:
# Unique per commit on main, so a merge burst cannot cancel a pending run
# before it starts. See the note below for what this is buying.
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}
# On a PR branch: latest-only, which is what cancel-in-progress buys.
#
# On main it buys less than it looks like. GitHub cancels any PENDING run in a
# concurrency group as soon as a newer one is queued, and cancel-in-progress
# governs only runs that are already EXECUTING ("any existing pending job or
# workflow in the same concurrency group will be canceled", workflow-syntax docs).
# So a burst of merges leaves only the tip's run alive: on 2026-08-17 four merges
# in 37 minutes cancelled three consecutive main runs of this workflow and none
# completed. Detection survives on the tip, since the tree is cumulative;
# per-commit attribution does not.
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
permissions:
contents: read
env:
# The oldest interpreter this code is expected to run on.
#
# Nothing executes it any more. The 3.10, 3.11 and 3.12 legs ran the same suite as 3.13
# and differed by exactly one skip marker, at 97 runner-minutes per push against a queue
# that has been observed 195 deep, so they are gone and the wall-clock they were costing
# every other workflow is gone with them.
#
# What they were really defending is that nothing reaches for a symbol newer than this,
# which is static, and scripts/lint_backend_python_floor.py now checks it here on every
# pull request in seconds across 1093 files. Raise this number only with a reason, and
# remember that raising it is a support decision, not a CI one.
#
# 3.10 rather than the 3.9 pyproject.toml declares, because 3.9 is not true today:
# unsloth/models/_utils.py uses dataclasses.dataclass(kw_only) and
# tempfile.TemporaryDirectory(ignore_cleanup_errors), both 3.10. Reconciling those two
# is worth doing separately; this lint is what made it visible.
PYTHON_FLOOR: '3.10'
jobs:
pytest:
name: (Python ${{ matrix.python }})
runs-on: ubuntu-latest
# 45 guards against a hang; it is not a performance budget. The comment this replaces
# claimed 14.1 to 15.0 minutes against a limit of 30, but this step now measures 22.5 to
# 29.2 minutes across the matrix, and it has been cancelling at the limit: across the six
# open PRs on this branch's stack, five had one leg cancelled at 30m9s while its siblings
# passed at 25 to 29 minutes. Which leg loses is luck, so every PR carries a red check
# that says nothing about the PR.
# Running it on all four cores would fix the duration properly and is worth ~79
# runner-minutes a push, but this suite is not order-independent enough yet. Measured
# under `-n 4` on a 4-vCPU runner it fails tests serial does not, in more than one way and
# not the same way twice. The repo job below IS order-independent and does run parallel;
# this one waits until those dependencies are found and fixed individually. Raising the
# limit costs nothing until a job actually needs it, and a cancelled leg costs the whole
# 30 minutes anyway.
#
# This cap is the last line, not the first one. It reports the word "cancelled" and
# names nothing, which is how a single deadlocked test held main red for forty hours
# (run 32385375389 onward) while the step timings said the suite itself still finished
# its work in six minutes. The per-test --timeout below is what actually names a hang;
# this stays as the backstop for a wedge the per-test timer cannot see, such as one in
# collection or in a worker that never comes up.
timeout-minutes: 45
strategy:
fail-fast: false
matrix:
# 3.13 only, on pull requests and on main alike.
#
# Measured on one runner over the same tree: the four legs collected the same
# 26,320 tests and differed by exactly ONE of them, the >= 3.12 gate on
# test_demonstrates_the_underlying_stdlib_regression. 3.10 and 3.11 reported
# 26193 passed / 127 skipped, 3.12 and 3.13 reported 26194 / 126. Four legs cost
# 97 runner-minutes per push (1663 + 1249 + 1419 + 1506 seconds) against a queue
# that has been observed 195 deep, to re-run one identical suite four times and
# learn the value of a single skip marker.
#
# What the older legs were really defending is that nothing in the tree reaches
# for a symbol newer than the floor, and that is a static property. It is now
# checked statically, on every pull request, in seconds, by
# scripts/lint_backend_python_floor.py, which reads stdlib API availability
# rather than syntax alone and covers 1093 shipped and executed files.
#
# A static check does not run anything, though, and that part cannot be waved
# away: two interpreters that both accept a line can still behave differently on
# it, and a sys.version_info branch is only ever parsed, never taken.
#
# So the branches were counted rather than assumed. Seven files in the backend
# carry one, at 3.10, 3.12 and 3.14. The 3.10 ones were never straddled even by
# the old matrix, whose oldest leg was 3.10, so every leg took the same side of
# them and removing legs loses nothing there. 3.14 is above every leg there has
# ever been. What is genuinely lost is the PRE-3.12 side of three files, and that
# is small enough to keep executing directly.
#
# Hence the second entry: 3.11, the newest version that still takes that side,
# running those three files and nothing else. Roughly 40 tests in under ten
# seconds, not a second copy of the suite, and it runs beside the full leg rather
# than in front of it, so the critical path is unchanged.
include:
- python: '3.13'
scope: full
- python: '3.11'
scope: floor-spot-check
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: '${{ matrix.python }}'
- name: Restore the pip cache
id: pip-cache
uses: ./.github/actions/pip-cache-restore
with:
key-files: |
pyproject.toml
studio/backend/requirements/*.txt
- name: Install backend test dependencies (CPU only)
run: |
python -m pip install --upgrade pip
# Unsloth's declared backend deps:
pip install -r studio/backend/requirements/studio.txt
# Extras that studio.txt does not list but the import chain needs
# (python-multipart for FastAPI form/file uploads, sqlalchemy/cryptography
# for the auth DB, yaml/jinja2 for utils.models.model_config, psutil for
# the orphan-cleanup process scan, etc.):
pip install \
python-multipart aiofiles sqlalchemy cryptography psutil \
pyyaml jinja2 mammoth unpdf requests \
'numpy<3' pytest pytest-asyncio pytest-xdist pytest-timeout httpx \
soundfile librosa
# pytest-timeout: see the --timeout flags below. Without it a test that never
# returns is indistinguishable from a slow suite, and the job reports only
# "cancelled".
# soundfile + librosa: test_audio_dataset_decode importorskips both, so without
# them the no-torchcodec decode path is silently untested here.
# Torch CPU + transformers are required by a chunk of the backend test
# suite (gpu_selection, kv_cache_estimation, utils). CPU-only torch
# keeps the install ~250 MB / ~1 min on a clean runner.
pip install --index-url https://download.pytorch.org/whl/cpu --extra-index-url https://pypi.org/simple 'torch>=2.4,<2.11'
pip install 'transformers>=4.51,<5.5'
# peft: core/inference/inference.py imports it at module scope, and it is listed
# only in extras-no-deps.txt and no-torch-runtime.txt, neither of which this job
# installs. Without it every importorskip of that module SKIPS instead of running,
# which is invisible: test_safetensors_reasoning_stream.py reported 10 passed and
# 3 skipped here, green. Stubbing it instead does not work -- transformers probes
# `importlib.util.find_spec("peft")`, which raises on a stub whose __spec__ is None.
# AFTER the CPU torch line above, not before it: peft depends on torch, so pip
# would resolve that from PyPI, and the CUDA wheel it picks satisfies the range
# the next line asks for, leaving this nominally CPU-only job carrying the CUDA
# distribution and its nvidia-* dependencies.
# The version comes from the requirements file rather than being written here:
# extras-no-deps.txt pins peft==0.18.1 deliberately (0.19.0 breaks Unsloth's
# export subprocess), and an unconstrained install takes the newest release,
# so this job would test against a peft production never ships and no other
# job installs. Read out of the file so the two cannot drift apart.
pip install "$(grep -m1 -E '^peft==' studio/backend/requirements/extras-no-deps.txt)"
# torchao: the pre-quant allowlist is built by importing the torchao tensor
# constructors a checkpoint names, and registration refuses unless at least one of
# them resolves. Without it every pre-quant load returns None and 21 tests across
# test_diffusion_prequant.py and test_diffusion_convrot.py fail on an install that is
# simply missing a dependency they need. CPU-only is fine: it declines to load its cpp
# extensions under torch < 2.11 and the pure-Python constructors still import.
pip install torchao
- name: Backend tests
if: matrix.scope == 'full'
working-directory: studio/backend
# Locally validated against this dep set: 831 passed, 5 skipped, 35 deselected.
# Deselections (all environment-specific, would never pass on a GPU-less
# `ubuntu-latest` runner regardless of code correctness):
# - llama_cpp_load_progress_live: spawns a real llama.cpp process
# - TestGpuAutoSelection / TestPreSpawnGpuResolution / TestPerGpuFitGuardAllCounts:
# require live transformers config introspection on real GPUs
# - TestTransformersIntrospection: same
# - test_returns_cuda_when_cuda_available / test_calls_cuda_cache_when_cuda:
# assume CUDA-capable GPU
#
# -n 4: 1322.6s -> 343.0s locally on the same 4-worker shape as the runner, with
# an IDENTICAL result -- 51 failed / 26190 passed serial against 51 failed /
# 26188 passed parallel, and the two failure sets compared equal name for name,
# so nothing here depends on the order it runs in. (Those 51 are a local
# environment missing peft and a diffusers pin; the point is that the two modes
# agree, not the count.) Same shape as repo-cpu-tests below, which has run -n 4
# since it measured 806s -> 220s.
#
# Unit tests, so this is CPU and not memory bound: the job installs --no-torch
# style CPU wheels and loads no model, unlike the inference smoke workflows
# where four workers on one runner would not fit.
#
# test_streaming_stripper is ignored here and run serially below, for the reason
# repo-cpu-tests already isolates load_freeze: it compares its own measured cost
# against a reference implementation timed in the same process, and a worker
# descheduled by the other three inflates one side of that ratio. Observed on
# staging, where the 3.10 leg reported "early markup cost 1.354s against the
# reference's 0.854s" while 3.13 passed the same commit.
#
# The other two assert ABSOLUTE elapsed time, and tightly: 50ms for a
# short-circuit that should not run at all, and 100ms for a regex backtracking
# guard. Bounds that small are inside one scheduler quantum, so under four
# workers they measure the scheduler as much as the code. Bounds of 0.5s and up
# are left in the parallel run; there are 22 files with elapsed-time bounds and
# serialising all of them would give back most of what this change buys. The
# isolation guard scans for the tight ones, so a new test below the threshold
# fails that guard instead of flaking here.
#
# --timeout: a per-test wall-clock cap, so a test that never returns fails with its
# own name in the summary instead of taking the job's 45 minutes down with it and
# reporting "cancelled". 330s comes off the measured distribution rather than taste:
# the slowest legitimate test in this suite is
# test_conversation_archive.py::test_the_newest_revision_survives_a_tied_run_LONGER_than_the_cap
# at 82.5s (measured under -n 4 on a contended box; 75.4s serially), and the next
# ones down are 76s and 25s, so the cap is 4x the worst real test. Nothing here is
# near it, and a deadlock is caught in minutes rather than never.
#
# --timeout arms inside each xdist worker, so a stuck CONTROLLER escapes it: when a
# worker wedges holding the pipe, the session waits forever and no per-test cap is
# consulted. On the #9473 branch this step sat silent from 95% for 36.8 minutes,
# three runs running, each burning the whole 45 minute budget and reporting
# "cancelled" -- unreadable against ordinary concurrency supersession.
#
# So bound the session from outside, where a wedged worker cannot defeat it. SIGINT
# first so pytest dumps where it was stuck, SIGKILL 60s later if it will not.
# Reproduced with a test forking a child that never exits: unwrapped it hangs
# silently, wrapped it exits 124 through execnet safe_terminate -> workerpool.waitall.
# Expect a stuck FRAME, not always a test name. 1500s is ~3x the 8-10 minute norm.
run: |
timeout --signal=INT --kill-after=60 1500 \
python -m pytest tests/ -q --tb=short -n 4 --timeout=330 \
--ignore=tests/test_studio_api.py \
--ignore=tests/test_streaming_stripper.py \
--ignore=tests/test_llama_cpp_wait_for_vram_settle.py \
--ignore=tests/test_tool_xml_strip.py \
--ignore=tests/test_diffusion_checkpoint_resume.py \
--ignore=tests/test_tool_output_streaming.py \
--ignore=tests/test_web_fetch_extraction.py \
--ignore=tests/test_tool_call_parser_strict.py \
--ignore=tests/test_tunnel_safe_long_post.py \
--ignore=tests/test_scan_loras_off_event_loop.py \
--ignore=tests/test_anthropic_messages.py \
--ignore=tests/test_profile_stats.py \
-k 'not llama_cpp_load_progress_live and not TestGpuAutoSelection and not TestPreSpawnGpuResolution and not TestPerGpuFitGuardAllCounts and not TestTransformersIntrospection and not test_returns_cuda_when_cuda_available and not test_calls_cuda_cache_when_cuda'
- name: Backend tests that cannot share a worker
if: matrix.scope == 'full'
working-directory: studio/backend
# Relative timing against a reference measured in the same process. Serial, so the
# comparison is between two implementations rather than between two schedulings.
run: |
python -m pytest -q --tb=short --timeout=330 \
tests/test_streaming_stripper.py \
tests/test_llama_cpp_wait_for_vram_settle.py \
tests/test_tool_xml_strip.py \
tests/test_diffusion_checkpoint_resume.py \
tests/test_tool_output_streaming.py \
tests/test_web_fetch_extraction.py \
tests/test_tool_call_parser_strict.py \
tests/test_tunnel_safe_long_post.py \
tests/test_scan_loras_off_event_loop.py \
tests/test_anthropic_messages.py \
tests/test_profile_stats.py
- name: Pre-3.12 branches, on the newest interpreter that takes them
if: matrix.scope == 'floor-spot-check'
working-directory: studio/backend
# Not a second copy of the suite. Seven backend files carry a sys.version_info
# branch, at 3.10, 3.12 and 3.14. The 3.10 ones were never straddled even by the
# old four-leg matrix, whose oldest leg WAS 3.10, so every leg took the same side
# and dropping legs loses nothing there. 3.14 is above every leg there has ever
# been. What a 3.13-only matrix genuinely stops executing is the pre-3.12 side of
# these files, and that is small enough to keep running rather than argue about.
#
# 3.11 because it is the newest version that still takes that side: closest to
# the ceiling, so anything it catches is about the boundary rather than about
# being old. Roughly 40 tests in under ten seconds, beside the full leg rather
# than in front of it, so the critical path is the full leg either way.
run: |
python -m pytest -q --tb=short --timeout=330 \
tests/test_third_party_source.py \
tests/test_recommended_folders_permission.py \
tests/test_hf_cache_settings.py
- name: Save the pip cache
if: always()
uses: ./.github/actions/pip-cache-save
with:
dir: ${{ steps.pip-cache.outputs.dir }}
key: ${{ steps.pip-cache.outputs.key }}
cache-hit: ${{ steps.pip-cache.outputs.cache-hit }}
repo-cpu-tests:
# Auto-discover everything under tests/ that is not GPU-bound by
# design. New tests added in covered directories are picked up
# without a workflow edit. Locally validated: 760 passed, 1 skipped,
# 23 deselected. tests/conftest.py (mirroring unsloth-zoo PR #624)
# pre-loads unsloth_zoo.device_type and unsloth.device_type under a
# mocked torch.cuda.is_available so the unsloth import chain
# succeeds on CPU.
name: Repo tests (CPU)
runs-on: ubuntu-latest
# Same reason as the matrix above: this one measured 13.4 minutes against the same 15.
timeout-minutes: 30
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: '3.12'
- name: Restore the pip cache
id: pip-cache
uses: ./.github/actions/pip-cache-restore
with:
key-files: |
pyproject.toml
studio/backend/requirements/*.txt
# node + uv unlock ~60 tests that previously skipped on CI:
# - 9 tests in test_chat_preset_builtin_invariants.py need node to
# compile a tiny TS harness against the frontend chat sources.
# - tests/python/* spawn fresh `uv venv`s to verify the no-torch
# install path; they self-skip when uv is missing.
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: '22'
- name: Install uv (for tests/python/* sandboxed venvs)
run: pip install uv
- name: Install deps (shared shape with backend pytest job)
run: |
python -m pip install --upgrade pip
pip install -r studio/backend/requirements/studio.txt
pip install \
python-multipart aiofiles sqlalchemy cryptography psutil \
pyyaml jinja2 mammoth unpdf requests typer \
'numpy<3' pytest pytest-asyncio pytest-xdist httpx
# torchvision: unsloth_zoo.vision_utils imports it at module scope.
pip install --index-url https://download.pytorch.org/whl/cpu --extra-index-url https://pypi.org/simple \
'torch>=2.4,<2.11' 'torchvision<0.26'
pip install 'transformers>=4.51,<5.5'
# bitsandbytes: hard import in unsloth/models/_utils.py. Recent
# versions ship a CPU build that imports cleanly on Linux.
pip install 'bitsandbytes>=0.45'
# unsloth.device_type imports unsloth_zoo.utils.Version at module
# scope, so the conftest preload needs unsloth_zoo. Pull from
# git main so this job sees the same zoo HEAD as Core / MLX /
# install.sh do (otherwise a fix on zoo main hides until release).
# No --no-deps: matches prior `pip install 'unsloth_zoo>=2026.5.1'`
# behaviour so triton etc. still come in for the Repo tests CPU
# collection imports.
for attempt in 1 2 3; do
if pip install "unsloth_zoo @ git+https://github.com/unslothai/unsloth-zoo"; then
break
fi
[ "$attempt" -eq 3 ] && { echo "::error::unsloth_zoo install failed after 3 attempts"; exit 1; }
sleep $((5 * attempt))
done
pip install -e . --no-deps
- name: Repo tests (CPU, auto-discovered)
env:
# tests/python/* import install_python_stack from studio/.
PYTHONPATH: ${{ github.workspace }}/studio
# Skip lazy compilation work the unsloth import chain wants to
# do at import time on a real GPU.
UNSLOTH_COMPILE_DISABLE: '1'
# --ignore: GPU-bound directories (qlora/saving need real weights;
# tests/sh is the shell suite the next step handles; tests/utils
# is a helpers folder); tests/vllm_compat + tests/version_compat
# are dedicated multi-version drift canaries with their own job
# in version-compat-ci.yml that installs the heavier dep set
# (torchcodec, full transformers/peft/bnb pins) those tests need.
# State-sensitive hardware-spoofing files run in isolation in the
# next step because they mutate hardware.py module globals, and
# tests/studio/load_freeze in the step after that because its
# assertions are wall-clock latency bounds.
# -m: honour markers from tests/python/conftest.py (`server` =
# needs studio venv, `e2e` = needs network).
# --deselect:
# - test_model_registration / test_all_model_registration:
# hit huggingface_hub for live model existence checks.
# - test_autoconfig_works_with_no_torch_runtime / test_autoconfig_succeeds:
# fail because no-torch-runtime.txt does not pin tokenizers
# and the latest tokenizers (0.23.1) is incompatible with the
# transformers it resolves to. Tracked separately; this is a
# real bug in the no-torch install path, not a CI issue.
#
# -n 4: 806.1s -> 219.7s locally on the same 4-worker shape as the runner, with an
# identical 2-failed / 8066-passed / 178-skipped / 40-subtest result. The three pytest
# steps below stay serial deliberately: the hardware-spoof step exists precisely
# because those files mutate hardware.py module globals, and all three already measure
# under 0.1 minutes on the runner, so there is nothing to win and isolation to lose.
run: |
python -m pytest tests/ -q --tb=short -n 4 \
--ignore=tests/qlora \
--ignore=tests/saving \
--ignore=tests/utils \
--ignore=tests/sh \
--ignore=tests/studio/load_freeze \
--ignore=tests/studio/test_hardware_dispatch_matrix.py \
--ignore=tests/studio/test_is_mlx_dispatch_gate.py \
--ignore=tests/studio/test_xpu_spoof_pipeline.py \
--ignore=tests/vllm_compat \
--ignore=tests/version_compat \
-m 'not server and not e2e' \
--deselect tests/test_model_registry.py::test_model_registration \
--deselect tests/test_model_registry.py::test_all_model_registration \
--deselect 'tests/python/test_tokenizers_and_torch_constraint.py::TestE2ETokenizersFix::test_autoconfig_works_with_no_torch_runtime' \
--deselect 'tests/python/test_tokenizers_and_torch_constraint.py::TestE2EFullNoTorchSandbox::test_autoconfig_succeeds'
- name: Hardware-spoof tests (state-sensitive, run in isolation)
env:
PYTHONPATH: ${{ github.workspace }}/studio
UNSLOTH_COMPILE_DISABLE: '1'
# These files mutate hardware.py module globals at runtime via the
# spoof fixtures (CUDA/ROCm/XPU/MLX/CPU), which leaks state into any
# other test that imports hardware. Run them in their own pytest
# invocation so the leak does not cross file boundaries.
run: |
python -m pytest -q --tb=short \
tests/studio/test_hardware_dispatch_matrix.py \
tests/studio/test_is_mlx_dispatch_gate.py \
tests/studio/test_xpu_spoof_pipeline.py
- name: Event-loop latency tests (wall clock, run without CPU contention)
env:
PYTHONPATH: ${{ github.workspace }}/studio
UNSLOTH_COMPILE_DISABLE: '1'
# These drive a live uvicorn server and assert upper bounds on real elapsed
# time: 50 concurrent probes under 15s, a fast-shim probe under 2s, five
# sequential probes under 10s, and a not-loaded short circuit under 50 ms.
# A descheduled worker inflates them, so they run alone.
#
# The two tightest bounds this step used to carry, 250 ms for a /health burst
# and 350 ms for a 100-request burst, are gone. Both were proxies for one
# property, that the route hands its blocking call to a thread and leaves the
# event loop free, and both could be falsified by a descheduled thread with the
# code entirely correct. Those two tests now hold the blocking call open on an
# event and require /health to answer while it is held, which asserts the
# property itself and does not move with load.
# 20s serially, against the ~10 minutes -n 4 saves on this job.
run: python -m pytest tests/studio/load_freeze -q --tb=short
- name: CLI tests (unsloth_cli)
# unsloth_cli/tests had no CI at all: `unsloth_cli/**` was only a paths
# trigger and a ruff target, so 673 tests covering the studio launcher,
# the pre-exposure gate and the auth secret writers ran nowhere, and
# four of them had been failing on main unnoticed.
# Own step, not folded into the tests/ discovery above: pyproject's
# testpaths is tests/, and this suite needs no PYTHONPATH or CUDA spoof
# (it self-bootstraps sys.path and imports neither unsloth nor torch).
run: python -m pytest unsloth_cli/tests -q --tb=short
- name: Shell installer tests
# Auto-discovered rather than allowlisted. The old hardcoded list had
# silently fallen seven files behind tests/run_all.sh, including
# test_strixhalo_wsl_reroute.sh -- the only shell coverage of the ROCm
# WSL reroute -- so that suite never ran on a PR. Skips are explicit,
# each with a reason, and tests/studio/test_ci_shell_suite_coverage.py
# fails if this step stops discovering the directory or the skip list
# grows without one.
#
# Skipped:
# test_install_rollback_lifecycle.sh: covered by cross-platform-parity-ci.yml.
run: |
set -e
skip="test_install_rollback_lifecycle.sh"
found=0
for s in tests/sh/test_*.sh; do
case " $skip " in
*" $(basename "$s") "*) echo "skipping $s (see workflow comment)"; continue ;;
esac
found=$((found + 1))
echo "::group::$s"
bash "$s"
echo "::endgroup::"
done
[ "$found" -gt 0 ] || { echo "::error::no shell tests discovered under tests/sh"; exit 1; }
echo "ran $found shell installer test files"
- name: Save the pip cache
if: always()
uses: ./.github/actions/pip-cache-save
with:
dir: ${{ steps.pip-cache.outputs.dir }}
key: ${{ steps.pip-cache.outputs.key }}
cache-hit: ${{ steps.pip-cache.outputs.cache-hit }}
Reference in a new issue View git blame Copy permalink