mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-25 08:42:25 +00:00
96cf275d71
* Run the Backend CI matrix in parallel, like its sibling job already does The matrix leg is the longest job in the repo at 23.3 minutes, and it was the only large pytest run still serial. repo-cpu-tests has run -n 4 since it measured 806s -> 220s. Measured over the same tree in the same environment, serial against -n 4: 1322.6s -> 343.0s, and the results are identical -- 51 failed / 26190 passed serial, 51 failed / 26188 passed parallel, with the two failure sets compared name for name and equal. So nothing in the backend suite depends on the order it runs in. (Those 51 are one local environment missing peft and a diffusers pin. The point is that the two modes agree.) This is CPU bound and not memory bound: the suite loads no model, unlike the inference smoke workflows where four workers on one runner would not fit. The existing isolation guard caught this change, correctly: it assumed exactly one parallel pytest run, and the two now run over different trees. The repo-root job runs tests/ from the checkout, the matrix job runs the backend's own suite from studio/backend, so tests/studio/load_freeze is not a path that exists for it and demanding those ignores would be nonsense. The guard now tells the two apart and applies the isolation rules to the repo-root run only, and a new test pins the matrix leg as parallel so losing the flag shows up as a failure rather than as CI slowly getting slower again. * Keep the relative-timing tests off the parallel workers Staging caught what the local comparison could not: the 3.10 leg reported 'early markup cost 1.354s against the reference's 0.854s' and 'incremental cost grew 7.0x vs the reference's 11.5x', while 3.13 passed the same commit in 9 minutes against the 23 it used to take. test_streaming_stripper times itself against a reference implementation measured in the same process. Under four workers on four vCPUs one side of that ratio gets descheduled and the other does not, so the comparison stops being between two implementations. It is the same reason repo-cpu-tests already keeps load_freeze out of its parallel run, and it does not reproduce on a machine with cores to spare, which is why the local run agreed serially and in parallel. So it is ignored from the parallel run and runs again in its own serial step, and the isolation guard now holds that pair together for the backend run the way it already does for the repo-root one: putting the file back in the parallel run fails one test, deleting the serial step fails another. * Find the tight elapsed-time bounds by scanning, not by remembering Two more files assert ABSOLUTE elapsed time, and tightly: 50ms for a short-circuit that should not run the probe at all, and 100ms for a regex backtracking guard. Bounds that small sit inside one scheduler quantum, so under four workers on four vCPUs they measure the scheduler as much as the code. Both passed on staging, which is the problem: they would have flaked later, on somebody else's change. Twenty-two backend files assert some elapsed bound, and serialising all of them would give back most of what -n 4 buys. So the line is drawn at 0.1s, where the measurement stops being about the code, and the three files at or below it are ignored from the parallel run and rerun serially, which costs 2.2s and 1.7s. The guard now finds them by scanning rather than by listing. It reads with ast, so the name has to be assigned from a difference of two clock readings: grepping for '< 0.05' would match a float tolerance, and grepping for 'elapsed' would match anything. A new test asserting a 20ms bound fails that guard instead of buying a flake, which is verified by adding one. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Run the Backend CI matrix in parallel, like its sibling job already does The matrix leg is the longest job in the repo at 23.3 minutes, and it was the only large pytest run still serial. repo-cpu-tests has run -n 4 since it measured 806s -> 220s. Measured over the same tree in the same environment, serial against -n 4: 1322.6s -> 343.0s, and the results are identical -- 51 failed / 26190 passed serial, 51 failed / 26188 passed parallel, with the two failure sets compared name for name and equal. So nothing in the backend suite depends on the order it runs in. (Those 51 are one local environment missing peft and a diffusers pin. The point is that the two modes agree.) This is CPU bound and not memory bound: the suite loads no model, unlike the inference smoke workflows where four workers on one runner would not fit. The existing isolation guard caught this change, correctly: it assumed exactly one parallel pytest run, and the two now run over different trees. The repo-root job runs tests/ from the checkout, the matrix job runs the backend's own suite from studio/backend, so tests/studio/load_freeze is not a path that exists for it and demanding those ignores would be nonsense. The guard now tells the two apart and applies the isolation rules to the repo-root run only, and a new test pins the matrix leg as parallel so losing the flag shows up as a failure rather than as CI slowly getting slower again. * Keep the relative-timing tests off the parallel workers Staging caught what the local comparison could not: the 3.10 leg reported 'early markup cost 1.354s against the reference's 0.854s' and 'incremental cost grew 7.0x vs the reference's 11.5x', while 3.13 passed the same commit in 9 minutes against the 23 it used to take. test_streaming_stripper times itself against a reference implementation measured in the same process. Under four workers on four vCPUs one side of that ratio gets descheduled and the other does not, so the comparison stops being between two implementations. It is the same reason repo-cpu-tests already keeps load_freeze out of its parallel run, and it does not reproduce on a machine with cores to spare, which is why the local run agreed serially and in parallel. So it is ignored from the parallel run and runs again in its own serial step, and the isolation guard now holds that pair together for the backend run the way it already does for the repo-root one: putting the file back in the parallel run fails one test, deleting the serial step fails another. * Find the tight elapsed-time bounds by scanning, not by remembering Two more files assert ABSOLUTE elapsed time, and tightly: 50ms for a short-circuit that should not run the probe at all, and 100ms for a regex backtracking guard. Bounds that small sit inside one scheduler quantum, so under four workers on four vCPUs they measure the scheduler as much as the code. Both passed on staging, which is the problem: they would have flaked later, on somebody else's change. Twenty-two backend files assert some elapsed bound, and serialising all of them would give back most of what -n 4 buys. So the line is drawn at 0.1s, where the measurement stops being about the code, and the three files at or below it are ignored from the parallel run and rerun serially, which costs 2.2s and 1.7s. The guard now finds them by scanning rather than by listing. It reads with ast, so the name has to be assigned from a difference of two clock readings: grepping for '< 0.05' would match a float tolerance, and grepping for 'elapsed' would match anything. A new test asserting a 20ms bound fails that guard instead of buying a flake, which is verified by adding one. * Teach the timing scan the two shapes it was blind to The scan only recognised a comparison whose left operand was a name assigned from a clock difference. Two forms in this suite are written differently and were silently skipped: time.monotonic() - started < 0.2 the difference inline (test_stt_download_followups) _elapsed(big) < 8 * _elapsed(small) a helper returning one (test_diffusion_checkpoint_resume) The second is not a near miss. It compares two wall-clock measurements taken in sequence, so descheduling one side and not the other breaks it at ANY magnitude, with no threshold to be under, which is the same reason test_streaming_stripper came out of the parallel run. It was still running under -n 4. So the scan now asks whether an expression IS a duration, however it was spelled: a name assigned from a difference, a difference written inline, or a call to a function that returns one, found by walking for a return of a clock difference at any nesting depth -- the helper in question is defined inside its own test. And a relative comparison is fragile regardless of magnitude, while an absolute one still has to be at or below the threshold. test_diffusion_checkpoint_resume joins the serial step, costing 8.9s. Adding either shape to a file that is not isolated fails the guard, both verified. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow the clock value through a container, and record what is benign The scan tracked names assigned a clock DIFFERENCE. test_tool_output_streaming compares first_seen_at[0] - started against finished - started - 0.5, where every term is an instant, no single name ever holds a duration, and one of them is parked in a list by a callback. Nothing in it looked timed, so the file kept running under -n 4 while asserting that a callback fired at least 0.5s before the child exited -- which collapses if the worker is descheduled while the child sleeps. So instants count now, not just differences, including one appended to a container, and the check walks the expression rather than reading its top node. That widened net found four more files, and only two are real: test_web_fetch_extraction compares parse time at two input sizes, and test_tool_output_streaming is the above. The other three are not performance claims at all. A sandwich, "before <= recorded <= after", cannot be falsified by widening the gap; a poll deadline inside a wait-for-condition loop is the pattern that replaces a guessed sleep; and "stamp < 0.0" compares against a sentinel. Those are in BENIGN_TIMING with their reasons, keyed on the enclosing function so an edit above them does not move the exemption onto something else. Keeping the net wide means a new benign pattern lands here too, so the failure message now says which of the three ways out applies rather than assuming the test is wrong. Verified: a stored-instant comparison added to an unisolated file fails the guard, and the two new files cost 39.5s and 13.3s in the serial step. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow a duration back through the name a helper returned it in The helper check required the return expression itself to read a clock. test_tool_call_parser_strict has def best_ms(depth): best = float("inf") for _ in range(5): t0 = time.perf_counter() ... best = min(best, time.perf_counter() - t0) return best t200 = best_ms(200) t400 = best_ms(400) assert t400 < t200 * 3.0 where the return reads no clock, and neither does the assignment that holds the result. Two links were missing, not one: a function counts as a timing helper if it returns any of its OWN timed names, and a name counts as timed if it was assigned from a call to such a helper. Helpers are resolved first for that reason, and to a fixpoint, so a helper built on another helper is found on the next pass rather than missed. Worth noting as a check on the scan rather than on this test: test_streaming_stripper is now found by the scan on its own, having been in the isolation list by hand since the run that started this. The rule and the list agree where before only the list knew. test_tool_call_parser_strict joins the serial step at 2.0s. A helper returning a duration through a local name, added to a file that is not isolated, fails the guard. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Isolate the keepalive test staging caught, and say what the scan cannot see test_tunnel_safe_long_post::test_the_route_still_pads_the_same_slow_load failed on a staging 3.13 leg that had been green. It patches the keepalive threshold to 0.05s and makes the work sleep 0.2s, then asserts the response starts with padding. Whether it passes turns on which of two timers fired first, and four times the threshold was not enough margin under four workers on four vCPUs. The scan did not find it and structurally cannot: it looks for assertions COMPARING clock-derived values, and this one has no clock in it at all. The dependency is implicit, between a patched constant and a sleep, and the assertion is on the result. Ten backend files pair a sub-second sleep with a small threshold. Since 4x margin already proved insufficient, the ratio is not a usable rule, and flagging all ten would serialise a large part of the suite on a guess. So that class stays with staging, which is the only thing that has ever caught one, and the limit is written next to BACKEND_ISOLATED rather than left for the next person to discover the same way. The file joins the serial step at 3.9s. * Isolate the heartbeat-count test, and correct the note about how this class is found test_scan_loras_off_event_loop counts how many times a heartbeat coroutine ticked during a 0.3s sleep and requires at least three. Descheduling the worker costs ticks without the scan being wrong, so it fails for a reason that has nothing to do with the code under test. Same class as the keepalive test in the previous commit, from the other direction: there the assertion was on a result, here it is on a COUNT, and either way there is no duration in the expression for the scan to find. The previous commit said this class was left to staging, "the only thing that has ever detected one". That was true when written and is not now: this one came from review, on a file staging had not yet failed on. Found by reading, not by scanning, is the accurate statement, and the note says that instead. The file joins the serial step at 1.6s. * Isolate the anthropic keepalive counts from the xdist workers Codex found test_scan_loras_off_event_loop by reading, not by running: it counts heartbeats across a 0.3s sleep, and the assertion compares a count rather than anything clock-derived, so the AST scan in this file is structurally unable to see it. That is a class, not a one-off, so I read the rest of the suite for the same shape instead of waiting for staging to hit it. One more: test_anthropic_messages asserts len(keepalives) >= 3 and >= 2 across a _time.sleep(0.24) stall past several shortened keepalive windows. A descheduled worker loses keepalives exactly as the heartbeat test loses ticks. It costs 2.2s to run serially. One false positive worth naming, because the grep that finds these is crude: test_diffusion_backend asserts len(staged) > 1 near a 0.2s sleep, but staged is a list comprehension over cached filenames with no timing in it. It also costs 152s, so matching on the pattern alone would have been expensive as well as wrong. * Read every link of a chained comparison, not just the first A tight bound is often written as a sandwich, and a sandwich is one Compare node whose leftmost operand is the literal floor. Requiring that operand to be timed made the scan skip the upper link entirely, so a file could hold an assertion of the shape it exists to find and stay in the -n 4 run with the guard green. test_llama_cpp_wait_for_vram_settle already writes a bound that way. Verified by running the scan over a file with 0.3 <= elapsed < 0.05, which it now reports and previously did not. The walk also reads Gt and GtE by swapping the operands, since a bound written backwards bounds the same thing. That turned up one live case, an access token asserted to expire after the wall clock. The margin is 600 seconds, so reading both sides late by whole seconds still leaves it true, and it goes in BENIGN_TIMING rather than into the serial step. * Stop the test stubs shadowing httpx once the suite runs in small pieces The 3.10 leg failed collection on two of the ten files in the new serial step, on module 'httpx' has no attribute 'Response', and it is worth being precise about what that is: httpx is installed on that job. Thirteen backend modules build a fake one and install it with sys.modules.setdefault, which reads as deferring to the real library and does not. sys.modules holds what has been IMPORTED, not what is installed, so in a process where nothing has touched httpx yet the stub wins and shadows it for the rest of the session. The stubs have no Response, starlette.testclient reads httpx.Response at import, and everything collected afterwards that reaches fastapi.testclient or routes.inference dies. In a 26,000-test run something always imports httpx before any of them is collected, which is why this has been invisible for as long as the suite ran as one process. Splitting the timing tests out removed the accident rather than introduced the bug, and any future split would have found the same landmine. All thirteen now try the real import first, the form test_llama_cpp_placement.py already uses. Reproduced before the change by collecting wait_for_vram_settle and diffusion_checkpoint_resume together, which errored, and after, which collects 154 tests. The full suite still collects 26391. The guard is scoped to the isolated files. Around fifty other modules stub structlog the same way and are load-bearing in a run that also imports the real one, so rewriting those is a separate change with its own risk. What has to hold here is that anything moved OUT of the parallel run stands on its own. * Propagate helpers through assigned results, and isolate one more tick count Two from review, both real. The fixpoint over timing helpers called _timed_names without the helpers it had already found, so a wrapper that assigns value = base() and returns value never learned that value was timed. base was discovered, the wrapper was not, and any relative benchmark built on the wrapper stayed invisible. The pass that learns a helper is not the pass that reads its callers, which is the whole reason this runs to a fixpoint, so the set has to go in as well as come out. Verified on a base/wrapper pair the scan now reports and did not before. test_profile_stats counts event-loop ticks during a 0.5s blocking call and needs more than ten of the roughly fifty that fit. That is the same shape as the two tick counts already isolated: no clock appears in the assertion, so the scan cannot see it, and a descheduled worker loses ticks. 12.8s serially. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
457 lines
16 KiB
Python
457 lines
16 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""Extended test matrix for ``LlamaCppBackend.load_progress()``.
|
|
|
|
Companion to ``test_llama_cpp_load_progress.py`` (basic contract). Covers
|
|
cross-platform edge cases: platform matrix (/proc absence), VmRSS parsing,
|
|
filesystem edges (HF-cache symlinks, broken/missing/relative paths), shard
|
|
aggregation, lifecycle races, concurrent sampling, and fraction bounds.
|
|
|
|
Linux-only in practice (``/proc`` stubbed where needed).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import io
|
|
import os
|
|
import sys
|
|
import threading
|
|
import types as _types
|
|
from pathlib import Path
|
|
from unittest.mock import patch
|
|
|
|
import pytest
|
|
|
|
# Stub heavy/unavailable deps before importing the module under test.
|
|
|
|
_BACKEND_DIR = str(Path(__file__).resolve().parent.parent)
|
|
if _BACKEND_DIR not in sys.path:
|
|
sys.path.insert(0, _BACKEND_DIR)
|
|
|
|
_loggers_stub = _types.ModuleType("loggers")
|
|
_loggers_stub.get_logger = lambda name: __import__("logging").getLogger(name)
|
|
sys.modules.setdefault("loggers", _loggers_stub)
|
|
|
|
_structlog_stub = _types.ModuleType("structlog")
|
|
sys.modules.setdefault("structlog", _structlog_stub)
|
|
|
|
_httpx_stub = _types.ModuleType("httpx")
|
|
for _exc_name in (
|
|
"ConnectError",
|
|
"TimeoutException",
|
|
"ReadTimeout",
|
|
"ReadError",
|
|
"RemoteProtocolError",
|
|
"CloseError",
|
|
):
|
|
setattr(_httpx_stub, _exc_name, type(_exc_name, (Exception,), {}))
|
|
|
|
|
|
class _FakeTimeout:
|
|
def __init__(self, *a, **kw):
|
|
pass
|
|
|
|
|
|
_httpx_stub.Timeout = _FakeTimeout
|
|
_httpx_stub.Client = type(
|
|
"Client",
|
|
(),
|
|
{
|
|
"__init__": lambda self, **kw: None,
|
|
"__enter__": lambda self: self,
|
|
"__exit__": lambda self, *a: None,
|
|
},
|
|
)
|
|
# Only when the real library is absent. sys.modules holds what has been IMPORTED, not
|
|
# what is installed, so setdefault does not defer to a real httpx that nothing in this
|
|
# process has touched yet: the stub wins and shadows it for the whole session. This stub
|
|
# has no Response, and starlette.testclient reads httpx.Response at import, so every
|
|
# module collected afterwards that reaches fastapi.testclient or routes.inference dies.
|
|
try:
|
|
import httpx # noqa: F401
|
|
except ImportError:
|
|
sys.modules.setdefault("httpx", _httpx_stub)
|
|
|
|
from core.inference.llama_cpp import LlamaCppBackend
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Helpers
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _make():
|
|
inst = LlamaCppBackend.__new__(LlamaCppBackend)
|
|
inst._process = None
|
|
inst._gguf_path = None
|
|
inst._healthy = False
|
|
return inst
|
|
|
|
|
|
class _Proc:
|
|
def __init__(self, pid):
|
|
self.pid = pid
|
|
|
|
|
|
def _sparse(path, size):
|
|
with open(path, "wb") as f:
|
|
if size > 0:
|
|
f.truncate(size)
|
|
|
|
|
|
def _fake_proc_reader(rss_kb):
|
|
"""An ``open()`` replacement faking /proc reads with a VmRSS line."""
|
|
|
|
def fake_open(path, *args, **kwargs):
|
|
if str(path).startswith("/proc/"):
|
|
return io.StringIO(f"VmRSS:\t{rss_kb}\tkB\n")
|
|
return open(path, *args, **kwargs)
|
|
|
|
return fake_open
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# A. Platform matrix
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestPlatformMatrix:
|
|
"""Linux-first via /proc. On macOS/Windows must degrade to None
|
|
rather than crash."""
|
|
|
|
def test_linux_live_proc_is_self_pid(self, tmp_path):
|
|
"""Self-pid /proc read uses the real kernel interface."""
|
|
gguf = tmp_path / "m.gguf"
|
|
_sparse(gguf, 1 * 1024**3)
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = str(gguf)
|
|
inst._healthy = False
|
|
out = inst.load_progress()
|
|
assert out is not None
|
|
assert out["phase"] == "mmap"
|
|
assert out["bytes_total"] == 1 * 1024**3
|
|
# Our process has some RSS -- sanity-check it's positive.
|
|
assert out["bytes_loaded"] > 0
|
|
|
|
def test_macos_no_proc_returns_none(self, tmp_path):
|
|
"""Simulate macOS: /proc open fails with FileNotFoundError."""
|
|
gguf = tmp_path / "m.gguf"
|
|
_sparse(gguf, 1 * 1024**3)
|
|
inst = _make()
|
|
inst._process = _Proc(pid = 12345)
|
|
inst._gguf_path = str(gguf)
|
|
|
|
def fake_open(path, *args, **kwargs):
|
|
if str(path).startswith("/proc/"):
|
|
raise FileNotFoundError(f"No such file: {path}")
|
|
return open(path, *args, **kwargs)
|
|
|
|
with patch("builtins.open", side_effect = fake_open):
|
|
out = inst.load_progress()
|
|
assert out is None
|
|
|
|
def test_windows_no_proc_returns_none(self, tmp_path):
|
|
"""Simulate Windows: opening /proc raises PermissionError or OSError."""
|
|
gguf = tmp_path / "m.gguf"
|
|
_sparse(gguf, 1 * 1024**3)
|
|
inst = _make()
|
|
inst._process = _Proc(pid = 4567)
|
|
inst._gguf_path = str(gguf)
|
|
|
|
def fake_open(path, *args, **kwargs):
|
|
if str(path).startswith("/proc/"):
|
|
raise PermissionError("access denied")
|
|
return open(path, *args, **kwargs)
|
|
|
|
with patch("builtins.open", side_effect = fake_open):
|
|
out = inst.load_progress()
|
|
assert out is None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# B. VmRSS parsing edge cases
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestVmRSSParsing:
|
|
def test_standard_tab_delimited(self, tmp_path):
|
|
gguf = tmp_path / "m.gguf"
|
|
_sparse(gguf, 4 * 1024**3)
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = str(gguf)
|
|
with patch("builtins.open", side_effect = _fake_proc_reader(2 * 1024**2)):
|
|
out = inst.load_progress()
|
|
assert out["bytes_loaded"] == 2 * 1024**3
|
|
|
|
def test_space_separated_fallback(self, tmp_path):
|
|
"""Some kernels emit a single space, not a tab."""
|
|
gguf = tmp_path / "m.gguf"
|
|
_sparse(gguf, 4 * 1024**3)
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = str(gguf)
|
|
|
|
def fake_open(path, *a, **kw):
|
|
if str(path).startswith("/proc/"):
|
|
return io.StringIO("VmRSS: 4194304 kB\n")
|
|
return open(path, *a, **kw)
|
|
|
|
with patch("builtins.open", side_effect = fake_open):
|
|
out = inst.load_progress()
|
|
assert out["bytes_loaded"] == 4 * 1024**3
|
|
|
|
def test_missing_vmrss_line(self, tmp_path):
|
|
"""Kernel with VmRSS stripped (zombie / kthread) -> 0."""
|
|
gguf = tmp_path / "m.gguf"
|
|
_sparse(gguf, 1 * 1024**3)
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = str(gguf)
|
|
|
|
def fake_open(path, *a, **kw):
|
|
if str(path).startswith("/proc/"):
|
|
return io.StringIO("Name:\ttest\nState:\tZ (zombie)\n")
|
|
return open(path, *a, **kw)
|
|
|
|
with patch("builtins.open", side_effect = fake_open):
|
|
out = inst.load_progress()
|
|
assert out is not None
|
|
assert out["bytes_loaded"] == 0
|
|
assert out["fraction"] == 0.0
|
|
|
|
def test_malformed_vmrss_value(self, tmp_path):
|
|
"""Non-integer VmRSS is treated like an absent line (ValueError
|
|
caught)."""
|
|
gguf = tmp_path / "m.gguf"
|
|
_sparse(gguf, 1 * 1024**3)
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = str(gguf)
|
|
|
|
def fake_open(path, *a, **kw):
|
|
if str(path).startswith("/proc/"):
|
|
return io.StringIO("VmRSS:\tXXXX\tkB\n")
|
|
return open(path, *a, **kw)
|
|
|
|
with patch("builtins.open", side_effect = fake_open):
|
|
out = inst.load_progress()
|
|
# int() ValueError is caught and returns None.
|
|
assert out is None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# C. Filesystem edge cases
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestFilesystemEdges:
|
|
def test_symlink_primary_follows_to_blob(self, tmp_path):
|
|
"""HF cache stores blobs under blobs/ and symlinks them from
|
|
snapshots/. Must follow the symlink."""
|
|
blob = tmp_path / "blob"
|
|
_sparse(blob, 12 * 1024**3)
|
|
snap = tmp_path / "snap"
|
|
snap.mkdir()
|
|
link = snap / "m.gguf"
|
|
link.symlink_to(blob)
|
|
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = str(link)
|
|
with patch("builtins.open", side_effect = _fake_proc_reader(6 * 1024**2)):
|
|
out = inst.load_progress()
|
|
assert out["bytes_total"] == 12 * 1024**3
|
|
|
|
def test_broken_symlink_skipped(self, tmp_path):
|
|
snap = tmp_path / "snap"
|
|
snap.mkdir()
|
|
link = snap / "m.gguf"
|
|
link.symlink_to(tmp_path / "missing-blob")
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = str(link)
|
|
with patch("builtins.open", side_effect = _fake_proc_reader(1024)):
|
|
out = inst.load_progress()
|
|
assert out["bytes_total"] == 0
|
|
assert out["bytes_loaded"] == 1024 * 1024
|
|
|
|
def test_nonexistent_path_skipped(self, tmp_path):
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = str(tmp_path / "ghost.gguf")
|
|
with patch("builtins.open", side_effect = _fake_proc_reader(1024)):
|
|
out = inst.load_progress()
|
|
assert out["bytes_total"] == 0
|
|
|
|
def test_relative_gguf_path(self, tmp_path):
|
|
"""Relative paths shouldn't crash; behaviour depends on CWD but
|
|
must not raise."""
|
|
cwd = os.getcwd()
|
|
try:
|
|
os.chdir(tmp_path)
|
|
_sparse(Path("rel.gguf"), 8 * 1024**3)
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = "rel.gguf"
|
|
with patch("builtins.open", side_effect = _fake_proc_reader(0)):
|
|
out = inst.load_progress()
|
|
assert out is not None
|
|
assert out["bytes_total"] == 8 * 1024**3
|
|
finally:
|
|
os.chdir(cwd)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# D. Shard aggregation
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestShardAggregation:
|
|
def test_partial_multi_shard_download(self, tmp_path):
|
|
"""Primary present but shards 2..N still ``.incomplete``. Sums
|
|
only the fully-arrived ``.gguf`` files."""
|
|
_sparse(tmp_path / "m-00001-of-00004.gguf", 30 * 1024**3)
|
|
_sparse(tmp_path / "m-00002-of-00004.gguf", 30 * 1024**3)
|
|
# 3 and 4 still downloading as .incomplete.
|
|
_sparse(tmp_path / "m-00003-of-00004.gguf.incomplete", 5 * 1024**3)
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = str(tmp_path / "m-00001-of-00004.gguf")
|
|
with patch("builtins.open", side_effect = _fake_proc_reader(0)):
|
|
out = inst.load_progress()
|
|
assert out["bytes_total"] == 60 * 1024**3 # only the .gguf siblings
|
|
|
|
def test_two_shard_series_in_same_dir(self, tmp_path):
|
|
"""Defensive: when two quant series share a dir, the prefix
|
|
filter sums only siblings of the chosen primary."""
|
|
for i in range(1, 3):
|
|
_sparse(tmp_path / f"m_q4-{i:05d}-of-00002.gguf", 10 * 1024**3)
|
|
_sparse(tmp_path / f"m_q8-{i:05d}-of-00002.gguf", 20 * 1024**3)
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = str(tmp_path / "m_q8-00001-of-00002.gguf")
|
|
with patch("builtins.open", side_effect = _fake_proc_reader(0)):
|
|
out = inst.load_progress()
|
|
assert out["bytes_total"] == 40 * 1024**3 # just q8 series
|
|
|
|
def test_mmproj_sibling_not_counted(self, tmp_path):
|
|
"""Vision models drop an ``mmproj-*.gguf`` alongside. For a
|
|
single-file (non-sharded) primary, count only the primary."""
|
|
_sparse(tmp_path / "m.gguf", 8 * 1024**3)
|
|
_sparse(tmp_path / "mmproj-BF16.gguf", 2 * 1024**3)
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = str(tmp_path / "m.gguf")
|
|
with patch("builtins.open", side_effect = _fake_proc_reader(0)):
|
|
out = inst.load_progress()
|
|
# Non-sharded: only the primary is counted.
|
|
assert out["bytes_total"] == 8 * 1024**3
|
|
|
|
def test_single_file_model(self, tmp_path):
|
|
"""Non-sharded model: primary only."""
|
|
_sparse(tmp_path / "small.gguf", 4 * 1024**3)
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = str(tmp_path / "small.gguf")
|
|
with patch("builtins.open", side_effect = _fake_proc_reader(2 * 1024**2)):
|
|
out = inst.load_progress()
|
|
assert out["bytes_total"] == 4 * 1024**3
|
|
assert out["bytes_loaded"] == 2 * 1024**3
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# E. Lifecycle races
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestLifecycleRaces:
|
|
def test_process_set_but_gguf_path_not_yet(self, tmp_path):
|
|
"""Window between Popen and self._gguf_path=model_path."""
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = None
|
|
with patch("builtins.open", side_effect = _fake_proc_reader(1024)):
|
|
out = inst.load_progress()
|
|
assert out is not None
|
|
assert out["phase"] == "mmap"
|
|
assert out["bytes_total"] == 0
|
|
assert out["bytes_loaded"] == 1024 * 1024
|
|
|
|
def test_process_died_mid_sample(self, tmp_path):
|
|
"""/proc/<pid> disappears -> None."""
|
|
_sparse(tmp_path / "m.gguf", 1 * 1024**3)
|
|
inst = _make()
|
|
inst._process = _Proc(pid = 999_999_999)
|
|
inst._gguf_path = str(tmp_path / "m.gguf")
|
|
assert inst.load_progress() is None
|
|
|
|
def test_healthy_true_ready_phase(self, tmp_path):
|
|
_sparse(tmp_path / "m.gguf", 1 * 1024**3)
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = str(tmp_path / "m.gguf")
|
|
inst._healthy = True
|
|
with patch("builtins.open", side_effect = _fake_proc_reader(1024)):
|
|
out = inst.load_progress()
|
|
assert out["phase"] == "ready"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# F. Concurrent sampling (simulates multiple browser tabs polling)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestConcurrentSampling:
|
|
def test_parallel_invocations_never_raise(self, tmp_path):
|
|
"""Many concurrent samplers on one backend must not raise.
|
|
|
|
No ``builtins.open`` patch: ``mock.patch`` isn't thread-safe and could
|
|
leak a Mock into ``open``. Each thread hits the real ``/proc/self/status``.
|
|
"""
|
|
_sparse(tmp_path / "m.gguf", 1 * 1024**3)
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = str(tmp_path / "m.gguf")
|
|
errors = []
|
|
|
|
def run():
|
|
try:
|
|
for _ in range(50):
|
|
inst.load_progress()
|
|
except Exception as e: # pragma: no cover
|
|
errors.append(e)
|
|
|
|
threads = [threading.Thread(target = run) for _ in range(10)]
|
|
for t in threads:
|
|
t.start()
|
|
for t in threads:
|
|
t.join()
|
|
assert not errors, errors
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# G. Fraction bounds
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestFractionBounds:
|
|
def test_fraction_capped_at_one(self, tmp_path):
|
|
_sparse(tmp_path / "m.gguf", 1 * 1024**3)
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = str(tmp_path / "m.gguf")
|
|
# RSS > total (post-paged-in + extra structures)
|
|
with patch("builtins.open", side_effect = _fake_proc_reader(2 * 1024**2)):
|
|
out = inst.load_progress()
|
|
assert 0.0 <= out["fraction"] <= 1.0
|
|
|
|
def test_fraction_zero_when_total_zero(self):
|
|
inst = _make()
|
|
inst._process = _Proc(os.getpid())
|
|
inst._gguf_path = None
|
|
with patch("builtins.open", side_effect = _fake_proc_reader(1024**2)):
|
|
out = inst.load_progress()
|
|
assert out["fraction"] == 0.0
|