2 commits

Author	SHA1	Message	Date
Daniel Han	d458029a32	Do not scan vendored packages that refuse to import on our floor (#8252) ... tests/test_python39_compatibility.py is a hard gate in Core CI and it is currently red on main for every open PR: studio/backend/vendor/truststore/_api.py:30: str | bytes (type alias) studio/backend/vendor/truststore/_api.py:31: str | bytes | ... (type alias) Those unions cannot be reached on 3.9. truststore/__init__.py raises ImportError below 3.10, and its only caller wraps the import in try/except, so the package is gone before _api.py is ever loaded. Skip files under a package whose __init__ declares a floor above ours, keyed on the guard rather than the path so unguarded code added to the same vendor directory is still scanned.	2026-08-09 04:06:19 -07:00
Daniel Han	c45b985390	Keep the package importable on the Python floor pyproject declares (#7840) ... * Keep the package importable on the Python floor pyproject declares requires-python says >=3.9, but importing unsloth.registry.registry on 3.9 fails: File "unsloth/registry/registry.py", line 90, in ModelMeta quant_types: list[QuantType] | dict[str, list[QuantType]] = ... TypeError: unsupported operand type(s) for |: 'types.GenericAlias' and 'types.GenericAlias' It is a dataclass field, so the annotation evaluates at class creation rather than lazily. benchmark_fused_moe.py has the same defect in two function signatures. Nothing caught either: every job in consolidated-tests-ci.yml pins 3.12, so the declared floor is never exercised. Both files get `from __future__ import annotations`. No annotation text changes. Safe because nothing in the package calls get_type_hints or reads __annotations__ back, and dataclasses handles string annotations natively. tests/test_python39_compatibility.py is a static AST check needing no torch, GPU or network. It reads the floor from pyproject.toml rather than hardcoding it, so raising requires-python relaxes it instead of turning it into a false alarm, and it runs as a HARD GATE in the consolidated job. Verified on a real CPython 3.9.25: ModelMeta imports and its 8 fields resolve. Depends on unslothai/unsloth-zoo#982 -- unsloth_zoo is itself un-importable on 3.9 until that lands, so this fix is not observable on its own. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the 3.9 gate precise: no false positives, no false negatives Carries over two fixes found reviewing the same gate in unsloth-zoo#982. Function-local variable annotations are never evaluated (PEP 526), but ast.walk visited nested AnnAssign nodes all the same, so the gate would have demanded a future import for valid 3.9 code. Scope tracking now flags variable annotations only at module and class level, while signature annotations stay flagged wherever their def is, including nested ones. Type aliases were missed entirely. `PathLike = str | Path` evaluates on import and raises below 3.10, and unlike an annotation `from __future__ import annotations` does NOT defer it, so the old "skip files with the future import" shortcut made the blind spot worse. Assigned values at module and class scope are now checked, filtered by a conservative type test that keeps bitwise flag arithmetic (re.DOTALL | re.MULTILINE, os.W_OK | os.X_OK) out. Proven both ways: a function-local annotation and bitwise flags pass, while a module-level annotation, a dataclass field shaped like the registry.py one this PR fixes, and a type alias behind the future import each fail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hold every packaged directory to the declared floor, not just unsloth/ pyproject ships unsloth*, unsloth_cli* and studio.backend* under one requires-python, but the gate only scanned unsloth/, so it could pass while a shipped module stayed unimportable on 3.9. studio/backend/main.py:382 is the concrete case: an unfuturized `dict[str, str] | None` return annotation. The roots are now read from pyproject's `include` list rather than hardcoded, so newly packaged directories cannot silently escape the guarantee. All three are held to the syntax check, which they pass today. studio's evaluated-union debt is ratcheted rather than cleared. 35 files carry it, and they include FastAPI routers and pydantic models. `from __future__ import annotations` is supported there but has real failure modes around class dependencies, where FastAPI resolves annotations as it builds each endpoint, so converting them needs Studio booted and its routes exercised rather than a bulk edit. The ratchet stops the debt growing meanwhile and names the offenders. Verified non-vacuous both ways: a `match` statement dropped into studio fails the syntax check, and one added union file fails the ratchet. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop the gate flagging dict merges, and catch classes nested in functions Both carried over from review of the same gate in unsloth-zoo#982. `not name.isupper()` treated any lowercase `|` as a type alias, so it flagged dict merges (PEP 584, valid on 3.9), set unions and lowercase flag arithmetic. A hard gate rejecting `merged = defaults | extra` would block unrelated PRs for writing ordinary 3.9 code. An operand must now be name-shaped and at least one must anchor to a real type: a builtin, None, or a name the module imported from typing / typing_extensions / collections.abc. A class body nested in a function inherited in_function=True and was skipped, but a class body is class scope wherever it sits and its annotations evaluate. Confirmed on 3.9.25 that the nested case raises. Probed both ways: dict merge, set union and function-local annotations pass, while a type alias and a class nested in a function fail. The studio ratchet count is unchanged at 35 under the more precise logic. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scan evaluated values in every packaged root, and close three gate holes studio/backend/hub/workers/hf_download.py:52 had a shipped `HfTokenArg = str | bool | None`, which is import-fatal on 3.9 and which the future import at the top of that file does not defer. It slipped through because the union check still used the default unsloth/ root while the studio ratchet looks at annotations only. Rewritten as typing.Union, and the evaluated-value scan now covers every packaged root. It was the only such alias in the distribution. Annotation checking stays scoped to unsloth/, since studio's 35 annotation files are the ratcheted debt; values are cheap and were fully closable, so they are. Three more holes, carried over from unsloth-zoo#982: Control flow. evaluated_values walked only direct children and ClassDef, so an alias under a module-level if, try, with or loop was invisible. It now descends compound statements while still stopping at function bodies. Defaults and decorators. `def parse(kind = str | Path)` and `@register(str | Path)` evaluate when the def executes, but neither was collected. Misplaced future imports. ast.parse accepts `from __future__ import annotations` after an executable statement; import does not, and has_future_annotations would then suppress the union check on a module that cannot import at all. test_every_packaged_module_compiles runs compile() over every packaged root. Probed both ways. Newly caught: the studio alias, an alias under an if, a union in a default, a union in a decorator, a misplaced future import. Still clean: a dict merge and a function-local annotation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Daniel Han <unslothshared@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-04 07:41:52 -07:00