mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-24 00:04:14 +00:00
54b6ca4c3f
* Stop spending the Actions cache budget on caches nothing reads The repo holds 50.0 GB of Actions cache against a 10 GB limit, so LRU eviction runs continuously and the entries CI actually depends on are the ones being thrown away. Measured: 28.9 GB 60 entries setup-python pip 58% of the budget, only 18 distinct keys 7.2 GB 8 entries v0-rust 11.8 GB 5 entries the GGUF / HF model caches CI depends on 25.9 GB 214 entries on PR refs, restorable only by re-runs of that same PR Two doors let that happen, and both are silent: nothing goes red when a cache is evicted, CI just re-downloads a 4.6 GB model and everyone assumes that is the cost. setup-python derives one pip key per interpreter from dependency files across the whole repo, so dozens of unrelated jobs share it and race to save under it. The entries measure 666-715 MB. Jobs that only pip-install huggingface_hub, pytest or playwright were paying that for the 0-7s their restore step takes. Those 24 call sites give the cache back; the 9 that install torch/transformers keep it. The Playwright browser caches in studio-ui-smoke.yml used the read-write actions/cache, which saves from its post-step on every ref, at ~470 MB per browser set. They now restore always and save on main only, matching what every model cache in this repo already does. This is the same thrash loop the GGUF caches were fixed for, arriving through a different door: PR misses, downloads, saves its own copy, evicts main's, next PR misses. Windows benefits most. Its model caches are among the evicted, and a Windows cache miss costs about 3x a Linux one. tests/studio/test_cache_budget_discipline.py asserts both doors stay shut, that cold-install lanes are never warmed by a cache, and that every setup-python step still pins an interpreter -- the last because removing cache: 'pip' from an inline-flow mapping by deleting the line takes python-version with it, which this change did once before it was caught. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Correct the cache budget figure: 50 GiB, not GitHub's 10GB default The budget comments across eleven workflows quoted GitHub's 10GB default. This repo's quota is 50 GiB, so the earlier framing (and the 33.3GB / 3.3x-over measurement they carry) understated the headroom and overstated the severity. Re-measured with the correct quota: 49.63 GiB across 258 entries, 99.3% full. The conclusion does not change, only the reason. Eviction runs at the margin, and what fills the budget is redundancy rather than useful payload: 20.74 GiB the SAME key held on several refs (42% of the whole cache) 6.67 GiB 13 copies setup-python ... python-3.13.15-pip-85e247d7... 6.50 GiB 11 copies setup-python ... python-3.11.15-pip-85e247d7... 3.83 GiB 10 copies setup-python ... python-3.12.13-pip-85e247d7... 0.91 GiB 3 copies ms-playwright-Linux-1.62.0-cfw-v1 10.70 GiB 84 entries written and never read again, 10.60 GiB of it setup-python 24.01 GiB 198 entries on PR refs, restorable only by re-runs of that same PR 0.00 GiB entries unread for 7+ days: nothing is idle, the cache is churning Every duplicated key already has a copy on main, which every PR can restore from, so the PR-scoped copies buy no hit rate and evict the copy that does. Worth recording for whoever adds the next cache: the duplication is a SYMPTOM of sitting at quota, not an independent cause. A PR only writes its own copy after missing, and it only misses because main's copy was evicted. Getting back under the quota stops the loop on its own, which is why this change reclaims claimants rather than restructuring the nine jobs that legitimately cache torch. * Scope every pip cache key to what its job actually installs Five of the nine jobs that keep a pip cache had no cache-dependency-path, so setup-python hashed dependency files across the whole repo to build their key. That is the second multiplier behind the 27.05 GiB those caches occupy: 16 distinct keys appeared in a week, because one requirements edit anywhere invalidates every interpreter's entry at once and orphans the old ones at ~700MB each. The four already-scoped jobs (consolidated-tests, studio-backend) show the pattern; it just was not applied consistently. mlx-ci installs from studio/backend/requirements/studio.txt, so it now points there like its siblings. The other four pin their dependencies inline in the workflow (a torch CPU index URL, pinned transformers/trl/peft), so there is no requirements file to name and their key was describing files they never read. For those the workflow file IS the dependency spec, so the key hashes that: it moves when the install actually changes and not otherwise. notebooks-ci's step had to be expanded from the inline-flow form (with: { python-version: '3.12', cache: 'pip' }) to take the new key. The guard test now asserts every allowed pip cache scopes its key, since an unscoped one costs budget quietly rather than failing. * Save the Playwright browsers only when the download succeeded The save ran under a bare always(), so a browser install that failed part-way still stored whatever had landed on disk. The key is pinned to the resolved Playwright version and never rolls over, so every later run would restore that partial tree, report a cache hit, skip the download and run install-deps against engines that are not there. Every UI job fails until someone deletes the entry by hand, and nothing in the log points at the cache. always() stays, so browsers that did download are not thrown away because an unrelated earlier step failed; the install step now carries an id and the save checks its outcome. Both jobs in the workflow had the same condition. * Point the scoped cache keys at the nested checkout, and run the guard on workflow-only PRs Three jobs check the repo out under `unsloth/` because they need a second repo beside it: notebooks-ci api-introspect, and version-compat-ci zoo-imports-under-spoof and grpo-fake-run. The cache-dependency-path added for them was workspace-root-relative, so it matched no file. setup-python treats that as fatal ("No file in ... matched to ..."), not as a reason to skip the cache, so all three jobs would have stopped before installing anything. Prefixed with the checkout directory, and guarded: the new test resolves every cache-dependency-path against its own job's checkout paths. The discipline test is also wired into workflow-trigger-lint, which carries no paths filter. Repo tests (CPU) does collect the file, but studio-backend-ci.yml's paths do not match a workflow-only PR, so the job never starts for exactly the change this guard exists to reject. Collection is not coverage if nothing triggers the workflow that collects it. * Count setup-python's implicit save, and drop a cache path with no cache The save scan only looked for actions/cache steps, so it reported clean while nine jobs wrote a PR-scoped entry on every run. setup-python's cache: input registers a post-step (post: dist/cache-save/index.js in its own action.yml) that saves after the job on whatever ref it ran on, with no condition to gate it. Those nine are now named in PIP_CACHE_JOBS_PENDING_CONVERSION rather than skipped silently, so a tenth joining them fails the check, and the follow-up that converts them to an explicit restore plus main-only save empties the set. lint-ci.yml also kept a cache-dependency-path after its cache: 'pip' was removed. That is inert, since setup-python only reads the path when caching is on, but it reads as a scoped cache key and the next person believes the job is cached. Removed, with the comment rewritten to say why the job has no cache, and a check for the same shape elsewhere. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Read the save condition and the cache path, rather than searching them Two guards in this file passed on text that looked right rather than on the property they claim. The main-only check was a substring search for "refs/heads/main". Three conditions contain that literal while still saving on every PR: a `!=` comparison, an `||` that admits another event, and the check appearing in only one branch of one. It now splits the expression on its top-level `||` and requires a positive `github.ref == 'refs/heads/main'` in every alternative, which is conservative in the safe direction. The splitter is itself tested against nine expressions, including a `||` inside a string. The cache-dependency-path check compared prefixes and skipped root-checkout jobs entirely, so `unsloth/.github/workflows/typo.yml` read as correct and a misspelling in a root-checkout job was never examined. Each entry is now resolved against the checkout it belongs to and globbed against the tree. Paths under a checkout of a DIFFERENT repository are skipped rather than reported: notebooks-ci api-introspect checks out unslothai/notebooks beside this one, and a path under it cannot be resolved here. Both matter because setup-python does not skip a cache it cannot resolve, it fails the job with "No file in ... matched" before anything is installed. All four shapes verified red: the inverted comparison, the widened `||`, a correctly prefixed but misspelled nested path, and a misspelled root path. * Save the pip cache on main only, via a restore/save action pair (#9165) * Save the pip cache on main only, via a restore/save action pair setup-python's built-in cache: 'pip' is the read-write form. It restores in the step and saves from its own post-step on whatever ref the job ran on, and exposes no condition to stop that. An entry written on a pull_request ref can only be restored by re-runs of that same pull request, so it buys no hit rate while competing for the shared 50 GiB budget against the copy on main that every PR can read. Measured on this repo before the change: setup-python entries were 19.45 GiB across 40 entries, 15.49 GiB of it on PR refs, with four interpreter keys duplicated four to six times each for 9.60 GiB of pure waste. None of it shows up as a failure. Over quota, GitHub evicts least-recently-used, so main's copy goes, the next PR misses, downloads and writes its own, and CI simply gets slower. The repo had already diagnosed and fixed this same loop for the GGUF caches and the Playwright browsers; setup-python was left doing it because the built-in cache has no save-gating knob. Splitting restore from save is how you get one. The nine jobs that genuinely install a torch/transformers-class dependency set now use .github/actions/pip-cache-restore and .github/actions/pip-cache-save, with the save gated on refs/heads/main. Defined once as a composite pair rather than inlined nine times, following install-unsloth-local: the gate is the whole point of the change and nine copies of it would drift. The action resolves pip's cache directory with pip cache dir rather than hardcoding a path, so the macOS and Windows call sites work unchanged, and it fails loudly when key-files matches nothing, since an empty hash would silently collapse every job onto one key. The guards move with the architecture. The save scan now reads composite actions too, which is where the save now lives; a job using the built-in cache at all is now a failure rather than an allowlist question; and each of the nine is checked to restore and save as a wired pair. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Point the local action references at the nested checkout `uses: ./...` resolves from GITHUB_WORKSPACE, not from the workflow file, and three jobs check the repo out under `unsloth/` because they need a second repo beside it. The unprefixed path is a directory that does not exist there, so all six references would have failed with "Can't find 'action.yml', 'action.yaml' or 'Dockerfile'" before installing anything. Same root cause as the cache key paths, one level out: those were fixed for these jobs and the action paths were not. The guard now resolves every local action reference against its own job's checkout paths. * Discover pip cache users, rather than only iterating the allowlist Replacing setup-python's built-in `cache: 'pip'` removed the mechanism that found claimants. Every remaining check is parametrized over PIP_CACHE_JOBS, so a new job adding the restore/save pair was never visited: it would get a ~700MB entry with no scoping check, no wiring check and no justification, and this file would stay green. The allowlist stopped being enforced and became a list to iterate. Two checks now read the workflows instead. One rejects any job using either half of the pair without being listed. The other re-applies the heavy-install requirement to the discovered set, so a job that keeps its cache after its torch-class install moves elsewhere is reported rather than grandfathered. Verified red by giving workflow-trigger-lint, which installs nothing heavy, a pip-cache-restore step. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| action.yml | ||