vrr/unsloth
2
0
Fork 0
mirror of https://github.com/unslothai/unsloth.git synced 2026-08-24 00:04:14 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 30
unsloth/.github/workflows/studio-api-smoke.yml
Daniel Han fdf83cbd6a
Make apt fail fast, so the retry has something to retry (#9260) 
* Make apt fail fast, so the retry has something to retry

#9256 bounded the stalls. This says why they happened and stops them.

Reading the four logs, every one ends the same way:

  04:47:02  Get:5 https://archive.ubuntu.com/ubuntu noble-security InRelease [126 kB]
  05:16:29  ##[error]The operation was canceled.

Twenty-nine minutes of silence mid-fetch of a 126 kB index file, and two
of the four hung on that same file. What came before it is the other
half: azure.archive.ubuntu.com was Ign'd four times over thirty seconds,
so apt had already failed over through /etc/apt/apt-mirrors.txt to the
public archive, which is not provisioned for this fleet.

apt did not treat any of that as an error. Acquire::http::Timeout
defaults to 120s and is an idle timeout, so a socket that is open and
trickling never trips it. Cutting it to 20s with internal retries turns
a 29 minute hang into a ~1 minute failure, and that matters beyond
speed: the wall-clock kill is what orphans the dpkg lock, so an apt that
gives up on its own is one we never have to kill.

Not pinning a mirror. The evidence does not support it: in one stall
Azure was dead and the public archive hung, in another Azure was serving
fine and the transfer stalled at a 13.6 MB package. Neither is reliably
better, and the mirrorlist failover is already the right mechanism. It
just needed to be allowed to give up.

The written config is validated against apt itself (apt-config dump
round-trips all four options), and the helper's retry, timeout, giving
up, recovery and no-command paths are exercised end to end.

* Wait on all four apt locks, and stop racing a slow mirror

Two bugs, both mine, both caught by CI on this branch.

1. apt-get update takes /var/lib/apt/lists/lock and nothing else. The
   helper waited on /var/lib/dpkg/lock-frontend only, so after an attempt
   was killed mid-update the wait saw a free lock, retried immediately,
   and produced

     E: Could not get lock /var/lib/apt/lists/lock. It is held by process 2420 (apt-get)

   twice in under two seconds. Three attempts, one real one -- exactly the
   'a retry that cannot succeed is worse than none' failure the header
   warns about, in the one case it was written for. It now waits on all
   four: dpkg's two, lists, and archives.

2. 150s per attempt was racing the mirror, not bounding it. The mirror
   was degraded rather than dead, so every attempt got killed at the same
   point and none finished. Two attempts of 360s instead of three of 150:
   the bound exists to stop an infinite hang, not to beat a slow server.
   studio-update-smoke's job budget goes 15 to 25 to fit the new worst
   case; the guard checks that arithmetic.

* Ask the image's package lists first, and refresh only if they miss

The mirror outage is still live. From this branch's own GGUF smoke:

  06:55:33 Ign:4 http://azure.archive.ubuntu.com/ubuntu noble-backports InRelease
  06:55:35 Get:5 https://archive.ubuntu.com/ubuntu noble-security InRelease [126 kB]
  07:01:11 (nothing, for 5m36s, then the attempt's cap)

Azure is unreachable and the public archive it fails over to is not
provisioned for this fleet. Acquire::http::Timeout does not save us there:
it is an idle timeout, and a server dribbling a byte every few seconds
never trips it, so the wall-clock bound is what ends the attempt.

The bound is working as intended -- the step now fails in 14 minutes with
the reason printed, where before it spent 30 and was reported as
"cancelled" with nothing said. But a bounded failure is still a failure,
and the operation that fails is one we mostly do not need.

`apt-get update` refreshes every index for every suite. The runner image
already ships populated lists, and these steps install a handful of
ordinary packages. So try the install first, and refresh only when it
misses:

  apt-get install -y X || { apt-get update && apt-get install -y X; }

In the common case the slowest and most failure-prone apt operation is
never performed at all. When the image's lists really are too stale --
the "Unable to locate package" case the runner-images maintainers warn
about -- the update still runs and the install is retried, so nothing is
traded away.

Not applied to clean-machine-install-ci: a fresh WSL image genuinely has
no lists, so install-first would always miss, and a bare machine is that
leg's whole premise.

The three outcomes were checked as shell rather than reasoned about:
resolvable (update never reached, rc=0), not resolvable (update runs,
rc=0), and both failing (rc=1, so the failure still propagates under the
`bash -e` GitHub runs steps with).
2026-08-19 00:19:50 -07:00

203 lines
9.2 KiB
YAML
Raw Permalink Blame History

# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
# Unsloth API & Auth Tests -- HTTP-level integration tests for the
# FastAPI surface. No Playwright, no model UI; tests/studio/test_studio_api_smoke.py
# runs ~30 s and asserts:
# - CORS hardening (no wildcard + credentials, no bootstrap leak)
# - /api/system + /api/system/hardware require auth
# - Auth state machine + JWT expiry
# - API key lifecycle E2E (create / list / use / delete / reject)
# - Auth file-mode hardening (Linux only)
# - Inference lifecycle (force reload, bogus variant, /v1/models, /v1/embeddings, /v1/responses)
# - Endpoint-by-endpoint auth audit
#
# Reuses the GGUF cache key from studio-ui-smoke.yml so the model
# download is one cache-hit on the second job.
name: Unsloth API CI
on:
pull_request:
paths:
- 'studio/**'
- 'unsloth/**'
- 'unsloth_cli/**'
- 'install.sh'
- 'pyproject.toml'
- 'tests/studio/**'
- '.github/workflows/studio-api-smoke.yml'
- '.github/scripts/retry-with-apt-lock.sh'
# Every server boot in this workflow shells out to that script, so an edit
# to it changes what this workflow actually runs.
- '.github/scripts/boot-studio-api-only.sh'
# Same for the /api/health wait that runs after a boot.
- '.github/scripts/wait-for-health.sh'
# The install step in this workflow is `uses:` on that composite action,
# so an edit to the action changes what this workflow actually runs.
- '.github/actions/install-unsloth-local/action.yml'
push:
branches: [main]
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}
# Latest-only on a PR branch. On main this does less than it reads like: it stops
# a RUNNING main job being killed, but GitHub cancels any PENDING run in the group
# the moment a newer one is queued, so a merge burst still leaves only the tip.
# See studio-backend-ci.yml, which is grouped per commit on main for that reason.
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
permissions:
contents: read
jobs:
api-smoke:
name: Unsloth API & Auth Tests
runs-on: ubuntu-latest
# Sized for the apt step's bounded worst case (13m) plus the smoke itself.
# Raised from 12, where the job budget was smaller than the retries the step
# authorises, so the job timeout would have fired first and reported nothing.
timeout-minutes: 20
env:
GGUF_REPO: unsloth/gemma-3-270m-it-GGUF
GGUF_VARIANT: UD-Q4_K_XL
GGUF_FILE: gemma-3-270m-it-UD-Q4_K_XL.gguf
STUDIO_PORT: '18893'
HF_HOME: ${{ github.workspace }}/hf-cache
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Linux deps
# Bounded and retried through the shared helper: an unbounded apt step does
# not fail, it spends the job's whole budget and is reported as "cancelled"
# with no reason and every later step skipped. update and install go as one
# unit, since retrying the install after a stalled update re-reads the same
# broken package list.
# Two long attempts, not three short ones. 150s killed apt mid-`update`
# against a mirror that was degraded rather than dead, and every attempt
# then hit the same wall -- three kills and no result. The bound exists to
# stop an infinite hang, not to race a slow mirror.
timeout-minutes: 15
env:
RETRY_ATTEMPTS: '2'
RETRY_ATTEMPT_TIMEOUT: '360'
run: |
bash .github/scripts/retry-with-apt-lock.sh sudo sh -c \
'apt-get install -y --no-install-recommends libcurl4-openssl-dev libssl-dev jq || { apt-get update && apt-get install -y --no-install-recommends libcurl4-openssl-dev libssl-dev jq; }'
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: '22'
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: '3.12'
# Cross-OS shared entry. The tree under `hf-cache` is byte-identical on
# Linux, macOS and Windows, so the key carries no `runner.os`, and
# enableCrossOsArchive lets Windows (which tars with --force-local) join it.
- name: Restore HF_HOME for ${{ env.GGUF_REPO }}
id: cache-hf
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
continue-on-error: true
with:
path: hf-cache
# Shared by every gemma-3-270m-it job in CI (Linux, macOS and
# Windows alike) so the model is downloaded once, not once per OS.
key: hf-${{ env.GGUF_REPO }}-${{ env.GGUF_VARIANT }}-v3
enableCrossOsArchive: true
- name: Prime HF_HOME with the GGUF
id: prime-hf
if: steps.cache-hf.outputs.cache-hit != 'true' || steps.cache-hf.outcome != 'success'
env:
# Withheld on PR: this step runs checked-out PR code; public GGUF still downloads.
HF_TOKEN: ${{ github.event_name != 'pull_request' && secrets.HF_TOKEN || '' }}
run: |
python -m pip install --upgrade huggingface_hub
mkdir -p hf-cache
bash .github/scripts/hf-download-with-retry.sh "$GGUF_REPO" "$GGUF_FILE"
bash .github/scripts/hf-download-with-retry.sh ggml-org/models tinyllamas/stories260K.gguf
- name: Save HF_HOME for ${{ env.GGUF_REPO }}
# Save on main only. Caches created on a PR ref are scoped to that
# merge ref -- per GitHub's docs they "can only be restored by re-runs
# of the pull request" -- while every PR *can* restore from the default
# branch. So a PR-scoped save helps almost nothing and competes for the
# per-repo cache budget, and when that budget is exceeded GitHub evicts
# by least-recently-used, which deletes main's copies that all PRs share.
# This repo's budget is 50 GiB, not GitHub's 10GB default, and it was
# measured at 49.63 GiB across 258 entries -- 99.3% full, so eviction runs
# at the margin. 20.74 GiB of that (42%) is the SAME key held on several
# refs, and every one of those keys already has a copy on main, so the
# PR-scoped duplicates are redundant by construction. That is the thrash
# loop: PR misses -> downloads -> saves its own copy -> evicts main's ->
# next PR misses.
if: always() && github.ref == 'refs/heads/main' && steps.prime-hf.outcome == 'success' && hashFiles('hf-cache/**/*.gguf') != ''
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: hf-cache
key: hf-${{ env.GGUF_REPO }}-${{ env.GGUF_VARIANT }}-v3
enableCrossOsArchive: true
- name: Install Unsloth (--local, --no-torch)
uses: ./.github/actions/install-unsloth-local
with:
gh-token: ${{ secrets.GITHUB_TOKEN }}
# Withheld on PR: this step runs checked-out PR code; public GGUF still downloads.
hf-token: ${{ github.event_name != 'pull_request' && secrets.HF_TOKEN || '' }}
- name: Install pyjwt for the JWT-expiry forge test
run: pip install 'pyjwt>=2.6'
- name: Reset auth + boot Unsloth (API-only)
run: |
# Wipe (not reset-password): the boot below must re-seed a fresh .bootstrap_password.
bash .github/scripts/boot-studio-api-only.sh --port "$STUDIO_PORT"
- name: Wait for /api/health
run: |
bash .github/scripts/wait-for-health.sh --port "$STUDIO_PORT"
- name: Pass bootstrap password + rotated targets to the test
# The test does its own bootstrap-login + rotation to exercise
# the auth state machine; we just pre-mint two random rotated
# passwords for it. Mask them so the log is clean.
run: |
OLD=$(cat ~/.unsloth/studio/auth/.bootstrap_password)
NEW="ApiSmoke-$(python -c 'import secrets; print(secrets.token_urlsafe(16))')"
NEW2="ApiSmoke-$(python -c 'import secrets; print(secrets.token_urlsafe(16))')"
echo "::add-mask::$OLD"
echo "::add-mask::$NEW"
echo "::add-mask::$NEW2"
echo "STUDIO_OLD_PW=$OLD" >> "$GITHUB_ENV"
echo "STUDIO_NEW_PW=$NEW" >> "$GITHUB_ENV"
echo "STUDIO_NEW2_PW=$NEW2" >> "$GITHUB_ENV"
- name: Run Unsloth API & Auth tests
# The script is named WITHOUT a `test_` prefix so it isn't
# auto-collected by pytest in Backend CI's `tests/` walk
# (which doesn't set BASE_URL and would crash at import).
env:
BASE_URL: http://127.0.0.1:18893
STUDIO_AUTH_DIR: /home/runner/.unsloth/studio/auth
run: python tests/studio/studio_api_smoke.py
- name: Stop Unsloth
if: always()
run: |
kill "${STUDIO_PID}" 2>/dev/null || true
sleep 2
- name: Upload API smoke logs
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: studio-api-smoke-log
path: |
logs/install.log
logs/studio.log
retention-days: 7
Reference in a new issue View git blame Copy permalink