mirror of
https://github.com/unslothai/unsloth.git
synced 2026-07-30 01:54:02 +00:00
bd3972804d
* Measure where Studio's startup time actually goes Nothing measured this. studio/backend/main.py logs 'lifespan startup completed in X ms' but no test or CI job ever asserted a budget, a repo-wide grep for startup_ms or time_to_ready matches only that one file, and studio_test_kit polls /healthz in a loop that discards the elapsed time it already computes. Its default healthz_timeout_s of 180 was the only recorded expectation. scripts/profile_startup.py breaks a launch into phases: import cost via python -X importtime in a subprocess (top cumulative contributors), process spawn to first output, and spawn to /healthz 200, over N repeats with median and p90. First numbers on Linux: importing the backend module costs 5.7 to 6.6 seconds before the server can even bind, and it dominates everything else. That is eager module-level imports pulled in by the routes package, not the hardware detection I first suspected: utils.hardware is 23ms and does not pull torch. --max-healthz-seconds exists so a budget can be enforced once per-platform numbers are agreed. It is not wired into a gate yet, deliberately: a threshold picked before the data is in would either be meaningless or flaky. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Profile the code under test, and let the profile fail Both installer calls omitted --local, so every phase measured the published PyPI backend and could not move when a PR edits main.py, run.py or routes. t_first_byte was a dead local, advertised in the docstring but never returned, and the reader could deadlock once the child filled the pipe. A failed launch and an impossible budget both produced a warning and exit 0, and the importtime parse reported the largest cumulative row, which is site, not main, so a raising import published a number as success. Pin the controller to the profiled venv's interpreter. * Stop the startup summary hiding failed launches The aggregates cover only the runs that reached healthz, so two dead launches and one fast one rendered as a normal fast startup, and an all-failed phase printed nothing at all. With continue-on-error and no budget wired, that summary is the only thing anyone sees. Say how many launches the number is made of, and say so explicitly when none came up. * Reject --repeats below 1 range(0) launches nothing, so the empty runs list reached the budget check as "no healthz measurement", warned and exited 0: a gate that cannot fail. The value comes straight from a dispatch input, so reject it loudly instead. * Run the startup profile when the imported startup tree changes The path filter listed main.py, run.py and routes/**, but the graph the profiler measures is far wider: main.py imports auth, core, hub, loggers, models, picker and utils at module scope, and routes/models.py imports utils.utils and utils.hidden_models. A change to any of those moved `import main` without ever running this job, so the regressions the workflow exists to catch went unmeasured. Cover studio/backend/** (tests excluded) and unsloth_cli/**, since the launch phase spawns `unsloth studio --api-only` and the CLI is on the process-to-healthz path. * Read the labelled main row and kill the Windows launcher tree total_seconds took by_cum[0], the largest cumulative row in -X importtime output. That output also carries the interpreter's own startup graph (site, encodings, whatever a venv sitecustomize pulls in), which is not part of import main, and the two are not ordered by construction. With a trivial main the old code reported site's 0.027s as "import main" while main actually cost 0.000249s. Today's backend dwarfs site so the published figures are unchanged, but the headline number must not silently become another module's cost once the backend imports get optimized, so read the row named main. profile_launch spawned Scripts/unsloth.exe on Windows. A pip console-script .exe is a distlib launcher stub that CreateProcess's the venv python and waits, so terminate() reaped the stub and left the backend holding the inherited stdout handle: the reader thread never saw EOF and burned the full 10s join, and with --repeats each iteration stranded another server on the shared UNSLOTH_STUDIO_HOME. Walk the tree with taskkill /T, matching the cleanup in unsloth_cli/commands/start.py and unsloth/dataprep/synthetic.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fail the startup budget when nothing was measured and fall back when taskkill fails * Trigger on installer inputs and harden the startup gate tests * Tighten comments in the startup profiler and its workflow * Trigger the startup profile on the Studio setup scripts install.sh --local runs the checkout's studio/setup.sh, install.ps1 reaches studio/setup.ps1 through the editable install, and both call install_python_stack.py, which decides the dependency set that gets imported. Editing any of them could change startup time with no measurement taken. * Shorten the startup profiler comments Comments and docstrings only. * Reject non-finite startup budgets and profile when the desktop argv changes --max-healthz-seconds nan or inf parses as a float but compares False against any median, so the gate reported success without bounding anything. Require a finite value. The profiler hardcodes the argv that process.rs::backend_args builds, but that file was not in the trigger paths, so a change to the desktop launch command scheduled no measurement. Add it, and anchor the two argv lists with a test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothai@gmail.com> |
||
|---|---|---|
| .. | ||
| data | ||
| build_whisper_cpp.sh | ||
| check_frontend_dep_removal.py | ||
| check_new_install_scripts.py | ||
| enforce_kwargs_spacing.py | ||
| install_gemma4_mlx.sh | ||
| install_qwen3_6_mlx.sh | ||
| install_rocm_wsl_strixhalo.sh | ||
| lint_workflow_triggers.py | ||
| lockfile_supply_chain_audit.py | ||
| notebook_to_python.py | ||
| notebook_validator.py | ||
| profile_startup.py | ||
| run_ruff_format.py | ||
| scan_npm_packages.py | ||
| scan_npm_packages_baseline.json | ||
| scan_packages.py | ||
| scan_packages_baseline.json | ||
| stamp_studio_release.py | ||
| sync_allow_scripts_pins.py | ||
| uninstall.ps1 | ||
| uninstall.sh | ||
| verify_comment_only_diff.py | ||
| verify_import_hoist.py | ||