mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-25 08:42:25 +00:00
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Eyera
|
0b147fdd2a
|
fix(studio): harden training setup, lifecycle, and audio loading (#8103)
* fix(studio): harden training setup and model loading Scan every model load root before approving remote code, and pin third-party codec sources to verified revisions. Align dataset option validation across the UI and backend, preserve manual drafts, and include edits in training start identity. Require job-scoped stop requests and retain bounded early-cancel tombstones without unsafe eviction. * fix(studio): harden training lifecycle and audio loading Preserve early start cancellations with bounded tombstones and explicit capacity handling for concurrent requests. Pin and verify third-party audio sources and codec artifacts with safe archive extraction and offline cache migration. Keep automatic evaluation data separate from every split included in a combined training instruction. Keep training summaries and GPU progress state aligned with the active configuration through public feature exports. Make route and lifecycle tests deterministic by isolating Hub reachability and executor behavior. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): correct completion masking and offline training Apply Alpaca response markers safely when completion-only training is enabled. Skip optional runtime dependency installs while Studio is offline. Limit causal-conv1d hooks to recognized model families without dropping supported architectures. Disable known-broken TileLang dispatch when offline repair is unavailable. Use cached GGUF and model-size metadata without offline Hub retries. * fix(studio): resolve causal conv kernels from model configs Detect causal-conv1d requirements from resolved model architectures before loading model code. Keep name matching as a fallback while excluding unrelated renamed checkpoints. * fix(studio): preserve registered training cancellation Allow registered start cancellations to reclaim the oldest expiring tombstone when unknown cancellation capacity is full. Preserve the hard capacity limit and 429 response for unregistered request IDs. * Fix reset job scoping, DAC fast path, scan target and subset splits for PR #8103 - /api/train/reset: an unscoped reset could force-terminate a run mid-cancel. The guard now refuses to touch a live run it cannot prove it owns. The field stays optional so pre-rework clients, which POST /reset with no body, keep working. - ensure_dac_speech_weights: install the download into the pinned destination, so later loads hit the fast path instead of re-downloading and re-hashing 295 MB under the install lock. - _requires_security_review_for_model: apply the same load_scan_target alias normalization the sibling remote-code check gained, else the Spark-TTS alias 404s and fails open to "no review needed". - _purge_package_bytecode: best effort. It runs without the install lock over a cache shared by the inference and training workers; 7 of 8 concurrent imports died on it. Also moved inside the try so a failure cannot strand the cache dir on sys.path, and snapshot sys.modules before the origin audit. - commitSubset: clear the backing splits too, else the render-phase draft sync reads the previous subset's split back into the boxes it just reset. - Fix two tests that fail on the branch: the DAC assertion pinned the old return value, and the causal-conv1d assertion matched call formatting. * Make the SSM runtime tests Windows aware ensure_ssm_runtime deliberately skips causal-conv1d on win32 (no prebuilt wheel), so the two install-order assertions only hold off Windows. Caught on a real windows-latest runner. * Update two frontend source contracts the branch moved - captureTrainingStartInputs now delegates to createTrainingStartInputIdentity, so the normalize/flags assertions belong against training-start-inputs.ts. - resetTraining takes a RequiredTrainingJobScope and always sends the body, which is stronger than the hasScope branch the contract pinned. Both fail on the branch today; caught by tests/studio, which the studio backend job does not cover. * Fix unscoped reset compat, pyc purge fail-open and DAC fallback for PR #8103 Corrects four things in my earlier commits on this branch. Unscoped /api/train/reset returned "superseded" (HTTP 200) for a live run. The pre-rework cancel dialog chains stopTrainingRun then a bodyless reset, so an older client read that 200 as success and cleared its UI while training kept running. Return "active" (409) instead: same answer a live run already gives, and one those clients already handle. It still never force-terminates, so a bodyless reset landing between current_job_id being set and _cancel_requested being cleared cannot kill the run that just started. The bytecode purge was made best-effort, but it is the only thing stopping a stale or planted .pyc shadowing a verified .py: the manifest skips __pycache__ and the origin audit reads __file__, which still names the .py. Tolerate only FileNotFoundError, the real concurrent-purge race, and let PermissionError fail the load again. The DAC fast path copies 295 MB inside the hub cache with only Timeout caught, so a full disk turned a hash-verified download into a hard failure. Fall back to the verified hub path on OSError. commitSubset cleared both splits, but setDatasetSubset already does that; the extra setDatasetSplit(null) only cost a runDatasetCheck against an assumed "train" split. * Restore the eval split reset and the cancelled-run dismiss for PR #8103 Two corrections to 4e5389c21. commitSubset: I removed setDatasetEvalSplit(null) because setDatasetSubset already nulls datasetEvalSplit. It does, but it never resets evalSteps, and setDatasetEvalSplit is not a plain setter: it zeroes evalSteps and runs streamingCompatiblePatch. Without it, changing the subset left evaluation armed with no split, which routes/training.py rejects with 422 once streaming is on, and which silently auto-detects an eval split otherwise. Restored. setDatasetSplit(null) stays out, since its only unique effect was a runDatasetCheck against an assumed "train". Unscoped reset: returning "active" for every live run was too broad. The pre-rework cancel dialog only dismisses after stopTrainingRun succeeded, so _cancel_requested is already set and clearing the UI is right; 409 there just wedged the overlay behind a "Training still active" toast. Now 409 only when no stop was requested, which is the stale-tab case the change was for. Still no force_terminate on an unscoped reset. Also covers the __pycache__ branch of the purge, which is the route a planted .pyc actually takes; the existing test only reached the top-level .pyc loop. * Keep live start cancellations and fall back on a full disk for PR #8103 Two fixes plus the regression tests the earlier lifecycle fixes shipped without. Cancelling the active start at tombstone capacity reclaimed a slot by deleting the soonest-expiring entry. Expired ones are already pruned a few lines above, so that entry was always live, and dropping it let a delayed /start spawn the job it had cancelled. Reserve capacity instead: only the owner of the active start reaches that branch and there is at most one, so the table lands at cap + 1 rather than forgetting a cancellation. Unregistered ids still hit the hard cap. ensure_dac_speech_weights migrated a pre-existing legacy file with an unguarded copy, so a hub cache that cannot absorb a second 295 MB copy failed weights that had already passed size and sha256. Same fallback the download branch below it already uses. Tests: renamed and local SSM checkpoints resolving from config rather than name, the owner of an active start staying cancellable at capacity, live cancellations surviving that cancel, and the full-disk legacy DAC fallback. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Opt the pinned source checkouts into Git long paths for PR #8103 Cross-OS CI caught 14 failures on windows-latest, all of them the pinned checkout dying with "error: unable to write file ...: Filename too long". Git for Windows still enforces MAX_PATH unless told otherwise, and the cache nests a 40-char revision, a staging dir and .git/objects under the studio home. A venv-inferred home already measures about 253 of the 260 characters, so a slightly longer user or install path fails on a normal Windows machine, not just under the deeper pytest tmp dir. Passed per invocation with -c so no user or system Git config is touched, and it is a no-op off Windows. * Bound pending cancels, reach legacy DAC weights, and delete read-only checkouts for PR #8103 Three fixes, two of them on my own previous commit. Moving the owner cancel from evict-oldest to a one-slot overshoot also changed what the hardcoded reclaim_capacity=True on the pending non-owner branch did: it used to evict, so the table stayed at the cap, and it started overshooting instead. Start plus cancel could then be repeated to grow it without bound (1224 entries against a cap of 1024). That branch now takes the plain reservation, so the extra slot belongs to the owner of the active run alone. The DAC legacy fallback sat behind destination.parent.mkdir() and the install lock, both of which need a writable cache, so a read-only or full hub cache raised before weights we can already verify were ever looked at. Fall back at both points, and only to an artifact that passes the same size and sha256 check. Cross-OS CI then caught replacing a pinned checkout failing on windows-latest with WinError 5: Git marks .git/objects read-only and Windows will not delete a read-only file, so any repair or revision change died there. Clear the attribute and retry, only when the path is genuinely not writable, so an open handle still surfaces. * Delete the cached checkout the Windows-safe way in the migration test for PR #8103 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <moonshotaisubstack@gmail.com> |
||
|
Eyera
|
b52d3b56ed
|
feat(studio): rework train page setup flow (#7633)
* feat(studio): rework training setup and cache handling * Rework train page resource selection * Improve train page workflows and resource selection * Fix train page picker behavior * Fix stale training errors after token updates * Fix training eval steps regression * Fix shared token use when resuming training * Harden train page resource workflows Unify model and dataset picker behavior, training start guards, and token handling. Validate cache provenance for local models, downloaded resources, and processed datasets. Prevent stale preview and history state while keeping the train page modular. * Fix training method race and test isolation * Restore training upload limit exports * Fix training config formatting contract * fix(studio): harden training model selection and preflight * fix training resume, cache, and lifecycle reliability * Fix training stop watchdog path stub * Fix training start state and train page UI * Unify train page selection controls * Fix train page validation, localization, and paging * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Clean up train locales and restore recipes link * fix(studio): correct training model selection and preflight Preserve freeform local model paths, reject remote GGUF-only repositories, and correct the training start snapshot contract. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): harden training resource selection Preserve local dataset path intent, localize setup-change errors, and remove unused picker tour hooks. Refresh model picker and PDF recipe contracts for the refactored training flow. * fix(studio): remove train tab scrolling * fix(studio): harden training resource preflight Reject missing local models and binary adapter artifacts. Preserve selected model and dataset cache pins through preflight and QLoRA loading. Detect cache path changes before training starts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix train page validation and cache handling Keep transient model checks nonblocking and bind cached models and datasets to their selected snapshots. Improve Hub auth errors, retries, task search, streaming consistency, locale formatting, reset handling, payload mapping, partial inventory filtering, theme accents, and regression coverage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): stabilize train page pickers Unify Hub validation, local inventory, picker styling, and dataset names. Localize training feedback and explain streaming modality changes. Add unit, contract, and cross-browser picker coverage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): align cached scans and variant state purge Scan selected cached snapshots during training security preflight. Purge manifests and cancel markers using their stored GGUF variant. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): harden train cache, consent, and UI Fix cache cleanup and cross-platform model path handling. Align resume and adapter consent scans with pinned load targets. Localize dataset flows and refine navigation, state cleanup, and theming. * fix(studio): harden training resource selection Preserve cached model configuration HTTP errors so stale cache references return their intended 404 responses. Wait for device inventory settlement before locking inferred picker tabs, while keeping known device items visible during scans and retries. Apply modality name heuristics only to the final model path component across supported platforms. Expose full model and dataset identities on truncated picker triggers. Add regression coverage for picker settlement, retry behavior, and modality inference. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix train picker flow and resume safety Scan the exact cached model snapshot before resume consent and preserve the actual repository identity. Keep cold pickers on Device until inventory settles, cap automatic Hub pagination, and provide a stable Load more action. Improve train control and history card accessibility, remove duplicate token labeling and dead styling, and correct picker spacing. Normalize Windows relative model paths and add coverage for resume pins, picker policy, pagination, path identity, and accessibility contracts. Fix the reported formatting and import ordering issues. * Fix cross-platform train dataset path detection Use the shared local path detector so persisted Windows, Unix, UNC, and home-relative dataset references do not render Hugging Face-only controls. Recognize Arrow dataset files and add regression coverage for supported local path formats and Hub repository identifiers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix training start recovery and dataset selection Reconcile failed fresh and resumed training starts with backend status before reporting an error. Treat the selected dataset source as authoritative so filename-like Hub repositories retain subset and split controls. Align dataset picker sorting and remove unused advisor template state with a persisted-state migration. Add regression coverage for transport recovery and Hub dataset selection. * Fix train picker validation and start recovery Update the resume training contract test for the asynchronous failure handler. Filter invalid Hugging Face search results and block invalid saved model and dataset selections. Restrict local dataset selection to inventoried application paths. Load training YAML through a bounded native file picker while keeping the browser fallback. Only recover uncertain training starts after network or response parsing failures, and match the active job to its request identifier. Treat cached full precision models as ready for QLoRA without showing a false download warning. Add regression coverage for the picker, config import, and training recovery behavior. * Harden training starts and config imports Track training start request IDs from preflight through pending, accepted, and rejected states so retries remain idempotent. Disable automatic retries for start posts and reconcile ambiguous client responses with server request status. Clear terminal training status on reset so later runs do not inherit stale state. Enforce the 1 MiB YAML limit for browser imports and keep read errors specific to the selected file label. Remove unused dataset picker translation entries across all locales. Add focused tests for request idempotency, start recovery, config limits, and picker contracts. * Harden train picker feedback and start handling Show explicit empty states for device dataset searches and invalid Hub model or dataset queries while keeping pagination sentinels mounted. Run the train picker Playwright suite across Chromium, Firefox, and WebKit in Windows CI. Keep training start outcomes owned by backend spawn completion so cancelled handlers cannot mark active jobs as rejected. Correct the Train section heading hierarchy and format the branch-owned native file dialog test. Add picker contracts, browser assertions, and cancellation race coverage for the updated behavior. * Fix training setup races and configuration state Keep hardware-based method selection within model-default loading so starts cannot race changing defaults. Clear start errors only after deliberate configuration or token edits while background reconciliation stays silent. Generate training request IDs safely for LAN HTTP access before acquiring the runtime lease. Filter persisted state, sanitize invalid methods, and guard preview metadata. Localize config size failures, use the toast wrapper, and remove obsolete training files. Add regression coverage for the updated start, persistence, and import behavior. * Refine training configuration and picker workflows Split the training configuration store and parameter panel into focused persistence, policy, LoRA, hyperparameter, memory, and MLX modules. Run picker coverage on platform-appropriate browser engines and parameterize the unmanaged dataset path. Surface configuration changes when streaming is disabled and clear stale dataset modality state after failed probes. Consolidate dataset format and AI-assisted mapping on hub routes with header-based Hugging Face tokens. Hold local path actions until model inventory loading settles. * Fix train picker state and preflight regressions Open online pickers on the Hub before inventory settles and keep the inferred tab stable for the session. Preserve parameter and VRAM metadata for pinned selected models. Reject failed backend starts once and cancel automatic modality corrections without showing a destructive setup error. Update persistence contracts to follow the extracted training configuration module. Move shared parameter option styling out of the React component module to preserve Fast Refresh. Add regression coverage for picker tab behavior, pinned model metadata, persistence contracts, and start rejection handling. * fix(train): harden model selection and start feedback Preserve cache metadata and model capabilities when freeform local paths resolve to discovered models. Return stable training start error codes through direct and recovered requests, then localize Hugging Face preflight failures across every supported locale. Show whether submitted advanced settings are default or non-default in Simple mode, and migrate the parameter mode preference to the standard storage key. Restore absent-key migration semantics and update the embedding security test for the extracted gate helper. Add focused coverage for structured errors, advanced setting summaries, start recovery, and security gates. * fix(train): harden picker validation and focus Align Hub ID validation with Hugging Face rules while accepting valid underscore boundaries. Reject persisted path-shaped datasets from Hugging Face-specific controls. Reuse model identity normalization for cache path comparisons. Move picker search focus into the popover focus lifecycle. Update source contract tests for extracted model selection and structured resume errors. * fix(train): refine picker behavior and dataset structure Wait for device inventory before locking inferred picker tabs while preserving explicit user choices. Align Hub resource ID validation with backend rules for leading and trailing underscores. Support ArrowDown navigation from picker tabs into available options. Split dataset selection, uploads, streaming settings, and inventory refresh logic into focused modules. * fix(studio): stabilize train picker contracts and state Update dataset contracts to follow the extracted selection, upload, and inventory modules. Keep the inferred picker tab stable while device inventory settles. Use the shared TrainingMethod type for VRAM estimation and remove redundant casts. * fix(train): align Hugging Face repo validation Allow underscores at repository segment boundaries to match Hugging Face repo ID rules. Keep model and dataset search queries unvalidated while preserving selection safeguards. Add regression coverage for accepted IDs and unrestricted picker searches. * Fix training resource compatibility and cache safety Restore legacy dataset format and mapping routes while moving training uploads and checks to the canonical Hub endpoints. Enforce configured upload limits, clean partial files after failures or cancellation, and support both upload route families in middleware. Align Hugging Face repository validation with accepted underscore boundaries and skip PyTorch dependent tests when the dependency is unavailable. Keep model format and cache metadata correct when the same repository is selected from another source. Only promote complete runnable model weights as cached training resources, align download notices with picker capabilities, and reject weightless local snapshots during start preflight. Cancel stale dataset mapping assistant requests before they can overwrite a newer selection. Preserve touch selection for training methods by separating tooltip toggling from Select item activation. Add focused backend, frontend, and picker regression coverage for these paths. * fix(studio): finish train page and onboarding rework Ignore cache-only fields when comparing training start inputs so reconciliation does not cancel valid runs. Route native dataset drops through Tauri path validation and lease-backed managed uploads. Use the production model and dataset selectors in onboarding and replace placeholder uploads with real imports. Require positive learning rates while keeping learning rate, epoch, and step inputs editable. Normalize model identities, improve dataset summary order, and guard persisted selection state. Recognize Windows rooted and drive-relative paths consistently in picker logic. Add MLX optimizer help and learning rate validation messages to every supported locale. Add frontend, backend, and native policy regression coverage for the updated behavior. * chore(ci): remove train picker smoke workflow changes Remove the train picker Playwright step from the macOS Studio smoke checks. Remove the train picker Playwright step from the Linux Studio smoke checks. Remove the train picker Playwright step from the Windows Studio smoke checks. Drop the matching artifact upload paths so the workflow files match upstream main. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): harden train picker behavior and boundaries Expose dataset drop helpers through the training public API and move local path detection into shared code. Block freeform Hub choices while offline for pointer and keyboard selection. Make native dataset drop handling safe for committed React renders. Keep recipe navigation working when session storage is unavailable. Preserve Windows drive root identities and cover the distinction from drive relative paths. Update picker and training input contracts to match the extracted implementations. Remove orphaned translations and align the train style block with the existing indentation. * Fix train picker cache and upload behavior Keep selected model cache paths strict and preserve fallback discovery only when no path is supplied. Offload multipart writes and native dataset copies from the backend event loop. Prewarm train picker inventories so device-first tab inference can settle before the first open. Use shared picker tab constants across the model and dataset selectors. Update picker contracts and add regression coverage for cache resolution and dataset uploads. * Refine train picker controls and component structure Split picker retry, error, and pagination UI into focused components. Add a shared segmented control and use native radio semantics for dataset source selection. Centralize segmented indicator positioning with RTL-aware transforms. Separate dataset advanced settings state wiring from its presentation and source toggle. Make picker arrow navigation move symmetrically from tabs through search and options. * Fix train page reconciliation and picker contracts Preserve tuned training parameters when model cache references change, while refreshing defaults when no user edits occurred. Keep streaming modality corrections visible through the Start CTA after preflight stops. Localize the new training parameter and dataset mapping text across all supported locales. Refresh picker structural contracts after the component split and apply Biome formatting to the changed files. * Fix train upload tests and desktop drop handling Pass an explicit empty native path lease in legacy upload limit tests so direct route calls match FastAPI request behavior. Track Tauri scale factor changes during dataset drag and drop, clean up both listeners safely, and cover runtime updates. Replace the run preview card's important border utility with an explicit dark theme card modifier. * fix(studio): harden training setup and history state Restore Hugging Face token entry and validation in the onboarding model and dataset steps. Apply the configured upload size preflight before onboarding dataset uploads. Update moved model and dataset cache references without showing false missing cache warnings. Parse SSE frame delimiters by their matched length so mixed line endings preserve event data. Match sensitive pid paths by exact segment while retaining database state protections. Clear deleted Data Recipe selections after inventory settlement while preserving direct uploads. Show the files deleted status only when a run previously recorded an output directory. Handle Data Recipe navigation failures and keep the native drop listener stable during uploads. Replace formatting-specific contract checks and add focused behavioral regression coverage. * Harden training start and model picker flows Split the train model picker into its own entry point to keep it out of shared route bundles. Preserve pending idempotent start reservations and reconcile them before accepting a run. Keep unconfirmed starts in a polling state with accurate localized feedback. Continue starts after automatically disabling unsupported multimodal dataset streaming. Localize the updated model picker controls across supported languages. Lock dataset uploads before asynchronous preflight to prevent concurrent selections. Add regression coverage for reservation states and unconfirmed runtime starts. * fix(studio): harden training picker edge cases Preserve POSIX model path identity while retaining Windows relative path normalization, and clarify the override migration behavior. Localize the onboarding model and dataset controls across every supported locale. Truncate native dataset filenames by Unicode code point so generated labels cannot contain lone surrogates. Cancel training after preflight disables streaming for image or audio datasets so users can review the changed setting before restarting. Update runtime contract assertions for the setStartPending rename and add regression coverage for path identity and Unicode truncation. * fix(studio): align training picker controls Restore full-height Browse and Amazon S3 selection with an accessible radiogroup that avoids fieldset sizing behavior. Align model and dataset picker tabs with the shared 36 px segmented control geometry and typography. Remove active borders and focus rings from picker search inputs while preserving keyboard focus indicators on other controls. Add contract coverage for segmented control sizing and picker search focus styling. * fix(studio): harden onboarding picker flows Classify native dataset drops from the full path before shortening display names. Keep onboarding model choices within the selected training type across Hub, device, and freeform selections. Retry model default loading when onboarding restores an interrupted model selection. Localize the Hugging Face token field across supported languages. Reset picker result scrolling when switching between Device and Hugging Face tabs. Add regression coverage for native filenames, model constraints, hydration, and picker scrolling. * fix(studio): reconcile multimodal streaming before training Disable streaming when dataset checks detect image or audio data, then recheck cached selections in non-streaming mode. Keep start-time modality detection as a safe fallback that updates the configuration and continues without an error or a second click. Build onboarding training method options from the shared method order and metadata so onboarding stays aligned with the train page. Add contract coverage for the streaming reconciliation and shared training method list. * Fix train picker contracts and dataset recovery Allow local models without reliable modality metadata to pass onboarding constraints while retaining explicit mismatch checks. Reconcile GPU selection without effect-driven state updates and mark intentional deep Hub imports for lint. Return and consume stable local dataset cache miss codes, and send the canonical train_split field. Move train-specific picker code into its owning feature and update the related contract coverage. Use the canonical Hub dataset progress route from Chat. * Harden training resource selection and offline handling Return an actionable preflight error when offline mode is enabled and the selected model is not cached. Validate evaluation dataset uploads with the shared extension policy and reuse the centralized accept list. Disable streaming for explicit on-device dataset selections while preserving Hugging Face selection intent. Keep dataset identifiers safety checked without blocking benign values before Hugging Face handles repository validation. Remove the branch-added source assertions and cover the new selection policy with behavioral tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix training setup validation and async state races Validate Hugging Face dataset IDs and repository access before training starts while keeping local paths on their filesystem validation flow. Scope model default race protection to fields that defaults can overwrite so unrelated dataset edits no longer suppress model initialization. Stop Hugging Face token edits from invalidating token-independent device inventory scans on every keystroke. Add regression coverage for dataset source validation, cached and remote Hub preflight, model default application, and token inventory behavior. * Tidy up train page layout, tooltips and dark mode Layout - Move the Browse / Amazon S3 toggle into the Dataset section header - Put Subset, Train Split and Evaluation Split on one row - Put Target Format, Train Split Start and Train Split End on one row - Pair Project Name with Max Steps, and Context Length with Learning Rate - Centre the hyperparameter tabs and give them a fixed width so the sliding indicator lines up with its segment - Give parameter rows a min height so slider and select rows share one vertical rhythm - Slightly more spacing below section headings and at card bottoms Tooltips - Add hints for Model, Method, Dataset and Project Name - Move field descriptions into the tooltips and drop the inline copy - Upload field keeps the accepted file types inline, with size limit and Learning Recipes note in the tooltip HF token - Show a masked preview of a saved token instead of a generic label - Read Not set when no token is stored Dark mode - Drop borders on the LoRA option cards and target module chips, and separate states with background fill instead - Use a lighter, less saturated green for the selected state Other - Expand LoRA Settings and Training Hyperparameters when Advanced opens - Shorten the upload label to Drop file or click to upload - Localise the new wizard tooltips across all locales * Fix training defaults, MLX validation, and dataset preflight Prevent late hardware recommendations from replacing configuration values loaded after model selection. Block unsupported MLX methods and embedding runs in the UI and backend before training is queued. Require live Hub access for streaming dataset preflight instead of accepting unrelated cached data. Compare advanced settings against applied model defaults and cover the updated behavior with focused tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix training picker validation and platform safeguards Update the Tauri YAML contract test to verify save filter behavior without depending on the Rust vector implementation. Align frontend Hub resource validation and training provenance normalization with the shared repository ID rules, including valid underscore and maximum length identifiers. Reuse one frontend training method policy so Apple Silicon onboarding and the Train page consistently disable CPT and prevent unsupported onboarding completion. Add regression coverage for rejected Hub IDs and exact resume provenance. * Fix MLX training capability validation Detect hardware before platform validation so unsupported MLX configurations fail before a job is queued. Reject audio dataset training during server preflight while preserving the worker guard. Share Apple Silicon capability checks across model selection, onboarding, readiness, submission, and LoRA controls. Cover audio, CPT, embedding, LoftQ, DoRA, and warm-start detection with regression tests. * Fix training quit protection, translations, and preference reset Protect desktop quits while a training start is pending or a run is active, and keep the warning text accurate. Localize dataset upload details across every supported locale and require those keys in parity checks. Centralize the training picker and parameter mode storage keys so resetting local preferences clears current and legacy values. Add regression coverage for training activity transitions and preference reset ownership. * Preserve multimodal training support Keep upstream training eligibility authoritative while preserving separate vision, audio, and embedding capabilities. Classify dual audio and vision models for vision training without changing inference support or enabling pure audio training on MLX. Add coverage for model type constraints, backend type resolution, and MLX validation. Apply the pending Biome formatting fixes to the dataset selector and training wizard. * Fix training defaults retry and clean stale translations Allow model defaults retries after failed cache reconciliation without overwriting user-edited settings. Ignore stale fallback vision checks once a newer defaults request starts. Remove unused parameter description keys from every locale and cover the retry behavior in the training contract tests. * Document direct model identity imports Explain why the model identity helpers bypass the Hub barrel beside each lint suppression. * Preserve training settings across reloads Persist applied model defaults and their advanced settings baseline while refreshing transient model metadata without overwriting tuned hyperparameters. Keep remote format probe tests runnable without an installed huggingface_hub package by supplying a fake module. Normalize tilde-prefixed path separators while preserving case-sensitive identities. Remove unmatched model and dataset tour anchors. Add regression coverage for persistence migration and model identity normalization. * Harden train page validation and persistence Validate CPT embedding learning rates against backend bounds and add localized client feedback. Keep invalid learning rate drafts out of persisted state and YAML exports. Notify users when multimodal model selection restores a non-S3 dataset source. Remove W&B tokens from training config persistence and migrate stored secrets safely. Update the model defaults contract test to cover metadata refresh without requiring the removed early return. Move segmented control styles into a non-component module to preserve Fast Refresh behavior. Add validation and persistence coverage for the new contracts. * Fix training model IDs, cache state, and localization Preserve root-level Hugging Face model IDs instead of rewriting them into the Unsloth namespace. Debounce token-scoped inventory reconciliation so token and inventory updates use stable request keys. Keep the selected training history run when the Studio view remounts. Localize dataset subset and split selectors across every supported locale. Complete Italian picker and training translations and enforce Studio-wide locale parity. * Fix training history errors and stale stop state Return structured artifact deletion errors and show localized messages that distinguish active training output from filesystem failures while keeping history intact. Clear superseded stop requests only when they belong to the current runtime generation, and cover start invalidation and stale stop handling with regression tests. Clarify that shared picker styles apply outside the Hub while Hub-only rules remain scoped. * Fix training method persistence and settings summaries Persist training method provenance so manual learning rates, model adapter rates, and pre-CPT dataset formats survive reloads and method changes. Deduplicate imported target modules and compare advanced-setting arrays by value counts so duplicate entries cannot hide non-default settings. Add migration and regression coverage for legacy state, rehydrated method transitions, restored learning rates, and duplicate target modules. * Fix persisted completion defaults and inventory retries Persist trainOnCompletions across reloads so model defaults and advanced settings summaries remain accurate. Defer missing legacy values until model metadata loads, preserving tuned settings while respecting streaming, raw text, CPT, embedding models, and explicit user changes. Settle partial inventory failures when usable rows exist so manual local model paths and Enter submission remain available. Add retry actions for partially failed model and dataset scans, including when filtering leaves no visible results. Add regression coverage for persistence migration, constrained completion defaults, inventory settlement, and picker empty states. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix training validation and route state regressions Retry newly detected image datasets with vision-aware validation, clear stale dataset check failures, and cover the failed background classification case. Restore Train history cleanup and centralize the learning recipes navigation intent key. Repair backend route stubs for canonical model IDs and path normalization, and isolate consent tests from shared package import state. * Preserve training cache pins during preflight Keep model and dataset cache flags and local paths in the pending start comparison so reconciliation cannot submit a stale snapshot. Cover changed cached copies and uncached transitions with focused regression assertions. * Preserve Hub identity for cached training exports Keep exact cached snapshots for training while restoring the standard Hub repository identity before PEFT saves. Recover repository identity in memory for legacy adapters so imatrix exports work without rewriting adapter configs. Cover repository mismatches, local models, and Windows cache paths with focused regressions. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix cached training, evaluation, and MLX resume behavior Limit cached dataset slice loads so small training ranges do not materialize full duplicate caches. Restore Hub model identity for cached Torch and MLX runs so saved adapters remain portable. Attest MLX model provenance across runtime and prequantized 4-bit formats so valid checkpoints can resume. Recognize missing cached eval splits, reload train and eval together, treat explicit eval split errors as fatal, and use deterministic held-out data for automatic fallback. Retain non-fatal evaluation warnings in training status and render them in the train UI. Apply pin and model format validation to every MLX worker entry point and keep the worker test fixture aligned with its imports. * Stabilize training progress updates across job handoffs Keep active and pending training ownership separate so status polling cannot switch identities during a running job. Scope metrics, progress streams, stop, and reset operations to the expected job and recheck ownership during handoffs. Make polling, stream parsing, and start reconciliation abortable, monotonic, and safe against stale overlapping requests. Preserve same-run interface state so live updates do not remount progress controls or dismiss the stop confirmation. Add backend and frontend regression coverage for ownership changes, stale events, request races, and stream cleanup. * Fix cached training fallback and dataset reconciliation Recognize incomplete SentencePiece tokenizer caches and retry online model loads through the Hub while preserving strict offline and resume pins. Prevent rejected dataset snapshots from being promoted again until the selection or inventory changes. Forward cancellation signals to dataset format requests so superseded checks stop at the transport layer. Add focused backend and frontend regression coverage for these cache paths. * Harden offline training and cache fallback behavior Validate cached model snapshots for tokenizer and processor support, then fall back to the pinned Hub revision when the cache is incomplete. Remember rejected dataset cache entries and cancel superseded checks so metadata-only snapshots cannot trigger request loops. Reject deleted local datasets before any Hub access and clear only the matching stale selection in background, preview, and start flows. Reserve training starts before validation so overlapping requests and GPU consumers cannot race the spawn window. Keep GGUF fallback metadata tied to the cache that supplied variants when Hub access fails. Filter DSpark and DFlash drafter companions across picker, loading, inventory, and deletion paths while preserving MTP behavior. * Harden training setup and picker behavior Keep training starts blocked while a stop request is pending and preserve the stop latch through backend failures. Skip malformed training progress events while validating SSE payloads and releasing stream readers safely. Enforce non-streaming state for upload and S3 datasets across source changes, persistence, and rehydration. Route Markdown dataset drops to Data Recipes across browser and native path formats. Give model and dataset source tablists localized accessible names. Refresh training start contracts and add focused regression coverage for the corrected behavior. * Fix offline dataset cache selection and token status Prefer finalized processed dataset caches for offline training while keeping raw cache paths available for scoped management. Treat malformed Hugging Face tokens as unset in Train and surface local validation feedback without unnecessary requests. Add regression coverage for cache coexistence, interrupted cache builds, path propagation, and malformed token presentation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Preserve sidecar Hub version during dataset cache checks Keep processed dataset cache discovery free of eager datasets imports so training does not preload the base Hugging Face Hub package. Preserve cache root discovery through HF_DATASETS_CACHE, HF_HOME, and XDG_CACHE_HOME before Transformers sidecar activation. * Add the AGPL-3.0 header to the two new studio training contract tests * Scan the fallback target and pin cached snapshots that hold weights Cache fallback scanned the pinned snapshot it was about to discard, then loaded the Hub repo unscanned. Scan after the pin is dropped instead, in all four training paths. Preflight probed the literal model name, so the registry bicodec alias Spark-TTS-0.5B/LLM 404'd and rejected a supported model. Probe the repo the trainer downloads and treat its load subdir as the weight root. An unpinned start still downloads its dataset, so a stray cached copy no longer skips Hub verification. Offline keeps accepting the cache. refs/main can point at a metadata-only revision, so the model pin now prefers a snapshot that carries weights before falling back. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Let the capability-cache guard see through the key type alias model_config.py names the cache key _CapabilityCacheKey, so matching the literal Dict[Tuple no longer finds it and the guard fails on a cache that is still correctly tuple-keyed. Resolve module-level aliases first. Checked by mutation: reverting a cache to Dict[str, ...] is still caught. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: point the extra-UI tour anchor check at studio-model-picker The train page rework renamed the model tour anchor from studio-model to studio-model-picker and updated tour/steps/base-model.tsx accordingly, but tests/studio/playwright_extra_ui.py still looked for the old anchor, so the Chat UI Tests job failed with "[data-tour='studio-model'] not found". Verified against two live Studio instances: the anchor is studio-model on main and studio-model-picker at this head; studio-dataset and studio-params are unchanged on both. * Studio: require metadata and weights together when pinning a model snapshot Pass 1 of _resolve_model_snapshot matched on weights alone, so two cases still selected a snapshot that /training/start then rejects with "does not contain trainable weights": - a newer weights-only fetch (interrupted download, or an allow_patterns pull that never took config.json) displaced an older complete sibling. That is a regression against the previous metadata-first ordering, which started the run; reproduced with a two-snapshot cache where the complete one is older. - consolidated.safetensors counted as weights for selection but is absent from _MODEL_WEIGHT_CANDIDATES in routes/training.py, and transformers 4.57.6 has no loader path for it (zero references in the package), so a config plus consolidated snapshot won over a loadable sibling. Pass 1 now demands metadata AND weights via a required_groups argument on latest_snapshot_from_cache_path, and consolidated.safetensors is dropped from the selection tuple so it matches what the route accepts. Pass 2 keeps the metadata-only fallback unchanged, so caches that never held weights resolve exactly as before. Both new tests in test_model_cache_snapshot.py fail without this change and pass with it; 401 tests across the cache, preflight, provenance and identity suites pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make the new cache tests pass on Windows Four of the PR's new tests fail on windows-latest while passing on ubuntu and macos. All four are defects in the tests, not in the production code. The three blob-symlink tests hardcode a POSIX relative target ("../../blobs/x"). Windows stores the reparse-point substitute name verbatim and resolves it in the object namespace, where / is not a separator, so the link is created but dangles: Path.resolve(strict=True) raises and the provenance and dataset-cache helpers correctly return None. huggingface_hub builds these targets with os.path.relpath (file_download.py _create_symlink), which yields ..\..\blobs\x on Windows, so a real cache never has this shape and no Windows user is affected. Building the target with os.path.relpath / os.path.join matches what huggingface_hub writes. The cross-snapshot rejection tests had the same POSIX targets, so on Windows they were passing for the wrong reason (dangling link rather than the escape check). They now use native separators too, so the rejection logic is actually exercised there. test_runtime_4bit_resume_reaches_worker_with_source_resource_pins compared model_local_path against str(Path); the route posix-normalizes that field via normalize_path, so the assertion now uses Path.as_posix(), which is a no-op on POSIX. 324 tests across the three files pass on Linux. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: bound the cached dataset re-check and stop adopting unconfirmed starts Two defects introduced by this PR's new modules. 1. Unbounded dataset re-check (#7853). training-config-store.ts re-runs the cached format check whenever rejectValidation() reports the in-flight token as stale, with no counter and no backoff. DatasetCacheRejectionTracker advances its generation on any inventory-fingerprint change, and the fingerprint includes sizeBytes, so a dataset that is still downloading invalidates every check in flight and the pair never converges. Measured at 480 requests in 60s; a probe driving the real tracker ran 500 iterations without settling while a stable inventory settles in 1. The retry now draws from a per-selection budget (dataset-recheck-budget.ts) and falls through to the uncached check once spent, so the answer still refreshes when the inventory genuinely changes but a churning one cannot spin. The budget resets when the dataset or split changes. Fixing the tracker fingerprint instead was rejected: existing tests pin the behaviour that a sizeBytes change makes a rejected cache retryable. 2. Unconfirmed starts adopted as recovered. reconcileTrainingStartTransportFailure ended with adoptAcceptedTrainingStart(pending.jobId, ...) and returned "recovered". The backend reserves the request id and job id before the heavy preflight, so a start still pending when the 30s window closes may yet be rejected. Adopting there reported success and pinned a job id that never became current_job_id: acknowledgeTrainingStartRequest was never sent, and dismissTrainingRun's expectedJobId check then returned "superseded", leaving the rejected state unclearable until a new start re-reserved it. It now calls the module's existing settleUnconfirmedTrainingStart and returns "unknown", which both callers already handle by warning startUnconfirmed and settling unconfirmed, and which preserves start_request_id for acknowledgement. New tests fail without each change and pass with it; removing the bound makes them fail rather than hang. typecheck, 598 frontend tests, build, and the 154 studio contract tests all pass. * Studio: retry tokenizer-less pinned snapshots, and skip Hub preflight when offline Two more defects introduced by this rework. Neither mechanism exists on main: worker.py there has 0 occurrences of local_files_only / model_snapshot_path / cache_artifact, and routes/training.py has 0 of model_info / dataset_info. 1. Tokenizer-less pinned snapshot is terminal (#7845). The pin only requires config.json plus a weights file, so a snapshot with no tokenizer pins clean, local_files_only is set, and AutoTokenizer then fails. Recovery is gated on _is_model_cache_artifact_error, whose marker list matches only three of the message shapes transformers actually emits. Sweeping all 159 tokenizer classes against a tokenizer-less snapshot: 37 failures were classified not retryable, of which 26 are genuine cache problems that got zero Hub retry, including XLMRoberta (BGE-M3, multilingual-e5, LaBSE), MBart, NLLB, Bloom, GPTNeoX, Cohere, Marian and the generic PreTrainedTokenizerFast. SentencePiece and BPE families resolve a missing vocab path to None and then dereference it, so the failure arrives as a bare AttributeError with no cache-specific text. Adding those four shapes takes the sweep from 37 misses to 11, and all 11 remaining are correctly fatal (missing optional Python dependency, or an unsupported tokenizer class), which no Hub retry can fix. Widening the classifier rather than validating tokenizers before pinning: the vocab filename space is as open-ended as the exception space across 159 classes, requiring a tokenizer would break the deliberate adapter_config.json pin path, and a real loadability check means loading a tokenizer inside the start request. The recovery path is already gated on offline and require_exact, so a false positive costs one Hub attempt while the current false negative fails the run. 2. Blocking Hub preflight with no reachability guard. Both preflight legs retry metadata at 5s then 10s with no reachability check, and requests applies the timeout per resolved address. Measured 30.0s added to a single POST /training/start with the Hub black-holed, scaling with the number of addresses, surfacing as 503. utils.utils already provides a bounded, memoised hf_unreachable/hf_dns_dead that the training worker subprocess uses one layer down; the route that spawns it did not consult it. The model leg raises inside the existing try, so the except HTTPException handler still runs _resolve_model_snapshot and a cached snapshot pins exactly as before, just without first burning the remote budget. The guard fails open, so an online start is unchanged. New tests fail without each change and pass with it, including a wiring contract that fails if the guard stops being consulted. 443 tests across the preflight, cached-start, provenance, snapshot and streaming suites pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the import-hoist lint honour __all__ re-exports Source lint failed on studio/backend/utils/security/__init__.py: [BLOCKER] HOISTED-IMPORT-UNUSED 'load_scan_target' (['from:utils.security.file_security:load_scan_target']) added but unused That file is a barrel __init__: every name it imports carries "# noqa: F401" and is listed in __all__, and routes/training.py imports load_scan_target from the package rather than the submodule, so the re-export is load-bearing. HOISTED-IMPORT-UNUSED only excluded re-exports that already existed before the change, so adding any new name to an existing barrel was an automatic blocker. The script's own docstring already records this shape as a known false positive for NEW-UNUSED-IMPORT. Skip module-level imports whose bound name appears in __all__, since those are loaded by importers rather than by the module itself. The rule keeps its teeth: a newly added import whose name is not in __all__ still blocks, and a dangling alias still trips UNRESOLVED-NEW. Verified with a four-case mutation matrix plus --self-test, and the full 79-file changed-set lint now reports OVERALL: PASS. * Keep studio off evaluated PEP 604 unions on the 3.9 floor tests/test_python39_compatibility.py failed on Core (HF=4.57.6, default and latest) and Repo tests (CPU): test_no_pep604_unions_are_evaluated_on_the_declared_floor model_config.py:937: Tuple[...] | Tuple[...] (type alias) test_studio_evaluated_unions_do_not_grow 36 studio files now evaluate PEP 604 unions, up from 35 Both offenders came from this branch. model_config.py:937 is a module-level type alias, so it is evaluated at import time and "from __future__ import annotations" cannot defer it; it needs typing.Union, which is what the test message prescribes. hub/utils/dataset_cache.py is the one file that pushed the count to 36, and all five of its unions are function annotations, so the future import is enough there. No behaviour change: the alias is only a cache-key type, and the annotations are unevaluated either way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the dataset re-check budget on the whole selection The budget added for #7853 keyed on dataset + split only, but a dataset cache usability identity has four user-chosen dimensions: dataset, subset, split and streaming. Changing subset or toggling streaming therefore kept the same key, so a genuinely different selection inherited an exhausted budget, skipped the cache-preferring re-check and dropped straight to a remote resolution with datasetKnownCached cleared. Key on all four instead, JSON-encoded so no delimiter can collide with a name containing it and null stays distinguishable from the string "null". cachePath is deliberately still excluded even though the usability identity carries it. It is derived state that moves as a download populates the cache, so feeding it into the key would mint a fresh budget on every poll and re-arm the exact non-terminating loop this module exists to bound. Tests: two new cases cover subset and streaming, and an inverse case pins that nothing outside the selection can refresh the budget. Reverting the key to dataset::split fails exactly the two new cases and no others. * Stop the pinned snapshot path reaching PEFT as the base model name A completed run wrote a machine-local path as the adapter's base model: base_model_name_or_path = '/home/user/.cache/huggingface/hub/ models--unsloth--Llama-3.2-1B-Instruct/snapshots/0123…' where main writes the Hub id. It lands in adapter_config.json, every checkpoint-*/adapter_config.json, the run card, export_metadata.json for merged and GGUF exports, and the model card push_to_hub uploads, none of which resolve on another machine. restore_hf_cache_repo_identity runs in UnslothTrainer.load_model before get_peft_model, so its peft_config branch has nothing to repair yet, and PEFT then derives the name itself: # peft/mapping_func.py new_name = model.__dict__.get("name_or_path", None) peft_config.base_model_name_or_path = new_name PreTrainedModel.__init__ copies config.name_or_path onto the instance, so restoring only config._name_or_path leaves that slot holding the snapshot path. Restore the instance attribute too. It goes through the same guard as the other fields, so an ordinary local model, an unrelated Hub id and a repo mismatch are all still left alone. Tests are behavioural: the existing model identity suite asserts the call site via AST and stays green with this bug present, which is why it was missed. Removing the new line fails 3 of the 6 added tests and none of the other 11. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve cached snapshots that load from a subdirectory unsloth/Spark-TTS-0.5B keeps everything trainable under LLM/: its snapshot root carries only README.md and config.yaml, no config.json and no weights. _resolve_model_snapshot only looked at the root, so a cached copy resolved to None. _apply_model_cache_pin then warned "Cached copy not found on disk; downloading", and offline the start route turned the same None into a 409 hf_model_not_cached_offline for a model that was sitting in the cache. The remote preflight already handles this: it expands load roots through load_scan_target. Only the cached path disagreed. Reuse security_load_subdirs, which already reports ("LLM",) for BiCodec, so both paths share one source of truth, and apply it to the resume and preflight pin lookups too rather than just the fresh-start one. Detection can raise offline or for a gated repo, so a failure degrades to root-only rather than propagating. Tests build the real models--org--name/snapshots/<rev> layout. Making the helper a no-op fails 3 of the 6, and a snapshot with nothing loadable in either place still resolves to None, so the widening cannot mask an empty cache. * Use Optional in the deprecated dataset alias signatures The rewritten alias module annotates two parameters as UploadFile | None and str | None with no postponed annotations, so they evaluate at import on the declared 3.9 floor while the rest of the file already uses Optional[...]. The file was an evaluated-union offender before this change too, so this is not a new break, just keeping the new code consistent with its own convention and off the debt list. * Apply the repo kwarg-spacing formatter to the new code pre-commit.ci flagged the ruff-format-with-kwargs hook on this branch. Running scripts/run_ruff_format.py locally keeps the fix in the authoring commits instead of trailing a separate bot commit. * Say why a resume is refused instead of blaming the checkpoint can_resume_run gained a provenance clause in this rework, so it now returns False for runs whose checkpoint is entirely intact, most realistically once the pinned model snapshot is evicted from the HF cache. The start route answered every False with "Resume checkpoint must belong to a stopped or errored run with complete saved trainer state." pointing the user at trainer state that is fine, while the real cause was the resource gate. exact_resume_resource_requirements already raises with a precise explanation and resource_provenance_allows_resume was discarding it. Add resource_provenance_resume_blocker, which returns that explanation (or None when the run is resumable), define allows_resume in terms of it so the two cannot drift, and use it in the start route when it is the actual cause. The gate itself is unchanged: refusing an unattested resume is deliberate, only the diagnosis was wrong. Tests are behavioural, with one narrow wiring contract for the branch that regressed. Replacing the precise reason with a generic string fails the message test and nothing else. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Honour load subdirs everywhere a cached snapshot is probed Follow-up to 590ac9f22. Resolving cached Spark-TTS/BiCodec snapshots from their LLM/ load root got a start past _resolve_model_snapshot, but three later probes still asked for root-level config.json only, so the same snapshot was accepted in one place and rejected in the next: - provenance.py attested incomplete model provenance, after which exact_resume_resource_requirements refused resume for a snapshot still sitting on disk; - the start preflight reported a valid cached model as having no trainable weights; - the worker cleared model_snapshot_path before loading, falling back to the Hub instead of the selected snapshot. Promote the helper to hub.utils.hf_cache_state.with_load_subdirs so there is one definition rather than four, and use it at every site. _has_trainable_local_weights takes subdirectory roots instead, since it probes directories rather than a filename list. Unchanged for ordinary models: with no load subdirs the helper returns its input, and detection failures still degrade to root-only. * Let the model routes see the same cached snapshots training does Two ways /api/models disagreed with the training resolver, both reachable from resume: _model_config_inspection_target probed only the snapshot root, so a cached Spark-TTS/BiCodec copy answered "Selected cached model is no longer available" for a cache the training resolver accepts, and the exact-snapshot remote-code scan could fail with it. It now uses the shared with_load_subdirs helper. The model_snapshot_repo_id guard used the owner/repo-only regex, so resuming or scanning a namespace-less Hub model such as gpt2 or bert-base-uncased returned 400 before the snapshot could be inspected, even though hub.utils.paths .is_valid_repo_id and the picker both allow the one-segment form. That call site now uses the shared validator. The other five uses of the local regex predate this branch and are left alone. Reverting either fix fails one of the new tests and nothing else. * Get the resume refusal reason all the way to the user 300fe6321 fixed the diagnosis on POST /api/train/start, but the History UI never reaches it. can_resume: false hides the Resume button outright, and in the one window where the server could answer, resume-training-run.ts throws its own "Only stopped or errored runs with a saved checkpoint can be resumed" before sending any request, so no start call is ever made. For a run whose checkpoint is intact and whose pinned snapshot was evicted, that sentence is the wrong diagnosis, which is what 300fe6321 removed server-side. Carry the reason on TrainingRunSummary and prefer it in that guard. The field is optional and defaults to None, so old clients are unaffected, and it is only computed for rows already known unresumable. A checkpoint problem still reports None, leaving the client's existing wording for that case, and a failure inside the gate is swallowed so History still renders. Reverting either half fails one of the new tests: the summary stops carrying the reason, or the client stops preferring it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Honour load subdirs on the resume pin and keep the selected model cache copy Two follow-ups to the cached-snapshot work. The resume branch of _reject_untrainable_model_request still probed the pinned snapshot with a hardcoded (config.json, adapter_config.json) tuple, so a repo that loads from a subdirectory (unsloth/Spark-TTS-0.5B keeps everything trainable under LLM/) resolved to None. Offline that became a 409 hf_model_not_cached_offline for a snapshot present on disk; online it fell through to the remote metadata round trip the pin exists to avoid. It now uses with_load_subdirs like the other cached-snapshot probes. The cache reconciliation effect took the first usable inventory row for the selected repo. A repo can be present under more than one HF cache root, so that silently retargeted an explicit selection at a different copy on the next inventory tick. The selection logic moved to a pure module and now prefers a row whose path matches the current selection, falling back to the first usable row when there is no selection or the selected copy has gone. Tests: test_resume_pin_load_subdirs.py (5) and model-cache-reference-selection.test.ts (8); reverting either fix fails them. * Revert the model cache reference preference; it cannot fire The frontend half of 2c2a95309 assumed the model inventory can return more than one usable row for a repo. It cannot. hub/services/models/cache_inventory.py::_scan_cached_models collects into seen_lower keyed by repo_id.lower() and resolves collisions with _prefer_cache_row, and _dedupe_local_models keys hf_cache rows by (model_id, model_format, format_variant). Both return one row per repo, and a live check with the same repo under four HF cache roots got exactly one row from /api/hub/cached-models and one from /api/hub/local. With a single usable row usable.find(pred) ?? usable[0] is identically usable[0], so the change was a no-op and its tests asserted an array shape the API cannot emit. The resume pin fix in the same commit stands: that one was reproduced against the real unsloth/Spark-TTS-0.5B cache and reverting its hunk restores the 409. * Report the resume refusal that actually happened 300fe6321 and a98e721f4 set out to stop a provenance refusal being reported as a checkpoint problem. They overshot. Both sites asked resource_provenance_resume_blocker whenever can_resume_run said no, but that function refuses for several reasons and the blocker is computed independently of which one fired. initialize_resource_provenance writes {version: 1, status: pending} at the start of every run, so the blocker answers "The model revision used by this run was not attested." for any Hub-model run, including one whose checkpoint is simply missing. That is the more common way to be unresumable, so the change traded one misdiagnosis for another and the new one covered the larger population. Both sites now gate on has_resume_state, the same discriminator can_resume_run short-circuits on: no saved trainer state means the checkpoint is the cause and the client's own wording is correct; an intact checkpoint means a refusal really is provenance's doing and the specific reason is worth surfacing. test_resume_reason_matches_cause.py pins it; reverting either guard fails it. The route contract assertion is over the AST rather than the source text, because a substring search is satisfied by the explanatory comment beside the code and passes with the guard deleted. Two assertions in test_resume_blocked_reason_surfaces.py encoded the old behaviour and are corrected, including the docstring claiming History survives a raising gate: can_resume_run calls the same gate unguarded one line earlier, so that only holds when it short-circuits first. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope the import-hoist __all__ skip to package __init__.py 68bce3934 skipped any name listed in __all__. The comment said "the whole point of a package __init__", but compare() never consulted the path, so the skip applied to every module defining __all__ -- 27 non-package modules exempting 224 names, unsloth/models/_utils.py alone 74. The cost is not just breadth. lint-ci.yml names rename-clash as one of the two bugs this tool exists to catch because ruff and pyflakes miss it, and the skip disabled that detection for any name in __all__. Adding a name there became a one-line, reviewer-invisible way to switch the check off. The self-test I cited did not cover this: --self-test reported ALL PASS with the skip hunk reverted, because every case ran through the "<name>" path placeholder and none defined __all__. Cases may now carry their own path, and three new ones pin the behaviour from both sides -- a re-export in a package __init__ is allowed, the same shape in an ordinary module is still blocked, and a new import absent from __all__ is still blocked in an __init__ too. Deleting the skip fails the first; widening it back fails the second. Removes two genuinely unused imports the correctly-scoped check then found in test_resume_blocked_reason_surfaces.py. * Make deleting a run's artifacts reversible until the row is gone DELETE /runs/{run_id}?delete_artifacts=true is new in this PR; on |
||
Daniel Han
|
ab2717afe0
|
Studio: persistent per-user trust_remote_code approval cache (#6551)
* Studio: persistent per-user trust_remote_code approval cache The consent gate pins each approval to a content fingerprint (sha256 over every repo .py), but nothing was persisted, so the dialog reappeared on every fresh load of the same unchanged repo. This adds an on-disk, per-user approval cache that lets the gate skip the dialog when the same user reloads the same code, while keeping the safety guarantees intact. Two-tier validation, both must hold or the user is re-prompted: - Commit SHA (cheap, one HfApi.model_info().sha, no download): a match means a byte-identical tree to the approved revision, so the scan/download is skipped. - Content fingerprint (authoritative): used whenever the SHA is unavailable (local path / offline) and always recomputed on a SHA miss. A new or edited .py changes both the SHA and the fingerprint, so it is caught in every mode. Safety: - Keyed per subject; one user's approval never auto-runs code for another. - CRITICAL is never stored or honored (guarded on both write and read), so a hand-edited store cannot smuggle in an auto-approval. - The malware (HF unsafe-file) gate stays unconditional. - Fail-safe: a corrupt store, an unresolvable SHA, or any error degrades to "ask again", never to "auto-approve". UNSLOTH_TRC_APPROVAL_CACHE_DISABLE=1 turns the cache off entirely. New module utils/security/remote_code_approvals.py holds the store (studio_root()/security/remote_code_approvals.json, atomic write, 0600, RLock) plus the SHA resolvers. Recording happens at the single gate chokepoint when the caller supplies the matching fingerprint, so subject is just threaded through inference/training/export (orchestrators, routes, workers). The scan endpoint returns already_approved so the frontend can skip the dialog on a cache hit. Tests: new tests/test_trc_approval_cache.py covers cache miss, SHA-match skip, SHA-moved re-scan, new-file re-consent, CRITICAL never cached (write + forged read), disable flag, subject isolation, combined adapter+base key, corrupt store, and no-subject bypass. Full security suite: 101 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: make the approval cache skip only the prompt, never the scan Codex found that the SHA "no-scan" fast path could run untrusted code without re-consent. Removed it; the gate now always re-scans and the cache only seeds the authoritative fingerprint check, so it can skip the dialog but never the scan. - CRITICAL is hard-blocked on every load (the scan always runs), so a hand-edited store that downgrades a CRITICAL repo's severity can no longer auto-run it (P2: do not trust editable severity for SHA approvals). - The fingerprint covers external auto_map repos, so changed third-party code always re-prompts even when the primary commit SHA is unchanged; there is no longer a SHA path that bypasses the fingerprint (P1: external auto_map repos). - resolve_commit_sha is resolved fresh on every call (no memoization), so a repo whose default branch moves after approval re-prompts instead of reusing a stale cached SHA (P1: revalidate mutable Hub SHAs). The SHA is now only a conservative secondary gate: a fresh resolvable SHA must match the approved revision, else the seed is withheld; a None (local/offline) falls back to the fingerprint. - Approvals record the scanner ruleset version (SCAN_RULES_VERSION); the gate ignores approvals from an older ruleset so reclassified bytes are re-scanned and re-shown instead of silently auto-approved (P2: invalidate on scan-policy change). Tests: test_trc_approval_cache.py rewritten around the prompt-skip semantics (unchanged repo still scans; SHA move / changed code / scanner-version bump / disable flag all re-prompt; forged downgraded severity still blocks CRITICAL). 105 passed with test_consent_gate.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim comments to be more succinct * Keep run-owner subject out of persisted config; serialize approval writes Threading subject (the run owner's username / API-key id) into the training config meant _sanitize_db_config persisted it into config_json, which training-history GET returns to any authenticated user, leaking who started a run in multi-user installs. Filter subject alongside the token fields; the worker still receives it from the live config. The approval store's RLock only guards one process, but approvals are recorded from separate inference/export/training subprocesses, so concurrent writers could clobber each other on os.replace and drop an approval (re-prompt). Hold a best-effort cross-process file lock around the read-modify-write. * Fail safe on a malformed approval store A store with the right version but a non-dict shape (e.g. a hand-edited "subjects": []) passed _load()'s check, then lookup chained .get() on a list and raised, breaking every remote-code load until the file was removed. Validate that subjects is a dict in _load(), and tolerate a non-dict per-subject entry in lookup/record/forget, so a corrupt store fails safe (re-prompt) instead. * Keep subject out of the MLX W&B run config _run_mlx_training uploads the whole training config to W&B minus a sensitive set that only listed hf_token/wandb_token/s3_config, so the authenticated subject (username / API-key id) was sent to W&B as run config even though DB history already strips it. Add subject to the W&B-sensitive filter, mirroring training._sanitize_db_config. * Tighten the W&B subject-filter comment --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
|
Daniel Han
|
0533efe3f8
|
Harden model fetching (#6391)
* Harden model fetching: consent gate for trust_remote_code Add a load-path consent gate that scans a model's auto_map repository code before it executes and blocks CRITICAL/HIGH findings unless the user pins approval of that exact code version. Capability detection stays code-free, reading raw config.json instead of AutoConfig. - Scan config.json and tokenizer_config.json auto_map, nested local helpers, and external owner/name--module repos; fail closed on partial downloads. - Gate inference, training, and export workers, including the MLX path and a LoRA's base model, and report requires_trust_remote_code from the raw config so chat and auto-load surface the dialog. - Verify trusted-org auto-enable against the Hub with the request token and key the verdict cache by token; reject local-path and spoofed names. - Add a consent dialog showing the flagged file, line, and surrounding code. - Thread hf_token through the scan and load paths for gated repos. * Address review: token handling, tokenizer/LoRA scan coverage, rollback - Send the HF token for remote-code scans in the POST body, not the URL, so it never lands in a log or browser history. - Collect tokenizer_config.json auto_map files directly instead of relying only on the repo file listing. - Resolve a LoRA's base model for the validate flag and the scan endpoint so the dialog scans the code the workers actually gate. - Pass the request token to the training YAML trusted-org auto-enable. - Resend a previously approved fingerprint when rolling back to a custom-code model after a failed switch. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Consent UX: drop legacy chat toggle, fix decline copy, purge declined downloads The per-model consent dialog is now the single approval path for custom (auto_map) code in chat, so three leftovers from before it existed are removed: - Remove the "Enable custom code" switch from Chat Settings and stop persisting trust_remote_code, so a previously saved blanket-on cannot linger and load a model without going through per-version review. The flag stays as an internal YAML/preset default (e.g. first-party auto-enable); the load path still gates every custom-code load on a fingerprint only the dialog produces. - Reword the decline message and the auto-load toast to describe approving the model's code from the dialog, not a missing settings toggle. - On decline, purge the repo the scan downloaded so untrusted code is not left on disk. A new /api/models/discard-remote-code endpoint deletes only a metadata-only cache entry the scan created; it refuses local paths, loaded models, and any repo with weight files cached, so a model the user already had or pre-downloaded is always left untouched. The frontend only calls it when the scan reported created_by_scan. Adds discard-endpoint tests (delete metadata-only, refuse on weights/gguf, refuse local, no-op when not cached) and a created_by_scan payload assertion. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Export: remove the user-facing trust remote code toggle The Export page kept a "Trust remote code" switch (default on) next to the HF token field. Like chat, custom (auto_map) code should be approved per model through the load-time review dialog, not a persistent blanket switch, so the toggle is removed. The export load path already routes through the same consent dialog: an HF source now starts with trust_remote_code off and only enables it when the user approves the scanned code in the dialog (a local checkpoint the user exported stays trusted by default). With the dialog unreachable and no approval, an HF source loads with trust_remote_code off, which fails closed rather than running unreviewed code. * Block loads of repos with unsafe files using Hugging Face's security scan The trust_remote_code consent gate covers one load-time RCE vector (a repo's auto_map Python). It does not cover the other: a malicious pickle inside a weight file (pytorch_model.bin, *.pkl, *.dat) deserializes during from_pretrained even with trust_remote_code False, so a repo with a normal config plus a poisoned pickle slips past the existing gate. Add a metadata-only malware gate that uses Hugging Face's own scan (picklescan + ClamAV), read via model_info(securityStatus=True).security_repo_status. It never downloads, opens, or unpickles the flagged files; it only reads the Hub's verdict and surfaces the flagged file names. New evaluate_file_security runs unconditionally (independent of trust_remote_code) in every load path (inference, training SFT/MLX, export), blocking the load when a file is flagged unsafe/suspicious/malicious. The /remote-code-scan preflight and the validate endpoint also report the result so the consent dialog opens as a hard block (no override) listing the flagged files, even for a repo with no custom code. Policy: hard block with no user override; fail open when the scan is unavailable (offline/unscanned) so legitimate loads are not broken; no first-party exemption (a poisoned pickle in a compromised trusted repo still blocks); local paths and GGUF are skipped (no Hub scan, non-pickle format). Blocking does not gate on scansDone, since that is often false for clean repos and a file already flagged unsafe is unsafe regardless. Adds test_file_security.py covering the block/allow/fail-open/skip matrix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: scan list-form tokenizer auto_map, gate unsafe files on all load paths Fixes from a 10-reviewer pass on the model-fetching hardening: - The remote-code scanner skipped tokenizer auto_map encoded as a [slow, fast] list (transformers' standard tokenizer shape, e.g. {"AutoTokenizer": ["owner/repo--tokenization_x.Slow", null]}). External tokenizer code in that form was never fetched, scanned, or fingerprinted, so an AutoTokenizer(trust_remote_code=True) load could run it. _auto_map_refs now flattens string, list, and nested values. Adds a regression test. - Compare-mode chat loads and background auto-load only gated on requires_trust_remote_code, so a repo flagged unsafe by the Hub scan but with no custom code skipped the hard-block dialog. Both now also gate on requires_security_review, matching the main chat path. - The /remote-code-scan and /validate routes collapsed a LoRA adapter to its base before the malware scan, so unsafe files in the adapter repo itself were missed in the pre-load review (the workers already scan both). Both routes now run the file-security scan over the adapter and the base. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Require approval for all HIGH remote code, fail closed when unscannable Tighten the load-time security gates based on review: Consent gate - HIGH-severity auto_map code now requires explicit, per-version approval for every repo, including first-party unsloth/nvidia. The org is no longer a blanket bypass: a compromised first-party repo with HIGH code still warrants review. CRITICAL stays a hard block; clean code still loads after the consent prompt. - Fail closed when auto_map code is present but cannot be fully fetched or listed to scan (gated, offline, transient, or a repo-listing failure that could hide an imported helper). We cannot fingerprint code we cannot see, so this is a non-approvable block, retryable once the repo is reachable. - Scan auto_map from every config that can carry one (model, tokenizer, image and feature processor, processor, video processor), not just config.json and tokenizer_config.json, so a custom-processor model is not missed. The file list is the single source of truth in remote_code_scan and is pinned to the transformers filename constants by a guard test. - Distinguish a genuine 404 (config truly absent) from a transient error: only the latter forces a scan, so a repo with no config is correctly a no-op. Malware gate - Scan a remote repo even when its name ends in .gguf; only local paths skip the Hub scan, so a repo cannot dodge the scan by naming itself "*.gguf". - Correct the docstring: a file already flagged unsafe blocks regardless of scansDone; the only fail-open path is an unavailable scan. Coverage - Resolve a remote LoRA adapter's base model (not just local directories) so the base, where the code and weights actually execute, is scanned in validate, the scan route, and the training and export workers. - Gate the embedding training path (FastSentenceTransformer) with the malware and consent checks, matching the other load paths. Tests updated and added for each change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope malware gate to the load-path vector; stop false-blocking first-party models Follow-up hardening from a second review pass + a broad live model matrix (unsloth/* , nvidia/* , third-party, and the eicar malware repo). Malware / unsafe-file gate - Scope the block to the actual RCE vector: a root-level file in a code-executing format. from_pretrained deserializes weight files at the repo ROOT, so a flag is only a load-path pickle vector there. Two exclusions, because neither is loaded: inert formats (safetensors is tensor-only, gguf is non-pickle, configs/text/ images) and files in subdirectories. This keeps eicar blocked (its *.pkl/*.dat/ eicar_test_file sit at the repo root) while no longer false-blocking legitimate first-party repos: nvidia/Nemotron-H-8B-Base-8K ships root safetensors plus NeMo pickle checkpoints under nemo/ that the loader never touches, and the Hub flags both; the gate previously hard-blocked it. - Unknown / future non-"safe" levels now fail closed (block) instead of being silently allowed, so Hub schema drift cannot introduce a bypass; in-progress ("pending"/"scanning"/"error") levels stay non-blocking to avoid false blocks. Consent gate - Ignore a STALE own-repo auto_map target that is absent from the repo listing (an older config pointing at a file the repo no longer ships) instead of failing the whole repo closed as unscannable. The present .py are still fully scanned, which is the stronger coverage, and a file that is not there cannot execute. This unblocks first-party models like unsloth/PaddleOCR-VL (its tokenizer_config.json names processing_ppocrvl.py while the repo ships processing_paddleocr_vl.py). A referenced .py that IS present but cannot be fetched, and a repo-listing failure, still fail closed. Remote LoRA base resolution - Distinguish a genuine 404 (not a LoRA / repo absent -> None) from a transient error: the transient case is retried once, then logged as a WARNING (a missed base is scanned by neither gate) rather than silently skipped. Discard endpoint - Treat .onnx and .ckpt as weights so a repo whose only heavy artifact is one of those is never eligible for the declined-download purge. Tests added for each: load-path scoping (safetensors/subdir/Nemotron-H shapes, unknown-level fail-closed, pending non-block), stale own-repo auto_map ref, remote LoRA transient retry, and the empty-config-list (all-404 -> []) semantics. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make LoRA-base transient-warning test robust to logging backend Assert on the logger object directly instead of capsys, so the test does not depend on whether the real structlog logger or the module-stub logger is active (which varies with test collection order). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Allow a repo with auto_map but no executable code (e.g. GGUF) instead of blocking A config can declare an auto_map yet the repo ship NO executable .py -- most commonly a GGUF repo whose config.json carries an auto_map copied from the original model (e.g. unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF references modeling_decilm.py, which the GGUF-only repo does not contain). A GGUF model loads through llama.cpp, which never executes auto_map, and transformers cannot run a file that is not present, so there is nothing to scan and trust_remote_code is a no-op. The fail-closed change treated this empty result the same as "code is present but we could not fetch it" and hard-blocked the load. Distinguish the two: repo_remote_code_files now RAISES RemoteCodeUnscannable when code is present but cannot be fully fetched or listed (offline / gated / transient / a present .py that 404s / a listing failure), and returns an empty dict only when the listing succeeded and the repo genuinely ships no executable .py. The consent gate blocks on the exception (fail closed) and allows the empty case as a no-op. Real unscannable code still hard-blocks; eicar and CRITICAL/HIGH custom code are unaffected. Verified against all 37 unsloth/*Nemotron* models (two GGUF repos were false-blocked, now load) and the existing matrix (eicar still blocks; DeepSeek-OCR / NVLM-D-72B still prompt approvable consent). Tests updated to expect the raise for unscannable cases and added for the no-executable-code no-op. * Ignore vestigial auto_map in GGUF repos (llama.cpp never runs it) A GGUF repo's config.json is often copied verbatim from the original transformers model, auto_map and all, but a GGUF load goes through llama.cpp which never executes auto_map, so the config is inert. Treat a direct .gguf reference, and a repo that ships .gguf weights with no .safetensors, as having no remote code so the consent flow is never triggered. A mixed repo with both .gguf and .safetensors is still gated, since the safetensors variant would load through transformers where auto_map does run. The check sits behind the existing auto_map-present gate so normal models pay no extra repo listing. * Add scanner-result copy to the remote-code consent dialog Make the consent dialog state the scan outcome in plain language for every model. When the static scan finds nothing, reassure the user with 'Our automatic scanner did not flag any worrying files, but please double check.' (shown only for the clean, approvable case). When the scan flags custom code or unsafe files, label the list with 'Our automatic scanner flagged issues including:'. The Hugging Face attribution for unsafe files stays in the dialog description. * Close GGUF-suffix consent bypass for repo ids ending in .gguf The .gguf short-circuit in _config_has_auto_map skipped the scan for any model name ending in .gguf, including a bare two-segment repo id like 'evil/model.gguf'. Such a repo can still ship safetensors plus auto_map Python that transformers would execute, so skipping the scan was an asymmetric bypass (file_security already scans those repos). Restrict the short-circuit to genuine direct GGUF file references via _is_direct_gguf_file_ref: a local .gguf path, or a remote repo_id plus filename (three or more segments). A two-segment repo id named *.gguf now falls through to the config scan and _is_gguf_repo file inspection, so it only skips consent when it actually ships .gguf weights and no safetensors. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Align consent dialog body with the title and fix narrow-width overflow The scan results (the 'Our automatic scanner...' label, finding/unsafe cards, and the clean-scan reassurance) sat at the dialog's left padding while the title and description were indented past the status icon, so the body did not line up under the description. Move the title, description and results into one column to the right of the icon so they share a left edge, and let that column fill its width so the description no longer wraps early. Also stop a wide code snippet from pushing the dialog off-screen on narrow viewports: AlertDialogHeader is a grid with place-items-center, which sized the content row to its content; give the row w-full so it fills the track, and add min-w-0 down the results chain so the snippet scrolls inside its card instead of widening the dialog. Verified aligned and contained from mobile portrait through ultrawide. * Treat a repo as GGUF-only only when it ships no transformers weights _is_gguf_repo excluded only .safetensors, so a repo with a .gguf and a pytorch_model.bin (or .pt/.pth/.h5/.msgpack/.onnx/.ckpt) and no safetensors was treated as GGUF-only and skipped the consent scan, even though transformers can load that weight set and execute the repo's auto_map code. Require the absence of ANY transformers-loadable weight before treating the repo as a llama.cpp-only GGUF load. A genuine GGUF-only repo (only .gguf) is still inert; a mixed repo with any pickle or safetensors weight is gated. Adds a regression test across all the non-safetensors weight formats. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Block flagged subdir weight shards referenced by a root index The malware gate treated every subdirectory file as non-loadable, but from_pretrained deserializes a subdir shard a root index references (pytorch_model.bin.index.json -> shards/...-00001-of-00002.bin). Read the root weight indexes and block a flagged subdir pickle the weight_map points at; a flagged subdir pickle no index lists (NeMo nemo/*.distcp) stays non-blocking, and an inconclusive index lookup fails closed. * Pass hf_token to the export checkpoint load ExportBackend.load_checkpoint scanned with hf_token in the worker but loaded the weights unauthenticated, so a gated/private checkpoint passed preflight then 401'd at from_pretrained. Add hf_token to load_checkpoint and forward token to every from_pretrained branch; the worker passes the command's hf_token. * Scope created_by_scan to every HF cache the discard searches created_by_scan used get_cache_path (active HF_HUB_CACHE only) while /discard-remote-code deletes across active, legacy, and default caches. A repo the user already had in a legacy/default cache was marked scan-created and deleted on decline. Check all three caches for the repo dir before declaring the scan created it. * Scan the full .py closure of external auto_map repos An auto_map cross-repo ref (owner/name--module.Class) only had its entry file downloaded, but transformers also fetches that file's relative imports from the same repo, so a dangerous helper.py was left outside the scanned fingerprint. List each external repo's .py and scan the whole set (plus the referenced entry files); fail closed if the repo cannot be listed or fetched. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fail closed when a weight index cannot be fully read _indexed_shard_paths treated a partial result as definitive: if one weight index read cleanly but another failed transiently, it returned the shard paths it did see. A flagged subdirectory pickle listed only by the index we could not read would then be classed as "not a load input" and skipped, re-opening the very fail-open this guard was added to close. Return None whenever any index read is inconclusive, even if another read cleanly, so the caller blocks the already-flagged subdir pickle. A repo that ships no index files raises EntryNotFoundError for each (never inconclusive) and still returns an empty set. * Match cached repos case-insensitively in the created_by_scan guard _repo_in_any_hf_cache resolved casing only against the active cache and then probed every cache with an exact directory name. A case-variant already present in a legacy or default cache (models--Unsloth--Foo for a scan of unsloth/foo) was missed, so the repo was marked created_by_scan and deleted on decline -- but discard_remote_code_download deletes case-insensitively, so that delete would hit the user's pre-existing cache entry. Detect case-insensitively too, mirroring the deletion path. * Skip remote-code and security review for selected GGUF variants validate_model ran the trust_remote_code and Hugging Face security-scan preflight against the repo even when the selected artifact is a .gguf. A GGUF loads through llama.cpp, which never executes the repo's auto_map Python and never deserializes root pickle weights, so repo-level Transformers artifacts (a config.json with auto_map, or an unsafe pytorch_model.bin next to the .gguf in a mixed repo) are inert for that load. Gating the GGUF on them is a false positive. Run both preflights only for non-GGUF loads. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope the malware gate to actual load roots and serialized files Two fixes to evaluate_file_security so it neither misses a load-path pickle nor false-blocks an inert file: - Honor subdirectory load roots. Spark-TTS / BiCodec call from_pretrained on the snapshot's LLM subdirectory, so a flagged pickle directly under it is a root-level load artifact there. A new load_subdirs parameter (set from the model's audio type via security_load_subdirs) reclassifies those files relative to the load root and looks for weight indexes under it, so a flagged shard in that subdir is no longer skipped as "not root-level". - Exempt source files. A root .py is never deserialized by from_pretrained; executable repo code runs only through auto_map, which the remote-code consent gate scans. Flagging a Python helper here would false-block a repo that merely ships a build or train script. * Scan a LoRA adapter and base as one consent unit, and gate MEDIUM code A LoRA load runs both the adapter's and the base's repo code. The consent gate scanned them separately and pinned one fingerprint per repo, so an adapter that shipped its own auto_map code was either never shown in the dialog (which only saw the base) or impossible to approve with the base's fingerprint. evaluate_remote_code_consent_for_targets now scans all of a load's repos as a single combined unit and pins ONE fingerprint over the union of their code, so approving the load approves every repo's code together. evaluate_remote_code_consent becomes a thin single-target wrapper, and an unscannable target fails the whole load closed. Also gate MEDIUM findings: like HIGH they now block pending pinned approval, so a direct API caller cannot run flagged code by setting trust_remote_code=True without consenting. Only a clean scan loads without a fingerprint. * Preflight a LoRA load's adapter and base as one combined consent scan scan_model_remote_code rewrote a LoRA adapter to its base and scanned only the base for remote code, so the dialog never surfaced an adapter's own auto_map code. Scan the adapter and base together through preflight_remote_code_consent_for_targets, which pins one combined fingerprint the worker gate accepts. The malware preflight is also scoped to each target's load subdirectories. * Apply combined consent and subdir-aware malware scan in load workers Each load worker (inference, export, training) evaluated remote-code consent once per target with a single shared fingerprint, so a LoRA adapter that ships its own auto_map code could not be approved by the base's fingerprint. They now scan the adapter and base together via evaluate_remote_code_consent_for_targets, which pins one combined fingerprint over the union of their code. The malware scan in each worker is also scoped to the model's load subdirectories so a flagged pickle under a from_pretrained load subdir is not missed. * Report a consistent trust_remote_code requirement after a model loads validate_model reports requires_trust_remote_code from the YAML default OR the raw auto_map, but the load, already-loaded, and status responses reported only the YAML default. A custom-code model approved and loaded via auto_map was then reported as not requiring trust_remote_code, so the frontend stored false and a later retry or rollback sent trust_remote_code=false and failed. A shared resolver reports the same requirement for a loaded model (a value stored at load time, else the trust_remote_code the load used, else the YAML default, else the raw auto_map check), and the load response persists it so the status and already-loaded paths stay consistent. The selected-GGUF security review is also scoped to the model's load subdirectories. * Run the consent gate on training resume and for YAML-only trust_remote_code Three frontend gaps left a model loading without the trust_remote_code it needs: - The shared consent helper returned early when the scan found no auto_map and no unsafe files, dropping a requirement that comes from a model's Studio YAML default (e.g. GLM-4.7-Flash). It now grants the caller's requirement with an empty pin instead of sending trust_remote_code=false. - Resume-from-history called startTraining directly with no consent gate, so a resumed run whose model needs custom code (or an old run with no approved fingerprint) hit the worker block with no dialog. It now runs the same gate as a fresh start. - HF export passed requiresTrustRemoteCode=false for every HF source, so a YAML-only model could not flip the flag before export. It now signals the requirement for HF sources. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover both LoRA repos in validate, report GGUF as inert, purge all declined repos Three follow-on gaps from the combined adapter+base consent work: - validate_model resolved requires_trust_remote_code from the base alone, so a LoRA adapter that ships its OWN auto_map code (with a plain base) was reported as not needing trust_remote_code and the consent dialog never opened. It now checks the [adapter, base] target set, matching the scan route and the workers (which already gate both) and the security review already running over both. - The already-loaded, loaded, and status responses for a selected GGUF reported requires_trust_remote_code from the model's YAML default. A GGUF loads through llama.cpp, which never executes the repo's auto_map Python, so the requirement is inert for that load. They now report False, matching validate_model (which already skips both gates for GGUF) so a status refresh cannot flip the flag back on. - The remote-code scan downloads both the adapter's and the base's config, but created_by_scan tracked only the primary, so a base the scan was first to pull into the cache was left on disk when the user declined. The scan now reports scan_created_repos (every repo it newly cached) and the decline cleanup purges each; created_by_scan stays for older clients. The frontend falls back to the primary flag when the list is absent. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scan the repo the load fetches, purge external code on decline, harden consent pins Six follow-on hardening fixes from a fresh review pass over the gate: - The malware gate scanned the literal "Spark-TTS-0.5B/LLM" alias, but the trainer downloads it as unsloth/Spark-TTS-0.5B and loads LLM/, so the alias 404'd and failed open, missing a flagged LLM/ pickle. evaluate_file_security now resolves the alias to the repo the loader fetches and scans LLM/ as a load root. - security_load_subdirs relied only on tokenizer detection, which fails on an unresolved alias or offline; it now also honors the Studio YAML audio_type default, so a BiCodec LLM/ load root is not missed. - The remote-code scan downloads external auto_map repos (owner/name--module.Class), but the decline cleanup tracked only the model/adapter/base, leaving the external untrusted code cached. The scan now enumerates external auto_map repos and reports the ones it created in scan_created_repos, so a decline purges them too. - External auto_map refs failed the whole load closed on a stale or mis-derived dotted ref (sub.mod.py vs the real sub/mod.py) even though the actual file was present and scanned. They now drop such refs when the repo listing is real, exactly like the own-repo path; an empty/incomplete listing still fetches and fails closed. - The combined consent fingerprint keyed code by the raw target string, so the scan endpoint's canonicalized casing and a worker's raw user input produced different pins for identical code, rejecting a valid approval. Hub repo ids are now folded to lowercase in the key (local paths stay case-sensitive), so the pin tracks the code. - Export threaded hf_token into the weight load but not into detect_audio_type / is_vision_model, so a gated multimodal base 404'd in detection and fell through to the text loader. Both probes now use the same token. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Thread the token through check-vision and guard the gate's parallel sites The /check-vision endpoint classified a model without the hf_token, so a gated or private vision model 404'd in the probe and was reported as a plain text model -- the same dropped-token shape as the export probes, at a sibling site. It now passes the token like the neighboring /check-embedding endpoint. Add deterministic consistency guards (tests/test_security_gate_consistency.py) that enumerate the gate's parallel sites mechanically instead of relying on a review to spot a missed sibling: every is_vision_model / is_embedding_model / detect_audio_type caller under routes/ and core/ must thread the token, every GGUF response must report trust_remote_code via the resolver or False (never the raw YAML default), and every load worker that runs the malware or consent gate must resolve the LoRA base. A new site that drops the token or mis-reports the requirement now fails CI directly. * Narrow the LLM alias rewrite and make audio detection token-aware Three fixes from the confirmatory review, one a regression from the previous round: - _load_scan_target rewrote EVERY remote repo ending in "/LLM" to unsloth/<parent>, so a real third-party repo named "<owner>/LLM" was scanned as unsloth/<owner> while the loader still fetched the real repo -- a fail-open hole introduced when the Spark-TTS alias handling was added. It now rewrites only a registry-known bicodec alias; every other "/LLM" repo is scanned as itself. - detect_audio_type cached results under the bare model name, so an unauthenticated probe of a gated/private repo cached None and poisoned a later authenticated call with the token. The cache is now keyed by (normalized_name, token_fingerprint), matching the vision cache. - The training fallback /check-vision call dropped the hf_token, misclassifying a gated/private VLM when the config endpoint failed. It now passes the token, like the getModelConfig call it falls back from; checkEmbeddingModel takes the token too. Extend the consistency guards: every capability cache must be keyed by a tuple including the token, so a cache re-declared as Dict[str, ...] fails CI. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Document the broad .py scan as deliberate and enforce it with a test The remote-code scanner scans every .py in a repo once an auto_map exists, not just the auto_map entry's static import closure. This is intentional: the entry module can reach a sibling via an absolute import, importlib, or exec, none of which a static relative-import closure follows, so closure-only scanning would be a real bypass of a load-time RCE gate. The broad scan never under-scans; the cost is that an unrelated benign script can over-block, which is the safe failure direction (HIGH stays approvable; only CRITICAL hard-blocks). Spell this out at both the local and remote scan sites so the choice reads as deliberate, and add a test asserting an unrelated, never-imported .py is still scanned -- so a future narrowing to the static closure fails CI. * Purge a declined remote LoRA adapter the scan downloaded scan_model_remote_code probed the created-by-scan state AFTER resolving the base, but get_base_model_from_lora_identifier downloads a remote adapter's own adapter_config.json, so the adapter looked already-cached and was dropped from scan_created_repos. On decline the adapter -- including the auto_map .py the preflight fetched -- was left on disk, defeating the "untrusted code is not left on disk" guarantee for the adapter itself. Snapshot the primary's cache state BEFORE base resolution and use it when marking the adapter scan-created; on any probe error treat it as pre-existing so a decline never deletes it. The base and external repos are unaffected (their configs are not downloaded before their own probe). Add a test that models the mid-scan download side effect, which the prior static-stub tests did not. * Clear remote-code approval when the training model changes Switching the training model from an approved custom-code model to a clean one kept the previous model's trust_remote_code=true and approved fingerprint in the store: setSelectedModel reset visionImageSize on a true switch but not the remote-code approval. The clean model then trained with trust_remote_code=true, which bypasses the compiler and disables fused cross-entropy. Reset trustRemoteCode and approvedRemoteCodeFingerprint on a true model switch. The new model's own YAML default is re-applied by loadAndApplyModelDefaults, and a custom-code model still re-opens the consent dialog before training starts, so the only change is that a clean model no longer inherits a stale approval. * Trim verbose comments across the model-fetching hardening changes Condense the explanatory comments and docstrings introduced across the trust_remote_code consent gate, the malware/unsafe-file gate, the remote-code scanner, the load workers, the model routes, and the security frontend into fewer, tighter lines while preserving every security rationale (fail-open vs fail-closed direction, the deliberate broad-scan anti-bypass note, the empty-vs-unscannable distinction, stale-ref handling, and the alias-rewrite spoof guard). Comments and docstrings only. No code, logic, identifiers, or test behaviour changed; verified comment-only via the AST/TypeScript checker (40/40), with the backend test suite and frontend tsc green. * Do not cache transient audio-detection failures detect_audio_type cached _detect_audio_from_tokenizer's result unconditionally, so a transient read failure (network error or 5xx, returned as None) poisoned the cache and the later successful probe never ran. Mirror the vision cache: _detect_audio_from_tokenizer now returns (audio_type, definitive) and the caller caches only definitive results. A read that succeeds with no audio tokens, or clean 404s for every tokenizer path, stays a cacheable None; only a genuine transient failure (connection error, timeout, 5xx, malformed body) skips the cache so the next call retries. --------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |