mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-24 16:23:51 +00:00
12 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
sts-change
|
8d6e969378
|
Studio: never pick a macOS AppleDouble sidecar as a GGUF (#8919)
--------- Co-authored-by: Lyxot <longyixing331@gmail.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
Eyera
|
0b147fdd2a
|
fix(studio): harden training setup, lifecycle, and audio loading (#8103)
* fix(studio): harden training setup and model loading Scan every model load root before approving remote code, and pin third-party codec sources to verified revisions. Align dataset option validation across the UI and backend, preserve manual drafts, and include edits in training start identity. Require job-scoped stop requests and retain bounded early-cancel tombstones without unsafe eviction. * fix(studio): harden training lifecycle and audio loading Preserve early start cancellations with bounded tombstones and explicit capacity handling for concurrent requests. Pin and verify third-party audio sources and codec artifacts with safe archive extraction and offline cache migration. Keep automatic evaluation data separate from every split included in a combined training instruction. Keep training summaries and GPU progress state aligned with the active configuration through public feature exports. Make route and lifecycle tests deterministic by isolating Hub reachability and executor behavior. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): correct completion masking and offline training Apply Alpaca response markers safely when completion-only training is enabled. Skip optional runtime dependency installs while Studio is offline. Limit causal-conv1d hooks to recognized model families without dropping supported architectures. Disable known-broken TileLang dispatch when offline repair is unavailable. Use cached GGUF and model-size metadata without offline Hub retries. * fix(studio): resolve causal conv kernels from model configs Detect causal-conv1d requirements from resolved model architectures before loading model code. Keep name matching as a fallback while excluding unrelated renamed checkpoints. * fix(studio): preserve registered training cancellation Allow registered start cancellations to reclaim the oldest expiring tombstone when unknown cancellation capacity is full. Preserve the hard capacity limit and 429 response for unregistered request IDs. * Fix reset job scoping, DAC fast path, scan target and subset splits for PR #8103 - /api/train/reset: an unscoped reset could force-terminate a run mid-cancel. The guard now refuses to touch a live run it cannot prove it owns. The field stays optional so pre-rework clients, which POST /reset with no body, keep working. - ensure_dac_speech_weights: install the download into the pinned destination, so later loads hit the fast path instead of re-downloading and re-hashing 295 MB under the install lock. - _requires_security_review_for_model: apply the same load_scan_target alias normalization the sibling remote-code check gained, else the Spark-TTS alias 404s and fails open to "no review needed". - _purge_package_bytecode: best effort. It runs without the install lock over a cache shared by the inference and training workers; 7 of 8 concurrent imports died on it. Also moved inside the try so a failure cannot strand the cache dir on sys.path, and snapshot sys.modules before the origin audit. - commitSubset: clear the backing splits too, else the render-phase draft sync reads the previous subset's split back into the boxes it just reset. - Fix two tests that fail on the branch: the DAC assertion pinned the old return value, and the causal-conv1d assertion matched call formatting. * Make the SSM runtime tests Windows aware ensure_ssm_runtime deliberately skips causal-conv1d on win32 (no prebuilt wheel), so the two install-order assertions only hold off Windows. Caught on a real windows-latest runner. * Update two frontend source contracts the branch moved - captureTrainingStartInputs now delegates to createTrainingStartInputIdentity, so the normalize/flags assertions belong against training-start-inputs.ts. - resetTraining takes a RequiredTrainingJobScope and always sends the body, which is stronger than the hasScope branch the contract pinned. Both fail on the branch today; caught by tests/studio, which the studio backend job does not cover. * Fix unscoped reset compat, pyc purge fail-open and DAC fallback for PR #8103 Corrects four things in my earlier commits on this branch. Unscoped /api/train/reset returned "superseded" (HTTP 200) for a live run. The pre-rework cancel dialog chains stopTrainingRun then a bodyless reset, so an older client read that 200 as success and cleared its UI while training kept running. Return "active" (409) instead: same answer a live run already gives, and one those clients already handle. It still never force-terminates, so a bodyless reset landing between current_job_id being set and _cancel_requested being cleared cannot kill the run that just started. The bytecode purge was made best-effort, but it is the only thing stopping a stale or planted .pyc shadowing a verified .py: the manifest skips __pycache__ and the origin audit reads __file__, which still names the .py. Tolerate only FileNotFoundError, the real concurrent-purge race, and let PermissionError fail the load again. The DAC fast path copies 295 MB inside the hub cache with only Timeout caught, so a full disk turned a hash-verified download into a hard failure. Fall back to the verified hub path on OSError. commitSubset cleared both splits, but setDatasetSubset already does that; the extra setDatasetSplit(null) only cost a runDatasetCheck against an assumed "train" split. * Restore the eval split reset and the cancelled-run dismiss for PR #8103 Two corrections to 4e5389c21. commitSubset: I removed setDatasetEvalSplit(null) because setDatasetSubset already nulls datasetEvalSplit. It does, but it never resets evalSteps, and setDatasetEvalSplit is not a plain setter: it zeroes evalSteps and runs streamingCompatiblePatch. Without it, changing the subset left evaluation armed with no split, which routes/training.py rejects with 422 once streaming is on, and which silently auto-detects an eval split otherwise. Restored. setDatasetSplit(null) stays out, since its only unique effect was a runDatasetCheck against an assumed "train". Unscoped reset: returning "active" for every live run was too broad. The pre-rework cancel dialog only dismisses after stopTrainingRun succeeded, so _cancel_requested is already set and clearing the UI is right; 409 there just wedged the overlay behind a "Training still active" toast. Now 409 only when no stop was requested, which is the stale-tab case the change was for. Still no force_terminate on an unscoped reset. Also covers the __pycache__ branch of the purge, which is the route a planted .pyc actually takes; the existing test only reached the top-level .pyc loop. * Keep live start cancellations and fall back on a full disk for PR #8103 Two fixes plus the regression tests the earlier lifecycle fixes shipped without. Cancelling the active start at tombstone capacity reclaimed a slot by deleting the soonest-expiring entry. Expired ones are already pruned a few lines above, so that entry was always live, and dropping it let a delayed /start spawn the job it had cancelled. Reserve capacity instead: only the owner of the active start reaches that branch and there is at most one, so the table lands at cap + 1 rather than forgetting a cancellation. Unregistered ids still hit the hard cap. ensure_dac_speech_weights migrated a pre-existing legacy file with an unguarded copy, so a hub cache that cannot absorb a second 295 MB copy failed weights that had already passed size and sha256. Same fallback the download branch below it already uses. Tests: renamed and local SSM checkpoints resolving from config rather than name, the owner of an active start staying cancellable at capacity, live cancellations surviving that cancel, and the full-disk legacy DAC fallback. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Opt the pinned source checkouts into Git long paths for PR #8103 Cross-OS CI caught 14 failures on windows-latest, all of them the pinned checkout dying with "error: unable to write file ...: Filename too long". Git for Windows still enforces MAX_PATH unless told otherwise, and the cache nests a 40-char revision, a staging dir and .git/objects under the studio home. A venv-inferred home already measures about 253 of the 260 characters, so a slightly longer user or install path fails on a normal Windows machine, not just under the deeper pytest tmp dir. Passed per invocation with -c so no user or system Git config is touched, and it is a no-op off Windows. * Bound pending cancels, reach legacy DAC weights, and delete read-only checkouts for PR #8103 Three fixes, two of them on my own previous commit. Moving the owner cancel from evict-oldest to a one-slot overshoot also changed what the hardcoded reclaim_capacity=True on the pending non-owner branch did: it used to evict, so the table stayed at the cap, and it started overshooting instead. Start plus cancel could then be repeated to grow it without bound (1224 entries against a cap of 1024). That branch now takes the plain reservation, so the extra slot belongs to the owner of the active run alone. The DAC legacy fallback sat behind destination.parent.mkdir() and the install lock, both of which need a writable cache, so a read-only or full hub cache raised before weights we can already verify were ever looked at. Fall back at both points, and only to an artifact that passes the same size and sha256 check. Cross-OS CI then caught replacing a pinned checkout failing on windows-latest with WinError 5: Git marks .git/objects read-only and Windows will not delete a read-only file, so any repair or revision change died there. Clear the attribute and retry, only when the path is genuinely not writable, so an open handle still surfaces. * Delete the cached checkout the Windows-safe way in the migration test for PR #8103 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <moonshotaisubstack@gmail.com> |
||
|
Eyera
|
b52d3b56ed
|
feat(studio): rework train page setup flow (#7633)
* feat(studio): rework training setup and cache handling * Rework train page resource selection * Improve train page workflows and resource selection * Fix train page picker behavior * Fix stale training errors after token updates * Fix training eval steps regression * Fix shared token use when resuming training * Harden train page resource workflows Unify model and dataset picker behavior, training start guards, and token handling. Validate cache provenance for local models, downloaded resources, and processed datasets. Prevent stale preview and history state while keeping the train page modular. * Fix training method race and test isolation * Restore training upload limit exports * Fix training config formatting contract * fix(studio): harden training model selection and preflight * fix training resume, cache, and lifecycle reliability * Fix training stop watchdog path stub * Fix training start state and train page UI * Unify train page selection controls * Fix train page validation, localization, and paging * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Clean up train locales and restore recipes link * fix(studio): correct training model selection and preflight Preserve freeform local model paths, reject remote GGUF-only repositories, and correct the training start snapshot contract. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): harden training resource selection Preserve local dataset path intent, localize setup-change errors, and remove unused picker tour hooks. Refresh model picker and PDF recipe contracts for the refactored training flow. * fix(studio): remove train tab scrolling * fix(studio): harden training resource preflight Reject missing local models and binary adapter artifacts. Preserve selected model and dataset cache pins through preflight and QLoRA loading. Detect cache path changes before training starts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix train page validation and cache handling Keep transient model checks nonblocking and bind cached models and datasets to their selected snapshots. Improve Hub auth errors, retries, task search, streaming consistency, locale formatting, reset handling, payload mapping, partial inventory filtering, theme accents, and regression coverage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): stabilize train page pickers Unify Hub validation, local inventory, picker styling, and dataset names. Localize training feedback and explain streaming modality changes. Add unit, contract, and cross-browser picker coverage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): align cached scans and variant state purge Scan selected cached snapshots during training security preflight. Purge manifests and cancel markers using their stored GGUF variant. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): harden train cache, consent, and UI Fix cache cleanup and cross-platform model path handling. Align resume and adapter consent scans with pinned load targets. Localize dataset flows and refine navigation, state cleanup, and theming. * fix(studio): harden training resource selection Preserve cached model configuration HTTP errors so stale cache references return their intended 404 responses. Wait for device inventory settlement before locking inferred picker tabs, while keeping known device items visible during scans and retries. Apply modality name heuristics only to the final model path component across supported platforms. Expose full model and dataset identities on truncated picker triggers. Add regression coverage for picker settlement, retry behavior, and modality inference. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix train picker flow and resume safety Scan the exact cached model snapshot before resume consent and preserve the actual repository identity. Keep cold pickers on Device until inventory settles, cap automatic Hub pagination, and provide a stable Load more action. Improve train control and history card accessibility, remove duplicate token labeling and dead styling, and correct picker spacing. Normalize Windows relative model paths and add coverage for resume pins, picker policy, pagination, path identity, and accessibility contracts. Fix the reported formatting and import ordering issues. * Fix cross-platform train dataset path detection Use the shared local path detector so persisted Windows, Unix, UNC, and home-relative dataset references do not render Hugging Face-only controls. Recognize Arrow dataset files and add regression coverage for supported local path formats and Hub repository identifiers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix training start recovery and dataset selection Reconcile failed fresh and resumed training starts with backend status before reporting an error. Treat the selected dataset source as authoritative so filename-like Hub repositories retain subset and split controls. Align dataset picker sorting and remove unused advisor template state with a persisted-state migration. Add regression coverage for transport recovery and Hub dataset selection. * Fix train picker validation and start recovery Update the resume training contract test for the asynchronous failure handler. Filter invalid Hugging Face search results and block invalid saved model and dataset selections. Restrict local dataset selection to inventoried application paths. Load training YAML through a bounded native file picker while keeping the browser fallback. Only recover uncertain training starts after network or response parsing failures, and match the active job to its request identifier. Treat cached full precision models as ready for QLoRA without showing a false download warning. Add regression coverage for the picker, config import, and training recovery behavior. * Harden training starts and config imports Track training start request IDs from preflight through pending, accepted, and rejected states so retries remain idempotent. Disable automatic retries for start posts and reconcile ambiguous client responses with server request status. Clear terminal training status on reset so later runs do not inherit stale state. Enforce the 1 MiB YAML limit for browser imports and keep read errors specific to the selected file label. Remove unused dataset picker translation entries across all locales. Add focused tests for request idempotency, start recovery, config limits, and picker contracts. * Harden train picker feedback and start handling Show explicit empty states for device dataset searches and invalid Hub model or dataset queries while keeping pagination sentinels mounted. Run the train picker Playwright suite across Chromium, Firefox, and WebKit in Windows CI. Keep training start outcomes owned by backend spawn completion so cancelled handlers cannot mark active jobs as rejected. Correct the Train section heading hierarchy and format the branch-owned native file dialog test. Add picker contracts, browser assertions, and cancellation race coverage for the updated behavior. * Fix training setup races and configuration state Keep hardware-based method selection within model-default loading so starts cannot race changing defaults. Clear start errors only after deliberate configuration or token edits while background reconciliation stays silent. Generate training request IDs safely for LAN HTTP access before acquiring the runtime lease. Filter persisted state, sanitize invalid methods, and guard preview metadata. Localize config size failures, use the toast wrapper, and remove obsolete training files. Add regression coverage for the updated start, persistence, and import behavior. * Refine training configuration and picker workflows Split the training configuration store and parameter panel into focused persistence, policy, LoRA, hyperparameter, memory, and MLX modules. Run picker coverage on platform-appropriate browser engines and parameterize the unmanaged dataset path. Surface configuration changes when streaming is disabled and clear stale dataset modality state after failed probes. Consolidate dataset format and AI-assisted mapping on hub routes with header-based Hugging Face tokens. Hold local path actions until model inventory loading settles. * Fix train picker state and preflight regressions Open online pickers on the Hub before inventory settles and keep the inferred tab stable for the session. Preserve parameter and VRAM metadata for pinned selected models. Reject failed backend starts once and cancel automatic modality corrections without showing a destructive setup error. Update persistence contracts to follow the extracted training configuration module. Move shared parameter option styling out of the React component module to preserve Fast Refresh. Add regression coverage for picker tab behavior, pinned model metadata, persistence contracts, and start rejection handling. * fix(train): harden model selection and start feedback Preserve cache metadata and model capabilities when freeform local paths resolve to discovered models. Return stable training start error codes through direct and recovered requests, then localize Hugging Face preflight failures across every supported locale. Show whether submitted advanced settings are default or non-default in Simple mode, and migrate the parameter mode preference to the standard storage key. Restore absent-key migration semantics and update the embedding security test for the extracted gate helper. Add focused coverage for structured errors, advanced setting summaries, start recovery, and security gates. * fix(train): harden picker validation and focus Align Hub ID validation with Hugging Face rules while accepting valid underscore boundaries. Reject persisted path-shaped datasets from Hugging Face-specific controls. Reuse model identity normalization for cache path comparisons. Move picker search focus into the popover focus lifecycle. Update source contract tests for extracted model selection and structured resume errors. * fix(train): refine picker behavior and dataset structure Wait for device inventory before locking inferred picker tabs while preserving explicit user choices. Align Hub resource ID validation with backend rules for leading and trailing underscores. Support ArrowDown navigation from picker tabs into available options. Split dataset selection, uploads, streaming settings, and inventory refresh logic into focused modules. * fix(studio): stabilize train picker contracts and state Update dataset contracts to follow the extracted selection, upload, and inventory modules. Keep the inferred picker tab stable while device inventory settles. Use the shared TrainingMethod type for VRAM estimation and remove redundant casts. * fix(train): align Hugging Face repo validation Allow underscores at repository segment boundaries to match Hugging Face repo ID rules. Keep model and dataset search queries unvalidated while preserving selection safeguards. Add regression coverage for accepted IDs and unrestricted picker searches. * Fix training resource compatibility and cache safety Restore legacy dataset format and mapping routes while moving training uploads and checks to the canonical Hub endpoints. Enforce configured upload limits, clean partial files after failures or cancellation, and support both upload route families in middleware. Align Hugging Face repository validation with accepted underscore boundaries and skip PyTorch dependent tests when the dependency is unavailable. Keep model format and cache metadata correct when the same repository is selected from another source. Only promote complete runnable model weights as cached training resources, align download notices with picker capabilities, and reject weightless local snapshots during start preflight. Cancel stale dataset mapping assistant requests before they can overwrite a newer selection. Preserve touch selection for training methods by separating tooltip toggling from Select item activation. Add focused backend, frontend, and picker regression coverage for these paths. * fix(studio): finish train page and onboarding rework Ignore cache-only fields when comparing training start inputs so reconciliation does not cancel valid runs. Route native dataset drops through Tauri path validation and lease-backed managed uploads. Use the production model and dataset selectors in onboarding and replace placeholder uploads with real imports. Require positive learning rates while keeping learning rate, epoch, and step inputs editable. Normalize model identities, improve dataset summary order, and guard persisted selection state. Recognize Windows rooted and drive-relative paths consistently in picker logic. Add MLX optimizer help and learning rate validation messages to every supported locale. Add frontend, backend, and native policy regression coverage for the updated behavior. * chore(ci): remove train picker smoke workflow changes Remove the train picker Playwright step from the macOS Studio smoke checks. Remove the train picker Playwright step from the Linux Studio smoke checks. Remove the train picker Playwright step from the Windows Studio smoke checks. Drop the matching artifact upload paths so the workflow files match upstream main. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): harden train picker behavior and boundaries Expose dataset drop helpers through the training public API and move local path detection into shared code. Block freeform Hub choices while offline for pointer and keyboard selection. Make native dataset drop handling safe for committed React renders. Keep recipe navigation working when session storage is unavailable. Preserve Windows drive root identities and cover the distinction from drive relative paths. Update picker and training input contracts to match the extracted implementations. Remove orphaned translations and align the train style block with the existing indentation. * Fix train picker cache and upload behavior Keep selected model cache paths strict and preserve fallback discovery only when no path is supplied. Offload multipart writes and native dataset copies from the backend event loop. Prewarm train picker inventories so device-first tab inference can settle before the first open. Use shared picker tab constants across the model and dataset selectors. Update picker contracts and add regression coverage for cache resolution and dataset uploads. * Refine train picker controls and component structure Split picker retry, error, and pagination UI into focused components. Add a shared segmented control and use native radio semantics for dataset source selection. Centralize segmented indicator positioning with RTL-aware transforms. Separate dataset advanced settings state wiring from its presentation and source toggle. Make picker arrow navigation move symmetrically from tabs through search and options. * Fix train page reconciliation and picker contracts Preserve tuned training parameters when model cache references change, while refreshing defaults when no user edits occurred. Keep streaming modality corrections visible through the Start CTA after preflight stops. Localize the new training parameter and dataset mapping text across all supported locales. Refresh picker structural contracts after the component split and apply Biome formatting to the changed files. * Fix train upload tests and desktop drop handling Pass an explicit empty native path lease in legacy upload limit tests so direct route calls match FastAPI request behavior. Track Tauri scale factor changes during dataset drag and drop, clean up both listeners safely, and cover runtime updates. Replace the run preview card's important border utility with an explicit dark theme card modifier. * fix(studio): harden training setup and history state Restore Hugging Face token entry and validation in the onboarding model and dataset steps. Apply the configured upload size preflight before onboarding dataset uploads. Update moved model and dataset cache references without showing false missing cache warnings. Parse SSE frame delimiters by their matched length so mixed line endings preserve event data. Match sensitive pid paths by exact segment while retaining database state protections. Clear deleted Data Recipe selections after inventory settlement while preserving direct uploads. Show the files deleted status only when a run previously recorded an output directory. Handle Data Recipe navigation failures and keep the native drop listener stable during uploads. Replace formatting-specific contract checks and add focused behavioral regression coverage. * Harden training start and model picker flows Split the train model picker into its own entry point to keep it out of shared route bundles. Preserve pending idempotent start reservations and reconcile them before accepting a run. Keep unconfirmed starts in a polling state with accurate localized feedback. Continue starts after automatically disabling unsupported multimodal dataset streaming. Localize the updated model picker controls across supported languages. Lock dataset uploads before asynchronous preflight to prevent concurrent selections. Add regression coverage for reservation states and unconfirmed runtime starts. * fix(studio): harden training picker edge cases Preserve POSIX model path identity while retaining Windows relative path normalization, and clarify the override migration behavior. Localize the onboarding model and dataset controls across every supported locale. Truncate native dataset filenames by Unicode code point so generated labels cannot contain lone surrogates. Cancel training after preflight disables streaming for image or audio datasets so users can review the changed setting before restarting. Update runtime contract assertions for the setStartPending rename and add regression coverage for path identity and Unicode truncation. * fix(studio): align training picker controls Restore full-height Browse and Amazon S3 selection with an accessible radiogroup that avoids fieldset sizing behavior. Align model and dataset picker tabs with the shared 36 px segmented control geometry and typography. Remove active borders and focus rings from picker search inputs while preserving keyboard focus indicators on other controls. Add contract coverage for segmented control sizing and picker search focus styling. * fix(studio): harden onboarding picker flows Classify native dataset drops from the full path before shortening display names. Keep onboarding model choices within the selected training type across Hub, device, and freeform selections. Retry model default loading when onboarding restores an interrupted model selection. Localize the Hugging Face token field across supported languages. Reset picker result scrolling when switching between Device and Hugging Face tabs. Add regression coverage for native filenames, model constraints, hydration, and picker scrolling. * fix(studio): reconcile multimodal streaming before training Disable streaming when dataset checks detect image or audio data, then recheck cached selections in non-streaming mode. Keep start-time modality detection as a safe fallback that updates the configuration and continues without an error or a second click. Build onboarding training method options from the shared method order and metadata so onboarding stays aligned with the train page. Add contract coverage for the streaming reconciliation and shared training method list. * Fix train picker contracts and dataset recovery Allow local models without reliable modality metadata to pass onboarding constraints while retaining explicit mismatch checks. Reconcile GPU selection without effect-driven state updates and mark intentional deep Hub imports for lint. Return and consume stable local dataset cache miss codes, and send the canonical train_split field. Move train-specific picker code into its owning feature and update the related contract coverage. Use the canonical Hub dataset progress route from Chat. * Harden training resource selection and offline handling Return an actionable preflight error when offline mode is enabled and the selected model is not cached. Validate evaluation dataset uploads with the shared extension policy and reuse the centralized accept list. Disable streaming for explicit on-device dataset selections while preserving Hugging Face selection intent. Keep dataset identifiers safety checked without blocking benign values before Hugging Face handles repository validation. Remove the branch-added source assertions and cover the new selection policy with behavioral tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix training setup validation and async state races Validate Hugging Face dataset IDs and repository access before training starts while keeping local paths on their filesystem validation flow. Scope model default race protection to fields that defaults can overwrite so unrelated dataset edits no longer suppress model initialization. Stop Hugging Face token edits from invalidating token-independent device inventory scans on every keystroke. Add regression coverage for dataset source validation, cached and remote Hub preflight, model default application, and token inventory behavior. * Tidy up train page layout, tooltips and dark mode Layout - Move the Browse / Amazon S3 toggle into the Dataset section header - Put Subset, Train Split and Evaluation Split on one row - Put Target Format, Train Split Start and Train Split End on one row - Pair Project Name with Max Steps, and Context Length with Learning Rate - Centre the hyperparameter tabs and give them a fixed width so the sliding indicator lines up with its segment - Give parameter rows a min height so slider and select rows share one vertical rhythm - Slightly more spacing below section headings and at card bottoms Tooltips - Add hints for Model, Method, Dataset and Project Name - Move field descriptions into the tooltips and drop the inline copy - Upload field keeps the accepted file types inline, with size limit and Learning Recipes note in the tooltip HF token - Show a masked preview of a saved token instead of a generic label - Read Not set when no token is stored Dark mode - Drop borders on the LoRA option cards and target module chips, and separate states with background fill instead - Use a lighter, less saturated green for the selected state Other - Expand LoRA Settings and Training Hyperparameters when Advanced opens - Shorten the upload label to Drop file or click to upload - Localise the new wizard tooltips across all locales * Fix training defaults, MLX validation, and dataset preflight Prevent late hardware recommendations from replacing configuration values loaded after model selection. Block unsupported MLX methods and embedding runs in the UI and backend before training is queued. Require live Hub access for streaming dataset preflight instead of accepting unrelated cached data. Compare advanced settings against applied model defaults and cover the updated behavior with focused tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix training picker validation and platform safeguards Update the Tauri YAML contract test to verify save filter behavior without depending on the Rust vector implementation. Align frontend Hub resource validation and training provenance normalization with the shared repository ID rules, including valid underscore and maximum length identifiers. Reuse one frontend training method policy so Apple Silicon onboarding and the Train page consistently disable CPT and prevent unsupported onboarding completion. Add regression coverage for rejected Hub IDs and exact resume provenance. * Fix MLX training capability validation Detect hardware before platform validation so unsupported MLX configurations fail before a job is queued. Reject audio dataset training during server preflight while preserving the worker guard. Share Apple Silicon capability checks across model selection, onboarding, readiness, submission, and LoRA controls. Cover audio, CPT, embedding, LoftQ, DoRA, and warm-start detection with regression tests. * Fix training quit protection, translations, and preference reset Protect desktop quits while a training start is pending or a run is active, and keep the warning text accurate. Localize dataset upload details across every supported locale and require those keys in parity checks. Centralize the training picker and parameter mode storage keys so resetting local preferences clears current and legacy values. Add regression coverage for training activity transitions and preference reset ownership. * Preserve multimodal training support Keep upstream training eligibility authoritative while preserving separate vision, audio, and embedding capabilities. Classify dual audio and vision models for vision training without changing inference support or enabling pure audio training on MLX. Add coverage for model type constraints, backend type resolution, and MLX validation. Apply the pending Biome formatting fixes to the dataset selector and training wizard. * Fix training defaults retry and clean stale translations Allow model defaults retries after failed cache reconciliation without overwriting user-edited settings. Ignore stale fallback vision checks once a newer defaults request starts. Remove unused parameter description keys from every locale and cover the retry behavior in the training contract tests. * Document direct model identity imports Explain why the model identity helpers bypass the Hub barrel beside each lint suppression. * Preserve training settings across reloads Persist applied model defaults and their advanced settings baseline while refreshing transient model metadata without overwriting tuned hyperparameters. Keep remote format probe tests runnable without an installed huggingface_hub package by supplying a fake module. Normalize tilde-prefixed path separators while preserving case-sensitive identities. Remove unmatched model and dataset tour anchors. Add regression coverage for persistence migration and model identity normalization. * Harden train page validation and persistence Validate CPT embedding learning rates against backend bounds and add localized client feedback. Keep invalid learning rate drafts out of persisted state and YAML exports. Notify users when multimodal model selection restores a non-S3 dataset source. Remove W&B tokens from training config persistence and migrate stored secrets safely. Update the model defaults contract test to cover metadata refresh without requiring the removed early return. Move segmented control styles into a non-component module to preserve Fast Refresh behavior. Add validation and persistence coverage for the new contracts. * Fix training model IDs, cache state, and localization Preserve root-level Hugging Face model IDs instead of rewriting them into the Unsloth namespace. Debounce token-scoped inventory reconciliation so token and inventory updates use stable request keys. Keep the selected training history run when the Studio view remounts. Localize dataset subset and split selectors across every supported locale. Complete Italian picker and training translations and enforce Studio-wide locale parity. * Fix training history errors and stale stop state Return structured artifact deletion errors and show localized messages that distinguish active training output from filesystem failures while keeping history intact. Clear superseded stop requests only when they belong to the current runtime generation, and cover start invalidation and stale stop handling with regression tests. Clarify that shared picker styles apply outside the Hub while Hub-only rules remain scoped. * Fix training method persistence and settings summaries Persist training method provenance so manual learning rates, model adapter rates, and pre-CPT dataset formats survive reloads and method changes. Deduplicate imported target modules and compare advanced-setting arrays by value counts so duplicate entries cannot hide non-default settings. Add migration and regression coverage for legacy state, rehydrated method transitions, restored learning rates, and duplicate target modules. * Fix persisted completion defaults and inventory retries Persist trainOnCompletions across reloads so model defaults and advanced settings summaries remain accurate. Defer missing legacy values until model metadata loads, preserving tuned settings while respecting streaming, raw text, CPT, embedding models, and explicit user changes. Settle partial inventory failures when usable rows exist so manual local model paths and Enter submission remain available. Add retry actions for partially failed model and dataset scans, including when filtering leaves no visible results. Add regression coverage for persistence migration, constrained completion defaults, inventory settlement, and picker empty states. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix training validation and route state regressions Retry newly detected image datasets with vision-aware validation, clear stale dataset check failures, and cover the failed background classification case. Restore Train history cleanup and centralize the learning recipes navigation intent key. Repair backend route stubs for canonical model IDs and path normalization, and isolate consent tests from shared package import state. * Preserve training cache pins during preflight Keep model and dataset cache flags and local paths in the pending start comparison so reconciliation cannot submit a stale snapshot. Cover changed cached copies and uncached transitions with focused regression assertions. * Preserve Hub identity for cached training exports Keep exact cached snapshots for training while restoring the standard Hub repository identity before PEFT saves. Recover repository identity in memory for legacy adapters so imatrix exports work without rewriting adapter configs. Cover repository mismatches, local models, and Windows cache paths with focused regressions. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix cached training, evaluation, and MLX resume behavior Limit cached dataset slice loads so small training ranges do not materialize full duplicate caches. Restore Hub model identity for cached Torch and MLX runs so saved adapters remain portable. Attest MLX model provenance across runtime and prequantized 4-bit formats so valid checkpoints can resume. Recognize missing cached eval splits, reload train and eval together, treat explicit eval split errors as fatal, and use deterministic held-out data for automatic fallback. Retain non-fatal evaluation warnings in training status and render them in the train UI. Apply pin and model format validation to every MLX worker entry point and keep the worker test fixture aligned with its imports. * Stabilize training progress updates across job handoffs Keep active and pending training ownership separate so status polling cannot switch identities during a running job. Scope metrics, progress streams, stop, and reset operations to the expected job and recheck ownership during handoffs. Make polling, stream parsing, and start reconciliation abortable, monotonic, and safe against stale overlapping requests. Preserve same-run interface state so live updates do not remount progress controls or dismiss the stop confirmation. Add backend and frontend regression coverage for ownership changes, stale events, request races, and stream cleanup. * Fix cached training fallback and dataset reconciliation Recognize incomplete SentencePiece tokenizer caches and retry online model loads through the Hub while preserving strict offline and resume pins. Prevent rejected dataset snapshots from being promoted again until the selection or inventory changes. Forward cancellation signals to dataset format requests so superseded checks stop at the transport layer. Add focused backend and frontend regression coverage for these cache paths. * Harden offline training and cache fallback behavior Validate cached model snapshots for tokenizer and processor support, then fall back to the pinned Hub revision when the cache is incomplete. Remember rejected dataset cache entries and cancel superseded checks so metadata-only snapshots cannot trigger request loops. Reject deleted local datasets before any Hub access and clear only the matching stale selection in background, preview, and start flows. Reserve training starts before validation so overlapping requests and GPU consumers cannot race the spawn window. Keep GGUF fallback metadata tied to the cache that supplied variants when Hub access fails. Filter DSpark and DFlash drafter companions across picker, loading, inventory, and deletion paths while preserving MTP behavior. * Harden training setup and picker behavior Keep training starts blocked while a stop request is pending and preserve the stop latch through backend failures. Skip malformed training progress events while validating SSE payloads and releasing stream readers safely. Enforce non-streaming state for upload and S3 datasets across source changes, persistence, and rehydration. Route Markdown dataset drops to Data Recipes across browser and native path formats. Give model and dataset source tablists localized accessible names. Refresh training start contracts and add focused regression coverage for the corrected behavior. * Fix offline dataset cache selection and token status Prefer finalized processed dataset caches for offline training while keeping raw cache paths available for scoped management. Treat malformed Hugging Face tokens as unset in Train and surface local validation feedback without unnecessary requests. Add regression coverage for cache coexistence, interrupted cache builds, path propagation, and malformed token presentation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Preserve sidecar Hub version during dataset cache checks Keep processed dataset cache discovery free of eager datasets imports so training does not preload the base Hugging Face Hub package. Preserve cache root discovery through HF_DATASETS_CACHE, HF_HOME, and XDG_CACHE_HOME before Transformers sidecar activation. * Add the AGPL-3.0 header to the two new studio training contract tests * Scan the fallback target and pin cached snapshots that hold weights Cache fallback scanned the pinned snapshot it was about to discard, then loaded the Hub repo unscanned. Scan after the pin is dropped instead, in all four training paths. Preflight probed the literal model name, so the registry bicodec alias Spark-TTS-0.5B/LLM 404'd and rejected a supported model. Probe the repo the trainer downloads and treat its load subdir as the weight root. An unpinned start still downloads its dataset, so a stray cached copy no longer skips Hub verification. Offline keeps accepting the cache. refs/main can point at a metadata-only revision, so the model pin now prefers a snapshot that carries weights before falling back. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Let the capability-cache guard see through the key type alias model_config.py names the cache key _CapabilityCacheKey, so matching the literal Dict[Tuple no longer finds it and the guard fails on a cache that is still correctly tuple-keyed. Resolve module-level aliases first. Checked by mutation: reverting a cache to Dict[str, ...] is still caught. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: point the extra-UI tour anchor check at studio-model-picker The train page rework renamed the model tour anchor from studio-model to studio-model-picker and updated tour/steps/base-model.tsx accordingly, but tests/studio/playwright_extra_ui.py still looked for the old anchor, so the Chat UI Tests job failed with "[data-tour='studio-model'] not found". Verified against two live Studio instances: the anchor is studio-model on main and studio-model-picker at this head; studio-dataset and studio-params are unchanged on both. * Studio: require metadata and weights together when pinning a model snapshot Pass 1 of _resolve_model_snapshot matched on weights alone, so two cases still selected a snapshot that /training/start then rejects with "does not contain trainable weights": - a newer weights-only fetch (interrupted download, or an allow_patterns pull that never took config.json) displaced an older complete sibling. That is a regression against the previous metadata-first ordering, which started the run; reproduced with a two-snapshot cache where the complete one is older. - consolidated.safetensors counted as weights for selection but is absent from _MODEL_WEIGHT_CANDIDATES in routes/training.py, and transformers 4.57.6 has no loader path for it (zero references in the package), so a config plus consolidated snapshot won over a loadable sibling. Pass 1 now demands metadata AND weights via a required_groups argument on latest_snapshot_from_cache_path, and consolidated.safetensors is dropped from the selection tuple so it matches what the route accepts. Pass 2 keeps the metadata-only fallback unchanged, so caches that never held weights resolve exactly as before. Both new tests in test_model_cache_snapshot.py fail without this change and pass with it; 401 tests across the cache, preflight, provenance and identity suites pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make the new cache tests pass on Windows Four of the PR's new tests fail on windows-latest while passing on ubuntu and macos. All four are defects in the tests, not in the production code. The three blob-symlink tests hardcode a POSIX relative target ("../../blobs/x"). Windows stores the reparse-point substitute name verbatim and resolves it in the object namespace, where / is not a separator, so the link is created but dangles: Path.resolve(strict=True) raises and the provenance and dataset-cache helpers correctly return None. huggingface_hub builds these targets with os.path.relpath (file_download.py _create_symlink), which yields ..\..\blobs\x on Windows, so a real cache never has this shape and no Windows user is affected. Building the target with os.path.relpath / os.path.join matches what huggingface_hub writes. The cross-snapshot rejection tests had the same POSIX targets, so on Windows they were passing for the wrong reason (dangling link rather than the escape check). They now use native separators too, so the rejection logic is actually exercised there. test_runtime_4bit_resume_reaches_worker_with_source_resource_pins compared model_local_path against str(Path); the route posix-normalizes that field via normalize_path, so the assertion now uses Path.as_posix(), which is a no-op on POSIX. 324 tests across the three files pass on Linux. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: bound the cached dataset re-check and stop adopting unconfirmed starts Two defects introduced by this PR's new modules. 1. Unbounded dataset re-check (#7853). training-config-store.ts re-runs the cached format check whenever rejectValidation() reports the in-flight token as stale, with no counter and no backoff. DatasetCacheRejectionTracker advances its generation on any inventory-fingerprint change, and the fingerprint includes sizeBytes, so a dataset that is still downloading invalidates every check in flight and the pair never converges. Measured at 480 requests in 60s; a probe driving the real tracker ran 500 iterations without settling while a stable inventory settles in 1. The retry now draws from a per-selection budget (dataset-recheck-budget.ts) and falls through to the uncached check once spent, so the answer still refreshes when the inventory genuinely changes but a churning one cannot spin. The budget resets when the dataset or split changes. Fixing the tracker fingerprint instead was rejected: existing tests pin the behaviour that a sizeBytes change makes a rejected cache retryable. 2. Unconfirmed starts adopted as recovered. reconcileTrainingStartTransportFailure ended with adoptAcceptedTrainingStart(pending.jobId, ...) and returned "recovered". The backend reserves the request id and job id before the heavy preflight, so a start still pending when the 30s window closes may yet be rejected. Adopting there reported success and pinned a job id that never became current_job_id: acknowledgeTrainingStartRequest was never sent, and dismissTrainingRun's expectedJobId check then returned "superseded", leaving the rejected state unclearable until a new start re-reserved it. It now calls the module's existing settleUnconfirmedTrainingStart and returns "unknown", which both callers already handle by warning startUnconfirmed and settling unconfirmed, and which preserves start_request_id for acknowledgement. New tests fail without each change and pass with it; removing the bound makes them fail rather than hang. typecheck, 598 frontend tests, build, and the 154 studio contract tests all pass. * Studio: retry tokenizer-less pinned snapshots, and skip Hub preflight when offline Two more defects introduced by this rework. Neither mechanism exists on main: worker.py there has 0 occurrences of local_files_only / model_snapshot_path / cache_artifact, and routes/training.py has 0 of model_info / dataset_info. 1. Tokenizer-less pinned snapshot is terminal (#7845). The pin only requires config.json plus a weights file, so a snapshot with no tokenizer pins clean, local_files_only is set, and AutoTokenizer then fails. Recovery is gated on _is_model_cache_artifact_error, whose marker list matches only three of the message shapes transformers actually emits. Sweeping all 159 tokenizer classes against a tokenizer-less snapshot: 37 failures were classified not retryable, of which 26 are genuine cache problems that got zero Hub retry, including XLMRoberta (BGE-M3, multilingual-e5, LaBSE), MBart, NLLB, Bloom, GPTNeoX, Cohere, Marian and the generic PreTrainedTokenizerFast. SentencePiece and BPE families resolve a missing vocab path to None and then dereference it, so the failure arrives as a bare AttributeError with no cache-specific text. Adding those four shapes takes the sweep from 37 misses to 11, and all 11 remaining are correctly fatal (missing optional Python dependency, or an unsupported tokenizer class), which no Hub retry can fix. Widening the classifier rather than validating tokenizers before pinning: the vocab filename space is as open-ended as the exception space across 159 classes, requiring a tokenizer would break the deliberate adapter_config.json pin path, and a real loadability check means loading a tokenizer inside the start request. The recovery path is already gated on offline and require_exact, so a false positive costs one Hub attempt while the current false negative fails the run. 2. Blocking Hub preflight with no reachability guard. Both preflight legs retry metadata at 5s then 10s with no reachability check, and requests applies the timeout per resolved address. Measured 30.0s added to a single POST /training/start with the Hub black-holed, scaling with the number of addresses, surfacing as 503. utils.utils already provides a bounded, memoised hf_unreachable/hf_dns_dead that the training worker subprocess uses one layer down; the route that spawns it did not consult it. The model leg raises inside the existing try, so the except HTTPException handler still runs _resolve_model_snapshot and a cached snapshot pins exactly as before, just without first burning the remote budget. The guard fails open, so an online start is unchanged. New tests fail without each change and pass with it, including a wiring contract that fails if the guard stops being consulted. 443 tests across the preflight, cached-start, provenance, snapshot and streaming suites pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the import-hoist lint honour __all__ re-exports Source lint failed on studio/backend/utils/security/__init__.py: [BLOCKER] HOISTED-IMPORT-UNUSED 'load_scan_target' (['from:utils.security.file_security:load_scan_target']) added but unused That file is a barrel __init__: every name it imports carries "# noqa: F401" and is listed in __all__, and routes/training.py imports load_scan_target from the package rather than the submodule, so the re-export is load-bearing. HOISTED-IMPORT-UNUSED only excluded re-exports that already existed before the change, so adding any new name to an existing barrel was an automatic blocker. The script's own docstring already records this shape as a known false positive for NEW-UNUSED-IMPORT. Skip module-level imports whose bound name appears in __all__, since those are loaded by importers rather than by the module itself. The rule keeps its teeth: a newly added import whose name is not in __all__ still blocks, and a dangling alias still trips UNRESOLVED-NEW. Verified with a four-case mutation matrix plus --self-test, and the full 79-file changed-set lint now reports OVERALL: PASS. * Keep studio off evaluated PEP 604 unions on the 3.9 floor tests/test_python39_compatibility.py failed on Core (HF=4.57.6, default and latest) and Repo tests (CPU): test_no_pep604_unions_are_evaluated_on_the_declared_floor model_config.py:937: Tuple[...] | Tuple[...] (type alias) test_studio_evaluated_unions_do_not_grow 36 studio files now evaluate PEP 604 unions, up from 35 Both offenders came from this branch. model_config.py:937 is a module-level type alias, so it is evaluated at import time and "from __future__ import annotations" cannot defer it; it needs typing.Union, which is what the test message prescribes. hub/utils/dataset_cache.py is the one file that pushed the count to 36, and all five of its unions are function annotations, so the future import is enough there. No behaviour change: the alias is only a cache-key type, and the annotations are unevaluated either way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the dataset re-check budget on the whole selection The budget added for #7853 keyed on dataset + split only, but a dataset cache usability identity has four user-chosen dimensions: dataset, subset, split and streaming. Changing subset or toggling streaming therefore kept the same key, so a genuinely different selection inherited an exhausted budget, skipped the cache-preferring re-check and dropped straight to a remote resolution with datasetKnownCached cleared. Key on all four instead, JSON-encoded so no delimiter can collide with a name containing it and null stays distinguishable from the string "null". cachePath is deliberately still excluded even though the usability identity carries it. It is derived state that moves as a download populates the cache, so feeding it into the key would mint a fresh budget on every poll and re-arm the exact non-terminating loop this module exists to bound. Tests: two new cases cover subset and streaming, and an inverse case pins that nothing outside the selection can refresh the budget. Reverting the key to dataset::split fails exactly the two new cases and no others. * Stop the pinned snapshot path reaching PEFT as the base model name A completed run wrote a machine-local path as the adapter's base model: base_model_name_or_path = '/home/user/.cache/huggingface/hub/ models--unsloth--Llama-3.2-1B-Instruct/snapshots/0123…' where main writes the Hub id. It lands in adapter_config.json, every checkpoint-*/adapter_config.json, the run card, export_metadata.json for merged and GGUF exports, and the model card push_to_hub uploads, none of which resolve on another machine. restore_hf_cache_repo_identity runs in UnslothTrainer.load_model before get_peft_model, so its peft_config branch has nothing to repair yet, and PEFT then derives the name itself: # peft/mapping_func.py new_name = model.__dict__.get("name_or_path", None) peft_config.base_model_name_or_path = new_name PreTrainedModel.__init__ copies config.name_or_path onto the instance, so restoring only config._name_or_path leaves that slot holding the snapshot path. Restore the instance attribute too. It goes through the same guard as the other fields, so an ordinary local model, an unrelated Hub id and a repo mismatch are all still left alone. Tests are behavioural: the existing model identity suite asserts the call site via AST and stays green with this bug present, which is why it was missed. Removing the new line fails 3 of the 6 added tests and none of the other 11. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve cached snapshots that load from a subdirectory unsloth/Spark-TTS-0.5B keeps everything trainable under LLM/: its snapshot root carries only README.md and config.yaml, no config.json and no weights. _resolve_model_snapshot only looked at the root, so a cached copy resolved to None. _apply_model_cache_pin then warned "Cached copy not found on disk; downloading", and offline the start route turned the same None into a 409 hf_model_not_cached_offline for a model that was sitting in the cache. The remote preflight already handles this: it expands load roots through load_scan_target. Only the cached path disagreed. Reuse security_load_subdirs, which already reports ("LLM",) for BiCodec, so both paths share one source of truth, and apply it to the resume and preflight pin lookups too rather than just the fresh-start one. Detection can raise offline or for a gated repo, so a failure degrades to root-only rather than propagating. Tests build the real models--org--name/snapshots/<rev> layout. Making the helper a no-op fails 3 of the 6, and a snapshot with nothing loadable in either place still resolves to None, so the widening cannot mask an empty cache. * Use Optional in the deprecated dataset alias signatures The rewritten alias module annotates two parameters as UploadFile | None and str | None with no postponed annotations, so they evaluate at import on the declared 3.9 floor while the rest of the file already uses Optional[...]. The file was an evaluated-union offender before this change too, so this is not a new break, just keeping the new code consistent with its own convention and off the debt list. * Apply the repo kwarg-spacing formatter to the new code pre-commit.ci flagged the ruff-format-with-kwargs hook on this branch. Running scripts/run_ruff_format.py locally keeps the fix in the authoring commits instead of trailing a separate bot commit. * Say why a resume is refused instead of blaming the checkpoint can_resume_run gained a provenance clause in this rework, so it now returns False for runs whose checkpoint is entirely intact, most realistically once the pinned model snapshot is evicted from the HF cache. The start route answered every False with "Resume checkpoint must belong to a stopped or errored run with complete saved trainer state." pointing the user at trainer state that is fine, while the real cause was the resource gate. exact_resume_resource_requirements already raises with a precise explanation and resource_provenance_allows_resume was discarding it. Add resource_provenance_resume_blocker, which returns that explanation (or None when the run is resumable), define allows_resume in terms of it so the two cannot drift, and use it in the start route when it is the actual cause. The gate itself is unchanged: refusing an unattested resume is deliberate, only the diagnosis was wrong. Tests are behavioural, with one narrow wiring contract for the branch that regressed. Replacing the precise reason with a generic string fails the message test and nothing else. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Honour load subdirs everywhere a cached snapshot is probed Follow-up to 590ac9f22. Resolving cached Spark-TTS/BiCodec snapshots from their LLM/ load root got a start past _resolve_model_snapshot, but three later probes still asked for root-level config.json only, so the same snapshot was accepted in one place and rejected in the next: - provenance.py attested incomplete model provenance, after which exact_resume_resource_requirements refused resume for a snapshot still sitting on disk; - the start preflight reported a valid cached model as having no trainable weights; - the worker cleared model_snapshot_path before loading, falling back to the Hub instead of the selected snapshot. Promote the helper to hub.utils.hf_cache_state.with_load_subdirs so there is one definition rather than four, and use it at every site. _has_trainable_local_weights takes subdirectory roots instead, since it probes directories rather than a filename list. Unchanged for ordinary models: with no load subdirs the helper returns its input, and detection failures still degrade to root-only. * Let the model routes see the same cached snapshots training does Two ways /api/models disagreed with the training resolver, both reachable from resume: _model_config_inspection_target probed only the snapshot root, so a cached Spark-TTS/BiCodec copy answered "Selected cached model is no longer available" for a cache the training resolver accepts, and the exact-snapshot remote-code scan could fail with it. It now uses the shared with_load_subdirs helper. The model_snapshot_repo_id guard used the owner/repo-only regex, so resuming or scanning a namespace-less Hub model such as gpt2 or bert-base-uncased returned 400 before the snapshot could be inspected, even though hub.utils.paths .is_valid_repo_id and the picker both allow the one-segment form. That call site now uses the shared validator. The other five uses of the local regex predate this branch and are left alone. Reverting either fix fails one of the new tests and nothing else. * Get the resume refusal reason all the way to the user 300fe6321 fixed the diagnosis on POST /api/train/start, but the History UI never reaches it. can_resume: false hides the Resume button outright, and in the one window where the server could answer, resume-training-run.ts throws its own "Only stopped or errored runs with a saved checkpoint can be resumed" before sending any request, so no start call is ever made. For a run whose checkpoint is intact and whose pinned snapshot was evicted, that sentence is the wrong diagnosis, which is what 300fe6321 removed server-side. Carry the reason on TrainingRunSummary and prefer it in that guard. The field is optional and defaults to None, so old clients are unaffected, and it is only computed for rows already known unresumable. A checkpoint problem still reports None, leaving the client's existing wording for that case, and a failure inside the gate is swallowed so History still renders. Reverting either half fails one of the new tests: the summary stops carrying the reason, or the client stops preferring it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Honour load subdirs on the resume pin and keep the selected model cache copy Two follow-ups to the cached-snapshot work. The resume branch of _reject_untrainable_model_request still probed the pinned snapshot with a hardcoded (config.json, adapter_config.json) tuple, so a repo that loads from a subdirectory (unsloth/Spark-TTS-0.5B keeps everything trainable under LLM/) resolved to None. Offline that became a 409 hf_model_not_cached_offline for a snapshot present on disk; online it fell through to the remote metadata round trip the pin exists to avoid. It now uses with_load_subdirs like the other cached-snapshot probes. The cache reconciliation effect took the first usable inventory row for the selected repo. A repo can be present under more than one HF cache root, so that silently retargeted an explicit selection at a different copy on the next inventory tick. The selection logic moved to a pure module and now prefers a row whose path matches the current selection, falling back to the first usable row when there is no selection or the selected copy has gone. Tests: test_resume_pin_load_subdirs.py (5) and model-cache-reference-selection.test.ts (8); reverting either fix fails them. * Revert the model cache reference preference; it cannot fire The frontend half of 2c2a95309 assumed the model inventory can return more than one usable row for a repo. It cannot. hub/services/models/cache_inventory.py::_scan_cached_models collects into seen_lower keyed by repo_id.lower() and resolves collisions with _prefer_cache_row, and _dedupe_local_models keys hf_cache rows by (model_id, model_format, format_variant). Both return one row per repo, and a live check with the same repo under four HF cache roots got exactly one row from /api/hub/cached-models and one from /api/hub/local. With a single usable row usable.find(pred) ?? usable[0] is identically usable[0], so the change was a no-op and its tests asserted an array shape the API cannot emit. The resume pin fix in the same commit stands: that one was reproduced against the real unsloth/Spark-TTS-0.5B cache and reverting its hunk restores the 409. * Report the resume refusal that actually happened 300fe6321 and a98e721f4 set out to stop a provenance refusal being reported as a checkpoint problem. They overshot. Both sites asked resource_provenance_resume_blocker whenever can_resume_run said no, but that function refuses for several reasons and the blocker is computed independently of which one fired. initialize_resource_provenance writes {version: 1, status: pending} at the start of every run, so the blocker answers "The model revision used by this run was not attested." for any Hub-model run, including one whose checkpoint is simply missing. That is the more common way to be unresumable, so the change traded one misdiagnosis for another and the new one covered the larger population. Both sites now gate on has_resume_state, the same discriminator can_resume_run short-circuits on: no saved trainer state means the checkpoint is the cause and the client's own wording is correct; an intact checkpoint means a refusal really is provenance's doing and the specific reason is worth surfacing. test_resume_reason_matches_cause.py pins it; reverting either guard fails it. The route contract assertion is over the AST rather than the source text, because a substring search is satisfied by the explanatory comment beside the code and passes with the guard deleted. Two assertions in test_resume_blocked_reason_surfaces.py encoded the old behaviour and are corrected, including the docstring claiming History survives a raising gate: can_resume_run calls the same gate unguarded one line earlier, so that only holds when it short-circuits first. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope the import-hoist __all__ skip to package __init__.py 68bce3934 skipped any name listed in __all__. The comment said "the whole point of a package __init__", but compare() never consulted the path, so the skip applied to every module defining __all__ -- 27 non-package modules exempting 224 names, unsloth/models/_utils.py alone 74. The cost is not just breadth. lint-ci.yml names rename-clash as one of the two bugs this tool exists to catch because ruff and pyflakes miss it, and the skip disabled that detection for any name in __all__. Adding a name there became a one-line, reviewer-invisible way to switch the check off. The self-test I cited did not cover this: --self-test reported ALL PASS with the skip hunk reverted, because every case ran through the "<name>" path placeholder and none defined __all__. Cases may now carry their own path, and three new ones pin the behaviour from both sides -- a re-export in a package __init__ is allowed, the same shape in an ordinary module is still blocked, and a new import absent from __all__ is still blocked in an __init__ too. Deleting the skip fails the first; widening it back fails the second. Removes two genuinely unused imports the correctly-scoped check then found in test_resume_blocked_reason_surfaces.py. * Make deleting a run's artifacts reversible until the row is gone DELETE /runs/{run_id}?delete_artifacts=true is new in this PR; on |
||
Michael Han
|
a00fe86c13
|
Studio: read model text as utf-8 so umlauts survive on Windows (#7467)
* Studio: read model text as utf-8 so umlauts survive on Windows Chat rejects or mangles non-ASCII on Windows: "ä ö ü" in a prompt, a chat template, or a model path comes back as mojibake, or the load dies with UnicodeDecodeError. open() and Path.read_text() fall back to locale.getencoding() when no encoding is passed. On Windows that is the ANSI codepage (cp1252, cp932, cp1251, ... by system locale), never UTF-8. Hugging Face writes these files as raw UTF-8, so every read of one decodes with the wrong codec: - tokenizer_config.json, which holds the chat template. Templates routinely carry -> arrows, smart quotes and CJK, so this is the common path into chat - config.json and adapter_config.json - modules.json, Ollama manifests, and the .py sources the remote-code scanner reads before a model is allowed to load The llama-server and embedding-server stdout readers have the same problem via subprocess(text = True); they now decode utf-8 with errors = "replace" so a stray byte cannot kill a log reader. Encoding arguments only, no logic changes. tests/test_chat_text_encoding.py covers a config.json and a chat template holding umlauts, arrows and CJK, plus the remote-code scanner reading a source file with umlauts. Those pass anywhere the locale is already UTF-8, so a fourth test re-runs the readers under -X warn_default_encoding and fails on any platform if an encoding argument goes missing again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: name utf-8 explicitly on the remaining text I/O, with an AST guard (#7465) * Studio: name utf-8 explicitly on the remaining text I/O Follow-up to the model-text reads in #7467, covering the rest of the backend: system probes (nvidia-smi, amd-smi, powershell, git, node), package installers, /proc and /sys readers, and internal marker files (pid, install id, bootstrap password, Colab credentials). Same reason as #7467. open(), Path.read_text()/write_text() and subprocess(text = True) fall back to locale.getencoding(), which on Windows is the ANSI codepage rather than UTF-8. These paths are mostly ASCII today, so this is hardening, not a live bug. Encoding arguments only, no logic changes. Adds tests/test_text_io_encoding.py: an AST guard walking every backend source and asserting text I/O names its encoding, so the class of bug cannot creep back in one call at a time. 275 files. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Catch aliased subprocess and positional Path.open, migrate legacy JSONL The guard only matched a receiver literally named subprocess, so worker.py's `import subprocess as _sp` hid three text = True installs that decode pip output with the ANSI codepage. It also skipped any .open() with more than one positional argument, though Path.open takes buffering/encoding/errors/newline positionally. Resuming a scrape written by an older release is the other half: those JSONL lines are in the locale codepage, so the UTF-8 preload raised, the dedup keys were silently forgotten and duplicates were appended to a now mixed-encoding file. Decode with the locale codepage as fallback and rewrite as UTF-8 before the append handle opens, since Windows cannot replace a file it holds open. * Stream the JSONL preload and keep a torn line from relabelling the shard Reading the whole shard to migrate it was wrong twice over. These files reach gigabytes on a large scrape, so the preload now streams line by line and the rewrite streams through a temp file. Worse, one interrupted append used to condemn the file: the whole-file UTF-8 decode failed, every byte was retried as cp1252, and the rewrite persisted mojibake over records that were fine. A line now counts as legacy only if the locale codepage both decodes it and yields valid JSON, which a torn UTF-8 line does not. Damaged lines are skipped and copied through byte for byte. When the rewrite cannot be written at all, the append handle opens with the legacy encoding rather than mixing UTF-8 into the file. install_wheel takes run = subprocess.run as a parameter, so the guard cannot see it. Both wheel installs there now name their encoding. * Decide the shard's encoding from the file, not one line at a time Some byte strings parse both ways. cp1251 `Р°` is D0 B0, which is also valid UTF-8 for `а`, so a UTF-8-first parse quietly showed the wrong text instead of migrating it. A line now yields both readings, and the file decides. Any line that parses under the codepage but not as UTF-8 is unambiguous evidence, and ambiguous lines then follow that verdict, which is enough for any real shard: ordinary Cyrillic or Japanese prose is invalid UTF-8 several times per line. Keys for ambiguous lines are re-derived from the legacy reading during the rewrite. A shard is undecidable only if every line is ambiguous, and nothing can tell those apart. latin-1 is also tried after the locale codepage, so a scrape carried from Windows to a UTF-8 machine still has a reading rather than none. Requiring valid JSON, not just a decode, keeps that from claiming torn lines. * Weigh the whole shard, and never lose a record on the fallback path One structurally valid JSON line carrying a stray 0x96 parses as cp1252, so a single-line verdict let it relabel a healthy shard and mojibake every good record in it. Each line with non-ASCII bytes now votes: parsing only under the codepage is evidence for legacy, parsing as UTF-8 is evidence against, since codepage text rarely forms valid multibyte UTF-8. Ties leave the file alone. When the migration cannot be written the append handle uses the legacy codepage, and errors = "replace" quietly turned characters it cannot hold into question marks while write() still reported success. That path now escapes to \uXXXX instead, which is ASCII, so every codepage holds it and json.loads returns the exact characters. Nothing needs replacing, so errors = "strict" is safe. stream_installer runs sys.executable, so its output is now decoded as UTF-8 by utf8_child_env rather than read as the ANSI codepage. * Only rewrite a shard we can attribute, and append ASCII when we cannot latin-1 was doing too much work. It reads any byte, so it gave a moved shard a reading, but it is the right text only for cp1252: cp1251 Привет came back as Ïðèâåò and the rewrite made that permanent. The codepage is now trusted only when it is the locale's, and an untrusted reading is never written back. That leaves three cases where the file holds bytes UTF-8 cannot read and we are not converting it: no codepage to attribute it to, ambiguous lines outvoting the unambiguous ones, and a preload that could not read the file at all. All three used to append UTF-8 into it. They now append pure ASCII, which every ASCII-compatible codepage stores identically, so the file keeps decoding exactly as it did and no record is lost. Keys from the two readings are also kept apart. A damaged line in a healthy shard was marked seen through its codepage reading, so the retry that would have replaced the unreadable record was refused as a duplicate. * Let the flash-attn install stub take the kwargs the installer now passes _run_kwargs gained encoding and errors, so the one stub in this file that spelled its signature out rejected the call. The other four here already take **kwargs; this one now matches. * Do not let a stuck temp file mask the migration failure unlink() on the failure path could raise in its own right, on a stale .utf8.tmp directory or a temp another process holds. That escaped the constructor instead of returning False, so the caller never reached the ASCII append fallback that keeps the shard single-encoding. The pip fallback in install_wheel also spawns a Python child, so it gets utf8_child_env like the probe above it already had. The uv and nvidia-smi children are native binaries, where PYTHONIOENCODING would do nothing. * Stop converting legacy shards; the encoding that wrote them is unknowable trusted only ever meant that the bytes parse under this machine's codepage, which for a single-byte codepage is nearly always true. A cp1251 shard opened on a cp1252 Windows box decodes cleanly and would have been rewritten with Привет as Ïðèâåò. That is the fourth way this rewrite could corrupt a shard, and the common cause is that a file's encoding cannot be recovered from its bytes. So the rewrite is gone. The shard is left exactly as found, and appends are pure ASCII whenever it holds bytes UTF-8 cannot read, which is what actually delivered the no-mixed-encoding guarantee the rewrite was added for. Dedup keys still come from whichever reading parses, since ids are ASCII either way. This also removes the temp file, so there is no longer any file mode or ACL to carry across. * Scan the sandbox shim; it is shipped code, not a build artifact sandbox_site is on the sandboxed child's PYTHONPATH for every Python run (tools.py:332, 2660), so excluding it let two unannotated text calls through in code we ship. Both read and write the remap sidecar, which holds file paths. The exclusion list is meant for build output only, so the directory comes off it and the two calls name their encoding. * Force the worker's pip children to UTF-8, and read DBCS keys with a DBCS codec The three installer calls run sys.executable -m pip with an inherited environment, so the parent decoded UTF-8 while the child emitted the ANSI codepage. They now go through utf8_child_env like the other Python children. Two tests asserted no env kwarg was passed as a stand-in for no HIP flag being injected. They now assert the flag itself, which is the guarantee they were written for and does not depend on how the env is delivered. Separately, latin-1 cannot stand in for a double-byte codepage while recovering dedup keys: cp932 表 is 95 5C, and the trail byte reads as a JSON backslash, so the record failed to parse and its id was forgotten, appending a duplicate on resume. cp932, cp936, cp949 and cp950 are tried too. The reading is still only ever used for keys, which are ASCII and identical whichever codec parses. * Require more than one legacy line before trusting its dedup keys A shard whose valid records are all ASCII casts no UTF-8 votes, so a single damaged line won the vote by itself, its key was remembered, and the retry that would have replaced the unreadable record was refused. One such line is genuinely undecidable: a legacy record with one accented character and an ASCII record with one stray byte are the same shape. Reading it as damage costs a duplicate; reading it as legacy loses the record for good. Only one of those is recoverable, so it is now read as damage. A real legacy shard has a legacy line for every record carrying an umlaut, so its dedup is unaffected. * Append ASCII whenever the shard already holds non-ASCII bytes The gate asked whether any line was undecodable as UTF-8, which misses a shard where every legacy line happens to be valid UTF-8 too. A cp1251 shard of Р° records is bytes D0 B0 throughout, so appending 世界 as UTF-8 left a file where cp1251 reads the old records correctly and the new one as mojibake, and UTF-8 does the reverse. No single decoding recovered the whole scrape. The gate is now simply whether the shard holds any non-ASCII byte at all, which covers both cases and is easier to reason about: if what is already there reads differently under different encodings, do not add more bytes that do. Appending ASCII costs only \uXXXX escapes, which json.loads turns back into the exact characters, and it leaves the new record correct under either reading. * Skip the two Linux-gated flash-attn tests off Linux _should_try_runtime_flash_attn_install ends in sys.platform.startswith( "linux"), and the threshold test one line above already asserts exactly that, so the two tests that drive _ensure_flash_attn_for_long_context past the gate cannot pass anywhere else: the call returns before it reports a status. They were written on Linux and only surface once the suite actually runs on Windows or macOS, where both fail on an empty status list. This PR is about making the backend behave on Windows, so its own suite should be runnable there. * Fail closed when a KFD topology node does not decode This PR pins that read to utf-8, which turns an undecodable byte into UnicodeDecodeError. That is a ValueError, not an OSError, so it slips past the handler one line below and escapes a helper whose docstring promises to fail closed on any unreadable node. The caller would then lose the whole HIP-order map on a machine that has AMD GPUs, and the reason the helper fails closed is that dropping a node shifts every later ordinal and lets a similar-capacity GPU pass the total-size guard while showing another card's usage. Widening the handler is the same one-line change main already made in #7487, so the two agree and the eventual merge is clean. * Tighten the comments added in this branch * Treat an undecodable marker and undecodable metadata as malformed, not fatal Two more places where pinning the decode changed the failure mode. A UnicodeDecodeError is a ValueError, so neither `except OSError` nor `except (JSONDecodeError, OSError)` catches it, and both sites had a documented fallback that stopped being reached. An undecodable .transport marker used to read as an unknown value, and the caller then safely purged and restarted the partial download. It now aborts prepare_cache_for_transport instead, so the transfer fails rather than retrying. Undecodable .meta.json used to fall back to the file's own name, the same way invalid JSON does. It now aborts URI construction for the entire unstructured seed, so one corrupt byte in original_filename takes out the whole dataset. Both handlers are widened, matching the KFD fix earlier on this branch. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Widen two more decode guards, and pin the kernel installer's pipe Same shape as the ones already fixed here: the read was pinned to UTF-8 while the handler around it still only catches OSError, and UnicodeDecodeError is a ValueError. hf_cache_snapshot_dir answers whether a model is already on disk, and the offline embedding checks turn a raise into a 500. A torn refs/main used to decode into a nonsense commit and miss the snapshot dir; it now skips that cache root and keeps looking. _remove_pid_file runs first in _graceful_shutdown, so a corrupt studio.pid raising there abandoned the inference, export, training and tunnel children the rest of that function exists to kill. ssm_runtime's source-build path builds its subprocess kwargs in a dict and splats them through _run_with_heartbeat, so neither the encoding guard nor the earlier sweep saw the text = True in it: pip's output was still decoded with the Windows ANSI codepage, where a non-ASCII path or a compiler diagnostic mojibakes or raises over an install that was going fine. It now pins the same utf-8/replace pair install_wheel uses, and the HIP branch extends that env rather than replacing it. The guard learned the dict-literal shape and reddens on the old code (ssm_runtime.py:253). * Tighten the comments around the UTF-8 text I/O pins Collapse the multi-line rationales added with the encoding pins down to a line or two each, drop what the code already says, and use one wording for the repeated child-env note. * Do not let an unreadable bootstrap password stop startup, and narrow the kwargs guard ensure_default_admin calls _load_bootstrap_password for every existing admin and the lifespan calls that with no handler, so pinning the decode turned a damaged or pre-pin .bootstrap_password file into a backend that will not start. We write that file ourselves in UTF-8, so a byte that will not decode belongs to a file whose plaintext is worthless anyway; it now reads as no bootstrap password, the same answer as an absent file. A readable one still loads. The new kwargs check also judged every dict literal in the tree, so an unrelated payload carrying "text": True would have been reported as subprocess configuration with a misleading message, and a dict that fills in its encoding on a later line would have been reported too. It now only judges a dict that actually reaches a call, either splatted through a name or written at the call site, and treats a later kw["encoding"] assignment as satisfying it. The ssm_runtime shape it was written for is still caught, and a test pins both directions. * Stop reading a UTF-8 record a second time _read_line always parsed the line under the codepage as well, even when it had already read as UTF-8. Both callers take the UTF-8 reading when there is one and never look at the other, so on a healthy shard the second parse is pure waste, and this file reads all of one on every resume of a scrape it expects to reach gigabytes. Measured on 200,000 records, 76 MB: 1.96s before, 0.81s after, so the double reading was costing 2.8x. The early return is limited to a record, since the key lookup deliberately falls through to the codepage reading when UTF-8 yields something that is not one. A line UTF-8 cannot read still tries the codepage, latin-1 and the double-byte encodings as before, which is what the second reading is for. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the scanned source fixture's line endings test_remote_code_scan_reads_non_ascii_sources compared a file's contents against the string it wrote, but wrote it in text mode, so Windows translated the line ends on the way out and the read back differed by a carriage return. That is the writer's doing, not the encoding the test is about, and it was the one failure on the Windows runner that belonged to this branch. The fixture now writes with newline = "" so the bytes on disk are the string on every platform. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the newer comments to their point Shorten the widened-guard and state store notes added since the last pass, and collapse the line-ending note on the scanned source fixture. * Read the scraper checkpoint as UTF-8 only, never as a codepage A checkpoint holds nothing but base64 cursors and booleans, so one written by an older locale-encoded release is byte-identical to a UTF-8 one and already reads back. The codepage fallback can therefore only ever contribute non-ASCII: if a single-byte reading of the file were all ASCII, the UTF-8 read would have succeeded first. So the only file it changes the answer for is a damaged one, and there it turns a safe reset into a resume on a mojibaked cursor. GitHub answers that with INVALID_CURSOR_ARGUMENTS at HTTP 200, gh_client returns the partial document, and the scraper reads zero nodes and an empty pageInfo, which marks the stream done. Every later resume then skips it entirely. Reading UTF-8 only restores the earlier behaviour of dropping a checkpoint that will not decode, which re-scrapes from the first page while the writers dedup the replay. The shard scan below keeps its codepage reading; those records do carry non-ASCII. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gate the remaining tilelang install tests to Linux _tilelang_platform_supported() returns False off Linux, so _ensure_tilelang_backend returns before the install and the subprocess mock these six assert on is never called. They fail on macOS runners for that reason alone. The rest of the file already carries this marker; these were missed. * Gate the Windows-incompatible worker and ROCm tests Two different gates, because the production code has two. The causal-conv1d and flash-linear-attention installers bail out on sys.platform == 'win32' alone and run everywhere else including macOS, so those cases get not_on_windows; marking them linux_only would skip tests that legitimately pass off Linux. The DRM and KFD readers return early unless platform.system() is Linux, and their fixtures build a fake sysfs tree needing PCI addresses like 0000:00:02.0 as directory names, which Windows cannot represent, so those get linux_only. The two visible-utilization cases failed for a different reason: on Windows get_visible_gpu_utilization takes the AMD adapter branch ahead of the torch fallback under test, and probing it imports torch, which the runner lacks. Stubbing that branch empty leaves every other platform unchanged. * Treat unparseable JSON nesting as a parse failure, and guard os.fdopen json.loads answers nesting it cannot descend with RecursionError, a RuntimeError, so _parse let it escape where the catch-all it replaced discarded the record. Both callers run _parse outside any further handler, so one damaged checkpoint or shard line aborted the scraper at startup. The encoding guard also missed os.fdopen, which is open() on a descriptor and takes the same locale default in text mode. It flags exactly the two text-mode calls that were left unencoded; the swap lock file's reader was already pinned to UTF-8 while its writer still used the codepage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Write the non-ASCII source fixture without a 3.10-only argument Path.write_text() only grew newline in 3.10, and pyproject declares requires-python >=3.9, so this raised TypeError there. open() takes the same argument on every supported version and pins the bytes on disk the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten encoding comments * Follow subprocess calls through callable aliases in the encoding guard --------- Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com> --------- Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com> |
||
Daniel Han
|
3fd948eb95
|
Pin utf-8 on shipping-code text I/O instead of the operator locale (#7486)
* Pin utf-8 on shipping-code text I/O instead of the operator locale 113 read_text/write_text/open call sites across unsloth, studio and unsloth_cli let locale.getencoding() decide the encoding. That is utf-8 on the Linux and macOS runners and cp1252 on a stock Windows install, so the same file decodes differently for a Windows user and silently produces mojibake or raises UnicodeDecodeError. Adds tests/test_runtime_text_encoding.py to keep it that way. It resolves openers through each file's own imports rather than a fixed list of module names, so an aliased tarfile.open or a local from PIL.Image import open is not asked for an encoding it does not take. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scan tracked files only and resolve the unbound Path calling forms * Honour PEP 263 when scanning sources and migrate a legacy JSONL before appending * Scope guard imports lexically and only migrate a legacy file when it round-trips * Leave a legacy JSONL untouched and resolve path aliases in the foreign-opener check * Tighten comments --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
|
Daniel Han
|
6e91d1dff8
|
Studio: scan HF cache snapshot loads by their repo id (#7398)
* Studio: scan HF cache snapshot loads by their repo id Inactive Hugging Face caches (legacy, default, and previously selected download locations) are loaded by their resolved snapshot path so they keep using the selected cache instead of re-downloading. That path is a local filesystem path, so evaluate_file_security exempted it with "local path; no Hub scan" and skipped Hugging Face's pickle/malware scan. Active caches load by repo id and are still scanned, so the same model could dodge the gate simply by being in an inactive cache. An HF cache snapshot keeps the canonical models--org--repo/snapshots/<rev> layout, so recover the repo id from that path and scan it instead of exempting it. Non-cache local paths (models directory, custom folders) still skip the scan, and a remote ref is still scanned by repo id. Adds a regression test that a flagged pickle in an inactive-cache snapshot path blocks the load. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: scan the exact cached commit for inactive HF caches An HF cache snapshot path encodes the commit, not just the repo id (models--org--repo/snapshots/<rev>). Recover the revision alongside the repo id and pass it to model_info and the shard-index lookup so the scan covers the exact files that will be deserialized, rather than the repo's default branch. Without this, a pickle in an older cached commit that was later removed from the branch would scan clean and still load. Extends the regression test to assert the recovered revision is forwarded to the Hub scan. --------- Co-authored-by: danielhanchen <unslothai@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
|
Daniel Han
|
c2114d64dd
|
Studio: fail closed on index-referenced nested pickle shards in the offline embedding gate (#7366)
* Studio: fail closed on index-referenced nested pickle shards in the offline embedding gate
The offline embedding security gate (HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE)
only scanned the direct files of each SentenceTransformer load root and never
parsed local weight indexes, so a cached snapshot whose pytorch_model.bin.index.json
maps a weight to a nested shard (e.g. shards/pytorch_model-00001-of-00001.bin) was
treated as inert and allowed. The loader then follows the index into the subdir and
unpickles the shard. The online gate already blocks index-referenced subdir pickles,
so the offline path was strictly weaker.
Parse each local weight index in a load root and follow weight_map into nested dirs,
flagging any referenced pickle-extension shard. Paths resolve lexically (normpath),
never Path.resolve(), since HF cache snapshot files symlink into blobs/ and resolving
would leave the snapshot dir and false-block every sharded model offline. An absolute
path, a .. traversal that escapes the snapshot, or an unreadable/invalid index fails
closed. The existing safetensors-sibling suppression is kept.
* Studio: classify offline indexed shards by torch.load path, not pickle extension
load_state_dict picks safetensors vs torch.load per shard by the shard's own
suffix, so two offline-gate gaps remained:
- A model.safetensors.index.json whose weight_map points at a .bin shard was
suppressed by has_base_safetensors (the index file itself matches the base
safetensors regex), yet Transformers still torch.loads that shard. Only the
pytorch index is superseded by a base safetensors now; a safetensors index is
the chosen archive, so its non-safetensors targets are always flagged.
- A pytorch index can map weights to arbitrary names (shards/payload,
weights.data); the loader torch.loads any target not ending in .safetensors.
Flag indexed shards by that rule instead of a pickle-extension allowlist.
Restrict the scan to the two torch-family indexes (tf/flax load via non-pickle
loaders). Add regression tests for both cases.
* Studio: match offline weight-index filenames case-insensitively
The index-name check compared the on-disk filename exactly, while the
surrounding weight and safetensors matches use case-insensitive rules. On a
case-insensitive volume (Windows or macOS) from_pretrained opens an oddly-cased
cache file such as PYTORCH_MODEL.BIN.INDEX.JSON when it requests the canonical
lowercase name, so the exact-case check skipped it and a nested pickle shard it
referenced was allowed through. Lower-case the index name before matching, as
the rest of the gate does, and add a regression test.
* Studio: match load_state_dict format/selection exactly in the offline index scan
Two edge cases in the offline weight-index scan:
- load_state_dict decides safetensors vs torch.load with a case-sensitive
endswith(".safetensors"), so a shard named payload.SAFETENSORS still
deserializes via torch.load. Classify indexed shard suffixes case-sensitively
to match, instead of lower-casing (which treated such a shard as inert).
- A complete direct model.safetensors is selected before either sharded index,
so a stale model.safetensors.index.json referencing a .bin shard never loads.
Skip both indexes when a direct model.safetensors is present, so an otherwise
loadable model is not over-blocked.
Add regression tests for both.
* Studio: read the offline weight index as UTF-8
Path.read_text() uses the locale default, which is cp1252 on Windows, so a
UTF-8 weight index with non-ASCII bytes raised UnicodeDecodeError and the gate
blocked an otherwise loadable model. JSON is UTF-8 by spec (and how the loader
reads it), so pin the encoding.
* Studio: resolve safetensors alternatives via the loader's own filename lookup
The offline gate decided a safetensors alternative existed by case-folding the
directory listing. On a case-sensitive filesystem that let an uppercase decoy
such as MODEL.SAFETENSORS suppress the pickle scan, yet from_pretrained asks for
the canonical lowercase model.safetensors, does not find the decoy, and selects
the pickle (a direct pytorch_model.bin or the pytorch index) and deserializes it.
Probe each alternative with (root / name).is_file() instead, mirroring the
loader: is_file() honors the platform's case rules, so a decoy suppresses only
where the loader would truly open it. Suppression must never fail open; detection
stays case-insensitive (fail closed). Add regression tests for the direct and
indexed pickle decoys (skipped on case-insensitive volumes, where no bypass
exists).
* Studio: resolve indexes and shards exactly as from_pretrained does
Two more loader-fidelity gaps in the offline index scan:
- Shard lookup normalized backslashes to forward slashes. On POSIX a backslash
is a literal filename character, so an index naming dir\payload.bin matches a
real pickle of that exact name that Transformers joins and deserializes, while
the normalized dir/payload.bin missed it. Join the raw weight_map value with
os.path.join so the probe mirrors the loader on each platform.
- Index detection case-folded the directory listing, so on a case-sensitive
filesystem an uppercase PYTORCH_MODEL.BIN.INDEX.JSON artifact the loader never
opens was treated as live and its shard blocked. Probe the canonical name with
the loader's own is_file lookup instead, so an index counts only where
from_pretrained would actually load it.
Update the uppercase-index tests to assert the correct per-filesystem behavior
and add a POSIX backslash-shard regression test.
---------
Co-authored-by: danielhanchen <unslothai@gmail.com>
|
||
oobabooga
|
dbb06ff60e
|
Studio: add configurable model download location (#7274)
Adds a configurable Hugging Face model download cache location to Unsloth Studio, selectable from Settings, with per-cache download manifests, scoped deletion, and read-only inventory of previously selected caches. |
||
Hakan Baysal
|
aa49c0710e
|
studio: classify embedding models from the HF cache and honor offline mode (#7218)
* studio: classify embedding models from the HF cache and honor offline mode is_embedding_model() went straight to huggingface_hub.model_info() for any repo id, so in offline mode (no DNS, or HF_HUB_OFFLINE set) selecting an already-downloaded model hung on network retries that could never succeed and training/export never started (#6817). Check the local HF cache first: a sentence-transformers repo carries modules.json in its snapshot (the same marker used for local paths), so a cached model is classified with no network call. When HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE is set, anything not positively an embedding model returns False without a network call instead of retrying a doomed request. Online, uncached lookups still fall through to model_info(), so tag-only embedding models (feature-extraction) are unaffected. Adds _embedding_marker_in_hf_cache() over the existing _iter_hf_cache_snapshots. * studio: judge the active cached revision, harden the cache probe, stop stub leaks Three review fixes on the cache-first embedding detection: 1. Prefer the revision refs/main resolves to. The HF cache keeps snapshots of older revisions, so an any-snapshot scan could classify a repo by a stale revision -- e.g. a repo that used to be a sentence-transformers model would short-circuit even the online lookup. When refs/main is recorded, only its snapshot is consulted; the newest-first scan remains the fallback for caches with no ref. 2. Keep the cache probe inside the detection error boundary. The snapshot iterator stat()s entries and could raise if a cached model is deleted concurrently, propagating a 500 out of the config/check-embedding routes. _embedding_marker_in_hf_cache now catches everything and reads as not-cached, so callers keep their normal Hub/offline fallback. 3. Stub loggers/structlog in the test only when the real modules are absent (try-import, mirroring test_windows_gpu_detection_mock), so collecting this file first can no longer shadow the real packages for later tests in the same pytest process. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: treat a missing active-ref snapshot as a cache miss, don't cache offline misses Two review fixes on the cache-first embedding detection: 1. When refs/main is recorded but points at a commit whose snapshot dir is absent (partial download / cache pruning), the recorded ref is still authoritative: return None (cache miss) instead of falling through to scan older snapshots, which could report a stale historical revision's modules.json as the active one -- the same stale-cache class this helper avoids. 2. Do not cache the offline negative. When HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE is set and the repo is not positively an ST model from modules.json, is_embedding_model stored False under the (model_name, hf_token) key shared with online lookups; after the env var cleared in the same process, a tag-only (feature-extraction) embedder returned the cached False and never reached model_info(). The offline negative is now returned without caching. * studio: defer online embedding detection to the Hub, re-probe offline The local modules.json marker short-circuited is_embedding_model() even online, so a repo that dropped (or added) the marker since it was cached was judged by its stale local revision instead of the current remote one. Online now treats model_info() as authoritative and uses the cache marker only as an uncached fallback when the Hub is unreachable, so a transient failure never poisons the memo. Offline re-probes the marker on every call without consulting or populating the memo, so a model downloaded later in the session (or a cached online negative that predates the download) is detected. _embedding_marker_in_hf_cache() now treats an unreadable refs/main (a non-FileNotFoundError OSError) as a cache miss rather than scanning stale history -- only a genuinely missing ref enables the fallback scan. * studio: harden offline embedding detection against empty refs, offline flips, and cache casing - _embedding_marker_in_hf_cache: an existing-but-empty/whitespace refs/main (a partial write or in-progress truncate-and-rewrite) now reads as a cache miss (None) instead of falling through to scan stale snapshots; only a genuinely missing ref enables the historical scan. - is_embedding_model: while offline, retain a positive already confirmed online this session (model_info only ever memoizes Hub-derived results), so _hf_offline_if_dns_dead() flipping the process to offline mid-load can't downgrade a verified tag-only embedder to False. Cached negatives are still bypassed and re-probed. - resolve_cached_repo_casing + settings route: persist the embedding model in the casing its local HF cache dir uses. Validation accepts a case-insensitive cache hit, but an offline SentenceTransformer load resolves the cache by exact case, so storing the requested spelling (baai/bge-m3 vs models--BAAI--bge-m3) made the model fail to load on a case-sensitive filesystem. * studio: reuse the exact-match-first case resolver and preserve the default Replace the ad-hoc resolve_cached_repo_casing with the existing resolve_cached_repo_id_case, which already prefers the exact-case cache dir before any case variant and tie-breaks variants deterministically -- so an exact requested id is never rewritten to a differently cased directory just because iterdir() happened to yield it first. Skip the normalization entirely when the submitted model equals the default: rewriting its casing would make set_rag_embedding_model()'s exact-string default comparison treat it as a custom override, pinning it so later changes to the configured default stop taking effect. * studio: don't let a stale cache marker mask a permanent Hub error is_embedding_model's Hub-failure fallback consulted the local modules.json marker for ANY model_info() exception, so a permanent error -- a deleted repo, a gated repo without credentials, or a typo that matches stale cache casing -- could pass online validation on a stale marker instead of returning the documented 409, and the persisted model could then fail when the loader refreshes from the Hub. Classify permanent Hub errors (RepositoryNotFound, GatedRepo, RevisionNotFound, EntryNotFound) as False, matching the nearby GGUF/vision detectors, and reserve the cache fallback for transient/5xx failures. * studio: honor TRANSFORMERS_OFFLINE in the embedding preflight, skip casing for local paths - The embedding-model save reached the offline-aware is_embedding_model() only after two preflight helpers made direct huggingface_hub calls that honor just HF_HUB_OFFLINE: _st_module_subdirs() downloads modules.json and the security scan fetches Hub metadata twice. In a TRANSFORMERS_OFFLINE-only session those blocked on network timeouts before the offline return, so saving an already cached model stalled. Both now consult a canonical hf_env_offline() helper -- the download passes local_files_only, and the metadata-only security scan short-circuits to its documented fail-open instead of burning both timeouts. - Skip cache-casing normalization for local paths: a relative directory such as "org/model" is loaded from disk, so rewriting it to a case-insensitive HF cache collision ("Org/model") would stop resolving to that directory and be read as a Hub repo id instead. * studio: never skip the security scan on TRANSFORMERS_OFFLINE alone The previous commit skipped the Hub security scan whenever either offline flag was set, but huggingface_hub honors only HF_HUB_OFFLINE: under a TRANSFORMERS_OFFLINE-only session the later SentenceTransformer load still reaches the network, so the scan was being skipped while the repo's pickle could still be downloaded and deserialized -- waving through exactly what _guard_model_security exists to block. Split the flags: hf_hub_offline() (HF_HUB_OFFLINE, the only one that actually prevents a fetch) gates the security short-circuit, while hf_env_offline() (either flag, the user's intent) is used only where local-only behavior is forced explicitly. The SentenceTransformer load now passes local_files_only from that intent, so TRANSFORMERS_OFFLINE genuinely stops the loader fetching instead of merely being assumed to. * studio: short-circuit the security preflight under either offline flag With the loader now pinned to the local cache by local_files_only = hf_env_offline(), a TRANSFORMERS_OFFLINE-only session can no longer fetch anything -- yet the preflight still fell through to two model_info() attempts on 10s and 20s timeouts, stalling every save and load of an already-cached embedder for half a minute before failing open anyway. Skip the metadata-only scan whenever either flag is set. The scan's job is to stop a poisoned pickle being downloaded and deserialized, and nothing can be downloaded under that predicate; the residual case -- a model cached BEFORE it was flagged -- is the same fail-open this function has always documented for an unavailable scan, and is exactly what HF_HUB_OFFLINE already did. That safety argument depends on every loader behind the gate honoring the same predicate, so it is pinned as a test invariant instead of a comment: removing local_files_only from the SentenceTransformer construction now fails the suite. Drops the short-lived hf_hub_offline() helper, which no longer has a caller. * studio: scope the offline scan bypass to callers that load local-only The previous commit put the offline short-circuit inside _fetch_security_status, which is the malware gate shared by every loader -- so TRANSFORMERS_OFFLINE=1 disabled it for all of them, while only the RAG embedder had been changed to pass local_files_only. MLX inference (core/inference/worker.py -> FastMLXModel .from_pretrained), training and export call from_pretrained with no local-only argument, and huggingface_hub ignores that flag, so those paths could still fetch and deserialize an unscanned model with the gate switched off. The bypass is now an explicit local_only_load argument, defaulting to False, and only the two RAG embedding callers -- whose loader is pinned to the local cache by the same predicate -- opt in. Tests pin both halves: the shared gate must still scan under either offline flag by default, and no other caller may pass local_only_load without constraining its loader. * studio: capture offline state once, and probe the ST cache root Two holes in the offline embedding path: - _get() read hf_env_offline() twice: once inside _guard_model_security and again for local_files_only. _hf_offline_if_dns_dead() mutates the process-wide offline vars and restores them on exit, so a concurrent load could see True in the guard -- skipping the Hub malware scan -- and False by the time the constructor ran, fetching and deserializing the unscanned repo and breaking the very invariant that licenses the bypass. The value is now read once in _get() and passed to both; _guard_model_security takes it as an argument instead of re-deriving it. - The cache probe searched only HF_HUB_CACHE. SentenceTransformer downloads into SENTENCE_TRANSFORMERS_HOME when that is set, using the same models--org--name/snapshots layout under a different root, so a model fully present there looked uncached and was rejected with a 409 offline even though the local-only loader could load it. Snapshot lookup now covers both roots. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: probe the cache the ST loader actually uses, and require it be loadable Adding SENTENCE_TRANSFORMERS_HOME to the shared snapshot iterator was too broad in one direction and too narrow in another: - _get() builds SentenceTransformer with no cache_folder, so with ST_HOME set it searches THAT root only, never the Hub cache. Probing the union let offline validation pass on a repo cached only in the Hub cache, after which the loader looked in ST_HOME and failed. The Sentence-Transformers probe now resolves to exactly one root: ST_HOME when set, the Hub cache otherwise. - The shared iterator is also used by the GGUF detectors, whose downloads go through hf_hub_download with no cache_dir and therefore really do use the Hub cache. It is back to Hub-cache-only so detection cannot pick a snapshot the GGUF load will not find. - Casing normalization ran through resolve_cached_repo_id_case, which scans the Hub cache, so with ST_HOME set the requested spelling was persisted unchanged and the exact-case offline load missed the differently cased directory that detection had just accepted. It now resolves against the same roots detection uses, exact match first. - A snapshot carrying only modules.json no longer counts as cached: the online security preflight downloads that single file itself, and a partial download leaves it behind, so validation passed for a snapshot with no weights and the first RAG load then failed. A hit now requires the marker plus a config and at least one weight file. * studio: thread the captured offline state into the module probe, fix the gate shard - _st_module_subdirs() re-read the process env for its local_files_only. With _hf_offline_if_dns_dead() flipping those vars from another thread, a load that captured local_only=False could still force this probe local-only, get () back because modules.json is not cached, and leave the scan with NO module load roots -- a Hub-flagged pickle under 0_Transformer/ would then pass as an unreferenced nested artifact while the loader fetched and deserialized it. It now takes the captured predicate as an argument, and the settings route reads the state once and uses that single value for both the probe and the scan. - Skip ST-cache casing on the llama-server backend. Nothing there loads through SentenceTransformer: the embedder derives a GGUF companion from the saved spelling and fetches it from the HUB cache, so normalizing to an ST_HOME spelling would point it at a repo _hf_gguf_backend_error() never validated (BAAI/bge-m3-GGUF instead of the checked baai/bge-m3-GGUF). - Fix the security-gate shard, which the signature change had broken: the direct _guard_model_security / _st_module_subdirs callers now pass the new argument (they were raising TypeError before reaching any assertion), and the casing tests patch utils.models.resolve_st_cached_repo_id_case, which the route actually calls, instead of the Hub-only resolver it no longer uses -- those patches were being silently ignored. * studio: accept only torch-loadable weights in the offline ST probe; fix re-export lint _snapshot_is_loadable_st_model accepted a cached snapshot whose only weights were .onnx (or .pt), but the RAG loader builds SentenceTransformer with the default torch backend, so such a snapshot passed offline validation and then failed on the first load, the exact validate-then-fail this helper exists to prevent. Restrict _ST_WEIGHT_SUFFIXES to .safetensors and .bin and add a regression test for an ONNX-only snapshot. Also teach scripts/verify_import_hoist.py that names listed in a module-level __all__ are uses, so the legitimately added resolve_st_cached_repo_id_case re-export in utils/models/__init__.py no longer trips HOISTED-IMPORT-UNUSED. Covered by two new self-test cases. * studio: probe the exact repo dir and revision an offline load resolves The cache probe modelled the cache loosely rather than modelling what SentenceTransformer actually does with local_files_only=True: - It merged snapshots across every case-variant repo dir and then read refs/main from whichever held the newest one. With both models--baai--bge-m3 and models--BAAI--bge-m3 present, a complete embedding snapshot in the directory the loader opens could be judged by a newer partial snapshot in the other, failing validation for a usable model. It now selects the ONE directory the loader opens, by the same exact-case-first rule resolve_st_cached_repo_id_case uses to choose the spelling that gets persisted. - It fell back to scanning historical snapshots when refs/main was absent. With local_files_only the default revision is resolved THROUGH that ref, so a snapshot directory alone is not discoverable: the settings request succeeded and the loader then failed at first indexing. A missing, empty or unreadable ref is now a cache miss, and the historical scan is gone. The tests exercise the real lookup against a built cache tree instead of patching the snapshot iterator, so they now cover the directory selection and ref resolution the loader depends on. * studio: record refs/main in the ONNX-only probe test The ONNX-only regression test predates the refs/main requirement, so after that change it returned None (a cache miss for want of a ref) before ever reaching the weight-format check it exists to make. Recording the ref restores its intent: the snapshot resolves, and the answer is False because an ONNX export is not loadable by the RAG loader's default Torch backend. * studio: recognize base-model weight files and gate the offline positive on a materialized snapshot _snapshot_is_loadable_st_model matched any .safetensors/.bin by suffix, so a partial cache carrying only a commonly published non-weight bin such as training_args.bin (or an adapter-only artifact) passed offline validation and then failed the local_files_only load at first indexing. Match recognized Torch base-model weight filenames (model / pytorch_model, including sharded) by name. is_embedding_model retained an online-confirmed positive offline even when no files were cached, so a metadata-only /check-embedding result let an uncached repo be saved and then fail at first indexing. Retain the positive only when the active revision is materialized locally, which still covers a downloaded tag-only embedder whose snapshot carries no modules.json. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: require a complete weight set offline and persist embedder verdicts across restarts Two follow-ups to the offline embedding-model classifier: - _snapshot_is_loadable_st_model now requires a COMPLETE Torch base-model weight set in one snapshot directory, not just any single recognized weight file. A partially downloaded sharded model (model-00001-of-00002 without its sibling) no longer passes offline validation and then fails at first indexing under local_files_only. Weight files are grouped by directory and a directory counts only when it holds a single model.safetensors / pytorch_model.bin or a full shard set whose indices cover 1..total. - Online-confirmed embedder verdicts are now recorded under the resolved Studio home (embedding_verdicts.json). The session memo is lost on exit, so a downloaded tag-only feature-extraction embedder (snapshot present but no modules.json) was misclassified as non-embedding the first offline call after a restart. The offline branch consults this durable allowlist in addition to the memo, still gated on the active revision being materialized on disk, so an uncached repo is never trusted. Writes are best-effort and only positive verdicts are stored. * studio: require complete weights (with shard index) and resolve default casing offline Follow-ups to the offline embedding-model classifier from the latest review: - Trust a recorded embedder verdict (session memo or persisted allowlist) offline only when the active snapshot carries a COMPLETE, loadable weight set, not merely that it is materialized. A partial download (config present, weights missing or an incomplete shard set) makes _embedding_marker_in_hf_cache read False rather than None, so the previous marker-is-not-None gate wrongly returned True and the local_files_only load then failed. Split out _snapshot_has_complete_weights (config plus complete weights, modules.json aside) and _active_snapshot_dir, and gate the known-embedder positive on the weight set. - Require a sharded checkpoint's index map (model.safetensors.index.json / pytorch_model.bin.index.json) in addition to every shard before accepting it: transformers discovers and wires shards through that index, so a complete shard set without it fails the local-only load. - Resolve the embedding model name to its exact cache casing in the RAG loader before constructing SentenceTransformer. The settings route persists that spelling for a custom override but deliberately leaves the configured default verbatim, so a default whose casing differs from the cache dir would miss it and fail offline. Resolving at load time covers the default too; a no-op for a local path or when nothing case-matching is cached, and idempotent for an already-normalized override. Adds regression tests for the partial-snapshot verdict, the missing shard index, and the loader casing resolution; updates the offline-invariant source assertion to the resolved-name variable. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: require a tokenizer, case-fold verdict ids, and serialize verdict writes Three follow-ups to the offline embedding-model classifier from the latest review: - _snapshot_has_complete_weights now also requires a tokenizer asset. A SentenceTransformer Transformer module builds an AutoTokenizer, so a snapshot with a complete weight set but no tokenizer.json / tokenizer_config.json / vocab still fails the local_files_only load. The check is a permissive union over the common fast-tokenizer, config, and WordPiece/BPE/SentencePiece assets, so an unusual but valid layout is not rejected -- only a genuinely tokenizer-less partial download. - The persisted embedder allowlist is now keyed case-insensitively. model_info() is queried under the requested casing while the settings route saves the cache-resolved casing, so an exact-string lookup missed the persisted positive after a restart (baai/model recorded, BAAI/model looked up) and a loadable tag-only embedder was rejected. Both persist and lookup case-fold the id. - _persist_embedder serializes its read-modify-write under a lock and writes through a per-thread temp file, so concurrent confirmations of different embedders no longer drop each other's entry or collide on the temp path. Cross-process writers stay best-effort (os.replace is atomic; a dropped verdict is only an optimization miss a later online re-confirmation heals). Adds regression tests for the missing-tokenizer reject, alternate tokenizer assets, cross-casing verdict match, and concurrent verdict writes; updates the snapshot test helpers to materialize a tokenizer alongside config and weights. * studio: tighten comments in the offline embedding-model classifier Comment-only pass over the PR's changed files. Collapse the long block comments and docstrings around is_embedding_model, the cache-snapshot and weight-completeness helpers, the embedder-verdict persistence, the offline security gate, and the offline/casing tests to short one- or two-line forms. Preserve the rationale (issue #6817, the local_files_only invariant, the casing and weight-gate reasons) in far fewer words. No code changes. * studio: drop redundant comments in the offline embedding-model classifier Second comment-reduction pass over the offline embedding-model cache work: delete comments and trailing notes that restate the adjacent code or an assertion, and trim the remaining docstrings and rationale comments to their load-bearing invariants. Comments and docstrings only; no code changes. * studio: pin embedder verdicts to a revision, canonicalize default aliases - A persisted verdict recorded that the Hub tagged ONE revision an embedder, but was stored per repo. Once refs/main advanced to a complete but non-embedding Transformer snapshot, the offline path still returned True: the settings route accepted the updated model without force and RAG could silently load it as an embedder. Verdicts now carry the commit they were confirmed at and are trusted only while the active revision matches. One confirmed before the repo was cached has no revision to compare, so the first revision observed afterwards is pinned then -- which is what lets a later advance be caught. The persisted file gains a {id: commit} form and still reads the previous list format. - tokenizer_config.json no longer counts as a tokenizer asset. It only DESCRIBES a tokenizer, so a snapshot with config, weights and just that file passed validation and then failed AutoTokenizer.from_pretrained(local_files_only=True) at first indexing for common BERT/GPT-style models. - A casing-only alias of the default is canonicalized to the default up front. Repo ids are case-insensitive but every gate here compares exact strings, so saving "Unsloth/bge-m3" against a default of "unsloth/bge-m3" ran the verification and scan for a custom model and then persisted an override -- after which later changes to the configured default stopped applying. - verify_import_hoist.py replays __all__ assignments in order instead of unioning them. Only the final value exports anything, so a later plain "=" that drops a name must leave its import counted as unused; "+=" still extends, and an unreadable rebind keeps the earlier names rather than flagging real re-exports. * studio: validate the real ST load root, and pin verdicts to the Hub revision Four ways the offline probe still disagreed with what the loader does: - Verdicts were pinned to the LOCAL refs/main, but model_info() describes the current HUB revision. With a stale cache the two differ, so an older snapshot nobody verified was allowlisted. The pin is now info.sha, taken from the ModelInfo that produced the positive. A verdict carrying no revision (a legacy entry) is no longer trusted at all -- trusting it meant pinning whatever happened to be cached, which is the same bug; the next online check re-records it properly. - config, tokenizer and weights had to exist somewhere in the snapshot, not together. modules.json can send SentenceTransformer at 0_Transformer/, which is loaded FROM that directory, so a cache with the config at the root and only 0_Transformer/model.safetensors passed and then failed the local-only load. Each directory is now checked as a complete load root, which covers both the plain HF layout and the ST module layout. - vocab.json and merges.txt counted independently, but BPE needs the pair unless a serialized tokenizer.json is present, so half a pair validated and then failed AutoTokenizer.from_pretrained(local_files_only=True). - A slashless short name like all-MiniLM-L6-v2 is a supported ST alias that the loader resolves through the sentence-transformers/ organization, so its snapshot is cached under that full id. Probing only the bare name reported a miss and 409'd a model that was cached and loadable; the bare id is still tried first, matching the loader's own order. * studio: fail closed for an offline security scan instead of failing open A local_only (offline) load cannot fetch Hugging Face's malware scan, and the previous behaviour skipped the scan and failed OPEN, so a cached repo with a poisoned pickle weight could deserialize under SentenceTransformer(local_files_only=True). Evaluate it fail-CLOSED against the cached files instead: block a base-model pickle weight the load would deserialize (pytorch_model.bin and its shards, in a directory with no safetensors alternative) and allow a pickle-free (safetensors / gguf are inert) cache. A cached pickle model must be reloaded online once to be scanned, or shipped as safetensors. Nothing cached is not a security event. _fetch_security_status no longer needs the local_only_load skip (the offline branch is handled in evaluate_file_security). Adds a regression test covering the safetensors-allow and pickle-block paths with no Hub call. * studio: only suppress an offline pickle when a loadable safetensors weight exists The offline security gate treated any .safetensors in a directory as covering a pickle weight, so a cache with pytorch_model.bin beside a bare adapter_model.safetensors (or an orphan shard with no index) passed the fail-closed check even though from_pretrained still selects and deserializes the pickle. Require a genuinely loadable safetensors weight -- an unsharded base file or a complete indexed shard set -- before treating the pickle as covered. Also make the import-hoist analyzer preserve uncertainty when __all__ is extended by a value it cannot read statically (__all__ += dynamic()), matching how it already handles an unreadable rebind, so a dynamically-supplied re-export is not flagged HOISTED-IMPORT-UNUSED. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: scope the offline pickle scan to load paths; reset __all__ opacity on rebind Address three review follow-ups on the offline security gate and the import-hoist analyzer: - The offline pickle scan walked the whole snapshot, so a stray pickle in a non-load subdirectory (archive/, nemo/) that SentenceTransformer never deserializes was blocked. Scope it to real from_pretrained load roots -- the snapshot root, or a subdir that holds its own config.json -- matching the online scan's load-path scoping. - _collect_dunder_all kept a sticky opaque flag: a readable replacing assignment after an unreadable extend (__all__ += dynamic(); __all__ = []) still credited every import, so a genuinely unused hoist went unreported. A replacing assignment now resets opacity. - A bare __all__: list[str] annotation has no runtime value; it was treated as an unreadable assignment and marked the export set opaque. Skip annotation-only declarations. * studio: recase slashless ST aliases and accept a pinned embedder after a transient failure Two offline-detection gaps on well-formed input: - resolve_st_cached_repo_id_case bailed on every slashless name, so a differently-cased short alias (all-minilm-l6-v2) validated case-insensitively but was loaded verbatim; the SentenceTransformer loader rewrites it to sentence-transformers/all-minilm-l6-v2 and looks it up case-sensitively, missing the canonical sentence-transformers/all-MiniLM-L6-v2 cache dir. Resolve through _st_cache_repo_dir, which follows the same org alias, and hand back the on-disk casing. - On a transient (non-permanent) Hub failure, is_embedding_model only accepted a cached modules.json marker, so a downloaded tag-only embedder (no modules.json) with a verdict pinned to the active revision was rejected even though the offline branch accepts the identical cache. Mirror the offline branch's pinned-verdict acceptance. * studio: scan modules.json-declared module roots in the offline pickle gate The offline pickle scan treated only the snapshot root and config.json-bearing subdirs as load roots, so a pickle in a non-Transformer SentenceTransformer module directory that has no config.json (e.g. a 0_WordEmbeddings/ module: wordembedding_config.json + pytorch_model.bin) was skipped even though the loader deserializes it. Parse modules.json (and thread through load_subdirs) to treat every declared module directory as a load root, so such a pickle is scanned and fail-closed offline. * studio: classify cached non-Transformer SentenceTransformer models offline _snapshot_has_complete_weights recognized only a Transformer-shaped load root (config + tokenizer + weights co-located), so a fully-cached model built from a non-Transformer module (0_WordEmbeddings uses wordembedding_config.json + embedding weights and its own tokenizer, no HF config.json; BoW keeps its vocab in config.json) was classified non-embedding offline and the settings endpoint returned 409. Add _snapshot_modules_all_loadable, which parses modules.json and accepts a snapshot when every declared module's path directory carries the files that module class's own load() reads (a Transformer/root module still needs the full HF load root; a WordEmbeddings module needs its config plus a complete weight set; other modules need their *_config.json), and at least one embedding-producing module is present. It is OR-ed after the Transformer check, so it only ever accepts more and cannot regress the existing path or reject a pruned cache. * studio: scan PEFT adapter pickle weights in the offline security gate from_pretrained auto-detects an adapter_config.json in the load root and deserializes the adapter weights on top of the base model, so adapter_model.bin is a separate pickle RCE vector that a safetensors base weight does not cover. The offline scan matched only base-model pickle names, so an offline local-only load with safetensors base weights plus a cached adapter_model.bin was allowed despite the live adapter pickle. Scan adapter pickles too, scoped to a load root where adapter_config.json is present and no adapter_model.safetensors exists. * studio: require weights for Dense/CNN/LSTM SentenceTransformer modules offline _module_dir_is_loadable accepted a Dense, CNN, or LSTM module dir with only its config, but those modules' load() hard-load model.safetensors else pytorch_model.bin (verified against sentence-transformers source: no fallback, raises if neither exists) -- exactly like WordEmbeddings. A cache with such a module's config but no weights would validate and then fail the local_files_only load. Require a complete weight set for every weighted module, not just WordEmbeddings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: scan root-index subdir pickle shards offline; handle __all__.append/.extend - The offline pickle scan followed only load-root directories, so a shard mapped by a root pytorch_model.bin.index.json into a non-root subdirectory was skipped even though from_pretrained follows the index weight_map and deserializes it (a layout an attacker can craft to evade the scanner). Read the local index and scan its referenced pickle shards, covered by a loadable base safetensors at the index root -- mirroring the online scan. - The import-hoist analyzer ignored __all__.append("X") / __all__.extend([...]) runtime re-export mutators, so an import added solely for one tripped HOISTED-IMPORT-UNUSED. Read their string args like +=, and treat any other __all__ method call as opaque. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: classify StaticEmbedding offline, require WordEmbeddings tokenizer, bound model_info - A StaticEmbedding module (e.g. sentence-transformers/static-retrieval-mrl-en-v1's 0_StaticEmbedding/) holds tokenizer.json + weights and NO config, so the config-gated non-Transformer path 409'd it offline. Recognize it by what StaticEmbedding.load() reads: a tokenizer.json plus a complete Torch weight set. - WordEmbeddings.load() rebuilds its tokenizer via the configured tokenizer_class.load() from the module dir, so a WordEmbeddings module now also requires a tokenizer artifact (whitespacetokenizer_config.json / phrasetokenizer_config.json, or a shared HF tokenizer asset), not just its config + weights. - With neither offline env var set, an unbounded model_info() could hang on connect/DNS retries for networkless users (the #6817 symptom). Bound it with a 15s timeout so a dead network fails fast and the existing transient-failure cache fallback resolves a cached model, while a reachable Hub still wins. (Documented caveat: a stalled DNS getaddrinfo may exceed this.) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve indexed safetensors shards relative to their index _safetensors_index_complete compared shard basenames against the flat set of files in the index directory, so an index whose weight_map names shards in a subdirectory was treated as incomplete whenever a legacy pytorch_model.bin sat beside it. That falsely blocked a snapshot whose pickle weights are fully covered by a complete, loadable safetensors shard set. Resolve each shard path relative to the index directory instead, and add a regression test for the subdir-mapped shard case. * Restrict offline weight-completeness check to declared load roots _snapshot_has_complete_weights scanned every directory in a snapshot and accepted it when ANY directory was a complete Transformer load root. When modules.json is present a SentenceTransformer load only opens the declared module paths, so a snapshot whose declared modules are incomplete but which happens to contain an unrelated complete directory was accepted offline and then failed at the first local_files_only load. Restrict the candidate directories to the roots a load actually opens: the snapshot root plus each modules.json module path. For a well-formed snapshot the verdict is unchanged; only a complete directory at an undeclared path no longer vouches for an otherwise-incomplete snapshot. * Scan SentenceTransformer Router child module weights offline A Router (legacy Asym) snapshot declares its child sub-modules only in router_config.json, not the top-level modules.json, and Router.load() deserializes each child's weights from its own subdir. A config.json-less child such as query_0_WordEmbeddings (wordembedding_config.json plus a pickle pytorch_model.bin loaded via torch.load) was therefore neither a modules.json-declared load root nor a config.json-bearing dir, so the offline gate skipped its pickle even though the loader deserializes it. Parse router_config.json at each load root and treat every declared child subdir as a load root (bounded BFS, so nested routers are covered), so those child pickles are scanned. Add Router regression tests: a pickle child blocks, a safetensors child is allowed, and a Router in a declared subfolder is followed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not treat an unreferenced config subdir as an offline load root The offline pickle gate skipped a directory only when it was neither a declared load root nor held a config.json. Because _st_load_roots already resolves every real load root (snapshot root, modules.json / load_subdirs dirs, Router children), the config.json fallback only ever promoted an UNREFERENCED subdir -- a nested checkpoint-500/ or archive/ that ships its own config.json + pytorch_model.bin -- to a load root. from_pretrained never descends into such a subdir and the online scan ignores the same unindexed pickle, so offline mode wrongly blocked a model the loader reads from a clean safetensors root. Scope the pickle to directory in roots only, and add a regression test (a stray checkpoint-500/ no longer blocks; a modules.json-declared module dir still does). * Classify a root Router (Asym) model as loadable offline _module_dir_is_loadable applied Transformer root requirements (config + tokenizer + weights) to every root module, so a Router saved at the snapshot root -- which carries only modules.json + router_config.json and loads its weights from child subdirs -- was classified not loadable offline, and is_embedding_model missed a cached Router embedder. Dispatch on the module class before the root Transformer fallback: a Router/Asym dir is loadable when router_config.json parses and every declared child subdir is loadable (validated recursively through _module_dir_is_loadable, so nested routers and every child type are covered) with at least one embedding-producing child. This also tightens a non-root Router, which previously validated on the mere presence of router_config.json without checking its children. Add Router regression tests (root and declared subfolder, complete and incomplete-child). * Require every declared module before accepting an offline cache _snapshot_is_loadable_st_model returned has_complete_weights OR modules_all_loadable, so a complete 0_Transformer short-circuited the or and vouched for the whole snapshot even when a declared sibling module was missing its serialized weights; SentenceTransformer builds every module in modules.json, so that snapshot passed offline validation and then failed the local-only load. When modules.json declares a non-empty list it is now authoritative (modules_all_loadable validates every declared module); has_complete_weights stays the fallback only for an empty/non-list modules.json (the plain from_pretrained root). Also add the weight-bearing modules whose load() hard-loads via load_torch_weights and previously fell to the config-only path -- LayerNorm, WeightedLayerPooling, SparseAutoEncoder -- to _ST_WEIGHTED_MODULE_NAMES, with source citations and the deliberate exclusions (Pooling/Normalize/BoW/WordWeights read no weights on load). Add parametrized regression tests over LayerNorm/WeightedLayerPooling/Dense (a weightless sibling rejects, a complete sibling accepts). * Reject self-referential Router children instead of recursing forever _router_dir_is_loadable validates each router_config.json child through _module_dir_is_loadable, which re-enters _router_dir_is_loadable for a Router child. A malformed types entry naming the router's own directory (a key of ".", which normalizes to the same dir) made that recursion never descend, so it looped until RecursionError -- breaking the documented never-raises contract and turning a crafted/corrupted cached model into a 500 from is_embedding_model instead of a graceful unverifiable result. A real child reference is a subdir and always resolves deeper, so reject any child whose resolved path is the router dir itself. Add a regression test (a router_config naming "." as a Router child returns False without raising). * Treat a destructuring __all__ assignment as opaque _collect_dunder_all detected __all__ only as a direct ast.Name assignment target, so a binding through a destructuring target (__all__, meta = [...], v -> an ast.Tuple) was skipped entirely, leaving an empty, non-opaque export set. A newly hoisted import re-exported only through that assignment was then falsely flagged HOISTED-IMPORT-UNUSED. Its value cannot be mapped statically, so mark the export set opaque when __all__ is reached only through a destructuring / item / attr target, matching how the collector already handles other unreadable __all__ forms. Add a self-test case. * Canonicalize declared module paths before scoping the offline pickle gate A repo could declare a traversing module path such as 0/../evil in modules.json (or a router_config child), which SentenceTransformer resolves to evil/ and deserializes evil/pytorch_model.bin. _st_load_roots recorded the raw snap/"0/../evil", which never equals the snap/evil that rglob yields, so the offline pickle gate skipped that directory and a malicious repo slipped a pickle past the newly added gate. Add _canonical_load_dir to collapse ./ and ../ components lexically and reject an upward escape, and route the modules.json paths, load_subdirs and router children through it so the gate scopes the same normalized directory the loader opens. Add regression tests for a traversing modules.json path and router child. * Close offline embedding-classification completeness gaps Five real offline misclassifications, each a false negative (the #6817 hang recurs) or false positive (accepted then 409s at the local_files_only load). Dispatch _module_dir_is_loadable on the module class before the root Transformer fallback. A module with save_in_root=True (every InputModule: WordEmbeddings, StaticEmbedding, SparseStaticEmbedding, Transformer, Router) is saved at the snapshot root, so a root WordEmbeddings was wrongly held to Transformer requirements (an HF tokenizer it never writes) and classified not loadable. CLIPModel is Transformer-shaped: CLIPModel.load() reads AutoModel weights plus AutoProcessor, so a config-only CLIP dir must not validate. SparseStaticEmbedding needs a tokenizer plus either idf.json or a complete torch weight set (conditionally weight-bearing); a config alone is not enough. A present but empty or malformed modules.json is not loadable and does not fall back to a root Transformer: with modules.json present the loader never takes the plain-Transformer path (base/model.py _load_config_modules). The tag-only no-modules.json embedder is classified separately via _snapshot_has_complete_weights. Validate a sharded weight index against its weight_map (every mapped shard present, resolved relative to the index dir) instead of trusting the index file's mere existence, mirroring the security-side check. Add regression tests for all five. * Close case-folding and online-traversal holes in the offline pickle gate Two gate bypasses where the security scan credited or scoped a path differently from what the loader actually resolves: The safetensors credit was case-folded. _cached_pickle_weight_files lowercases every filename, and the loadable-safetensors and adapter checks tested those folded keys against the exact-lowercase names. On a case-sensitive filesystem (Linux, the Studio default) a crafted repo shipping Model.SafeTensors plus a malicious pytorch_model.bin makes transformers and sentence-transformers miss the exact-name model.safetensors and deserialize the pickle, while the gate credited an inert safetensors and did not block. Credit safetensors case-sensitively against real filenames, and drop pytorch_model.safetensors from the credit set (transformers loads only model.safetensors, never that name). Pickle matching stays case-insensitive (over-blocking a mis-cased pickle the loader would not load is the safe direction). The online scan did not canonicalize traversing paths while the offline gate did. A repo-controlled modules.json path (threaded into the online scan via the RAG guard) or a weight_map shard entry like 0/../evil / ../evil was compared verbatim, so a flagged evil/pytorch_model.bin never matched and evaded the online scan though the loader resolves and deserializes it. Canonicalize the repo-controlled load-subdir prefixes and weight_map shards the same way the offline gate does, so offline and online agree. Add regression tests for both bypasses. * Treat a conditional __all__ mutation as opaque in the import-hoist linter _collect_dunder_all replayed only top-level module statements, so an __all__ assignment or mutation inside a module-level if / try / for / while / with / match (or a deeper scope) was ignored, leaving the export set understated. A newly hoisted import re-exported only through such a conditional __all__ was then falsely flagged HOISTED-IMPORT-UNUSED, blocking a valid change. A conditional value cannot be replayed statically, so mark the export set opaque when __all__ is bound or mutated anywhere other than a top-level statement. Add a self-test case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope Router child sub-modules as load roots in the online embedding scan The RAG embedding security guard unions the SentenceTransformer module dirs from modules.json into the load roots it scopes for the Hub scan, so a flagged pickle directly under a Transformer module blocks. A Router (legacy Asym) module declares its child sub-modules only in router_config.json, not in modules.json, and Router.load() deserializes each child from its own subdir. The online scan therefore dropped a flagged child pickle (for example query_0_WordEmbeddings/pytorch_model.bin) as an unreferenced nested shard while the loader still deserialized it, the counterpart to the offline gate which already expands router children via _router_child_dirs. _st_module_subdirs now reads router_config.json for any Router-typed module and adds each declared child (joined onto the module path, canonicalized so a traversing entry is dropped) to the load roots. The config is read only for a Router-typed module, so a plain embedder pays no extra fetch, and every failure path still returns () so the guard never bricks the embedder. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Allow a recorded-clean pickle embedder to load offline The offline embedding security gate is fail-closed: with no network to reach Hugging Face's scan, a cached pickle weight cannot be verified, so it is blocked and a model the user already downloaded and used online will not load offline. This adds a persistent cache of clean Hub verdicts so that exact content can load offline, without weakening the gate for an unknown or never-scanned pickle. When an embedding repo is loaded online and HF's scan returns a completed clean verdict, the load roots are hashed and recorded under the scanned commit as an exact map of snapshot-relative pickle name to sha256, in a per-user JSON store at studio_root()/security/embedding_scan_verdicts.json (atomic write, 0600, thread and cross-process locked, 30-day TTL). Offline, a cached pickle model loads only when the active cached commit and every load-root pickle's sha256 match the recorded verdict; a missing record, moved commit, changed or added pickle, expired record, or any error keeps blocking. Online loads always re-query the Hub and an authoritative unsafe verdict deletes any stale record, so a now-flagged commit cannot keep loading on an old clean record. The store binds repo id, full commit, and a per-file sha256 map so a locally swapped pickle at the same commit, a branch advance, or an added load-relevant pickle is detected. A same-user attacker who can rewrite the model cache or the store is outside the enforceable boundary and this is documented; the sha256 is computed just before load, so a narrow verify-to-load window remains, and a Hub scanner false negative is recorded faithfully (safetensors stays the stronger defense). Recording is triggered post-load in the RAG embedder because the settings route only validates and the pre-load guard runs before the constructor downloads; recording is skipped when the loaded commit differs from the scanned commit. The blocked-pickle enumerator now returns snapshot-relative Paths so two module dirs that ship the same pickle basename are hashed and reported distinctly. * Harden the embedding verdict cache against review findings Tighten the offline verdict cache and its enumeration so every uncertain or malformed input fails closed and the recorded hashes always match the files the loader reads: - Hash every case-colliding pickle in a load root, not one representative. On a case-sensitive filesystem pytorch_model.bin and PYTORCH_MODEL.BIN are distinct files; keying by lowered name dropped one and could hash a decoy instead of the loader's target. The enumerator now returns every variant Path. - Only persist a clean verdict for a COMPLETED, entirely-benign scan. Require scansDone to be the boolean True (not a truthy string), filesWithIssues to be a well-formed list, and every flagged file to be a definitively-safe level; a pending, error, unknown, or malformed entry no longer records as clean. The online block decision is unchanged. - Fail closed when the offline cache cannot be inspected: an rglob error now propagates and blocks instead of reading as pickle-free, and a snapshot that errors on resolution (vs a clean not-cached) blocks. The offline guard also raises instead of returning when its own inspection throws, so the constructor never deserializes an unverified cached pickle. - Expand online Router children recursively (bounded BFS with a seen set), mirroring the offline load-root expansion, so a flagged grandchild pickle is scoped online and cannot be recorded clean. - Reject absolute and drive/UNC declared paths in the load-root canonicalizers; the loader would resolve them outside the snapshot, so collapsing them to an in-snapshot relative dir scoped the wrong place. - Pin verdict recording to the scanned commit's snapshot and take the offline verify commit from the snapshot directory name, removing a second refs/main read and the skew it allowed. - Drop the now-unused pickle-name wrapper. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten offline embedding classification and the pickle gate Close a set of offline edge cases where validation accepted a cache the local_files_only load then rejects, and one gate bypass: - Credit a sharded model.safetensors.index.json for a pickle sibling only at a from_pretrained root. A non-Transformer SentenceTransformer module (Dense, WordEmbeddings, StaticEmbedding) loads via Module.load_torch_weights, which reads model.safetensors then pytorch_model.bin and never the index, so a sharded safetensors index in such a module dir must not vouch for its pytorch_model.bin. - Stop counting pytorch_model.safetensors as loadable in the offline classifier: the loader probes model.safetensors (then its index) or pytorch_model.bin, never pytorch_model.safetensors, matching the gate that already treats it as a decoy. - Treat a present but unreadable weight index as incomplete: transformers opens and parses any present index, so a malformed one or one without a weight_map fails the load rather than falling back to filename-numbered shards. - Require the CLIP image-processor config (preprocessor_config.json) for a CLIP module: CLIPModel.load builds a CLIPProcessor that needs it, so a tokenizer alone is not enough. - Require a SparseStaticEmbedding config to actually select idf.json (a path ending .json) or ship loadable weights; a bare idf.json the config does not name falls through to load_torch_weights and raises. - Do not use the tag-only recorded-verdict fallback when modules.json is present: with the file present the loader takes the modules.json path, so a present but empty or malformed manifest must not be validated as a plain root Transformer. - Import-hoist linter: only a module-level conditional mutation or a function that declares global __all__ makes the export set opaque; a __all__ bound as a local in a nested function or class no longer masks a genuinely unused hoisted import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope Router-child pickles to their deepest load root and gate the ST offline kwarg The online scan stripped the first matching load-subdir prefix from a flagged file, so a nested Router child pickle (0_Router/query_0_WordEmbeddings/pytorch_model.bin) matched the parent 0_Router root, looked like an unreferenced nested shard, and slipped the gate even though Router.load() deserializes that child directly. Match the deepest (longest) load subdir instead, so the child becomes root-level under its own load root and blocks. pyproject sets no lower bound on sentence-transformers and the local_files_only constructor arg is absent on older releases, so always forwarding it broke every embedder warm on those installs. Pass it only for an offline load; an online warm never forwards it and works as before, while the offline capability still requires a version that supports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject snapshot-escaping shard paths and credit Transformer submodule safetensors The offline pickle enumerator joined a weight-index weight_map value straight to the load root and followed it, so a repo-controlled index mapping "../.." into a sibling snapshot made an offline from_pretrained deserialize an out-of-snapshot pickle, and an online load would then hash and record that external file as the scanned commit's clean content. Reject any shard path that escapes the snapshot root and fail closed, mirroring the canonical-root check the online shard scan already applies. A complete model.safetensors.index.json was credited over a sibling pickle only at the snapshot root, but a Transformer module subdirectory (0_Transformer/) is loaded via AutoModel.from_pretrained, which honors that shard set and never reads the pickle. Credit the sharded index for Transformer-typed modules declared in modules.json so a cached model that ships both a sharded safetensors checkpoint and an unused PyTorch checkpoint is no longer falsely blocked offline. Non-Transformer modules (Dense, WordEmbeddings, StaticEmbedding) read a flat weight with no index and keep their pickle blocked. Limit the import-hoist verifier's global __all__ scan to the declaring function's own scope so a nested inner-scope local __all__ no longer marks the module export set opaque and mask an unused hoisted import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope Router children against the snapshot and mirror the ST alias rewrite Router.load resolves each child at Path(subfolder, model_id) relative to the Router dir, so a nested 1_Router with a "../evil" child points at evil/ inside the snapshot and the loader deserializes evil/pytorch_model.bin. The offline enumerator canonicalized the child against the Router dir alone and dropped anything with "..", so that pickle was never scanned and the gate reported the cache pickle-free. Canonicalize router children against the snapshot, retaining in-snapshot siblings as load roots and failing closed on a child that escapes the snapshot itself, matching the online scan which already joins the prefix before normalizing. The security gate resolved a slashless model id by probing the bare cache dir first, but the SentenceTransformer constructor rewrites a non-basic slashless name to sentence-transformers/ <name> and loads THAT snapshot (only the basic ORIGINAL_TRANSFORMER_MODELS load bare). With both models--<name> and models--sentence-transformers--<name> cached, the gate inspected the bare dir while the loader read the namespaced one, so a pickle there bypassed the local-only gate. Mirror the constructor: try the namespaced candidate first for non-basic slashless names. Add the same not (snapshot / modules.json).is_file() guard to the transient-Hub-failure tag-only fallback that the offline branch already carries, so a cache whose present manifest is empty or malformed is no longer reported as a loadable embedder. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten root shard credit, module-path escapes, and weight-set probe order Credit the sharded safetensors index at the snapshot ROOT only when the root is actually loaded through an AutoModel/from_pretrained path. A modules.json root module of a non-Transformer type (StaticEmbedding / WordEmbeddings / Dense) loads via load_torch_weights, which reads pytorch_model.bin and ignores the index, so crediting a root shard index there suppressed a live root pickle and let the offline gate report the cache pickle-free. Recognize the Transformer subclasses CLIPModel and MLMTransformer as index-honoring load roots (they load via from_pretrained), so a sharded-safetensors CLIP/MLM submodule with a legacy pytorch_model.bin sibling is no longer falsely blocked offline. Mirrors the classifier dispatch. Fail closed on an absolute or snapshot-escaping modules.json module path (or load_subdirs entry) instead of silently dropping it: SentenceTransformer resolves such a path outside the snapshot and would deserialize an external pytorch_model.bin the gate cannot scan. On the classifier side, walk the weight set in the exact from_pretrained probe order (model.safetensors, its index, pytorch_model.bin, its index) so a pickle behind a malformed safetensors index is no longer accepted as complete, and restrict shard names to the loader-probed stem/ext pairs so a decoy model-*.bin / pytorch_model-*.safetensors set is not treated as loadable. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore scripts/verify_import_hoist.py to main The offline embedding cache fix does not depend on the __all__ scope handling that had accumulated in this linter, so revert the file to its main version and keep the PR focused on the feature. The feature modules still pass the existing import hoist check unchanged. * Reuse a shared HF cache skeleton in the offline classification tests Extract _mk_repo and _activate helpers for the repeated snapshot cache setup that every per-type builder duplicated, and fold the two StaticEmbedding missing-asset cases into one parametrized test. Same 125 collected items, all still passing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reclassify embedding models from the cache on every offline call is_embedding_model consulted its process memo before the offline branch, so an online lookup that memoized True from tags (without caching any weights) was returned unchanged once the session went offline -- the studio flips HF_HUB_OFFLINE in-process on a dead DNS, and the ungated check-embedding route can populate the memo. Settings would then accept a repo the offline loader cannot open. Run the offline cache-marker reclassification ahead of the memo and never record it, so an offline verdict always reflects the local cache and a later cache materialization is not masked by a stale negative. Add regression tests. * Tighten comments on the offline embedding path Condense the offline-embedding helper docstrings and inline comments added in this PR to fewer, clearer lines, keeping the non-obvious security and offline rationale. Comments and docstrings only; no code change. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothai@gmail.com> |
||
|
Daniel Han
|
ab2717afe0
|
Studio: persistent per-user trust_remote_code approval cache (#6551)
* Studio: persistent per-user trust_remote_code approval cache The consent gate pins each approval to a content fingerprint (sha256 over every repo .py), but nothing was persisted, so the dialog reappeared on every fresh load of the same unchanged repo. This adds an on-disk, per-user approval cache that lets the gate skip the dialog when the same user reloads the same code, while keeping the safety guarantees intact. Two-tier validation, both must hold or the user is re-prompted: - Commit SHA (cheap, one HfApi.model_info().sha, no download): a match means a byte-identical tree to the approved revision, so the scan/download is skipped. - Content fingerprint (authoritative): used whenever the SHA is unavailable (local path / offline) and always recomputed on a SHA miss. A new or edited .py changes both the SHA and the fingerprint, so it is caught in every mode. Safety: - Keyed per subject; one user's approval never auto-runs code for another. - CRITICAL is never stored or honored (guarded on both write and read), so a hand-edited store cannot smuggle in an auto-approval. - The malware (HF unsafe-file) gate stays unconditional. - Fail-safe: a corrupt store, an unresolvable SHA, or any error degrades to "ask again", never to "auto-approve". UNSLOTH_TRC_APPROVAL_CACHE_DISABLE=1 turns the cache off entirely. New module utils/security/remote_code_approvals.py holds the store (studio_root()/security/remote_code_approvals.json, atomic write, 0600, RLock) plus the SHA resolvers. Recording happens at the single gate chokepoint when the caller supplies the matching fingerprint, so subject is just threaded through inference/training/export (orchestrators, routes, workers). The scan endpoint returns already_approved so the frontend can skip the dialog on a cache hit. Tests: new tests/test_trc_approval_cache.py covers cache miss, SHA-match skip, SHA-moved re-scan, new-file re-consent, CRITICAL never cached (write + forged read), disable flag, subject isolation, combined adapter+base key, corrupt store, and no-subject bypass. Full security suite: 101 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: make the approval cache skip only the prompt, never the scan Codex found that the SHA "no-scan" fast path could run untrusted code without re-consent. Removed it; the gate now always re-scans and the cache only seeds the authoritative fingerprint check, so it can skip the dialog but never the scan. - CRITICAL is hard-blocked on every load (the scan always runs), so a hand-edited store that downgrades a CRITICAL repo's severity can no longer auto-run it (P2: do not trust editable severity for SHA approvals). - The fingerprint covers external auto_map repos, so changed third-party code always re-prompts even when the primary commit SHA is unchanged; there is no longer a SHA path that bypasses the fingerprint (P1: external auto_map repos). - resolve_commit_sha is resolved fresh on every call (no memoization), so a repo whose default branch moves after approval re-prompts instead of reusing a stale cached SHA (P1: revalidate mutable Hub SHAs). The SHA is now only a conservative secondary gate: a fresh resolvable SHA must match the approved revision, else the seed is withheld; a None (local/offline) falls back to the fingerprint. - Approvals record the scanner ruleset version (SCAN_RULES_VERSION); the gate ignores approvals from an older ruleset so reclassified bytes are re-scanned and re-shown instead of silently auto-approved (P2: invalidate on scan-policy change). Tests: test_trc_approval_cache.py rewritten around the prompt-skip semantics (unchanged repo still scans; SHA move / changed code / scanner-version bump / disable flag all re-prompt; forged downgraded severity still blocks CRITICAL). 105 passed with test_consent_gate.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim comments to be more succinct * Keep run-owner subject out of persisted config; serialize approval writes Threading subject (the run owner's username / API-key id) into the training config meant _sanitize_db_config persisted it into config_json, which training-history GET returns to any authenticated user, leaking who started a run in multi-user installs. Filter subject alongside the token fields; the worker still receives it from the live config. The approval store's RLock only guards one process, but approvals are recorded from separate inference/export/training subprocesses, so concurrent writers could clobber each other on os.replace and drop an approval (re-prompt). Hold a best-effort cross-process file lock around the read-modify-write. * Fail safe on a malformed approval store A store with the right version but a non-dict shape (e.g. a hand-edited "subjects": []) passed _load()'s check, then lookup chained .get() on a list and raised, breaking every remote-code load until the file was removed. Validate that subjects is a dict in _load(), and tolerate a non-dict per-subject entry in lookup/record/forget, so a corrupt store fails safe (re-prompt) instead. * Keep subject out of the MLX W&B run config _run_mlx_training uploads the whole training config to W&B minus a sensitive set that only listed hf_token/wandb_token/s3_config, so the authenticated subject (username / API-key id) was sent to W&B as run config even though DB history already strips it. Add subject to the W&B-sensitive filter, mirroring training._sanitize_db_config. * Tighten the W&B subject-filter comment --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
|
Daniel Han
|
1582d2854c
|
Harden trust_remote_code consent: scan GGUF-only auto_map and drop pre-set TRC defaults (#6478)
* Scan auto_map for GGUF-only repo ids in the consent gate The trust_remote_code consent gate treated any repo classified GGUF-only (ships .gguf, no transformers-loadable weight) as having no remote code, so _config_has_auto_map returned False even when a config declared an auto_map and the repo shipped the referenced .py. The evaluator then skipped the scan/fingerprint for that target entirely. GGUF-inertness is a property of the loader, not the repo. A GGUF selection loads via llama.cpp, which never reads config.json/auto_map, and that case is already short-circuited upstream by the caller's is_gguf check (the inference route skips the remote-code preflight for a GGUF load). Every path that reaches this helper (export, training, non-GGUF inference) loads through transformers/Unsloth from_pretrained, which DOES import auto_map even for a repo that only ships .gguf weights: the custom module runs before from_pretrained fails on the missing transformers weights. The export path has no is_gguf guard and passes the source straight to FastLanguageModel.from_pretrained(trust_remote_code=True), so the in-helper GGUF skip let a repo with config.json (auto_map) + modeling_x.py + only a .gguf run unreviewed code during export. Drop the redundant repo-level GGUF short-circuit (and the now-unused _is_gguf_repo helper). A direct .gguf file reference stays inert via _is_direct_gguf_file_ref because that genuinely is a single-file llama.cpp load; repo ids are always scanned. A GGUF repo whose auto_map ships no .py still allows via the existing empty-code path, so legitimate GGUF loads are unaffected (and GGUF inference never reaches this helper at all). Only a repo that actually contains a .gguf can change behavior here; non-GGUF repos (safetensors, MLX) are byte-identical before and after. Update the GGUF auto_map test to expect a scan, and add two regression tests: a GGUF-only repo shipping auto_map Python is scanned and blocked, and a transformers-style repo (safetensors / MLX .npz) with auto_map stays scanned and blocked. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove trust_remote_code config defaults; consent dialog is the only enabler trust_remote_code is a per-load decision that must go through the remote-code consent dialog, which scans the auto_map code and pins the exact version. Two pre-set paths could still enable it without the user reviewing any code, and the GGUF consent bypass rode one of them into the export flow: - 4 model_defaults YAMLs shipped trust_remote_code: true (GLM-4.7-Flash, Nemotron-3-Nano-30B-A3B, PaddleOCR-VL, ERNIE-4.5-VL). - The frontend consent hook silently enabled trust_remote_code on a clean scan whenever the caller flagged the model as needing it. Remove every trust_remote_code key from the model_defaults YAMLs (the loaders already default to False when the key is absent) and delete the frontend silent auto-enable, so trust_remote_code is only turned on after the user approves the scanned code in the dialog. The three models that genuinely run custom code ship auto_map, which the consent gate detects on its own via _config_has_auto_map, so the dialog still fires for them in inference, training, and export (Nemotron is also re-granted by the trusted-org auto-enable in the workers). GLM-4.7-Flash has no auto_map: glm4_moe_lite is native in transformers 5.0+ and it loads with trust_remote_code=False, so its YAML flag was a no-op. Adds test_yaml_trust_remote_code_removed.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop YAML sections emptied by trust_remote_code removal Removing trust_remote_code from a model YAML whose section had no other key left a bare `inference:` header, which PyYAML parses as None; load_inference_config() then does `model_config.get("inference", {}).get(...)` and crashes on the None. Drop those now-empty section headers (24 model defaults, all the `inference:` section) so callers fall back to family/default inference params, which is the same result those models had before (their only inference override was trust_remote_code). Strengthens test_yaml_trust_remote_code_removed.py to forbid any empty/None top-level section and to load the affected models' inference config end to end. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add sweep asserting every model YAML loads via training + inference paths Loads all model_defaults YAMLs through load_model_defaults (training) and load_inference_config (inference) with the exact .get() access patterns the routes use, so a malformed/None section that crashes either loader is caught. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Assert ex-TRC auto_map models still surface the consent dialog Removing the trust_remote_code YAML default must not suppress the dialog for the models that genuinely run custom code. The dialog is driven by the repo's auto_map (via preflight_remote_code_consent_for_targets -> _config_has_auto_map), not the YAML flag, so Nemotron/PaddleOCR-VL/ERNIE-4.5-VL still require consent; GLM-4.7-Flash (no auto_map) takes no dialog and loads natively. Mocks only the Hub config + .py reader. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten comments in consent-gate changes * Trim comments to be more succinct --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <michaelhan2050@gmail.com> |
||
|
Daniel Han
|
0533efe3f8
|
Harden model fetching (#6391)
* Harden model fetching: consent gate for trust_remote_code Add a load-path consent gate that scans a model's auto_map repository code before it executes and blocks CRITICAL/HIGH findings unless the user pins approval of that exact code version. Capability detection stays code-free, reading raw config.json instead of AutoConfig. - Scan config.json and tokenizer_config.json auto_map, nested local helpers, and external owner/name--module repos; fail closed on partial downloads. - Gate inference, training, and export workers, including the MLX path and a LoRA's base model, and report requires_trust_remote_code from the raw config so chat and auto-load surface the dialog. - Verify trusted-org auto-enable against the Hub with the request token and key the verdict cache by token; reject local-path and spoofed names. - Add a consent dialog showing the flagged file, line, and surrounding code. - Thread hf_token through the scan and load paths for gated repos. * Address review: token handling, tokenizer/LoRA scan coverage, rollback - Send the HF token for remote-code scans in the POST body, not the URL, so it never lands in a log or browser history. - Collect tokenizer_config.json auto_map files directly instead of relying only on the repo file listing. - Resolve a LoRA's base model for the validate flag and the scan endpoint so the dialog scans the code the workers actually gate. - Pass the request token to the training YAML trusted-org auto-enable. - Resend a previously approved fingerprint when rolling back to a custom-code model after a failed switch. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Consent UX: drop legacy chat toggle, fix decline copy, purge declined downloads The per-model consent dialog is now the single approval path for custom (auto_map) code in chat, so three leftovers from before it existed are removed: - Remove the "Enable custom code" switch from Chat Settings and stop persisting trust_remote_code, so a previously saved blanket-on cannot linger and load a model without going through per-version review. The flag stays as an internal YAML/preset default (e.g. first-party auto-enable); the load path still gates every custom-code load on a fingerprint only the dialog produces. - Reword the decline message and the auto-load toast to describe approving the model's code from the dialog, not a missing settings toggle. - On decline, purge the repo the scan downloaded so untrusted code is not left on disk. A new /api/models/discard-remote-code endpoint deletes only a metadata-only cache entry the scan created; it refuses local paths, loaded models, and any repo with weight files cached, so a model the user already had or pre-downloaded is always left untouched. The frontend only calls it when the scan reported created_by_scan. Adds discard-endpoint tests (delete metadata-only, refuse on weights/gguf, refuse local, no-op when not cached) and a created_by_scan payload assertion. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Export: remove the user-facing trust remote code toggle The Export page kept a "Trust remote code" switch (default on) next to the HF token field. Like chat, custom (auto_map) code should be approved per model through the load-time review dialog, not a persistent blanket switch, so the toggle is removed. The export load path already routes through the same consent dialog: an HF source now starts with trust_remote_code off and only enables it when the user approves the scanned code in the dialog (a local checkpoint the user exported stays trusted by default). With the dialog unreachable and no approval, an HF source loads with trust_remote_code off, which fails closed rather than running unreviewed code. * Block loads of repos with unsafe files using Hugging Face's security scan The trust_remote_code consent gate covers one load-time RCE vector (a repo's auto_map Python). It does not cover the other: a malicious pickle inside a weight file (pytorch_model.bin, *.pkl, *.dat) deserializes during from_pretrained even with trust_remote_code False, so a repo with a normal config plus a poisoned pickle slips past the existing gate. Add a metadata-only malware gate that uses Hugging Face's own scan (picklescan + ClamAV), read via model_info(securityStatus=True).security_repo_status. It never downloads, opens, or unpickles the flagged files; it only reads the Hub's verdict and surfaces the flagged file names. New evaluate_file_security runs unconditionally (independent of trust_remote_code) in every load path (inference, training SFT/MLX, export), blocking the load when a file is flagged unsafe/suspicious/malicious. The /remote-code-scan preflight and the validate endpoint also report the result so the consent dialog opens as a hard block (no override) listing the flagged files, even for a repo with no custom code. Policy: hard block with no user override; fail open when the scan is unavailable (offline/unscanned) so legitimate loads are not broken; no first-party exemption (a poisoned pickle in a compromised trusted repo still blocks); local paths and GGUF are skipped (no Hub scan, non-pickle format). Blocking does not gate on scansDone, since that is often false for clean repos and a file already flagged unsafe is unsafe regardless. Adds test_file_security.py covering the block/allow/fail-open/skip matrix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: scan list-form tokenizer auto_map, gate unsafe files on all load paths Fixes from a 10-reviewer pass on the model-fetching hardening: - The remote-code scanner skipped tokenizer auto_map encoded as a [slow, fast] list (transformers' standard tokenizer shape, e.g. {"AutoTokenizer": ["owner/repo--tokenization_x.Slow", null]}). External tokenizer code in that form was never fetched, scanned, or fingerprinted, so an AutoTokenizer(trust_remote_code=True) load could run it. _auto_map_refs now flattens string, list, and nested values. Adds a regression test. - Compare-mode chat loads and background auto-load only gated on requires_trust_remote_code, so a repo flagged unsafe by the Hub scan but with no custom code skipped the hard-block dialog. Both now also gate on requires_security_review, matching the main chat path. - The /remote-code-scan and /validate routes collapsed a LoRA adapter to its base before the malware scan, so unsafe files in the adapter repo itself were missed in the pre-load review (the workers already scan both). Both routes now run the file-security scan over the adapter and the base. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Require approval for all HIGH remote code, fail closed when unscannable Tighten the load-time security gates based on review: Consent gate - HIGH-severity auto_map code now requires explicit, per-version approval for every repo, including first-party unsloth/nvidia. The org is no longer a blanket bypass: a compromised first-party repo with HIGH code still warrants review. CRITICAL stays a hard block; clean code still loads after the consent prompt. - Fail closed when auto_map code is present but cannot be fully fetched or listed to scan (gated, offline, transient, or a repo-listing failure that could hide an imported helper). We cannot fingerprint code we cannot see, so this is a non-approvable block, retryable once the repo is reachable. - Scan auto_map from every config that can carry one (model, tokenizer, image and feature processor, processor, video processor), not just config.json and tokenizer_config.json, so a custom-processor model is not missed. The file list is the single source of truth in remote_code_scan and is pinned to the transformers filename constants by a guard test. - Distinguish a genuine 404 (config truly absent) from a transient error: only the latter forces a scan, so a repo with no config is correctly a no-op. Malware gate - Scan a remote repo even when its name ends in .gguf; only local paths skip the Hub scan, so a repo cannot dodge the scan by naming itself "*.gguf". - Correct the docstring: a file already flagged unsafe blocks regardless of scansDone; the only fail-open path is an unavailable scan. Coverage - Resolve a remote LoRA adapter's base model (not just local directories) so the base, where the code and weights actually execute, is scanned in validate, the scan route, and the training and export workers. - Gate the embedding training path (FastSentenceTransformer) with the malware and consent checks, matching the other load paths. Tests updated and added for each change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope malware gate to the load-path vector; stop false-blocking first-party models Follow-up hardening from a second review pass + a broad live model matrix (unsloth/* , nvidia/* , third-party, and the eicar malware repo). Malware / unsafe-file gate - Scope the block to the actual RCE vector: a root-level file in a code-executing format. from_pretrained deserializes weight files at the repo ROOT, so a flag is only a load-path pickle vector there. Two exclusions, because neither is loaded: inert formats (safetensors is tensor-only, gguf is non-pickle, configs/text/ images) and files in subdirectories. This keeps eicar blocked (its *.pkl/*.dat/ eicar_test_file sit at the repo root) while no longer false-blocking legitimate first-party repos: nvidia/Nemotron-H-8B-Base-8K ships root safetensors plus NeMo pickle checkpoints under nemo/ that the loader never touches, and the Hub flags both; the gate previously hard-blocked it. - Unknown / future non-"safe" levels now fail closed (block) instead of being silently allowed, so Hub schema drift cannot introduce a bypass; in-progress ("pending"/"scanning"/"error") levels stay non-blocking to avoid false blocks. Consent gate - Ignore a STALE own-repo auto_map target that is absent from the repo listing (an older config pointing at a file the repo no longer ships) instead of failing the whole repo closed as unscannable. The present .py are still fully scanned, which is the stronger coverage, and a file that is not there cannot execute. This unblocks first-party models like unsloth/PaddleOCR-VL (its tokenizer_config.json names processing_ppocrvl.py while the repo ships processing_paddleocr_vl.py). A referenced .py that IS present but cannot be fetched, and a repo-listing failure, still fail closed. Remote LoRA base resolution - Distinguish a genuine 404 (not a LoRA / repo absent -> None) from a transient error: the transient case is retried once, then logged as a WARNING (a missed base is scanned by neither gate) rather than silently skipped. Discard endpoint - Treat .onnx and .ckpt as weights so a repo whose only heavy artifact is one of those is never eligible for the declined-download purge. Tests added for each: load-path scoping (safetensors/subdir/Nemotron-H shapes, unknown-level fail-closed, pending non-block), stale own-repo auto_map ref, remote LoRA transient retry, and the empty-config-list (all-404 -> []) semantics. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make LoRA-base transient-warning test robust to logging backend Assert on the logger object directly instead of capsys, so the test does not depend on whether the real structlog logger or the module-stub logger is active (which varies with test collection order). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Allow a repo with auto_map but no executable code (e.g. GGUF) instead of blocking A config can declare an auto_map yet the repo ship NO executable .py -- most commonly a GGUF repo whose config.json carries an auto_map copied from the original model (e.g. unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF references modeling_decilm.py, which the GGUF-only repo does not contain). A GGUF model loads through llama.cpp, which never executes auto_map, and transformers cannot run a file that is not present, so there is nothing to scan and trust_remote_code is a no-op. The fail-closed change treated this empty result the same as "code is present but we could not fetch it" and hard-blocked the load. Distinguish the two: repo_remote_code_files now RAISES RemoteCodeUnscannable when code is present but cannot be fully fetched or listed (offline / gated / transient / a present .py that 404s / a listing failure), and returns an empty dict only when the listing succeeded and the repo genuinely ships no executable .py. The consent gate blocks on the exception (fail closed) and allows the empty case as a no-op. Real unscannable code still hard-blocks; eicar and CRITICAL/HIGH custom code are unaffected. Verified against all 37 unsloth/*Nemotron* models (two GGUF repos were false-blocked, now load) and the existing matrix (eicar still blocks; DeepSeek-OCR / NVLM-D-72B still prompt approvable consent). Tests updated to expect the raise for unscannable cases and added for the no-executable-code no-op. * Ignore vestigial auto_map in GGUF repos (llama.cpp never runs it) A GGUF repo's config.json is often copied verbatim from the original transformers model, auto_map and all, but a GGUF load goes through llama.cpp which never executes auto_map, so the config is inert. Treat a direct .gguf reference, and a repo that ships .gguf weights with no .safetensors, as having no remote code so the consent flow is never triggered. A mixed repo with both .gguf and .safetensors is still gated, since the safetensors variant would load through transformers where auto_map does run. The check sits behind the existing auto_map-present gate so normal models pay no extra repo listing. * Add scanner-result copy to the remote-code consent dialog Make the consent dialog state the scan outcome in plain language for every model. When the static scan finds nothing, reassure the user with 'Our automatic scanner did not flag any worrying files, but please double check.' (shown only for the clean, approvable case). When the scan flags custom code or unsafe files, label the list with 'Our automatic scanner flagged issues including:'. The Hugging Face attribution for unsafe files stays in the dialog description. * Close GGUF-suffix consent bypass for repo ids ending in .gguf The .gguf short-circuit in _config_has_auto_map skipped the scan for any model name ending in .gguf, including a bare two-segment repo id like 'evil/model.gguf'. Such a repo can still ship safetensors plus auto_map Python that transformers would execute, so skipping the scan was an asymmetric bypass (file_security already scans those repos). Restrict the short-circuit to genuine direct GGUF file references via _is_direct_gguf_file_ref: a local .gguf path, or a remote repo_id plus filename (three or more segments). A two-segment repo id named *.gguf now falls through to the config scan and _is_gguf_repo file inspection, so it only skips consent when it actually ships .gguf weights and no safetensors. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Align consent dialog body with the title and fix narrow-width overflow The scan results (the 'Our automatic scanner...' label, finding/unsafe cards, and the clean-scan reassurance) sat at the dialog's left padding while the title and description were indented past the status icon, so the body did not line up under the description. Move the title, description and results into one column to the right of the icon so they share a left edge, and let that column fill its width so the description no longer wraps early. Also stop a wide code snippet from pushing the dialog off-screen on narrow viewports: AlertDialogHeader is a grid with place-items-center, which sized the content row to its content; give the row w-full so it fills the track, and add min-w-0 down the results chain so the snippet scrolls inside its card instead of widening the dialog. Verified aligned and contained from mobile portrait through ultrawide. * Treat a repo as GGUF-only only when it ships no transformers weights _is_gguf_repo excluded only .safetensors, so a repo with a .gguf and a pytorch_model.bin (or .pt/.pth/.h5/.msgpack/.onnx/.ckpt) and no safetensors was treated as GGUF-only and skipped the consent scan, even though transformers can load that weight set and execute the repo's auto_map code. Require the absence of ANY transformers-loadable weight before treating the repo as a llama.cpp-only GGUF load. A genuine GGUF-only repo (only .gguf) is still inert; a mixed repo with any pickle or safetensors weight is gated. Adds a regression test across all the non-safetensors weight formats. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Block flagged subdir weight shards referenced by a root index The malware gate treated every subdirectory file as non-loadable, but from_pretrained deserializes a subdir shard a root index references (pytorch_model.bin.index.json -> shards/...-00001-of-00002.bin). Read the root weight indexes and block a flagged subdir pickle the weight_map points at; a flagged subdir pickle no index lists (NeMo nemo/*.distcp) stays non-blocking, and an inconclusive index lookup fails closed. * Pass hf_token to the export checkpoint load ExportBackend.load_checkpoint scanned with hf_token in the worker but loaded the weights unauthenticated, so a gated/private checkpoint passed preflight then 401'd at from_pretrained. Add hf_token to load_checkpoint and forward token to every from_pretrained branch; the worker passes the command's hf_token. * Scope created_by_scan to every HF cache the discard searches created_by_scan used get_cache_path (active HF_HUB_CACHE only) while /discard-remote-code deletes across active, legacy, and default caches. A repo the user already had in a legacy/default cache was marked scan-created and deleted on decline. Check all three caches for the repo dir before declaring the scan created it. * Scan the full .py closure of external auto_map repos An auto_map cross-repo ref (owner/name--module.Class) only had its entry file downloaded, but transformers also fetches that file's relative imports from the same repo, so a dangerous helper.py was left outside the scanned fingerprint. List each external repo's .py and scan the whole set (plus the referenced entry files); fail closed if the repo cannot be listed or fetched. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fail closed when a weight index cannot be fully read _indexed_shard_paths treated a partial result as definitive: if one weight index read cleanly but another failed transiently, it returned the shard paths it did see. A flagged subdirectory pickle listed only by the index we could not read would then be classed as "not a load input" and skipped, re-opening the very fail-open this guard was added to close. Return None whenever any index read is inconclusive, even if another read cleanly, so the caller blocks the already-flagged subdir pickle. A repo that ships no index files raises EntryNotFoundError for each (never inconclusive) and still returns an empty set. * Match cached repos case-insensitively in the created_by_scan guard _repo_in_any_hf_cache resolved casing only against the active cache and then probed every cache with an exact directory name. A case-variant already present in a legacy or default cache (models--Unsloth--Foo for a scan of unsloth/foo) was missed, so the repo was marked created_by_scan and deleted on decline -- but discard_remote_code_download deletes case-insensitively, so that delete would hit the user's pre-existing cache entry. Detect case-insensitively too, mirroring the deletion path. * Skip remote-code and security review for selected GGUF variants validate_model ran the trust_remote_code and Hugging Face security-scan preflight against the repo even when the selected artifact is a .gguf. A GGUF loads through llama.cpp, which never executes the repo's auto_map Python and never deserializes root pickle weights, so repo-level Transformers artifacts (a config.json with auto_map, or an unsafe pytorch_model.bin next to the .gguf in a mixed repo) are inert for that load. Gating the GGUF on them is a false positive. Run both preflights only for non-GGUF loads. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope the malware gate to actual load roots and serialized files Two fixes to evaluate_file_security so it neither misses a load-path pickle nor false-blocks an inert file: - Honor subdirectory load roots. Spark-TTS / BiCodec call from_pretrained on the snapshot's LLM subdirectory, so a flagged pickle directly under it is a root-level load artifact there. A new load_subdirs parameter (set from the model's audio type via security_load_subdirs) reclassifies those files relative to the load root and looks for weight indexes under it, so a flagged shard in that subdir is no longer skipped as "not root-level". - Exempt source files. A root .py is never deserialized by from_pretrained; executable repo code runs only through auto_map, which the remote-code consent gate scans. Flagging a Python helper here would false-block a repo that merely ships a build or train script. * Scan a LoRA adapter and base as one consent unit, and gate MEDIUM code A LoRA load runs both the adapter's and the base's repo code. The consent gate scanned them separately and pinned one fingerprint per repo, so an adapter that shipped its own auto_map code was either never shown in the dialog (which only saw the base) or impossible to approve with the base's fingerprint. evaluate_remote_code_consent_for_targets now scans all of a load's repos as a single combined unit and pins ONE fingerprint over the union of their code, so approving the load approves every repo's code together. evaluate_remote_code_consent becomes a thin single-target wrapper, and an unscannable target fails the whole load closed. Also gate MEDIUM findings: like HIGH they now block pending pinned approval, so a direct API caller cannot run flagged code by setting trust_remote_code=True without consenting. Only a clean scan loads without a fingerprint. * Preflight a LoRA load's adapter and base as one combined consent scan scan_model_remote_code rewrote a LoRA adapter to its base and scanned only the base for remote code, so the dialog never surfaced an adapter's own auto_map code. Scan the adapter and base together through preflight_remote_code_consent_for_targets, which pins one combined fingerprint the worker gate accepts. The malware preflight is also scoped to each target's load subdirectories. * Apply combined consent and subdir-aware malware scan in load workers Each load worker (inference, export, training) evaluated remote-code consent once per target with a single shared fingerprint, so a LoRA adapter that ships its own auto_map code could not be approved by the base's fingerprint. They now scan the adapter and base together via evaluate_remote_code_consent_for_targets, which pins one combined fingerprint over the union of their code. The malware scan in each worker is also scoped to the model's load subdirectories so a flagged pickle under a from_pretrained load subdir is not missed. * Report a consistent trust_remote_code requirement after a model loads validate_model reports requires_trust_remote_code from the YAML default OR the raw auto_map, but the load, already-loaded, and status responses reported only the YAML default. A custom-code model approved and loaded via auto_map was then reported as not requiring trust_remote_code, so the frontend stored false and a later retry or rollback sent trust_remote_code=false and failed. A shared resolver reports the same requirement for a loaded model (a value stored at load time, else the trust_remote_code the load used, else the YAML default, else the raw auto_map check), and the load response persists it so the status and already-loaded paths stay consistent. The selected-GGUF security review is also scoped to the model's load subdirectories. * Run the consent gate on training resume and for YAML-only trust_remote_code Three frontend gaps left a model loading without the trust_remote_code it needs: - The shared consent helper returned early when the scan found no auto_map and no unsafe files, dropping a requirement that comes from a model's Studio YAML default (e.g. GLM-4.7-Flash). It now grants the caller's requirement with an empty pin instead of sending trust_remote_code=false. - Resume-from-history called startTraining directly with no consent gate, so a resumed run whose model needs custom code (or an old run with no approved fingerprint) hit the worker block with no dialog. It now runs the same gate as a fresh start. - HF export passed requiresTrustRemoteCode=false for every HF source, so a YAML-only model could not flip the flag before export. It now signals the requirement for HF sources. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover both LoRA repos in validate, report GGUF as inert, purge all declined repos Three follow-on gaps from the combined adapter+base consent work: - validate_model resolved requires_trust_remote_code from the base alone, so a LoRA adapter that ships its OWN auto_map code (with a plain base) was reported as not needing trust_remote_code and the consent dialog never opened. It now checks the [adapter, base] target set, matching the scan route and the workers (which already gate both) and the security review already running over both. - The already-loaded, loaded, and status responses for a selected GGUF reported requires_trust_remote_code from the model's YAML default. A GGUF loads through llama.cpp, which never executes the repo's auto_map Python, so the requirement is inert for that load. They now report False, matching validate_model (which already skips both gates for GGUF) so a status refresh cannot flip the flag back on. - The remote-code scan downloads both the adapter's and the base's config, but created_by_scan tracked only the primary, so a base the scan was first to pull into the cache was left on disk when the user declined. The scan now reports scan_created_repos (every repo it newly cached) and the decline cleanup purges each; created_by_scan stays for older clients. The frontend falls back to the primary flag when the list is absent. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scan the repo the load fetches, purge external code on decline, harden consent pins Six follow-on hardening fixes from a fresh review pass over the gate: - The malware gate scanned the literal "Spark-TTS-0.5B/LLM" alias, but the trainer downloads it as unsloth/Spark-TTS-0.5B and loads LLM/, so the alias 404'd and failed open, missing a flagged LLM/ pickle. evaluate_file_security now resolves the alias to the repo the loader fetches and scans LLM/ as a load root. - security_load_subdirs relied only on tokenizer detection, which fails on an unresolved alias or offline; it now also honors the Studio YAML audio_type default, so a BiCodec LLM/ load root is not missed. - The remote-code scan downloads external auto_map repos (owner/name--module.Class), but the decline cleanup tracked only the model/adapter/base, leaving the external untrusted code cached. The scan now enumerates external auto_map repos and reports the ones it created in scan_created_repos, so a decline purges them too. - External auto_map refs failed the whole load closed on a stale or mis-derived dotted ref (sub.mod.py vs the real sub/mod.py) even though the actual file was present and scanned. They now drop such refs when the repo listing is real, exactly like the own-repo path; an empty/incomplete listing still fetches and fails closed. - The combined consent fingerprint keyed code by the raw target string, so the scan endpoint's canonicalized casing and a worker's raw user input produced different pins for identical code, rejecting a valid approval. Hub repo ids are now folded to lowercase in the key (local paths stay case-sensitive), so the pin tracks the code. - Export threaded hf_token into the weight load but not into detect_audio_type / is_vision_model, so a gated multimodal base 404'd in detection and fell through to the text loader. Both probes now use the same token. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Thread the token through check-vision and guard the gate's parallel sites The /check-vision endpoint classified a model without the hf_token, so a gated or private vision model 404'd in the probe and was reported as a plain text model -- the same dropped-token shape as the export probes, at a sibling site. It now passes the token like the neighboring /check-embedding endpoint. Add deterministic consistency guards (tests/test_security_gate_consistency.py) that enumerate the gate's parallel sites mechanically instead of relying on a review to spot a missed sibling: every is_vision_model / is_embedding_model / detect_audio_type caller under routes/ and core/ must thread the token, every GGUF response must report trust_remote_code via the resolver or False (never the raw YAML default), and every load worker that runs the malware or consent gate must resolve the LoRA base. A new site that drops the token or mis-reports the requirement now fails CI directly. * Narrow the LLM alias rewrite and make audio detection token-aware Three fixes from the confirmatory review, one a regression from the previous round: - _load_scan_target rewrote EVERY remote repo ending in "/LLM" to unsloth/<parent>, so a real third-party repo named "<owner>/LLM" was scanned as unsloth/<owner> while the loader still fetched the real repo -- a fail-open hole introduced when the Spark-TTS alias handling was added. It now rewrites only a registry-known bicodec alias; every other "/LLM" repo is scanned as itself. - detect_audio_type cached results under the bare model name, so an unauthenticated probe of a gated/private repo cached None and poisoned a later authenticated call with the token. The cache is now keyed by (normalized_name, token_fingerprint), matching the vision cache. - The training fallback /check-vision call dropped the hf_token, misclassifying a gated/private VLM when the config endpoint failed. It now passes the token, like the getModelConfig call it falls back from; checkEmbeddingModel takes the token too. Extend the consistency guards: every capability cache must be keyed by a tuple including the token, so a cache re-declared as Dict[str, ...] fails CI. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Document the broad .py scan as deliberate and enforce it with a test The remote-code scanner scans every .py in a repo once an auto_map exists, not just the auto_map entry's static import closure. This is intentional: the entry module can reach a sibling via an absolute import, importlib, or exec, none of which a static relative-import closure follows, so closure-only scanning would be a real bypass of a load-time RCE gate. The broad scan never under-scans; the cost is that an unrelated benign script can over-block, which is the safe failure direction (HIGH stays approvable; only CRITICAL hard-blocks). Spell this out at both the local and remote scan sites so the choice reads as deliberate, and add a test asserting an unrelated, never-imported .py is still scanned -- so a future narrowing to the static closure fails CI. * Purge a declined remote LoRA adapter the scan downloaded scan_model_remote_code probed the created-by-scan state AFTER resolving the base, but get_base_model_from_lora_identifier downloads a remote adapter's own adapter_config.json, so the adapter looked already-cached and was dropped from scan_created_repos. On decline the adapter -- including the auto_map .py the preflight fetched -- was left on disk, defeating the "untrusted code is not left on disk" guarantee for the adapter itself. Snapshot the primary's cache state BEFORE base resolution and use it when marking the adapter scan-created; on any probe error treat it as pre-existing so a decline never deletes it. The base and external repos are unaffected (their configs are not downloaded before their own probe). Add a test that models the mid-scan download side effect, which the prior static-stub tests did not. * Clear remote-code approval when the training model changes Switching the training model from an approved custom-code model to a clean one kept the previous model's trust_remote_code=true and approved fingerprint in the store: setSelectedModel reset visionImageSize on a true switch but not the remote-code approval. The clean model then trained with trust_remote_code=true, which bypasses the compiler and disables fused cross-entropy. Reset trustRemoteCode and approvedRemoteCodeFingerprint on a true model switch. The new model's own YAML default is re-applied by loadAndApplyModelDefaults, and a custom-code model still re-opens the consent dialog before training starts, so the only change is that a clean model no longer inherits a stale approval. * Trim verbose comments across the model-fetching hardening changes Condense the explanatory comments and docstrings introduced across the trust_remote_code consent gate, the malware/unsafe-file gate, the remote-code scanner, the load workers, the model routes, and the security frontend into fewer, tighter lines while preserving every security rationale (fail-open vs fail-closed direction, the deliberate broad-scan anti-bypass note, the empty-vs-unscannable distinction, stale-ref handling, and the alias-rewrite spoof guard). Comments and docstrings only. No code, logic, identifiers, or test behaviour changed; verified comment-only via the AST/TypeScript checker (40/40), with the backend test suite and frontend tsc green. * Do not cache transient audio-detection failures detect_audio_type cached _detect_audio_from_tokenizer's result unconditionally, so a transient read failure (network error or 5xx, returned as None) poisoned the cache and the later successful probe never ran. Mirror the vision cache: _detect_audio_from_tokenizer now returns (audio_type, definitive) and the caller caches only definitive results. A read that succeeds with no audio tokens, or clean 404s for every tokenizer path, stays a cacheable None; only a genuine transient failure (connection error, timeout, 5xx, malformed body) skips the cache so the next call retries. --------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |