mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-25 00:33:49 +00:00
23 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
sts-change
|
8d6e969378
|
Studio: never pick a macOS AppleDouble sidecar as a GGUF (#8919)
--------- Co-authored-by: Lyxot <longyixing331@gmail.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
Long Yixing
|
ca7c83fafd
|
Studio: stop an unreachable Hugging Face endpoint from stalling the backend (#8799)
* fix(studio): stop an unreachable hub from stalling the RAG embedder Naming the embedder's GGUF went straight to `list_repo_files`, which has no timeout parameter and whose pagination layer passes an explicit `timeout=None` that overrides any client-level default. `socket.create_connection` applies a connect timeout per address and races no address families, so on a host whose route to the hub blackholes that call blocks until the kernel exhausts its SYN retries — while `_lifecycle_lock` is held, so every reader queues behind it. Resolve from the local Hugging Face cache first. `_model_path` is per-process, so every restart previously paid a hub call to name a file already on disk; the cache answers in tens of milliseconds and never touches the network. Only the revision `refs/main` names is used, since that is what a download would serve, and a hit pins the embedder to it: `embedding_identity` does not record a revision, so silently adopting republished weights would leave a persisted index answering queries from one model with documents embedded by another. On a genuine miss the hub is still used, now under a wall-clock deadline and inside the same forced-offline-when-unreachable guard the chat GGUF path uses. If the hub cannot name a file, a cached GGUF of another variant is adopted with a warning rather than leaving the install with no embedder; a transfer that fails for its own reasons still surfaces. Fixes #8778 * fix(studio): bound the prebuilt release-freshness fetch `urlopen(timeout = 5.0)` looks capped, but `socket.create_connection` applies that timeout once per address as it walks the `getaddrinfo` results, and races no address families. A host whose leading addresses blackhole therefore pays five seconds for each one before reaching a working address, so the effective cost is the timeout multiplied by the address count rather than the timeout itself. `/api/inference/status` reads this fetch, so that multiplication becomes the route's response time. Failed lookups are only memoised for 60s, which is shorter than the stall itself, so back-to-back status reads each pay it again. Run the fetch under a wall-clock deadline. A missed deadline is a failure like any other: it feeds the existing failure cache and the last-good disk value, so the freshness banner fails open exactly as it already does offline. * fix(studio): keep cache-first embedder resolution faithful to the listing Review of the cache-first path surfaced four ways a local snapshot could answer differently from the full repo listing it stands in for. Companion-repository precedence is restored. A custom model resolves through its derived `-GGUF` companion first and reaches the model repo only when the companion has no GGUF; consulting the cache for both candidates let a file cached under the fallback pre-empt a companion the hub could still resolve, and then tagged it as current. The cache is now consulted for the preferred repo only, while the offline degrade still reaches both. Selection no longer depends on arrival order. A listing arrives ordered and a directory scan does not, so among equal-length names the tiebreak now falls back to the name itself; without it a complete cached shard set could yield shard 2, which lacks the metadata llama-server needs. Split sets are verified before use. A snapshot holding part of a set cannot serve it, so the winner is checked and, if unservable, dropped and the pick retried — failing the lookup outright would let an incomplete family shadow a complete one that merely sorts later. Verifying per winner rather than per file keeps this to a single sibling walk. MTP drafters join mmproj in the exclusion. A drafter is a companion rather than a model, and a cache subset holding only the companion was read as holding the embedder. Repo ids now resolve through `resolve_cached_repo_id_case`, so a repo id typed with different casing than its cache folder still finds its GGUF instead of re-downloading, or failing outright when the hub is unreachable. |
||
Eyera
|
b52d3b56ed
|
feat(studio): rework train page setup flow (#7633)
* feat(studio): rework training setup and cache handling * Rework train page resource selection * Improve train page workflows and resource selection * Fix train page picker behavior * Fix stale training errors after token updates * Fix training eval steps regression * Fix shared token use when resuming training * Harden train page resource workflows Unify model and dataset picker behavior, training start guards, and token handling. Validate cache provenance for local models, downloaded resources, and processed datasets. Prevent stale preview and history state while keeping the train page modular. * Fix training method race and test isolation * Restore training upload limit exports * Fix training config formatting contract * fix(studio): harden training model selection and preflight * fix training resume, cache, and lifecycle reliability * Fix training stop watchdog path stub * Fix training start state and train page UI * Unify train page selection controls * Fix train page validation, localization, and paging * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Clean up train locales and restore recipes link * fix(studio): correct training model selection and preflight Preserve freeform local model paths, reject remote GGUF-only repositories, and correct the training start snapshot contract. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): harden training resource selection Preserve local dataset path intent, localize setup-change errors, and remove unused picker tour hooks. Refresh model picker and PDF recipe contracts for the refactored training flow. * fix(studio): remove train tab scrolling * fix(studio): harden training resource preflight Reject missing local models and binary adapter artifacts. Preserve selected model and dataset cache pins through preflight and QLoRA loading. Detect cache path changes before training starts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix train page validation and cache handling Keep transient model checks nonblocking and bind cached models and datasets to their selected snapshots. Improve Hub auth errors, retries, task search, streaming consistency, locale formatting, reset handling, payload mapping, partial inventory filtering, theme accents, and regression coverage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): stabilize train page pickers Unify Hub validation, local inventory, picker styling, and dataset names. Localize training feedback and explain streaming modality changes. Add unit, contract, and cross-browser picker coverage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): align cached scans and variant state purge Scan selected cached snapshots during training security preflight. Purge manifests and cancel markers using their stored GGUF variant. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): harden train cache, consent, and UI Fix cache cleanup and cross-platform model path handling. Align resume and adapter consent scans with pinned load targets. Localize dataset flows and refine navigation, state cleanup, and theming. * fix(studio): harden training resource selection Preserve cached model configuration HTTP errors so stale cache references return their intended 404 responses. Wait for device inventory settlement before locking inferred picker tabs, while keeping known device items visible during scans and retries. Apply modality name heuristics only to the final model path component across supported platforms. Expose full model and dataset identities on truncated picker triggers. Add regression coverage for picker settlement, retry behavior, and modality inference. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix train picker flow and resume safety Scan the exact cached model snapshot before resume consent and preserve the actual repository identity. Keep cold pickers on Device until inventory settles, cap automatic Hub pagination, and provide a stable Load more action. Improve train control and history card accessibility, remove duplicate token labeling and dead styling, and correct picker spacing. Normalize Windows relative model paths and add coverage for resume pins, picker policy, pagination, path identity, and accessibility contracts. Fix the reported formatting and import ordering issues. * Fix cross-platform train dataset path detection Use the shared local path detector so persisted Windows, Unix, UNC, and home-relative dataset references do not render Hugging Face-only controls. Recognize Arrow dataset files and add regression coverage for supported local path formats and Hub repository identifiers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix training start recovery and dataset selection Reconcile failed fresh and resumed training starts with backend status before reporting an error. Treat the selected dataset source as authoritative so filename-like Hub repositories retain subset and split controls. Align dataset picker sorting and remove unused advisor template state with a persisted-state migration. Add regression coverage for transport recovery and Hub dataset selection. * Fix train picker validation and start recovery Update the resume training contract test for the asynchronous failure handler. Filter invalid Hugging Face search results and block invalid saved model and dataset selections. Restrict local dataset selection to inventoried application paths. Load training YAML through a bounded native file picker while keeping the browser fallback. Only recover uncertain training starts after network or response parsing failures, and match the active job to its request identifier. Treat cached full precision models as ready for QLoRA without showing a false download warning. Add regression coverage for the picker, config import, and training recovery behavior. * Harden training starts and config imports Track training start request IDs from preflight through pending, accepted, and rejected states so retries remain idempotent. Disable automatic retries for start posts and reconcile ambiguous client responses with server request status. Clear terminal training status on reset so later runs do not inherit stale state. Enforce the 1 MiB YAML limit for browser imports and keep read errors specific to the selected file label. Remove unused dataset picker translation entries across all locales. Add focused tests for request idempotency, start recovery, config limits, and picker contracts. * Harden train picker feedback and start handling Show explicit empty states for device dataset searches and invalid Hub model or dataset queries while keeping pagination sentinels mounted. Run the train picker Playwright suite across Chromium, Firefox, and WebKit in Windows CI. Keep training start outcomes owned by backend spawn completion so cancelled handlers cannot mark active jobs as rejected. Correct the Train section heading hierarchy and format the branch-owned native file dialog test. Add picker contracts, browser assertions, and cancellation race coverage for the updated behavior. * Fix training setup races and configuration state Keep hardware-based method selection within model-default loading so starts cannot race changing defaults. Clear start errors only after deliberate configuration or token edits while background reconciliation stays silent. Generate training request IDs safely for LAN HTTP access before acquiring the runtime lease. Filter persisted state, sanitize invalid methods, and guard preview metadata. Localize config size failures, use the toast wrapper, and remove obsolete training files. Add regression coverage for the updated start, persistence, and import behavior. * Refine training configuration and picker workflows Split the training configuration store and parameter panel into focused persistence, policy, LoRA, hyperparameter, memory, and MLX modules. Run picker coverage on platform-appropriate browser engines and parameterize the unmanaged dataset path. Surface configuration changes when streaming is disabled and clear stale dataset modality state after failed probes. Consolidate dataset format and AI-assisted mapping on hub routes with header-based Hugging Face tokens. Hold local path actions until model inventory loading settles. * Fix train picker state and preflight regressions Open online pickers on the Hub before inventory settles and keep the inferred tab stable for the session. Preserve parameter and VRAM metadata for pinned selected models. Reject failed backend starts once and cancel automatic modality corrections without showing a destructive setup error. Update persistence contracts to follow the extracted training configuration module. Move shared parameter option styling out of the React component module to preserve Fast Refresh. Add regression coverage for picker tab behavior, pinned model metadata, persistence contracts, and start rejection handling. * fix(train): harden model selection and start feedback Preserve cache metadata and model capabilities when freeform local paths resolve to discovered models. Return stable training start error codes through direct and recovered requests, then localize Hugging Face preflight failures across every supported locale. Show whether submitted advanced settings are default or non-default in Simple mode, and migrate the parameter mode preference to the standard storage key. Restore absent-key migration semantics and update the embedding security test for the extracted gate helper. Add focused coverage for structured errors, advanced setting summaries, start recovery, and security gates. * fix(train): harden picker validation and focus Align Hub ID validation with Hugging Face rules while accepting valid underscore boundaries. Reject persisted path-shaped datasets from Hugging Face-specific controls. Reuse model identity normalization for cache path comparisons. Move picker search focus into the popover focus lifecycle. Update source contract tests for extracted model selection and structured resume errors. * fix(train): refine picker behavior and dataset structure Wait for device inventory before locking inferred picker tabs while preserving explicit user choices. Align Hub resource ID validation with backend rules for leading and trailing underscores. Support ArrowDown navigation from picker tabs into available options. Split dataset selection, uploads, streaming settings, and inventory refresh logic into focused modules. * fix(studio): stabilize train picker contracts and state Update dataset contracts to follow the extracted selection, upload, and inventory modules. Keep the inferred picker tab stable while device inventory settles. Use the shared TrainingMethod type for VRAM estimation and remove redundant casts. * fix(train): align Hugging Face repo validation Allow underscores at repository segment boundaries to match Hugging Face repo ID rules. Keep model and dataset search queries unvalidated while preserving selection safeguards. Add regression coverage for accepted IDs and unrestricted picker searches. * Fix training resource compatibility and cache safety Restore legacy dataset format and mapping routes while moving training uploads and checks to the canonical Hub endpoints. Enforce configured upload limits, clean partial files after failures or cancellation, and support both upload route families in middleware. Align Hugging Face repository validation with accepted underscore boundaries and skip PyTorch dependent tests when the dependency is unavailable. Keep model format and cache metadata correct when the same repository is selected from another source. Only promote complete runnable model weights as cached training resources, align download notices with picker capabilities, and reject weightless local snapshots during start preflight. Cancel stale dataset mapping assistant requests before they can overwrite a newer selection. Preserve touch selection for training methods by separating tooltip toggling from Select item activation. Add focused backend, frontend, and picker regression coverage for these paths. * fix(studio): finish train page and onboarding rework Ignore cache-only fields when comparing training start inputs so reconciliation does not cancel valid runs. Route native dataset drops through Tauri path validation and lease-backed managed uploads. Use the production model and dataset selectors in onboarding and replace placeholder uploads with real imports. Require positive learning rates while keeping learning rate, epoch, and step inputs editable. Normalize model identities, improve dataset summary order, and guard persisted selection state. Recognize Windows rooted and drive-relative paths consistently in picker logic. Add MLX optimizer help and learning rate validation messages to every supported locale. Add frontend, backend, and native policy regression coverage for the updated behavior. * chore(ci): remove train picker smoke workflow changes Remove the train picker Playwright step from the macOS Studio smoke checks. Remove the train picker Playwright step from the Linux Studio smoke checks. Remove the train picker Playwright step from the Windows Studio smoke checks. Drop the matching artifact upload paths so the workflow files match upstream main. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): harden train picker behavior and boundaries Expose dataset drop helpers through the training public API and move local path detection into shared code. Block freeform Hub choices while offline for pointer and keyboard selection. Make native dataset drop handling safe for committed React renders. Keep recipe navigation working when session storage is unavailable. Preserve Windows drive root identities and cover the distinction from drive relative paths. Update picker and training input contracts to match the extracted implementations. Remove orphaned translations and align the train style block with the existing indentation. * Fix train picker cache and upload behavior Keep selected model cache paths strict and preserve fallback discovery only when no path is supplied. Offload multipart writes and native dataset copies from the backend event loop. Prewarm train picker inventories so device-first tab inference can settle before the first open. Use shared picker tab constants across the model and dataset selectors. Update picker contracts and add regression coverage for cache resolution and dataset uploads. * Refine train picker controls and component structure Split picker retry, error, and pagination UI into focused components. Add a shared segmented control and use native radio semantics for dataset source selection. Centralize segmented indicator positioning with RTL-aware transforms. Separate dataset advanced settings state wiring from its presentation and source toggle. Make picker arrow navigation move symmetrically from tabs through search and options. * Fix train page reconciliation and picker contracts Preserve tuned training parameters when model cache references change, while refreshing defaults when no user edits occurred. Keep streaming modality corrections visible through the Start CTA after preflight stops. Localize the new training parameter and dataset mapping text across all supported locales. Refresh picker structural contracts after the component split and apply Biome formatting to the changed files. * Fix train upload tests and desktop drop handling Pass an explicit empty native path lease in legacy upload limit tests so direct route calls match FastAPI request behavior. Track Tauri scale factor changes during dataset drag and drop, clean up both listeners safely, and cover runtime updates. Replace the run preview card's important border utility with an explicit dark theme card modifier. * fix(studio): harden training setup and history state Restore Hugging Face token entry and validation in the onboarding model and dataset steps. Apply the configured upload size preflight before onboarding dataset uploads. Update moved model and dataset cache references without showing false missing cache warnings. Parse SSE frame delimiters by their matched length so mixed line endings preserve event data. Match sensitive pid paths by exact segment while retaining database state protections. Clear deleted Data Recipe selections after inventory settlement while preserving direct uploads. Show the files deleted status only when a run previously recorded an output directory. Handle Data Recipe navigation failures and keep the native drop listener stable during uploads. Replace formatting-specific contract checks and add focused behavioral regression coverage. * Harden training start and model picker flows Split the train model picker into its own entry point to keep it out of shared route bundles. Preserve pending idempotent start reservations and reconcile them before accepting a run. Keep unconfirmed starts in a polling state with accurate localized feedback. Continue starts after automatically disabling unsupported multimodal dataset streaming. Localize the updated model picker controls across supported languages. Lock dataset uploads before asynchronous preflight to prevent concurrent selections. Add regression coverage for reservation states and unconfirmed runtime starts. * fix(studio): harden training picker edge cases Preserve POSIX model path identity while retaining Windows relative path normalization, and clarify the override migration behavior. Localize the onboarding model and dataset controls across every supported locale. Truncate native dataset filenames by Unicode code point so generated labels cannot contain lone surrogates. Cancel training after preflight disables streaming for image or audio datasets so users can review the changed setting before restarting. Update runtime contract assertions for the setStartPending rename and add regression coverage for path identity and Unicode truncation. * fix(studio): align training picker controls Restore full-height Browse and Amazon S3 selection with an accessible radiogroup that avoids fieldset sizing behavior. Align model and dataset picker tabs with the shared 36 px segmented control geometry and typography. Remove active borders and focus rings from picker search inputs while preserving keyboard focus indicators on other controls. Add contract coverage for segmented control sizing and picker search focus styling. * fix(studio): harden onboarding picker flows Classify native dataset drops from the full path before shortening display names. Keep onboarding model choices within the selected training type across Hub, device, and freeform selections. Retry model default loading when onboarding restores an interrupted model selection. Localize the Hugging Face token field across supported languages. Reset picker result scrolling when switching between Device and Hugging Face tabs. Add regression coverage for native filenames, model constraints, hydration, and picker scrolling. * fix(studio): reconcile multimodal streaming before training Disable streaming when dataset checks detect image or audio data, then recheck cached selections in non-streaming mode. Keep start-time modality detection as a safe fallback that updates the configuration and continues without an error or a second click. Build onboarding training method options from the shared method order and metadata so onboarding stays aligned with the train page. Add contract coverage for the streaming reconciliation and shared training method list. * Fix train picker contracts and dataset recovery Allow local models without reliable modality metadata to pass onboarding constraints while retaining explicit mismatch checks. Reconcile GPU selection without effect-driven state updates and mark intentional deep Hub imports for lint. Return and consume stable local dataset cache miss codes, and send the canonical train_split field. Move train-specific picker code into its owning feature and update the related contract coverage. Use the canonical Hub dataset progress route from Chat. * Harden training resource selection and offline handling Return an actionable preflight error when offline mode is enabled and the selected model is not cached. Validate evaluation dataset uploads with the shared extension policy and reuse the centralized accept list. Disable streaming for explicit on-device dataset selections while preserving Hugging Face selection intent. Keep dataset identifiers safety checked without blocking benign values before Hugging Face handles repository validation. Remove the branch-added source assertions and cover the new selection policy with behavioral tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix training setup validation and async state races Validate Hugging Face dataset IDs and repository access before training starts while keeping local paths on their filesystem validation flow. Scope model default race protection to fields that defaults can overwrite so unrelated dataset edits no longer suppress model initialization. Stop Hugging Face token edits from invalidating token-independent device inventory scans on every keystroke. Add regression coverage for dataset source validation, cached and remote Hub preflight, model default application, and token inventory behavior. * Tidy up train page layout, tooltips and dark mode Layout - Move the Browse / Amazon S3 toggle into the Dataset section header - Put Subset, Train Split and Evaluation Split on one row - Put Target Format, Train Split Start and Train Split End on one row - Pair Project Name with Max Steps, and Context Length with Learning Rate - Centre the hyperparameter tabs and give them a fixed width so the sliding indicator lines up with its segment - Give parameter rows a min height so slider and select rows share one vertical rhythm - Slightly more spacing below section headings and at card bottoms Tooltips - Add hints for Model, Method, Dataset and Project Name - Move field descriptions into the tooltips and drop the inline copy - Upload field keeps the accepted file types inline, with size limit and Learning Recipes note in the tooltip HF token - Show a masked preview of a saved token instead of a generic label - Read Not set when no token is stored Dark mode - Drop borders on the LoRA option cards and target module chips, and separate states with background fill instead - Use a lighter, less saturated green for the selected state Other - Expand LoRA Settings and Training Hyperparameters when Advanced opens - Shorten the upload label to Drop file or click to upload - Localise the new wizard tooltips across all locales * Fix training defaults, MLX validation, and dataset preflight Prevent late hardware recommendations from replacing configuration values loaded after model selection. Block unsupported MLX methods and embedding runs in the UI and backend before training is queued. Require live Hub access for streaming dataset preflight instead of accepting unrelated cached data. Compare advanced settings against applied model defaults and cover the updated behavior with focused tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix training picker validation and platform safeguards Update the Tauri YAML contract test to verify save filter behavior without depending on the Rust vector implementation. Align frontend Hub resource validation and training provenance normalization with the shared repository ID rules, including valid underscore and maximum length identifiers. Reuse one frontend training method policy so Apple Silicon onboarding and the Train page consistently disable CPT and prevent unsupported onboarding completion. Add regression coverage for rejected Hub IDs and exact resume provenance. * Fix MLX training capability validation Detect hardware before platform validation so unsupported MLX configurations fail before a job is queued. Reject audio dataset training during server preflight while preserving the worker guard. Share Apple Silicon capability checks across model selection, onboarding, readiness, submission, and LoRA controls. Cover audio, CPT, embedding, LoftQ, DoRA, and warm-start detection with regression tests. * Fix training quit protection, translations, and preference reset Protect desktop quits while a training start is pending or a run is active, and keep the warning text accurate. Localize dataset upload details across every supported locale and require those keys in parity checks. Centralize the training picker and parameter mode storage keys so resetting local preferences clears current and legacy values. Add regression coverage for training activity transitions and preference reset ownership. * Preserve multimodal training support Keep upstream training eligibility authoritative while preserving separate vision, audio, and embedding capabilities. Classify dual audio and vision models for vision training without changing inference support or enabling pure audio training on MLX. Add coverage for model type constraints, backend type resolution, and MLX validation. Apply the pending Biome formatting fixes to the dataset selector and training wizard. * Fix training defaults retry and clean stale translations Allow model defaults retries after failed cache reconciliation without overwriting user-edited settings. Ignore stale fallback vision checks once a newer defaults request starts. Remove unused parameter description keys from every locale and cover the retry behavior in the training contract tests. * Document direct model identity imports Explain why the model identity helpers bypass the Hub barrel beside each lint suppression. * Preserve training settings across reloads Persist applied model defaults and their advanced settings baseline while refreshing transient model metadata without overwriting tuned hyperparameters. Keep remote format probe tests runnable without an installed huggingface_hub package by supplying a fake module. Normalize tilde-prefixed path separators while preserving case-sensitive identities. Remove unmatched model and dataset tour anchors. Add regression coverage for persistence migration and model identity normalization. * Harden train page validation and persistence Validate CPT embedding learning rates against backend bounds and add localized client feedback. Keep invalid learning rate drafts out of persisted state and YAML exports. Notify users when multimodal model selection restores a non-S3 dataset source. Remove W&B tokens from training config persistence and migrate stored secrets safely. Update the model defaults contract test to cover metadata refresh without requiring the removed early return. Move segmented control styles into a non-component module to preserve Fast Refresh behavior. Add validation and persistence coverage for the new contracts. * Fix training model IDs, cache state, and localization Preserve root-level Hugging Face model IDs instead of rewriting them into the Unsloth namespace. Debounce token-scoped inventory reconciliation so token and inventory updates use stable request keys. Keep the selected training history run when the Studio view remounts. Localize dataset subset and split selectors across every supported locale. Complete Italian picker and training translations and enforce Studio-wide locale parity. * Fix training history errors and stale stop state Return structured artifact deletion errors and show localized messages that distinguish active training output from filesystem failures while keeping history intact. Clear superseded stop requests only when they belong to the current runtime generation, and cover start invalidation and stale stop handling with regression tests. Clarify that shared picker styles apply outside the Hub while Hub-only rules remain scoped. * Fix training method persistence and settings summaries Persist training method provenance so manual learning rates, model adapter rates, and pre-CPT dataset formats survive reloads and method changes. Deduplicate imported target modules and compare advanced-setting arrays by value counts so duplicate entries cannot hide non-default settings. Add migration and regression coverage for legacy state, rehydrated method transitions, restored learning rates, and duplicate target modules. * Fix persisted completion defaults and inventory retries Persist trainOnCompletions across reloads so model defaults and advanced settings summaries remain accurate. Defer missing legacy values until model metadata loads, preserving tuned settings while respecting streaming, raw text, CPT, embedding models, and explicit user changes. Settle partial inventory failures when usable rows exist so manual local model paths and Enter submission remain available. Add retry actions for partially failed model and dataset scans, including when filtering leaves no visible results. Add regression coverage for persistence migration, constrained completion defaults, inventory settlement, and picker empty states. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix training validation and route state regressions Retry newly detected image datasets with vision-aware validation, clear stale dataset check failures, and cover the failed background classification case. Restore Train history cleanup and centralize the learning recipes navigation intent key. Repair backend route stubs for canonical model IDs and path normalization, and isolate consent tests from shared package import state. * Preserve training cache pins during preflight Keep model and dataset cache flags and local paths in the pending start comparison so reconciliation cannot submit a stale snapshot. Cover changed cached copies and uncached transitions with focused regression assertions. * Preserve Hub identity for cached training exports Keep exact cached snapshots for training while restoring the standard Hub repository identity before PEFT saves. Recover repository identity in memory for legacy adapters so imatrix exports work without rewriting adapter configs. Cover repository mismatches, local models, and Windows cache paths with focused regressions. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix cached training, evaluation, and MLX resume behavior Limit cached dataset slice loads so small training ranges do not materialize full duplicate caches. Restore Hub model identity for cached Torch and MLX runs so saved adapters remain portable. Attest MLX model provenance across runtime and prequantized 4-bit formats so valid checkpoints can resume. Recognize missing cached eval splits, reload train and eval together, treat explicit eval split errors as fatal, and use deterministic held-out data for automatic fallback. Retain non-fatal evaluation warnings in training status and render them in the train UI. Apply pin and model format validation to every MLX worker entry point and keep the worker test fixture aligned with its imports. * Stabilize training progress updates across job handoffs Keep active and pending training ownership separate so status polling cannot switch identities during a running job. Scope metrics, progress streams, stop, and reset operations to the expected job and recheck ownership during handoffs. Make polling, stream parsing, and start reconciliation abortable, monotonic, and safe against stale overlapping requests. Preserve same-run interface state so live updates do not remount progress controls or dismiss the stop confirmation. Add backend and frontend regression coverage for ownership changes, stale events, request races, and stream cleanup. * Fix cached training fallback and dataset reconciliation Recognize incomplete SentencePiece tokenizer caches and retry online model loads through the Hub while preserving strict offline and resume pins. Prevent rejected dataset snapshots from being promoted again until the selection or inventory changes. Forward cancellation signals to dataset format requests so superseded checks stop at the transport layer. Add focused backend and frontend regression coverage for these cache paths. * Harden offline training and cache fallback behavior Validate cached model snapshots for tokenizer and processor support, then fall back to the pinned Hub revision when the cache is incomplete. Remember rejected dataset cache entries and cancel superseded checks so metadata-only snapshots cannot trigger request loops. Reject deleted local datasets before any Hub access and clear only the matching stale selection in background, preview, and start flows. Reserve training starts before validation so overlapping requests and GPU consumers cannot race the spawn window. Keep GGUF fallback metadata tied to the cache that supplied variants when Hub access fails. Filter DSpark and DFlash drafter companions across picker, loading, inventory, and deletion paths while preserving MTP behavior. * Harden training setup and picker behavior Keep training starts blocked while a stop request is pending and preserve the stop latch through backend failures. Skip malformed training progress events while validating SSE payloads and releasing stream readers safely. Enforce non-streaming state for upload and S3 datasets across source changes, persistence, and rehydration. Route Markdown dataset drops to Data Recipes across browser and native path formats. Give model and dataset source tablists localized accessible names. Refresh training start contracts and add focused regression coverage for the corrected behavior. * Fix offline dataset cache selection and token status Prefer finalized processed dataset caches for offline training while keeping raw cache paths available for scoped management. Treat malformed Hugging Face tokens as unset in Train and surface local validation feedback without unnecessary requests. Add regression coverage for cache coexistence, interrupted cache builds, path propagation, and malformed token presentation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Preserve sidecar Hub version during dataset cache checks Keep processed dataset cache discovery free of eager datasets imports so training does not preload the base Hugging Face Hub package. Preserve cache root discovery through HF_DATASETS_CACHE, HF_HOME, and XDG_CACHE_HOME before Transformers sidecar activation. * Add the AGPL-3.0 header to the two new studio training contract tests * Scan the fallback target and pin cached snapshots that hold weights Cache fallback scanned the pinned snapshot it was about to discard, then loaded the Hub repo unscanned. Scan after the pin is dropped instead, in all four training paths. Preflight probed the literal model name, so the registry bicodec alias Spark-TTS-0.5B/LLM 404'd and rejected a supported model. Probe the repo the trainer downloads and treat its load subdir as the weight root. An unpinned start still downloads its dataset, so a stray cached copy no longer skips Hub verification. Offline keeps accepting the cache. refs/main can point at a metadata-only revision, so the model pin now prefers a snapshot that carries weights before falling back. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Let the capability-cache guard see through the key type alias model_config.py names the cache key _CapabilityCacheKey, so matching the literal Dict[Tuple no longer finds it and the guard fails on a cache that is still correctly tuple-keyed. Resolve module-level aliases first. Checked by mutation: reverting a cache to Dict[str, ...] is still caught. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: point the extra-UI tour anchor check at studio-model-picker The train page rework renamed the model tour anchor from studio-model to studio-model-picker and updated tour/steps/base-model.tsx accordingly, but tests/studio/playwright_extra_ui.py still looked for the old anchor, so the Chat UI Tests job failed with "[data-tour='studio-model'] not found". Verified against two live Studio instances: the anchor is studio-model on main and studio-model-picker at this head; studio-dataset and studio-params are unchanged on both. * Studio: require metadata and weights together when pinning a model snapshot Pass 1 of _resolve_model_snapshot matched on weights alone, so two cases still selected a snapshot that /training/start then rejects with "does not contain trainable weights": - a newer weights-only fetch (interrupted download, or an allow_patterns pull that never took config.json) displaced an older complete sibling. That is a regression against the previous metadata-first ordering, which started the run; reproduced with a two-snapshot cache where the complete one is older. - consolidated.safetensors counted as weights for selection but is absent from _MODEL_WEIGHT_CANDIDATES in routes/training.py, and transformers 4.57.6 has no loader path for it (zero references in the package), so a config plus consolidated snapshot won over a loadable sibling. Pass 1 now demands metadata AND weights via a required_groups argument on latest_snapshot_from_cache_path, and consolidated.safetensors is dropped from the selection tuple so it matches what the route accepts. Pass 2 keeps the metadata-only fallback unchanged, so caches that never held weights resolve exactly as before. Both new tests in test_model_cache_snapshot.py fail without this change and pass with it; 401 tests across the cache, preflight, provenance and identity suites pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make the new cache tests pass on Windows Four of the PR's new tests fail on windows-latest while passing on ubuntu and macos. All four are defects in the tests, not in the production code. The three blob-symlink tests hardcode a POSIX relative target ("../../blobs/x"). Windows stores the reparse-point substitute name verbatim and resolves it in the object namespace, where / is not a separator, so the link is created but dangles: Path.resolve(strict=True) raises and the provenance and dataset-cache helpers correctly return None. huggingface_hub builds these targets with os.path.relpath (file_download.py _create_symlink), which yields ..\..\blobs\x on Windows, so a real cache never has this shape and no Windows user is affected. Building the target with os.path.relpath / os.path.join matches what huggingface_hub writes. The cross-snapshot rejection tests had the same POSIX targets, so on Windows they were passing for the wrong reason (dangling link rather than the escape check). They now use native separators too, so the rejection logic is actually exercised there. test_runtime_4bit_resume_reaches_worker_with_source_resource_pins compared model_local_path against str(Path); the route posix-normalizes that field via normalize_path, so the assertion now uses Path.as_posix(), which is a no-op on POSIX. 324 tests across the three files pass on Linux. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: bound the cached dataset re-check and stop adopting unconfirmed starts Two defects introduced by this PR's new modules. 1. Unbounded dataset re-check (#7853). training-config-store.ts re-runs the cached format check whenever rejectValidation() reports the in-flight token as stale, with no counter and no backoff. DatasetCacheRejectionTracker advances its generation on any inventory-fingerprint change, and the fingerprint includes sizeBytes, so a dataset that is still downloading invalidates every check in flight and the pair never converges. Measured at 480 requests in 60s; a probe driving the real tracker ran 500 iterations without settling while a stable inventory settles in 1. The retry now draws from a per-selection budget (dataset-recheck-budget.ts) and falls through to the uncached check once spent, so the answer still refreshes when the inventory genuinely changes but a churning one cannot spin. The budget resets when the dataset or split changes. Fixing the tracker fingerprint instead was rejected: existing tests pin the behaviour that a sizeBytes change makes a rejected cache retryable. 2. Unconfirmed starts adopted as recovered. reconcileTrainingStartTransportFailure ended with adoptAcceptedTrainingStart(pending.jobId, ...) and returned "recovered". The backend reserves the request id and job id before the heavy preflight, so a start still pending when the 30s window closes may yet be rejected. Adopting there reported success and pinned a job id that never became current_job_id: acknowledgeTrainingStartRequest was never sent, and dismissTrainingRun's expectedJobId check then returned "superseded", leaving the rejected state unclearable until a new start re-reserved it. It now calls the module's existing settleUnconfirmedTrainingStart and returns "unknown", which both callers already handle by warning startUnconfirmed and settling unconfirmed, and which preserves start_request_id for acknowledgement. New tests fail without each change and pass with it; removing the bound makes them fail rather than hang. typecheck, 598 frontend tests, build, and the 154 studio contract tests all pass. * Studio: retry tokenizer-less pinned snapshots, and skip Hub preflight when offline Two more defects introduced by this rework. Neither mechanism exists on main: worker.py there has 0 occurrences of local_files_only / model_snapshot_path / cache_artifact, and routes/training.py has 0 of model_info / dataset_info. 1. Tokenizer-less pinned snapshot is terminal (#7845). The pin only requires config.json plus a weights file, so a snapshot with no tokenizer pins clean, local_files_only is set, and AutoTokenizer then fails. Recovery is gated on _is_model_cache_artifact_error, whose marker list matches only three of the message shapes transformers actually emits. Sweeping all 159 tokenizer classes against a tokenizer-less snapshot: 37 failures were classified not retryable, of which 26 are genuine cache problems that got zero Hub retry, including XLMRoberta (BGE-M3, multilingual-e5, LaBSE), MBart, NLLB, Bloom, GPTNeoX, Cohere, Marian and the generic PreTrainedTokenizerFast. SentencePiece and BPE families resolve a missing vocab path to None and then dereference it, so the failure arrives as a bare AttributeError with no cache-specific text. Adding those four shapes takes the sweep from 37 misses to 11, and all 11 remaining are correctly fatal (missing optional Python dependency, or an unsupported tokenizer class), which no Hub retry can fix. Widening the classifier rather than validating tokenizers before pinning: the vocab filename space is as open-ended as the exception space across 159 classes, requiring a tokenizer would break the deliberate adapter_config.json pin path, and a real loadability check means loading a tokenizer inside the start request. The recovery path is already gated on offline and require_exact, so a false positive costs one Hub attempt while the current false negative fails the run. 2. Blocking Hub preflight with no reachability guard. Both preflight legs retry metadata at 5s then 10s with no reachability check, and requests applies the timeout per resolved address. Measured 30.0s added to a single POST /training/start with the Hub black-holed, scaling with the number of addresses, surfacing as 503. utils.utils already provides a bounded, memoised hf_unreachable/hf_dns_dead that the training worker subprocess uses one layer down; the route that spawns it did not consult it. The model leg raises inside the existing try, so the except HTTPException handler still runs _resolve_model_snapshot and a cached snapshot pins exactly as before, just without first burning the remote budget. The guard fails open, so an online start is unchanged. New tests fail without each change and pass with it, including a wiring contract that fails if the guard stops being consulted. 443 tests across the preflight, cached-start, provenance, snapshot and streaming suites pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the import-hoist lint honour __all__ re-exports Source lint failed on studio/backend/utils/security/__init__.py: [BLOCKER] HOISTED-IMPORT-UNUSED 'load_scan_target' (['from:utils.security.file_security:load_scan_target']) added but unused That file is a barrel __init__: every name it imports carries "# noqa: F401" and is listed in __all__, and routes/training.py imports load_scan_target from the package rather than the submodule, so the re-export is load-bearing. HOISTED-IMPORT-UNUSED only excluded re-exports that already existed before the change, so adding any new name to an existing barrel was an automatic blocker. The script's own docstring already records this shape as a known false positive for NEW-UNUSED-IMPORT. Skip module-level imports whose bound name appears in __all__, since those are loaded by importers rather than by the module itself. The rule keeps its teeth: a newly added import whose name is not in __all__ still blocks, and a dangling alias still trips UNRESOLVED-NEW. Verified with a four-case mutation matrix plus --self-test, and the full 79-file changed-set lint now reports OVERALL: PASS. * Keep studio off evaluated PEP 604 unions on the 3.9 floor tests/test_python39_compatibility.py failed on Core (HF=4.57.6, default and latest) and Repo tests (CPU): test_no_pep604_unions_are_evaluated_on_the_declared_floor model_config.py:937: Tuple[...] | Tuple[...] (type alias) test_studio_evaluated_unions_do_not_grow 36 studio files now evaluate PEP 604 unions, up from 35 Both offenders came from this branch. model_config.py:937 is a module-level type alias, so it is evaluated at import time and "from __future__ import annotations" cannot defer it; it needs typing.Union, which is what the test message prescribes. hub/utils/dataset_cache.py is the one file that pushed the count to 36, and all five of its unions are function annotations, so the future import is enough there. No behaviour change: the alias is only a cache-key type, and the annotations are unevaluated either way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the dataset re-check budget on the whole selection The budget added for #7853 keyed on dataset + split only, but a dataset cache usability identity has four user-chosen dimensions: dataset, subset, split and streaming. Changing subset or toggling streaming therefore kept the same key, so a genuinely different selection inherited an exhausted budget, skipped the cache-preferring re-check and dropped straight to a remote resolution with datasetKnownCached cleared. Key on all four instead, JSON-encoded so no delimiter can collide with a name containing it and null stays distinguishable from the string "null". cachePath is deliberately still excluded even though the usability identity carries it. It is derived state that moves as a download populates the cache, so feeding it into the key would mint a fresh budget on every poll and re-arm the exact non-terminating loop this module exists to bound. Tests: two new cases cover subset and streaming, and an inverse case pins that nothing outside the selection can refresh the budget. Reverting the key to dataset::split fails exactly the two new cases and no others. * Stop the pinned snapshot path reaching PEFT as the base model name A completed run wrote a machine-local path as the adapter's base model: base_model_name_or_path = '/home/user/.cache/huggingface/hub/ models--unsloth--Llama-3.2-1B-Instruct/snapshots/0123…' where main writes the Hub id. It lands in adapter_config.json, every checkpoint-*/adapter_config.json, the run card, export_metadata.json for merged and GGUF exports, and the model card push_to_hub uploads, none of which resolve on another machine. restore_hf_cache_repo_identity runs in UnslothTrainer.load_model before get_peft_model, so its peft_config branch has nothing to repair yet, and PEFT then derives the name itself: # peft/mapping_func.py new_name = model.__dict__.get("name_or_path", None) peft_config.base_model_name_or_path = new_name PreTrainedModel.__init__ copies config.name_or_path onto the instance, so restoring only config._name_or_path leaves that slot holding the snapshot path. Restore the instance attribute too. It goes through the same guard as the other fields, so an ordinary local model, an unrelated Hub id and a repo mismatch are all still left alone. Tests are behavioural: the existing model identity suite asserts the call site via AST and stays green with this bug present, which is why it was missed. Removing the new line fails 3 of the 6 added tests and none of the other 11. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve cached snapshots that load from a subdirectory unsloth/Spark-TTS-0.5B keeps everything trainable under LLM/: its snapshot root carries only README.md and config.yaml, no config.json and no weights. _resolve_model_snapshot only looked at the root, so a cached copy resolved to None. _apply_model_cache_pin then warned "Cached copy not found on disk; downloading", and offline the start route turned the same None into a 409 hf_model_not_cached_offline for a model that was sitting in the cache. The remote preflight already handles this: it expands load roots through load_scan_target. Only the cached path disagreed. Reuse security_load_subdirs, which already reports ("LLM",) for BiCodec, so both paths share one source of truth, and apply it to the resume and preflight pin lookups too rather than just the fresh-start one. Detection can raise offline or for a gated repo, so a failure degrades to root-only rather than propagating. Tests build the real models--org--name/snapshots/<rev> layout. Making the helper a no-op fails 3 of the 6, and a snapshot with nothing loadable in either place still resolves to None, so the widening cannot mask an empty cache. * Use Optional in the deprecated dataset alias signatures The rewritten alias module annotates two parameters as UploadFile | None and str | None with no postponed annotations, so they evaluate at import on the declared 3.9 floor while the rest of the file already uses Optional[...]. The file was an evaluated-union offender before this change too, so this is not a new break, just keeping the new code consistent with its own convention and off the debt list. * Apply the repo kwarg-spacing formatter to the new code pre-commit.ci flagged the ruff-format-with-kwargs hook on this branch. Running scripts/run_ruff_format.py locally keeps the fix in the authoring commits instead of trailing a separate bot commit. * Say why a resume is refused instead of blaming the checkpoint can_resume_run gained a provenance clause in this rework, so it now returns False for runs whose checkpoint is entirely intact, most realistically once the pinned model snapshot is evicted from the HF cache. The start route answered every False with "Resume checkpoint must belong to a stopped or errored run with complete saved trainer state." pointing the user at trainer state that is fine, while the real cause was the resource gate. exact_resume_resource_requirements already raises with a precise explanation and resource_provenance_allows_resume was discarding it. Add resource_provenance_resume_blocker, which returns that explanation (or None when the run is resumable), define allows_resume in terms of it so the two cannot drift, and use it in the start route when it is the actual cause. The gate itself is unchanged: refusing an unattested resume is deliberate, only the diagnosis was wrong. Tests are behavioural, with one narrow wiring contract for the branch that regressed. Replacing the precise reason with a generic string fails the message test and nothing else. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Honour load subdirs everywhere a cached snapshot is probed Follow-up to 590ac9f22. Resolving cached Spark-TTS/BiCodec snapshots from their LLM/ load root got a start past _resolve_model_snapshot, but three later probes still asked for root-level config.json only, so the same snapshot was accepted in one place and rejected in the next: - provenance.py attested incomplete model provenance, after which exact_resume_resource_requirements refused resume for a snapshot still sitting on disk; - the start preflight reported a valid cached model as having no trainable weights; - the worker cleared model_snapshot_path before loading, falling back to the Hub instead of the selected snapshot. Promote the helper to hub.utils.hf_cache_state.with_load_subdirs so there is one definition rather than four, and use it at every site. _has_trainable_local_weights takes subdirectory roots instead, since it probes directories rather than a filename list. Unchanged for ordinary models: with no load subdirs the helper returns its input, and detection failures still degrade to root-only. * Let the model routes see the same cached snapshots training does Two ways /api/models disagreed with the training resolver, both reachable from resume: _model_config_inspection_target probed only the snapshot root, so a cached Spark-TTS/BiCodec copy answered "Selected cached model is no longer available" for a cache the training resolver accepts, and the exact-snapshot remote-code scan could fail with it. It now uses the shared with_load_subdirs helper. The model_snapshot_repo_id guard used the owner/repo-only regex, so resuming or scanning a namespace-less Hub model such as gpt2 or bert-base-uncased returned 400 before the snapshot could be inspected, even though hub.utils.paths .is_valid_repo_id and the picker both allow the one-segment form. That call site now uses the shared validator. The other five uses of the local regex predate this branch and are left alone. Reverting either fix fails one of the new tests and nothing else. * Get the resume refusal reason all the way to the user 300fe6321 fixed the diagnosis on POST /api/train/start, but the History UI never reaches it. can_resume: false hides the Resume button outright, and in the one window where the server could answer, resume-training-run.ts throws its own "Only stopped or errored runs with a saved checkpoint can be resumed" before sending any request, so no start call is ever made. For a run whose checkpoint is intact and whose pinned snapshot was evicted, that sentence is the wrong diagnosis, which is what 300fe6321 removed server-side. Carry the reason on TrainingRunSummary and prefer it in that guard. The field is optional and defaults to None, so old clients are unaffected, and it is only computed for rows already known unresumable. A checkpoint problem still reports None, leaving the client's existing wording for that case, and a failure inside the gate is swallowed so History still renders. Reverting either half fails one of the new tests: the summary stops carrying the reason, or the client stops preferring it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Honour load subdirs on the resume pin and keep the selected model cache copy Two follow-ups to the cached-snapshot work. The resume branch of _reject_untrainable_model_request still probed the pinned snapshot with a hardcoded (config.json, adapter_config.json) tuple, so a repo that loads from a subdirectory (unsloth/Spark-TTS-0.5B keeps everything trainable under LLM/) resolved to None. Offline that became a 409 hf_model_not_cached_offline for a snapshot present on disk; online it fell through to the remote metadata round trip the pin exists to avoid. It now uses with_load_subdirs like the other cached-snapshot probes. The cache reconciliation effect took the first usable inventory row for the selected repo. A repo can be present under more than one HF cache root, so that silently retargeted an explicit selection at a different copy on the next inventory tick. The selection logic moved to a pure module and now prefers a row whose path matches the current selection, falling back to the first usable row when there is no selection or the selected copy has gone. Tests: test_resume_pin_load_subdirs.py (5) and model-cache-reference-selection.test.ts (8); reverting either fix fails them. * Revert the model cache reference preference; it cannot fire The frontend half of 2c2a95309 assumed the model inventory can return more than one usable row for a repo. It cannot. hub/services/models/cache_inventory.py::_scan_cached_models collects into seen_lower keyed by repo_id.lower() and resolves collisions with _prefer_cache_row, and _dedupe_local_models keys hf_cache rows by (model_id, model_format, format_variant). Both return one row per repo, and a live check with the same repo under four HF cache roots got exactly one row from /api/hub/cached-models and one from /api/hub/local. With a single usable row usable.find(pred) ?? usable[0] is identically usable[0], so the change was a no-op and its tests asserted an array shape the API cannot emit. The resume pin fix in the same commit stands: that one was reproduced against the real unsloth/Spark-TTS-0.5B cache and reverting its hunk restores the 409. * Report the resume refusal that actually happened 300fe6321 and a98e721f4 set out to stop a provenance refusal being reported as a checkpoint problem. They overshot. Both sites asked resource_provenance_resume_blocker whenever can_resume_run said no, but that function refuses for several reasons and the blocker is computed independently of which one fired. initialize_resource_provenance writes {version: 1, status: pending} at the start of every run, so the blocker answers "The model revision used by this run was not attested." for any Hub-model run, including one whose checkpoint is simply missing. That is the more common way to be unresumable, so the change traded one misdiagnosis for another and the new one covered the larger population. Both sites now gate on has_resume_state, the same discriminator can_resume_run short-circuits on: no saved trainer state means the checkpoint is the cause and the client's own wording is correct; an intact checkpoint means a refusal really is provenance's doing and the specific reason is worth surfacing. test_resume_reason_matches_cause.py pins it; reverting either guard fails it. The route contract assertion is over the AST rather than the source text, because a substring search is satisfied by the explanatory comment beside the code and passes with the guard deleted. Two assertions in test_resume_blocked_reason_surfaces.py encoded the old behaviour and are corrected, including the docstring claiming History survives a raising gate: can_resume_run calls the same gate unguarded one line earlier, so that only holds when it short-circuits first. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope the import-hoist __all__ skip to package __init__.py 68bce3934 skipped any name listed in __all__. The comment said "the whole point of a package __init__", but compare() never consulted the path, so the skip applied to every module defining __all__ -- 27 non-package modules exempting 224 names, unsloth/models/_utils.py alone 74. The cost is not just breadth. lint-ci.yml names rename-clash as one of the two bugs this tool exists to catch because ruff and pyflakes miss it, and the skip disabled that detection for any name in __all__. Adding a name there became a one-line, reviewer-invisible way to switch the check off. The self-test I cited did not cover this: --self-test reported ALL PASS with the skip hunk reverted, because every case ran through the "<name>" path placeholder and none defined __all__. Cases may now carry their own path, and three new ones pin the behaviour from both sides -- a re-export in a package __init__ is allowed, the same shape in an ordinary module is still blocked, and a new import absent from __all__ is still blocked in an __init__ too. Deleting the skip fails the first; widening it back fails the second. Removes two genuinely unused imports the correctly-scoped check then found in test_resume_blocked_reason_surfaces.py. * Make deleting a run's artifacts reversible until the row is gone DELETE /runs/{run_id}?delete_artifacts=true is new in this PR; on |
||
Michael Han
|
3c400a67ba
|
Offline: detect an unreachable hub, not just dead DNS (#7591)
* Offline: detect an unreachable hub, not just dead DNS Loading an already-downloaded model with no internet took 11 minutes. The offline guard only checked whether huggingface.co resolved, so the common offline shapes where DNS still answers (WAN down behind a live router, captive portal, stale DNS cache) were treated as online and every hub call burned its full retry backoff. Two fixes: - Escalate from the DNS check to the bounded, proxy-aware reachability probe already used by export, memoised for 60s and opt-outable with UNSLOTH_OFFLINE_PROBE=0. - Force offline in-process, not just via env vars. huggingface_hub and transformers read their offline constants at import and hub sessions cache a non-offline adapter, so setting the env mid-process left the calls retrying anyway. Also guards the metadata routes that had none (/models/config, /models/check-vision, /picker/chat-template, the per-request vision probe) and applies the same detection in the training worker. Measured on a cached GGUF repo with the endpoint blackholed: POST /inference/load 686s -> 4s GET /models/config 378s -> 0s GET /models/check-vision 28s -> 0s Online is unchanged: reachable endpoints skip the guard entirely, and a fresh download still resolves, downloads and loads normally. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: endpoint-aware DNS check, shorter memo, strict gateway mode Four issues raised on the first commit, all reproduced before fixing: - The DNS pre-check hardcoded huggingface.co, so a reachable HF_ENDPOINT mirror was forced offline whenever huggingface.co did not resolve. It now follows the configured endpoint. - The reachability verdict was memoised for 60s, and a stale "reachable" hid the user pulling the plug right after a download, which is the exact workflow this fix targets. Window is now 5s in both directions: long enough to dedupe the probes within one load, short enough that neither direction goes stale. - hf_endpoint_unreachable counts 502/503/504 as offline, and the training worker used it to set flags for the whole job, so a momentary hub blip blocked every download for the rest of the run. Added gateway_errors_offline=False for callers setting lifetime flags; scoped callers keep the existing behaviour. - Dropped the guard from _target_is_vision. The resolver only yields local paths there, so it returns from the mmproj filesystem branch without touching the hub, and the probe only added latency per request. Verified unchanged offline: load 686s -> 5s, /models/config 378s -> 0s, /models/check-vision 28s -> 0s. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review round 2: proxy-aware detection, shared worker probe, local skip All four reproduced before fixing, and re-measured after. - Proxy-only egress was declared offline. With HTTP(S)_PROXY set, the proxy resolves the hub host, so a failing local lookup says nothing. The DNS shortcut now stands down whenever a proxy applies (and honours NO_PROXY), letting the proxy-aware probe decide. Measured: endpoint probe reachable through the proxy while the guard still forced offline. - The training worker kept its own inline probe hardcoded to huggingface.co, so a reachable HF_ENDPOINT mirror set lifetime offline flags. It now uses the shared endpoint- and proxy-aware helper. - /models/check-vision, /models/config and /picker/chat-template ran the probe even for local paths, which never reach the hub. Measured 0.9s of pure latency per request; now skipped via _hf_offline_if_unreachable_for. DNS/endpoint/proxy helpers now live in utils.utils so llama_cpp and the training worker share one implementation instead of three copies. The static pin in test_offline_inference_parent moved with the probe: the worker block must delegate to the shared helper and must not hardcode a host, and the daemon-thread/no-setdefaulttimeout property is pinned on dns_host_dead where it now lives. Offline path unchanged: load 686s -> 6s, /models/config 378s -> 0s, and a local-path vision check is back to 0s. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review round 3: slow links, blank endpoint, IPv6 resolution - A reachable endpoint that answers slower than the probe deadline was classified offline, so an uncached load failed instead of merely being slow. A clean socket timeout is now resolved with a bounded TCP connect: a loaded server still completes the handshake, a blackholed route does not. A refused connection counts as egress. - An empty or whitespace HF_ENDPOINT made the DNS shortcut fall back to the default hub while the HTTP probe probed "https://" and reported offline. Both stages now share one normaliser. - dns_host_dead used gethostbyname, which is IPv4-only and called an AAAA-only mirror or an IPv6 literal dead. It now uses getaddrinfo. A hang past the deadline still counts as unreachable: the real hub calls would hang the same way, so cache-only is the useful answer there. That distinction is what test_hung_probe_is_bounded pins, and it caught an earlier version of this change that treated every deadline overrun as inconclusive. Verified: slow endpoint reachable, blackholed route still offline, blank endpoint falls back, IPv6 literal resolves. Offline path unchanged, load 686s -> 9s and /models/config 378s -> 0s, chat still answers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review round 4: refcount concurrent guards, distrust proxy TCP - Overlapping requests lost offline mid-flight. A later guard saw the HF_HUB_OFFLINE that an earlier one had set and took the no-op branch, so when the earlier guard exited it restored the constants and sessions while the later request was still resolving hub files, dropping it back onto the retry path. Each guard now holds its own reference on the refcounted force_hf_offline window. A user-supplied offline variable is still left untouched, told apart via force_hf_offline_active(). - The socket-timeout fallback trusted a TCP handshake to the proxy, which only proves the proxy is up, not that it can reach the hub. A live proxy with a blackholed upstream therefore read as reachable. With a proxy configured the timeout now stays unreachable; the TCP check is only evidence when connecting to the endpoint directly. Verified: second guard engages and offline survives the first guard's exit, state fully restored after both; dead-upstream proxy reads unreachable while a slow direct endpoint still reads reachable; 9 concurrent metadata requests against an unreachable hub all return 200 in 5.1s total. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review round 5: probe through all_proxy, honour TRANSFORMERS_OFFLINE Resolve the hub proxy the way requests does (scheme-specific, then all_proxy, NO_PROXY wins) and issue the reachability HEAD through it. urllib ignores all_proxy, so a proxy-only setup failed the probe's direct lookup and was called offline while real hub calls would have succeeded. Skip the probe when TRANSFORMERS_OFFLINE alone is truthy: that is still an offline request, and the hub does not read it. Engage the guard directly instead of putting a DNS lookup and a HEAD in an explicitly offline process. HF_HUB_OFFLINE=0 remains an explicit stay-online opt-out. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix offline guard concurrency and proxy handling * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix shared offline probe edge cases * Handle proxy reachability edge cases * Offline guard: proxy detection without requests, fail-open probe, and two unguarded routes Follow-ups from simulating this path across a dependency matrix, emulated platforms and an edge-case sweep. All measured against a blackholed endpoint. Proxy detection no longer goes blind when requests is missing. huggingface_hub 1.x moved to httpx and dropped requests, so `from requests.utils import ...` raises there and hf_proxy_for_endpoint silently answered "no proxy". hf_dns_dead stands down only when a proxy is configured, so a proxy-only machine whose DNS cannot resolve the hub was forced offline while the hub was reachable through the proxy. This is a supported combination: transformers 5.x requires hub 1.x, and no-torch-runtime.txt allows huggingface_hub>=0.34.0 with transformers<=5.3.0. Added a stdlib fallback over urllib.request.getproxies that reproduces select_proxy's order and also covers macOS SystemConfiguration and the Windows registry. Verified with a real loopback proxy: before, the guard engaged offline; after, it correctly stays online. A socks5:// proxy is now inconclusive rather than offline. urllib cannot route through SOCKS and fails in 20ms with "unknown url type", which read as no egress even though the Hub client reaches the hub through that proxy. The probe fails open on non-network errors, as its docstring already claimed. A malformed HF_ENDPOINT, a bad proxy string or a bug in the probe itself returned "unreachable" and got memoised for 5s, quietly pinning every load to the cache. Socket errors, including URLError with a socket reason, still mean offline; blackhole and NXDOMAIN verdicts are unchanged. child_env(base=...) scrubs the scoped offline flags too. It took the mapping verbatim, so the vision-check sidecar, which builds its env from a raw os.environ copy and can run inside the /models/config window, inherited HF_HUB_OFFLINE=1 and stayed cache-only for its whole life. A user-set value still propagates. /models/check-embedding takes the same guard as its /check-vision twin. Against an unreachable hub it cost the full 15s model_info timeout and then answered False. Measured 15.02s to 5.01s. /inference/validate takes the guard /inference/load already had. The frontend validates before it loads, so the stall simply moved there. /models/config, /models/check-vision and /models/check-embedding run the guard in a worker thread. They are async handlers calling a blocking DNS + HEAD + TCP probe, which froze the whole API. Worst event-loop stall drops from 8014ms to under 1ms. An https:// proxy with no explicit port now defaults to 443 rather than 80. Two pre-existing tests moved from gethostbyname to getaddrinfo, which is what the probe calls; the wedged-resolver assertion was passing vacuously off the real NXDOMAIN for .invalid. test_unresolvable_host_still_dead now mocks the resolver rather than trusting the runner's, which fails under NXDOMAIN hijacking. Dropped an unused import of _hf_offline_if_unreachable in core/training/trainer.py. 17 regression tests added. Offline suites 407 passed, ruff clean. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Offline probe: route metadata reads through the hub proxy, treat resets as egress, hold offline across a spawn Three review follow-ups, each reproduced against real sockets before and after. Metadata reads now use the same proxy the hub client would. The probe became proxy-aware but _remote_lora_base, _check_tokenizer_config_needs_v5 and _load_config_json stayed on bare urlopen, which ignores ALL_PROXY. On a proxy-only machine the guard therefore reported online through the proxy while every raw config read went direct and returned None, dropping sidecar tier selection to substring matching on the repo name. Measured with a loopback proxy and HF_ENDPOINT on an unresolvable host: the probe reached the proxy, the three reads reached nothing. Added _hf_proxy_opener/_hf_urlopen and pointed the probe and the three readers at them, so they cannot diverge again. The empty ProxyHandler branch for a requests-side NO_PROXY bypass now covers the readers too. A socks proxy yields no opener, so those paths are unchanged. A connection reset now counts as egress. urllib only wraps OSErrors raised while sending, so a reset from getresponse arrives raw as http.client.RemoteDisconnected, a ConnectionResetError subclass, and fell through to the bare OSError handler as no egress. Against a loopback server that accepts then closes, and one that sends RST, the probe returned unreachable; the training worker turns that verdict into HF_HUB_OFFLINE for the whole job, so one transient reset stranded the run. Widened both branches to ConnectionError, the same "the wire answered" family as the refused case already there. A blackholed route raises gaierror or ENETUNREACH, neither of which is a ConnectionError, so genuine no-egress is unaffected. The offline gate now holds across a spawn. hf_environment_restored_for_spawn restores the user's own values into the parent os.environ for the whole Process.start() window so the child inherits their intent, but the raw metadata readers gate on _env_offline, which reads os.environ. Reproduced with three threads: guard held, spawn window open, and a _remote_lora_base call on a third thread reached the network with the patched hub constant still offline. Both _env_offline and hf_env_offline now also honour an open force_hf_offline window. force_hf_offline_active drops its lock so an offline check cannot block for the duration of a spawn; the depth is only raised after env and constants are already offline, so a lock-free read never reports offline early. Ten regression tests added, several driving real sockets rather than mocking the function under test. Offline suites 417 passed, wider studio backend suite unchanged at 17 pre-existing failures, ruff clean. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rename the offline guard refs main added after the rename, for PR #7591 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Offline guard: fail open on slow proxies, cover the validate preflights, isolate proxy env in tests for PR #7591 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Offline guard: key the window on what is read, spare local loads, keep a slow resolver and TRANSFORMERS_OFFLINE from forcing offline for PR #7591 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Read sources as utf-8 in the offline guard tests so Windows does not decode them as cp1252 for PR #7591 * Model-config offline predicate honours an open guard so the raw audio fetch stays offline across a spawn for PR #7591 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Offline guard: cover a local model's remote base, the validate security scan, and let route patches intercept for PR #7591 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Read hub metadata through the resolve route so mirrors work, and skip the guard when a local GGUF has no base for PR #7591 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Inference worker probes for an unreachable hub like the training and export workers for PR #7591 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Include a local LoRA's remote base in the guard targets, and tighten the offline comments for PR #7591 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hung probe honours fail-open behind a proxy, and sibling guards reuse one reachability verdict for PR #7591 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Clear the reachability memo between tests so a neighbour's verdict cannot short-circuit the stubs for PR #7591 * Skip the worker reachability probe for filesystem-only jobs for PR #7591 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Check local LoRA bases in the training gate and resolve /load config off the loop for PR #7591 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Include full-checkpoint bases in both worker probe gates for PR #7591 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Point the merged tunnel-safe load test at the renamed offline guards for PR #7591 * Mirror the adapter-only dir-name base in the probe gates for PR #7591 * Probe module availability instead of importing it in the offline test stubs for PR #7591 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
|
Michael Han
|
a00fe86c13
|
Studio: read model text as utf-8 so umlauts survive on Windows (#7467)
* Studio: read model text as utf-8 so umlauts survive on Windows Chat rejects or mangles non-ASCII on Windows: "ä ö ü" in a prompt, a chat template, or a model path comes back as mojibake, or the load dies with UnicodeDecodeError. open() and Path.read_text() fall back to locale.getencoding() when no encoding is passed. On Windows that is the ANSI codepage (cp1252, cp932, cp1251, ... by system locale), never UTF-8. Hugging Face writes these files as raw UTF-8, so every read of one decodes with the wrong codec: - tokenizer_config.json, which holds the chat template. Templates routinely carry -> arrows, smart quotes and CJK, so this is the common path into chat - config.json and adapter_config.json - modules.json, Ollama manifests, and the .py sources the remote-code scanner reads before a model is allowed to load The llama-server and embedding-server stdout readers have the same problem via subprocess(text = True); they now decode utf-8 with errors = "replace" so a stray byte cannot kill a log reader. Encoding arguments only, no logic changes. tests/test_chat_text_encoding.py covers a config.json and a chat template holding umlauts, arrows and CJK, plus the remote-code scanner reading a source file with umlauts. Those pass anywhere the locale is already UTF-8, so a fourth test re-runs the readers under -X warn_default_encoding and fails on any platform if an encoding argument goes missing again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: name utf-8 explicitly on the remaining text I/O, with an AST guard (#7465) * Studio: name utf-8 explicitly on the remaining text I/O Follow-up to the model-text reads in #7467, covering the rest of the backend: system probes (nvidia-smi, amd-smi, powershell, git, node), package installers, /proc and /sys readers, and internal marker files (pid, install id, bootstrap password, Colab credentials). Same reason as #7467. open(), Path.read_text()/write_text() and subprocess(text = True) fall back to locale.getencoding(), which on Windows is the ANSI codepage rather than UTF-8. These paths are mostly ASCII today, so this is hardening, not a live bug. Encoding arguments only, no logic changes. Adds tests/test_text_io_encoding.py: an AST guard walking every backend source and asserting text I/O names its encoding, so the class of bug cannot creep back in one call at a time. 275 files. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Catch aliased subprocess and positional Path.open, migrate legacy JSONL The guard only matched a receiver literally named subprocess, so worker.py's `import subprocess as _sp` hid three text = True installs that decode pip output with the ANSI codepage. It also skipped any .open() with more than one positional argument, though Path.open takes buffering/encoding/errors/newline positionally. Resuming a scrape written by an older release is the other half: those JSONL lines are in the locale codepage, so the UTF-8 preload raised, the dedup keys were silently forgotten and duplicates were appended to a now mixed-encoding file. Decode with the locale codepage as fallback and rewrite as UTF-8 before the append handle opens, since Windows cannot replace a file it holds open. * Stream the JSONL preload and keep a torn line from relabelling the shard Reading the whole shard to migrate it was wrong twice over. These files reach gigabytes on a large scrape, so the preload now streams line by line and the rewrite streams through a temp file. Worse, one interrupted append used to condemn the file: the whole-file UTF-8 decode failed, every byte was retried as cp1252, and the rewrite persisted mojibake over records that were fine. A line now counts as legacy only if the locale codepage both decodes it and yields valid JSON, which a torn UTF-8 line does not. Damaged lines are skipped and copied through byte for byte. When the rewrite cannot be written at all, the append handle opens with the legacy encoding rather than mixing UTF-8 into the file. install_wheel takes run = subprocess.run as a parameter, so the guard cannot see it. Both wheel installs there now name their encoding. * Decide the shard's encoding from the file, not one line at a time Some byte strings parse both ways. cp1251 `Р°` is D0 B0, which is also valid UTF-8 for `а`, so a UTF-8-first parse quietly showed the wrong text instead of migrating it. A line now yields both readings, and the file decides. Any line that parses under the codepage but not as UTF-8 is unambiguous evidence, and ambiguous lines then follow that verdict, which is enough for any real shard: ordinary Cyrillic or Japanese prose is invalid UTF-8 several times per line. Keys for ambiguous lines are re-derived from the legacy reading during the rewrite. A shard is undecidable only if every line is ambiguous, and nothing can tell those apart. latin-1 is also tried after the locale codepage, so a scrape carried from Windows to a UTF-8 machine still has a reading rather than none. Requiring valid JSON, not just a decode, keeps that from claiming torn lines. * Weigh the whole shard, and never lose a record on the fallback path One structurally valid JSON line carrying a stray 0x96 parses as cp1252, so a single-line verdict let it relabel a healthy shard and mojibake every good record in it. Each line with non-ASCII bytes now votes: parsing only under the codepage is evidence for legacy, parsing as UTF-8 is evidence against, since codepage text rarely forms valid multibyte UTF-8. Ties leave the file alone. When the migration cannot be written the append handle uses the legacy codepage, and errors = "replace" quietly turned characters it cannot hold into question marks while write() still reported success. That path now escapes to \uXXXX instead, which is ASCII, so every codepage holds it and json.loads returns the exact characters. Nothing needs replacing, so errors = "strict" is safe. stream_installer runs sys.executable, so its output is now decoded as UTF-8 by utf8_child_env rather than read as the ANSI codepage. * Only rewrite a shard we can attribute, and append ASCII when we cannot latin-1 was doing too much work. It reads any byte, so it gave a moved shard a reading, but it is the right text only for cp1252: cp1251 Привет came back as Ïðèâåò and the rewrite made that permanent. The codepage is now trusted only when it is the locale's, and an untrusted reading is never written back. That leaves three cases where the file holds bytes UTF-8 cannot read and we are not converting it: no codepage to attribute it to, ambiguous lines outvoting the unambiguous ones, and a preload that could not read the file at all. All three used to append UTF-8 into it. They now append pure ASCII, which every ASCII-compatible codepage stores identically, so the file keeps decoding exactly as it did and no record is lost. Keys from the two readings are also kept apart. A damaged line in a healthy shard was marked seen through its codepage reading, so the retry that would have replaced the unreadable record was refused as a duplicate. * Let the flash-attn install stub take the kwargs the installer now passes _run_kwargs gained encoding and errors, so the one stub in this file that spelled its signature out rejected the call. The other four here already take **kwargs; this one now matches. * Do not let a stuck temp file mask the migration failure unlink() on the failure path could raise in its own right, on a stale .utf8.tmp directory or a temp another process holds. That escaped the constructor instead of returning False, so the caller never reached the ASCII append fallback that keeps the shard single-encoding. The pip fallback in install_wheel also spawns a Python child, so it gets utf8_child_env like the probe above it already had. The uv and nvidia-smi children are native binaries, where PYTHONIOENCODING would do nothing. * Stop converting legacy shards; the encoding that wrote them is unknowable trusted only ever meant that the bytes parse under this machine's codepage, which for a single-byte codepage is nearly always true. A cp1251 shard opened on a cp1252 Windows box decodes cleanly and would have been rewritten with Привет as Ïðèâåò. That is the fourth way this rewrite could corrupt a shard, and the common cause is that a file's encoding cannot be recovered from its bytes. So the rewrite is gone. The shard is left exactly as found, and appends are pure ASCII whenever it holds bytes UTF-8 cannot read, which is what actually delivered the no-mixed-encoding guarantee the rewrite was added for. Dedup keys still come from whichever reading parses, since ids are ASCII either way. This also removes the temp file, so there is no longer any file mode or ACL to carry across. * Scan the sandbox shim; it is shipped code, not a build artifact sandbox_site is on the sandboxed child's PYTHONPATH for every Python run (tools.py:332, 2660), so excluding it let two unannotated text calls through in code we ship. Both read and write the remap sidecar, which holds file paths. The exclusion list is meant for build output only, so the directory comes off it and the two calls name their encoding. * Force the worker's pip children to UTF-8, and read DBCS keys with a DBCS codec The three installer calls run sys.executable -m pip with an inherited environment, so the parent decoded UTF-8 while the child emitted the ANSI codepage. They now go through utf8_child_env like the other Python children. Two tests asserted no env kwarg was passed as a stand-in for no HIP flag being injected. They now assert the flag itself, which is the guarantee they were written for and does not depend on how the env is delivered. Separately, latin-1 cannot stand in for a double-byte codepage while recovering dedup keys: cp932 表 is 95 5C, and the trail byte reads as a JSON backslash, so the record failed to parse and its id was forgotten, appending a duplicate on resume. cp932, cp936, cp949 and cp950 are tried too. The reading is still only ever used for keys, which are ASCII and identical whichever codec parses. * Require more than one legacy line before trusting its dedup keys A shard whose valid records are all ASCII casts no UTF-8 votes, so a single damaged line won the vote by itself, its key was remembered, and the retry that would have replaced the unreadable record was refused. One such line is genuinely undecidable: a legacy record with one accented character and an ASCII record with one stray byte are the same shape. Reading it as damage costs a duplicate; reading it as legacy loses the record for good. Only one of those is recoverable, so it is now read as damage. A real legacy shard has a legacy line for every record carrying an umlaut, so its dedup is unaffected. * Append ASCII whenever the shard already holds non-ASCII bytes The gate asked whether any line was undecodable as UTF-8, which misses a shard where every legacy line happens to be valid UTF-8 too. A cp1251 shard of Р° records is bytes D0 B0 throughout, so appending 世界 as UTF-8 left a file where cp1251 reads the old records correctly and the new one as mojibake, and UTF-8 does the reverse. No single decoding recovered the whole scrape. The gate is now simply whether the shard holds any non-ASCII byte at all, which covers both cases and is easier to reason about: if what is already there reads differently under different encodings, do not add more bytes that do. Appending ASCII costs only \uXXXX escapes, which json.loads turns back into the exact characters, and it leaves the new record correct under either reading. * Skip the two Linux-gated flash-attn tests off Linux _should_try_runtime_flash_attn_install ends in sys.platform.startswith( "linux"), and the threshold test one line above already asserts exactly that, so the two tests that drive _ensure_flash_attn_for_long_context past the gate cannot pass anywhere else: the call returns before it reports a status. They were written on Linux and only surface once the suite actually runs on Windows or macOS, where both fail on an empty status list. This PR is about making the backend behave on Windows, so its own suite should be runnable there. * Fail closed when a KFD topology node does not decode This PR pins that read to utf-8, which turns an undecodable byte into UnicodeDecodeError. That is a ValueError, not an OSError, so it slips past the handler one line below and escapes a helper whose docstring promises to fail closed on any unreadable node. The caller would then lose the whole HIP-order map on a machine that has AMD GPUs, and the reason the helper fails closed is that dropping a node shifts every later ordinal and lets a similar-capacity GPU pass the total-size guard while showing another card's usage. Widening the handler is the same one-line change main already made in #7487, so the two agree and the eventual merge is clean. * Tighten the comments added in this branch * Treat an undecodable marker and undecodable metadata as malformed, not fatal Two more places where pinning the decode changed the failure mode. A UnicodeDecodeError is a ValueError, so neither `except OSError` nor `except (JSONDecodeError, OSError)` catches it, and both sites had a documented fallback that stopped being reached. An undecodable .transport marker used to read as an unknown value, and the caller then safely purged and restarted the partial download. It now aborts prepare_cache_for_transport instead, so the transfer fails rather than retrying. Undecodable .meta.json used to fall back to the file's own name, the same way invalid JSON does. It now aborts URI construction for the entire unstructured seed, so one corrupt byte in original_filename takes out the whole dataset. Both handlers are widened, matching the KFD fix earlier on this branch. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Widen two more decode guards, and pin the kernel installer's pipe Same shape as the ones already fixed here: the read was pinned to UTF-8 while the handler around it still only catches OSError, and UnicodeDecodeError is a ValueError. hf_cache_snapshot_dir answers whether a model is already on disk, and the offline embedding checks turn a raise into a 500. A torn refs/main used to decode into a nonsense commit and miss the snapshot dir; it now skips that cache root and keeps looking. _remove_pid_file runs first in _graceful_shutdown, so a corrupt studio.pid raising there abandoned the inference, export, training and tunnel children the rest of that function exists to kill. ssm_runtime's source-build path builds its subprocess kwargs in a dict and splats them through _run_with_heartbeat, so neither the encoding guard nor the earlier sweep saw the text = True in it: pip's output was still decoded with the Windows ANSI codepage, where a non-ASCII path or a compiler diagnostic mojibakes or raises over an install that was going fine. It now pins the same utf-8/replace pair install_wheel uses, and the HIP branch extends that env rather than replacing it. The guard learned the dict-literal shape and reddens on the old code (ssm_runtime.py:253). * Tighten the comments around the UTF-8 text I/O pins Collapse the multi-line rationales added with the encoding pins down to a line or two each, drop what the code already says, and use one wording for the repeated child-env note. * Do not let an unreadable bootstrap password stop startup, and narrow the kwargs guard ensure_default_admin calls _load_bootstrap_password for every existing admin and the lifespan calls that with no handler, so pinning the decode turned a damaged or pre-pin .bootstrap_password file into a backend that will not start. We write that file ourselves in UTF-8, so a byte that will not decode belongs to a file whose plaintext is worthless anyway; it now reads as no bootstrap password, the same answer as an absent file. A readable one still loads. The new kwargs check also judged every dict literal in the tree, so an unrelated payload carrying "text": True would have been reported as subprocess configuration with a misleading message, and a dict that fills in its encoding on a later line would have been reported too. It now only judges a dict that actually reaches a call, either splatted through a name or written at the call site, and treats a later kw["encoding"] assignment as satisfying it. The ssm_runtime shape it was written for is still caught, and a test pins both directions. * Stop reading a UTF-8 record a second time _read_line always parsed the line under the codepage as well, even when it had already read as UTF-8. Both callers take the UTF-8 reading when there is one and never look at the other, so on a healthy shard the second parse is pure waste, and this file reads all of one on every resume of a scrape it expects to reach gigabytes. Measured on 200,000 records, 76 MB: 1.96s before, 0.81s after, so the double reading was costing 2.8x. The early return is limited to a record, since the key lookup deliberately falls through to the codepage reading when UTF-8 yields something that is not one. A line UTF-8 cannot read still tries the codepage, latin-1 and the double-byte encodings as before, which is what the second reading is for. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the scanned source fixture's line endings test_remote_code_scan_reads_non_ascii_sources compared a file's contents against the string it wrote, but wrote it in text mode, so Windows translated the line ends on the way out and the read back differed by a carriage return. That is the writer's doing, not the encoding the test is about, and it was the one failure on the Windows runner that belonged to this branch. The fixture now writes with newline = "" so the bytes on disk are the string on every platform. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the newer comments to their point Shorten the widened-guard and state store notes added since the last pass, and collapse the line-ending note on the scanned source fixture. * Read the scraper checkpoint as UTF-8 only, never as a codepage A checkpoint holds nothing but base64 cursors and booleans, so one written by an older locale-encoded release is byte-identical to a UTF-8 one and already reads back. The codepage fallback can therefore only ever contribute non-ASCII: if a single-byte reading of the file were all ASCII, the UTF-8 read would have succeeded first. So the only file it changes the answer for is a damaged one, and there it turns a safe reset into a resume on a mojibaked cursor. GitHub answers that with INVALID_CURSOR_ARGUMENTS at HTTP 200, gh_client returns the partial document, and the scraper reads zero nodes and an empty pageInfo, which marks the stream done. Every later resume then skips it entirely. Reading UTF-8 only restores the earlier behaviour of dropping a checkpoint that will not decode, which re-scrapes from the first page while the writers dedup the replay. The shard scan below keeps its codepage reading; those records do carry non-ASCII. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gate the remaining tilelang install tests to Linux _tilelang_platform_supported() returns False off Linux, so _ensure_tilelang_backend returns before the install and the subprocess mock these six assert on is never called. They fail on macOS runners for that reason alone. The rest of the file already carries this marker; these were missed. * Gate the Windows-incompatible worker and ROCm tests Two different gates, because the production code has two. The causal-conv1d and flash-linear-attention installers bail out on sys.platform == 'win32' alone and run everywhere else including macOS, so those cases get not_on_windows; marking them linux_only would skip tests that legitimately pass off Linux. The DRM and KFD readers return early unless platform.system() is Linux, and their fixtures build a fake sysfs tree needing PCI addresses like 0000:00:02.0 as directory names, which Windows cannot represent, so those get linux_only. The two visible-utilization cases failed for a different reason: on Windows get_visible_gpu_utilization takes the AMD adapter branch ahead of the torch fallback under test, and probing it imports torch, which the runner lacks. Stubbing that branch empty leaves every other platform unchanged. * Treat unparseable JSON nesting as a parse failure, and guard os.fdopen json.loads answers nesting it cannot descend with RecursionError, a RuntimeError, so _parse let it escape where the catch-all it replaced discarded the record. Both callers run _parse outside any further handler, so one damaged checkpoint or shard line aborted the scraper at startup. The encoding guard also missed os.fdopen, which is open() on a descriptor and takes the same locale default in text mode. It flags exactly the two text-mode calls that were left unencoded; the swap lock file's reader was already pinned to UTF-8 while its writer still used the codepage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Write the non-ASCII source fixture without a 3.10-only argument Path.write_text() only grew newline in 3.10, and pyproject declares requires-python >=3.9, so this raised TypeError there. open() takes the same argument on every supported version and pins the bytes on disk the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten encoding comments * Follow subprocess calls through callable aliases in the encoding guard --------- Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com> --------- Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com> |
||
Daniel Han
|
1daaa5cbb4
|
Let a decode failure degrade instead of escaping a fail-closed helper (#7487)
* Let a decode failure degrade instead of escaping a fail-closed helper Pinning utf-8 makes a read that used to return mojibake on Windows raise instead. 33 of those reads sit under a handler catching OSError or json.JSONDecodeError but not UnicodeDecodeError, which subclasses ValueError, so a corrupt file would now escape a helper written to return a default. Adds UnicodeDecodeError to those tuples only. * Treat an undecodable install lock as stale instead of retrying forever |
||
|
Daniel Han
|
3fd948eb95
|
Pin utf-8 on shipping-code text I/O instead of the operator locale (#7486)
* Pin utf-8 on shipping-code text I/O instead of the operator locale 113 read_text/write_text/open call sites across unsloth, studio and unsloth_cli let locale.getencoding() decide the encoding. That is utf-8 on the Linux and macOS runners and cp1252 on a stock Windows install, so the same file decodes differently for a Windows user and silently produces mojibake or raises UnicodeDecodeError. Adds tests/test_runtime_text_encoding.py to keep it that way. It resolves openers through each file's own imports rather than a fixed list of module names, so an aliased tarfile.open or a local from PIL.Image import open is not asked for an encoding it does not take. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scan tracked files only and resolve the unbound Path calling forms * Honour PEP 263 when scanning sources and migrate a legacy JSONL before appending * Scope guard imports lexically and only migrate a legacy file when it round-trips * Leave a legacy JSONL untouched and resolve path aliases in the foreign-opener check * Tighten comments --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
Lei Zhenyuan
|
47fa4ca6c1
|
Add Intel XPU support to Unsloth Studio (#4724)
Some checks failed
Wheel CI / Wheel build + content sanity + import smoke (push) Waiting to run
Mac Studio GGUF CI / OpenAI, Anthropic API tests (push) Waiting to run
Mac Studio GGUF CI / Tool calling Tests (push) Waiting to run
Mac Studio GGUF CI / JSON, images (push) Waiting to run
Mac Studio Install Matrix CI / Install + load (macos-14) (push) Waiting to run
Mac Studio Install Matrix CI / Install + load (macos-15) (push) Waiting to run
Mac Studio Install Matrix CI / Install + load (macos-26) (push) Waiting to run
Mac Studio Install Matrix CI / Install + load (macos-15-intel) (push) Waiting to run
Mac Studio Install Matrix CI / Install + load (macos-26-intel) (push) Waiting to run
Mac Studio UI CI / Chat UI Tests (push) Waiting to run
Mac Studio Update CI / Unsloth Updating Tests (push) Waiting to run
Unsloth Tauri CI / Tauri Linux debug build (no codesign) (push) Waiting to run
Unsloth UI CI / Chat UI Tests (push) Waiting to run
Unsloth Update CI / Unsloth Updating Tests (push) Waiting to run
Windows Unsloth API CI / Unsloth API & Auth Tests (push) Waiting to run
Windows Unsloth GGUF CI / OpenAI, Anthropic API tests (push) Waiting to run
Windows Unsloth GGUF CI / Tool calling Tests (push) Waiting to run
Windows Unsloth GGUF CI / JSON, images (push) Waiting to run
Windows Unsloth GGUF CI / Unsloth install + inference without Visual Studio (push) Waiting to run
Windows Unsloth GGUF CI / GPU prebuilt resolves without Visual Studio (push) Waiting to run
Windows Unsloth GGUF CI / setup.ps1 unit tests (VS 2026 / CMake guard) (push) Waiting to run
Windows Unsloth GGUF CI / real-VS detection (VS 2022) (push) Waiting to run
Windows Unsloth GGUF CI / real-VS detection (VS 2026) (push) Waiting to run
Windows Unsloth GGUF CI / VC++ runtime detect + install round-trip (windows-2025-vs2026) (push) Waiting to run
Windows Unsloth GGUF CI / VC++ runtime detect + install round-trip (windows-latest) (push) Waiting to run
Windows Unsloth UI CI / Chat UI Tests (push) Waiting to run
Windows Unsloth Update CI / Unsloth Updating Tests (push) Waiting to run
Cross-platform parity / parity (macos-latest) (push) Has been cancelled
Cross-platform parity / parity (ubuntu-latest) (push) Has been cancelled
Cross-platform parity / parity (windows-latest) (push) Has been cancelled
--------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
oobabooga
|
dbb06ff60e
|
Studio: add configurable model download location (#7274)
Adds a configurable Hugging Face model download cache location to Unsloth Studio, selectable from Settings, with per-cache download manifests, scoped deletion, and read-only inventory of previously selected caches. |
||
Hakan Baysal
|
aa49c0710e
|
studio: classify embedding models from the HF cache and honor offline mode (#7218)
* studio: classify embedding models from the HF cache and honor offline mode is_embedding_model() went straight to huggingface_hub.model_info() for any repo id, so in offline mode (no DNS, or HF_HUB_OFFLINE set) selecting an already-downloaded model hung on network retries that could never succeed and training/export never started (#6817). Check the local HF cache first: a sentence-transformers repo carries modules.json in its snapshot (the same marker used for local paths), so a cached model is classified with no network call. When HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE is set, anything not positively an embedding model returns False without a network call instead of retrying a doomed request. Online, uncached lookups still fall through to model_info(), so tag-only embedding models (feature-extraction) are unaffected. Adds _embedding_marker_in_hf_cache() over the existing _iter_hf_cache_snapshots. * studio: judge the active cached revision, harden the cache probe, stop stub leaks Three review fixes on the cache-first embedding detection: 1. Prefer the revision refs/main resolves to. The HF cache keeps snapshots of older revisions, so an any-snapshot scan could classify a repo by a stale revision -- e.g. a repo that used to be a sentence-transformers model would short-circuit even the online lookup. When refs/main is recorded, only its snapshot is consulted; the newest-first scan remains the fallback for caches with no ref. 2. Keep the cache probe inside the detection error boundary. The snapshot iterator stat()s entries and could raise if a cached model is deleted concurrently, propagating a 500 out of the config/check-embedding routes. _embedding_marker_in_hf_cache now catches everything and reads as not-cached, so callers keep their normal Hub/offline fallback. 3. Stub loggers/structlog in the test only when the real modules are absent (try-import, mirroring test_windows_gpu_detection_mock), so collecting this file first can no longer shadow the real packages for later tests in the same pytest process. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: treat a missing active-ref snapshot as a cache miss, don't cache offline misses Two review fixes on the cache-first embedding detection: 1. When refs/main is recorded but points at a commit whose snapshot dir is absent (partial download / cache pruning), the recorded ref is still authoritative: return None (cache miss) instead of falling through to scan older snapshots, which could report a stale historical revision's modules.json as the active one -- the same stale-cache class this helper avoids. 2. Do not cache the offline negative. When HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE is set and the repo is not positively an ST model from modules.json, is_embedding_model stored False under the (model_name, hf_token) key shared with online lookups; after the env var cleared in the same process, a tag-only (feature-extraction) embedder returned the cached False and never reached model_info(). The offline negative is now returned without caching. * studio: defer online embedding detection to the Hub, re-probe offline The local modules.json marker short-circuited is_embedding_model() even online, so a repo that dropped (or added) the marker since it was cached was judged by its stale local revision instead of the current remote one. Online now treats model_info() as authoritative and uses the cache marker only as an uncached fallback when the Hub is unreachable, so a transient failure never poisons the memo. Offline re-probes the marker on every call without consulting or populating the memo, so a model downloaded later in the session (or a cached online negative that predates the download) is detected. _embedding_marker_in_hf_cache() now treats an unreadable refs/main (a non-FileNotFoundError OSError) as a cache miss rather than scanning stale history -- only a genuinely missing ref enables the fallback scan. * studio: harden offline embedding detection against empty refs, offline flips, and cache casing - _embedding_marker_in_hf_cache: an existing-but-empty/whitespace refs/main (a partial write or in-progress truncate-and-rewrite) now reads as a cache miss (None) instead of falling through to scan stale snapshots; only a genuinely missing ref enables the historical scan. - is_embedding_model: while offline, retain a positive already confirmed online this session (model_info only ever memoizes Hub-derived results), so _hf_offline_if_dns_dead() flipping the process to offline mid-load can't downgrade a verified tag-only embedder to False. Cached negatives are still bypassed and re-probed. - resolve_cached_repo_casing + settings route: persist the embedding model in the casing its local HF cache dir uses. Validation accepts a case-insensitive cache hit, but an offline SentenceTransformer load resolves the cache by exact case, so storing the requested spelling (baai/bge-m3 vs models--BAAI--bge-m3) made the model fail to load on a case-sensitive filesystem. * studio: reuse the exact-match-first case resolver and preserve the default Replace the ad-hoc resolve_cached_repo_casing with the existing resolve_cached_repo_id_case, which already prefers the exact-case cache dir before any case variant and tie-breaks variants deterministically -- so an exact requested id is never rewritten to a differently cased directory just because iterdir() happened to yield it first. Skip the normalization entirely when the submitted model equals the default: rewriting its casing would make set_rag_embedding_model()'s exact-string default comparison treat it as a custom override, pinning it so later changes to the configured default stop taking effect. * studio: don't let a stale cache marker mask a permanent Hub error is_embedding_model's Hub-failure fallback consulted the local modules.json marker for ANY model_info() exception, so a permanent error -- a deleted repo, a gated repo without credentials, or a typo that matches stale cache casing -- could pass online validation on a stale marker instead of returning the documented 409, and the persisted model could then fail when the loader refreshes from the Hub. Classify permanent Hub errors (RepositoryNotFound, GatedRepo, RevisionNotFound, EntryNotFound) as False, matching the nearby GGUF/vision detectors, and reserve the cache fallback for transient/5xx failures. * studio: honor TRANSFORMERS_OFFLINE in the embedding preflight, skip casing for local paths - The embedding-model save reached the offline-aware is_embedding_model() only after two preflight helpers made direct huggingface_hub calls that honor just HF_HUB_OFFLINE: _st_module_subdirs() downloads modules.json and the security scan fetches Hub metadata twice. In a TRANSFORMERS_OFFLINE-only session those blocked on network timeouts before the offline return, so saving an already cached model stalled. Both now consult a canonical hf_env_offline() helper -- the download passes local_files_only, and the metadata-only security scan short-circuits to its documented fail-open instead of burning both timeouts. - Skip cache-casing normalization for local paths: a relative directory such as "org/model" is loaded from disk, so rewriting it to a case-insensitive HF cache collision ("Org/model") would stop resolving to that directory and be read as a Hub repo id instead. * studio: never skip the security scan on TRANSFORMERS_OFFLINE alone The previous commit skipped the Hub security scan whenever either offline flag was set, but huggingface_hub honors only HF_HUB_OFFLINE: under a TRANSFORMERS_OFFLINE-only session the later SentenceTransformer load still reaches the network, so the scan was being skipped while the repo's pickle could still be downloaded and deserialized -- waving through exactly what _guard_model_security exists to block. Split the flags: hf_hub_offline() (HF_HUB_OFFLINE, the only one that actually prevents a fetch) gates the security short-circuit, while hf_env_offline() (either flag, the user's intent) is used only where local-only behavior is forced explicitly. The SentenceTransformer load now passes local_files_only from that intent, so TRANSFORMERS_OFFLINE genuinely stops the loader fetching instead of merely being assumed to. * studio: short-circuit the security preflight under either offline flag With the loader now pinned to the local cache by local_files_only = hf_env_offline(), a TRANSFORMERS_OFFLINE-only session can no longer fetch anything -- yet the preflight still fell through to two model_info() attempts on 10s and 20s timeouts, stalling every save and load of an already-cached embedder for half a minute before failing open anyway. Skip the metadata-only scan whenever either flag is set. The scan's job is to stop a poisoned pickle being downloaded and deserialized, and nothing can be downloaded under that predicate; the residual case -- a model cached BEFORE it was flagged -- is the same fail-open this function has always documented for an unavailable scan, and is exactly what HF_HUB_OFFLINE already did. That safety argument depends on every loader behind the gate honoring the same predicate, so it is pinned as a test invariant instead of a comment: removing local_files_only from the SentenceTransformer construction now fails the suite. Drops the short-lived hf_hub_offline() helper, which no longer has a caller. * studio: scope the offline scan bypass to callers that load local-only The previous commit put the offline short-circuit inside _fetch_security_status, which is the malware gate shared by every loader -- so TRANSFORMERS_OFFLINE=1 disabled it for all of them, while only the RAG embedder had been changed to pass local_files_only. MLX inference (core/inference/worker.py -> FastMLXModel .from_pretrained), training and export call from_pretrained with no local-only argument, and huggingface_hub ignores that flag, so those paths could still fetch and deserialize an unscanned model with the gate switched off. The bypass is now an explicit local_only_load argument, defaulting to False, and only the two RAG embedding callers -- whose loader is pinned to the local cache by the same predicate -- opt in. Tests pin both halves: the shared gate must still scan under either offline flag by default, and no other caller may pass local_only_load without constraining its loader. * studio: capture offline state once, and probe the ST cache root Two holes in the offline embedding path: - _get() read hf_env_offline() twice: once inside _guard_model_security and again for local_files_only. _hf_offline_if_dns_dead() mutates the process-wide offline vars and restores them on exit, so a concurrent load could see True in the guard -- skipping the Hub malware scan -- and False by the time the constructor ran, fetching and deserializing the unscanned repo and breaking the very invariant that licenses the bypass. The value is now read once in _get() and passed to both; _guard_model_security takes it as an argument instead of re-deriving it. - The cache probe searched only HF_HUB_CACHE. SentenceTransformer downloads into SENTENCE_TRANSFORMERS_HOME when that is set, using the same models--org--name/snapshots layout under a different root, so a model fully present there looked uncached and was rejected with a 409 offline even though the local-only loader could load it. Snapshot lookup now covers both roots. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: probe the cache the ST loader actually uses, and require it be loadable Adding SENTENCE_TRANSFORMERS_HOME to the shared snapshot iterator was too broad in one direction and too narrow in another: - _get() builds SentenceTransformer with no cache_folder, so with ST_HOME set it searches THAT root only, never the Hub cache. Probing the union let offline validation pass on a repo cached only in the Hub cache, after which the loader looked in ST_HOME and failed. The Sentence-Transformers probe now resolves to exactly one root: ST_HOME when set, the Hub cache otherwise. - The shared iterator is also used by the GGUF detectors, whose downloads go through hf_hub_download with no cache_dir and therefore really do use the Hub cache. It is back to Hub-cache-only so detection cannot pick a snapshot the GGUF load will not find. - Casing normalization ran through resolve_cached_repo_id_case, which scans the Hub cache, so with ST_HOME set the requested spelling was persisted unchanged and the exact-case offline load missed the differently cased directory that detection had just accepted. It now resolves against the same roots detection uses, exact match first. - A snapshot carrying only modules.json no longer counts as cached: the online security preflight downloads that single file itself, and a partial download leaves it behind, so validation passed for a snapshot with no weights and the first RAG load then failed. A hit now requires the marker plus a config and at least one weight file. * studio: thread the captured offline state into the module probe, fix the gate shard - _st_module_subdirs() re-read the process env for its local_files_only. With _hf_offline_if_dns_dead() flipping those vars from another thread, a load that captured local_only=False could still force this probe local-only, get () back because modules.json is not cached, and leave the scan with NO module load roots -- a Hub-flagged pickle under 0_Transformer/ would then pass as an unreferenced nested artifact while the loader fetched and deserialized it. It now takes the captured predicate as an argument, and the settings route reads the state once and uses that single value for both the probe and the scan. - Skip ST-cache casing on the llama-server backend. Nothing there loads through SentenceTransformer: the embedder derives a GGUF companion from the saved spelling and fetches it from the HUB cache, so normalizing to an ST_HOME spelling would point it at a repo _hf_gguf_backend_error() never validated (BAAI/bge-m3-GGUF instead of the checked baai/bge-m3-GGUF). - Fix the security-gate shard, which the signature change had broken: the direct _guard_model_security / _st_module_subdirs callers now pass the new argument (they were raising TypeError before reaching any assertion), and the casing tests patch utils.models.resolve_st_cached_repo_id_case, which the route actually calls, instead of the Hub-only resolver it no longer uses -- those patches were being silently ignored. * studio: accept only torch-loadable weights in the offline ST probe; fix re-export lint _snapshot_is_loadable_st_model accepted a cached snapshot whose only weights were .onnx (or .pt), but the RAG loader builds SentenceTransformer with the default torch backend, so such a snapshot passed offline validation and then failed on the first load, the exact validate-then-fail this helper exists to prevent. Restrict _ST_WEIGHT_SUFFIXES to .safetensors and .bin and add a regression test for an ONNX-only snapshot. Also teach scripts/verify_import_hoist.py that names listed in a module-level __all__ are uses, so the legitimately added resolve_st_cached_repo_id_case re-export in utils/models/__init__.py no longer trips HOISTED-IMPORT-UNUSED. Covered by two new self-test cases. * studio: probe the exact repo dir and revision an offline load resolves The cache probe modelled the cache loosely rather than modelling what SentenceTransformer actually does with local_files_only=True: - It merged snapshots across every case-variant repo dir and then read refs/main from whichever held the newest one. With both models--baai--bge-m3 and models--BAAI--bge-m3 present, a complete embedding snapshot in the directory the loader opens could be judged by a newer partial snapshot in the other, failing validation for a usable model. It now selects the ONE directory the loader opens, by the same exact-case-first rule resolve_st_cached_repo_id_case uses to choose the spelling that gets persisted. - It fell back to scanning historical snapshots when refs/main was absent. With local_files_only the default revision is resolved THROUGH that ref, so a snapshot directory alone is not discoverable: the settings request succeeded and the loader then failed at first indexing. A missing, empty or unreadable ref is now a cache miss, and the historical scan is gone. The tests exercise the real lookup against a built cache tree instead of patching the snapshot iterator, so they now cover the directory selection and ref resolution the loader depends on. * studio: record refs/main in the ONNX-only probe test The ONNX-only regression test predates the refs/main requirement, so after that change it returned None (a cache miss for want of a ref) before ever reaching the weight-format check it exists to make. Recording the ref restores its intent: the snapshot resolves, and the answer is False because an ONNX export is not loadable by the RAG loader's default Torch backend. * studio: recognize base-model weight files and gate the offline positive on a materialized snapshot _snapshot_is_loadable_st_model matched any .safetensors/.bin by suffix, so a partial cache carrying only a commonly published non-weight bin such as training_args.bin (or an adapter-only artifact) passed offline validation and then failed the local_files_only load at first indexing. Match recognized Torch base-model weight filenames (model / pytorch_model, including sharded) by name. is_embedding_model retained an online-confirmed positive offline even when no files were cached, so a metadata-only /check-embedding result let an uncached repo be saved and then fail at first indexing. Retain the positive only when the active revision is materialized locally, which still covers a downloaded tag-only embedder whose snapshot carries no modules.json. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: require a complete weight set offline and persist embedder verdicts across restarts Two follow-ups to the offline embedding-model classifier: - _snapshot_is_loadable_st_model now requires a COMPLETE Torch base-model weight set in one snapshot directory, not just any single recognized weight file. A partially downloaded sharded model (model-00001-of-00002 without its sibling) no longer passes offline validation and then fails at first indexing under local_files_only. Weight files are grouped by directory and a directory counts only when it holds a single model.safetensors / pytorch_model.bin or a full shard set whose indices cover 1..total. - Online-confirmed embedder verdicts are now recorded under the resolved Studio home (embedding_verdicts.json). The session memo is lost on exit, so a downloaded tag-only feature-extraction embedder (snapshot present but no modules.json) was misclassified as non-embedding the first offline call after a restart. The offline branch consults this durable allowlist in addition to the memo, still gated on the active revision being materialized on disk, so an uncached repo is never trusted. Writes are best-effort and only positive verdicts are stored. * studio: require complete weights (with shard index) and resolve default casing offline Follow-ups to the offline embedding-model classifier from the latest review: - Trust a recorded embedder verdict (session memo or persisted allowlist) offline only when the active snapshot carries a COMPLETE, loadable weight set, not merely that it is materialized. A partial download (config present, weights missing or an incomplete shard set) makes _embedding_marker_in_hf_cache read False rather than None, so the previous marker-is-not-None gate wrongly returned True and the local_files_only load then failed. Split out _snapshot_has_complete_weights (config plus complete weights, modules.json aside) and _active_snapshot_dir, and gate the known-embedder positive on the weight set. - Require a sharded checkpoint's index map (model.safetensors.index.json / pytorch_model.bin.index.json) in addition to every shard before accepting it: transformers discovers and wires shards through that index, so a complete shard set without it fails the local-only load. - Resolve the embedding model name to its exact cache casing in the RAG loader before constructing SentenceTransformer. The settings route persists that spelling for a custom override but deliberately leaves the configured default verbatim, so a default whose casing differs from the cache dir would miss it and fail offline. Resolving at load time covers the default too; a no-op for a local path or when nothing case-matching is cached, and idempotent for an already-normalized override. Adds regression tests for the partial-snapshot verdict, the missing shard index, and the loader casing resolution; updates the offline-invariant source assertion to the resolved-name variable. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: require a tokenizer, case-fold verdict ids, and serialize verdict writes Three follow-ups to the offline embedding-model classifier from the latest review: - _snapshot_has_complete_weights now also requires a tokenizer asset. A SentenceTransformer Transformer module builds an AutoTokenizer, so a snapshot with a complete weight set but no tokenizer.json / tokenizer_config.json / vocab still fails the local_files_only load. The check is a permissive union over the common fast-tokenizer, config, and WordPiece/BPE/SentencePiece assets, so an unusual but valid layout is not rejected -- only a genuinely tokenizer-less partial download. - The persisted embedder allowlist is now keyed case-insensitively. model_info() is queried under the requested casing while the settings route saves the cache-resolved casing, so an exact-string lookup missed the persisted positive after a restart (baai/model recorded, BAAI/model looked up) and a loadable tag-only embedder was rejected. Both persist and lookup case-fold the id. - _persist_embedder serializes its read-modify-write under a lock and writes through a per-thread temp file, so concurrent confirmations of different embedders no longer drop each other's entry or collide on the temp path. Cross-process writers stay best-effort (os.replace is atomic; a dropped verdict is only an optimization miss a later online re-confirmation heals). Adds regression tests for the missing-tokenizer reject, alternate tokenizer assets, cross-casing verdict match, and concurrent verdict writes; updates the snapshot test helpers to materialize a tokenizer alongside config and weights. * studio: tighten comments in the offline embedding-model classifier Comment-only pass over the PR's changed files. Collapse the long block comments and docstrings around is_embedding_model, the cache-snapshot and weight-completeness helpers, the embedder-verdict persistence, the offline security gate, and the offline/casing tests to short one- or two-line forms. Preserve the rationale (issue #6817, the local_files_only invariant, the casing and weight-gate reasons) in far fewer words. No code changes. * studio: drop redundant comments in the offline embedding-model classifier Second comment-reduction pass over the offline embedding-model cache work: delete comments and trailing notes that restate the adjacent code or an assertion, and trim the remaining docstrings and rationale comments to their load-bearing invariants. Comments and docstrings only; no code changes. * studio: pin embedder verdicts to a revision, canonicalize default aliases - A persisted verdict recorded that the Hub tagged ONE revision an embedder, but was stored per repo. Once refs/main advanced to a complete but non-embedding Transformer snapshot, the offline path still returned True: the settings route accepted the updated model without force and RAG could silently load it as an embedder. Verdicts now carry the commit they were confirmed at and are trusted only while the active revision matches. One confirmed before the repo was cached has no revision to compare, so the first revision observed afterwards is pinned then -- which is what lets a later advance be caught. The persisted file gains a {id: commit} form and still reads the previous list format. - tokenizer_config.json no longer counts as a tokenizer asset. It only DESCRIBES a tokenizer, so a snapshot with config, weights and just that file passed validation and then failed AutoTokenizer.from_pretrained(local_files_only=True) at first indexing for common BERT/GPT-style models. - A casing-only alias of the default is canonicalized to the default up front. Repo ids are case-insensitive but every gate here compares exact strings, so saving "Unsloth/bge-m3" against a default of "unsloth/bge-m3" ran the verification and scan for a custom model and then persisted an override -- after which later changes to the configured default stopped applying. - verify_import_hoist.py replays __all__ assignments in order instead of unioning them. Only the final value exports anything, so a later plain "=" that drops a name must leave its import counted as unused; "+=" still extends, and an unreadable rebind keeps the earlier names rather than flagging real re-exports. * studio: validate the real ST load root, and pin verdicts to the Hub revision Four ways the offline probe still disagreed with what the loader does: - Verdicts were pinned to the LOCAL refs/main, but model_info() describes the current HUB revision. With a stale cache the two differ, so an older snapshot nobody verified was allowlisted. The pin is now info.sha, taken from the ModelInfo that produced the positive. A verdict carrying no revision (a legacy entry) is no longer trusted at all -- trusting it meant pinning whatever happened to be cached, which is the same bug; the next online check re-records it properly. - config, tokenizer and weights had to exist somewhere in the snapshot, not together. modules.json can send SentenceTransformer at 0_Transformer/, which is loaded FROM that directory, so a cache with the config at the root and only 0_Transformer/model.safetensors passed and then failed the local-only load. Each directory is now checked as a complete load root, which covers both the plain HF layout and the ST module layout. - vocab.json and merges.txt counted independently, but BPE needs the pair unless a serialized tokenizer.json is present, so half a pair validated and then failed AutoTokenizer.from_pretrained(local_files_only=True). - A slashless short name like all-MiniLM-L6-v2 is a supported ST alias that the loader resolves through the sentence-transformers/ organization, so its snapshot is cached under that full id. Probing only the bare name reported a miss and 409'd a model that was cached and loadable; the bare id is still tried first, matching the loader's own order. * studio: fail closed for an offline security scan instead of failing open A local_only (offline) load cannot fetch Hugging Face's malware scan, and the previous behaviour skipped the scan and failed OPEN, so a cached repo with a poisoned pickle weight could deserialize under SentenceTransformer(local_files_only=True). Evaluate it fail-CLOSED against the cached files instead: block a base-model pickle weight the load would deserialize (pytorch_model.bin and its shards, in a directory with no safetensors alternative) and allow a pickle-free (safetensors / gguf are inert) cache. A cached pickle model must be reloaded online once to be scanned, or shipped as safetensors. Nothing cached is not a security event. _fetch_security_status no longer needs the local_only_load skip (the offline branch is handled in evaluate_file_security). Adds a regression test covering the safetensors-allow and pickle-block paths with no Hub call. * studio: only suppress an offline pickle when a loadable safetensors weight exists The offline security gate treated any .safetensors in a directory as covering a pickle weight, so a cache with pytorch_model.bin beside a bare adapter_model.safetensors (or an orphan shard with no index) passed the fail-closed check even though from_pretrained still selects and deserializes the pickle. Require a genuinely loadable safetensors weight -- an unsharded base file or a complete indexed shard set -- before treating the pickle as covered. Also make the import-hoist analyzer preserve uncertainty when __all__ is extended by a value it cannot read statically (__all__ += dynamic()), matching how it already handles an unreadable rebind, so a dynamically-supplied re-export is not flagged HOISTED-IMPORT-UNUSED. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: scope the offline pickle scan to load paths; reset __all__ opacity on rebind Address three review follow-ups on the offline security gate and the import-hoist analyzer: - The offline pickle scan walked the whole snapshot, so a stray pickle in a non-load subdirectory (archive/, nemo/) that SentenceTransformer never deserializes was blocked. Scope it to real from_pretrained load roots -- the snapshot root, or a subdir that holds its own config.json -- matching the online scan's load-path scoping. - _collect_dunder_all kept a sticky opaque flag: a readable replacing assignment after an unreadable extend (__all__ += dynamic(); __all__ = []) still credited every import, so a genuinely unused hoist went unreported. A replacing assignment now resets opacity. - A bare __all__: list[str] annotation has no runtime value; it was treated as an unreadable assignment and marked the export set opaque. Skip annotation-only declarations. * studio: recase slashless ST aliases and accept a pinned embedder after a transient failure Two offline-detection gaps on well-formed input: - resolve_st_cached_repo_id_case bailed on every slashless name, so a differently-cased short alias (all-minilm-l6-v2) validated case-insensitively but was loaded verbatim; the SentenceTransformer loader rewrites it to sentence-transformers/all-minilm-l6-v2 and looks it up case-sensitively, missing the canonical sentence-transformers/all-MiniLM-L6-v2 cache dir. Resolve through _st_cache_repo_dir, which follows the same org alias, and hand back the on-disk casing. - On a transient (non-permanent) Hub failure, is_embedding_model only accepted a cached modules.json marker, so a downloaded tag-only embedder (no modules.json) with a verdict pinned to the active revision was rejected even though the offline branch accepts the identical cache. Mirror the offline branch's pinned-verdict acceptance. * studio: scan modules.json-declared module roots in the offline pickle gate The offline pickle scan treated only the snapshot root and config.json-bearing subdirs as load roots, so a pickle in a non-Transformer SentenceTransformer module directory that has no config.json (e.g. a 0_WordEmbeddings/ module: wordembedding_config.json + pytorch_model.bin) was skipped even though the loader deserializes it. Parse modules.json (and thread through load_subdirs) to treat every declared module directory as a load root, so such a pickle is scanned and fail-closed offline. * studio: classify cached non-Transformer SentenceTransformer models offline _snapshot_has_complete_weights recognized only a Transformer-shaped load root (config + tokenizer + weights co-located), so a fully-cached model built from a non-Transformer module (0_WordEmbeddings uses wordembedding_config.json + embedding weights and its own tokenizer, no HF config.json; BoW keeps its vocab in config.json) was classified non-embedding offline and the settings endpoint returned 409. Add _snapshot_modules_all_loadable, which parses modules.json and accepts a snapshot when every declared module's path directory carries the files that module class's own load() reads (a Transformer/root module still needs the full HF load root; a WordEmbeddings module needs its config plus a complete weight set; other modules need their *_config.json), and at least one embedding-producing module is present. It is OR-ed after the Transformer check, so it only ever accepts more and cannot regress the existing path or reject a pruned cache. * studio: scan PEFT adapter pickle weights in the offline security gate from_pretrained auto-detects an adapter_config.json in the load root and deserializes the adapter weights on top of the base model, so adapter_model.bin is a separate pickle RCE vector that a safetensors base weight does not cover. The offline scan matched only base-model pickle names, so an offline local-only load with safetensors base weights plus a cached adapter_model.bin was allowed despite the live adapter pickle. Scan adapter pickles too, scoped to a load root where adapter_config.json is present and no adapter_model.safetensors exists. * studio: require weights for Dense/CNN/LSTM SentenceTransformer modules offline _module_dir_is_loadable accepted a Dense, CNN, or LSTM module dir with only its config, but those modules' load() hard-load model.safetensors else pytorch_model.bin (verified against sentence-transformers source: no fallback, raises if neither exists) -- exactly like WordEmbeddings. A cache with such a module's config but no weights would validate and then fail the local_files_only load. Require a complete weight set for every weighted module, not just WordEmbeddings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: scan root-index subdir pickle shards offline; handle __all__.append/.extend - The offline pickle scan followed only load-root directories, so a shard mapped by a root pytorch_model.bin.index.json into a non-root subdirectory was skipped even though from_pretrained follows the index weight_map and deserializes it (a layout an attacker can craft to evade the scanner). Read the local index and scan its referenced pickle shards, covered by a loadable base safetensors at the index root -- mirroring the online scan. - The import-hoist analyzer ignored __all__.append("X") / __all__.extend([...]) runtime re-export mutators, so an import added solely for one tripped HOISTED-IMPORT-UNUSED. Read their string args like +=, and treat any other __all__ method call as opaque. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: classify StaticEmbedding offline, require WordEmbeddings tokenizer, bound model_info - A StaticEmbedding module (e.g. sentence-transformers/static-retrieval-mrl-en-v1's 0_StaticEmbedding/) holds tokenizer.json + weights and NO config, so the config-gated non-Transformer path 409'd it offline. Recognize it by what StaticEmbedding.load() reads: a tokenizer.json plus a complete Torch weight set. - WordEmbeddings.load() rebuilds its tokenizer via the configured tokenizer_class.load() from the module dir, so a WordEmbeddings module now also requires a tokenizer artifact (whitespacetokenizer_config.json / phrasetokenizer_config.json, or a shared HF tokenizer asset), not just its config + weights. - With neither offline env var set, an unbounded model_info() could hang on connect/DNS retries for networkless users (the #6817 symptom). Bound it with a 15s timeout so a dead network fails fast and the existing transient-failure cache fallback resolves a cached model, while a reachable Hub still wins. (Documented caveat: a stalled DNS getaddrinfo may exceed this.) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve indexed safetensors shards relative to their index _safetensors_index_complete compared shard basenames against the flat set of files in the index directory, so an index whose weight_map names shards in a subdirectory was treated as incomplete whenever a legacy pytorch_model.bin sat beside it. That falsely blocked a snapshot whose pickle weights are fully covered by a complete, loadable safetensors shard set. Resolve each shard path relative to the index directory instead, and add a regression test for the subdir-mapped shard case. * Restrict offline weight-completeness check to declared load roots _snapshot_has_complete_weights scanned every directory in a snapshot and accepted it when ANY directory was a complete Transformer load root. When modules.json is present a SentenceTransformer load only opens the declared module paths, so a snapshot whose declared modules are incomplete but which happens to contain an unrelated complete directory was accepted offline and then failed at the first local_files_only load. Restrict the candidate directories to the roots a load actually opens: the snapshot root plus each modules.json module path. For a well-formed snapshot the verdict is unchanged; only a complete directory at an undeclared path no longer vouches for an otherwise-incomplete snapshot. * Scan SentenceTransformer Router child module weights offline A Router (legacy Asym) snapshot declares its child sub-modules only in router_config.json, not the top-level modules.json, and Router.load() deserializes each child's weights from its own subdir. A config.json-less child such as query_0_WordEmbeddings (wordembedding_config.json plus a pickle pytorch_model.bin loaded via torch.load) was therefore neither a modules.json-declared load root nor a config.json-bearing dir, so the offline gate skipped its pickle even though the loader deserializes it. Parse router_config.json at each load root and treat every declared child subdir as a load root (bounded BFS, so nested routers are covered), so those child pickles are scanned. Add Router regression tests: a pickle child blocks, a safetensors child is allowed, and a Router in a declared subfolder is followed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not treat an unreferenced config subdir as an offline load root The offline pickle gate skipped a directory only when it was neither a declared load root nor held a config.json. Because _st_load_roots already resolves every real load root (snapshot root, modules.json / load_subdirs dirs, Router children), the config.json fallback only ever promoted an UNREFERENCED subdir -- a nested checkpoint-500/ or archive/ that ships its own config.json + pytorch_model.bin -- to a load root. from_pretrained never descends into such a subdir and the online scan ignores the same unindexed pickle, so offline mode wrongly blocked a model the loader reads from a clean safetensors root. Scope the pickle to directory in roots only, and add a regression test (a stray checkpoint-500/ no longer blocks; a modules.json-declared module dir still does). * Classify a root Router (Asym) model as loadable offline _module_dir_is_loadable applied Transformer root requirements (config + tokenizer + weights) to every root module, so a Router saved at the snapshot root -- which carries only modules.json + router_config.json and loads its weights from child subdirs -- was classified not loadable offline, and is_embedding_model missed a cached Router embedder. Dispatch on the module class before the root Transformer fallback: a Router/Asym dir is loadable when router_config.json parses and every declared child subdir is loadable (validated recursively through _module_dir_is_loadable, so nested routers and every child type are covered) with at least one embedding-producing child. This also tightens a non-root Router, which previously validated on the mere presence of router_config.json without checking its children. Add Router regression tests (root and declared subfolder, complete and incomplete-child). * Require every declared module before accepting an offline cache _snapshot_is_loadable_st_model returned has_complete_weights OR modules_all_loadable, so a complete 0_Transformer short-circuited the or and vouched for the whole snapshot even when a declared sibling module was missing its serialized weights; SentenceTransformer builds every module in modules.json, so that snapshot passed offline validation and then failed the local-only load. When modules.json declares a non-empty list it is now authoritative (modules_all_loadable validates every declared module); has_complete_weights stays the fallback only for an empty/non-list modules.json (the plain from_pretrained root). Also add the weight-bearing modules whose load() hard-loads via load_torch_weights and previously fell to the config-only path -- LayerNorm, WeightedLayerPooling, SparseAutoEncoder -- to _ST_WEIGHTED_MODULE_NAMES, with source citations and the deliberate exclusions (Pooling/Normalize/BoW/WordWeights read no weights on load). Add parametrized regression tests over LayerNorm/WeightedLayerPooling/Dense (a weightless sibling rejects, a complete sibling accepts). * Reject self-referential Router children instead of recursing forever _router_dir_is_loadable validates each router_config.json child through _module_dir_is_loadable, which re-enters _router_dir_is_loadable for a Router child. A malformed types entry naming the router's own directory (a key of ".", which normalizes to the same dir) made that recursion never descend, so it looped until RecursionError -- breaking the documented never-raises contract and turning a crafted/corrupted cached model into a 500 from is_embedding_model instead of a graceful unverifiable result. A real child reference is a subdir and always resolves deeper, so reject any child whose resolved path is the router dir itself. Add a regression test (a router_config naming "." as a Router child returns False without raising). * Treat a destructuring __all__ assignment as opaque _collect_dunder_all detected __all__ only as a direct ast.Name assignment target, so a binding through a destructuring target (__all__, meta = [...], v -> an ast.Tuple) was skipped entirely, leaving an empty, non-opaque export set. A newly hoisted import re-exported only through that assignment was then falsely flagged HOISTED-IMPORT-UNUSED. Its value cannot be mapped statically, so mark the export set opaque when __all__ is reached only through a destructuring / item / attr target, matching how the collector already handles other unreadable __all__ forms. Add a self-test case. * Canonicalize declared module paths before scoping the offline pickle gate A repo could declare a traversing module path such as 0/../evil in modules.json (or a router_config child), which SentenceTransformer resolves to evil/ and deserializes evil/pytorch_model.bin. _st_load_roots recorded the raw snap/"0/../evil", which never equals the snap/evil that rglob yields, so the offline pickle gate skipped that directory and a malicious repo slipped a pickle past the newly added gate. Add _canonical_load_dir to collapse ./ and ../ components lexically and reject an upward escape, and route the modules.json paths, load_subdirs and router children through it so the gate scopes the same normalized directory the loader opens. Add regression tests for a traversing modules.json path and router child. * Close offline embedding-classification completeness gaps Five real offline misclassifications, each a false negative (the #6817 hang recurs) or false positive (accepted then 409s at the local_files_only load). Dispatch _module_dir_is_loadable on the module class before the root Transformer fallback. A module with save_in_root=True (every InputModule: WordEmbeddings, StaticEmbedding, SparseStaticEmbedding, Transformer, Router) is saved at the snapshot root, so a root WordEmbeddings was wrongly held to Transformer requirements (an HF tokenizer it never writes) and classified not loadable. CLIPModel is Transformer-shaped: CLIPModel.load() reads AutoModel weights plus AutoProcessor, so a config-only CLIP dir must not validate. SparseStaticEmbedding needs a tokenizer plus either idf.json or a complete torch weight set (conditionally weight-bearing); a config alone is not enough. A present but empty or malformed modules.json is not loadable and does not fall back to a root Transformer: with modules.json present the loader never takes the plain-Transformer path (base/model.py _load_config_modules). The tag-only no-modules.json embedder is classified separately via _snapshot_has_complete_weights. Validate a sharded weight index against its weight_map (every mapped shard present, resolved relative to the index dir) instead of trusting the index file's mere existence, mirroring the security-side check. Add regression tests for all five. * Close case-folding and online-traversal holes in the offline pickle gate Two gate bypasses where the security scan credited or scoped a path differently from what the loader actually resolves: The safetensors credit was case-folded. _cached_pickle_weight_files lowercases every filename, and the loadable-safetensors and adapter checks tested those folded keys against the exact-lowercase names. On a case-sensitive filesystem (Linux, the Studio default) a crafted repo shipping Model.SafeTensors plus a malicious pytorch_model.bin makes transformers and sentence-transformers miss the exact-name model.safetensors and deserialize the pickle, while the gate credited an inert safetensors and did not block. Credit safetensors case-sensitively against real filenames, and drop pytorch_model.safetensors from the credit set (transformers loads only model.safetensors, never that name). Pickle matching stays case-insensitive (over-blocking a mis-cased pickle the loader would not load is the safe direction). The online scan did not canonicalize traversing paths while the offline gate did. A repo-controlled modules.json path (threaded into the online scan via the RAG guard) or a weight_map shard entry like 0/../evil / ../evil was compared verbatim, so a flagged evil/pytorch_model.bin never matched and evaded the online scan though the loader resolves and deserializes it. Canonicalize the repo-controlled load-subdir prefixes and weight_map shards the same way the offline gate does, so offline and online agree. Add regression tests for both bypasses. * Treat a conditional __all__ mutation as opaque in the import-hoist linter _collect_dunder_all replayed only top-level module statements, so an __all__ assignment or mutation inside a module-level if / try / for / while / with / match (or a deeper scope) was ignored, leaving the export set understated. A newly hoisted import re-exported only through such a conditional __all__ was then falsely flagged HOISTED-IMPORT-UNUSED, blocking a valid change. A conditional value cannot be replayed statically, so mark the export set opaque when __all__ is bound or mutated anywhere other than a top-level statement. Add a self-test case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope Router child sub-modules as load roots in the online embedding scan The RAG embedding security guard unions the SentenceTransformer module dirs from modules.json into the load roots it scopes for the Hub scan, so a flagged pickle directly under a Transformer module blocks. A Router (legacy Asym) module declares its child sub-modules only in router_config.json, not in modules.json, and Router.load() deserializes each child from its own subdir. The online scan therefore dropped a flagged child pickle (for example query_0_WordEmbeddings/pytorch_model.bin) as an unreferenced nested shard while the loader still deserialized it, the counterpart to the offline gate which already expands router children via _router_child_dirs. _st_module_subdirs now reads router_config.json for any Router-typed module and adds each declared child (joined onto the module path, canonicalized so a traversing entry is dropped) to the load roots. The config is read only for a Router-typed module, so a plain embedder pays no extra fetch, and every failure path still returns () so the guard never bricks the embedder. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Allow a recorded-clean pickle embedder to load offline The offline embedding security gate is fail-closed: with no network to reach Hugging Face's scan, a cached pickle weight cannot be verified, so it is blocked and a model the user already downloaded and used online will not load offline. This adds a persistent cache of clean Hub verdicts so that exact content can load offline, without weakening the gate for an unknown or never-scanned pickle. When an embedding repo is loaded online and HF's scan returns a completed clean verdict, the load roots are hashed and recorded under the scanned commit as an exact map of snapshot-relative pickle name to sha256, in a per-user JSON store at studio_root()/security/embedding_scan_verdicts.json (atomic write, 0600, thread and cross-process locked, 30-day TTL). Offline, a cached pickle model loads only when the active cached commit and every load-root pickle's sha256 match the recorded verdict; a missing record, moved commit, changed or added pickle, expired record, or any error keeps blocking. Online loads always re-query the Hub and an authoritative unsafe verdict deletes any stale record, so a now-flagged commit cannot keep loading on an old clean record. The store binds repo id, full commit, and a per-file sha256 map so a locally swapped pickle at the same commit, a branch advance, or an added load-relevant pickle is detected. A same-user attacker who can rewrite the model cache or the store is outside the enforceable boundary and this is documented; the sha256 is computed just before load, so a narrow verify-to-load window remains, and a Hub scanner false negative is recorded faithfully (safetensors stays the stronger defense). Recording is triggered post-load in the RAG embedder because the settings route only validates and the pre-load guard runs before the constructor downloads; recording is skipped when the loaded commit differs from the scanned commit. The blocked-pickle enumerator now returns snapshot-relative Paths so two module dirs that ship the same pickle basename are hashed and reported distinctly. * Harden the embedding verdict cache against review findings Tighten the offline verdict cache and its enumeration so every uncertain or malformed input fails closed and the recorded hashes always match the files the loader reads: - Hash every case-colliding pickle in a load root, not one representative. On a case-sensitive filesystem pytorch_model.bin and PYTORCH_MODEL.BIN are distinct files; keying by lowered name dropped one and could hash a decoy instead of the loader's target. The enumerator now returns every variant Path. - Only persist a clean verdict for a COMPLETED, entirely-benign scan. Require scansDone to be the boolean True (not a truthy string), filesWithIssues to be a well-formed list, and every flagged file to be a definitively-safe level; a pending, error, unknown, or malformed entry no longer records as clean. The online block decision is unchanged. - Fail closed when the offline cache cannot be inspected: an rglob error now propagates and blocks instead of reading as pickle-free, and a snapshot that errors on resolution (vs a clean not-cached) blocks. The offline guard also raises instead of returning when its own inspection throws, so the constructor never deserializes an unverified cached pickle. - Expand online Router children recursively (bounded BFS with a seen set), mirroring the offline load-root expansion, so a flagged grandchild pickle is scoped online and cannot be recorded clean. - Reject absolute and drive/UNC declared paths in the load-root canonicalizers; the loader would resolve them outside the snapshot, so collapsing them to an in-snapshot relative dir scoped the wrong place. - Pin verdict recording to the scanned commit's snapshot and take the offline verify commit from the snapshot directory name, removing a second refs/main read and the skew it allowed. - Drop the now-unused pickle-name wrapper. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten offline embedding classification and the pickle gate Close a set of offline edge cases where validation accepted a cache the local_files_only load then rejects, and one gate bypass: - Credit a sharded model.safetensors.index.json for a pickle sibling only at a from_pretrained root. A non-Transformer SentenceTransformer module (Dense, WordEmbeddings, StaticEmbedding) loads via Module.load_torch_weights, which reads model.safetensors then pytorch_model.bin and never the index, so a sharded safetensors index in such a module dir must not vouch for its pytorch_model.bin. - Stop counting pytorch_model.safetensors as loadable in the offline classifier: the loader probes model.safetensors (then its index) or pytorch_model.bin, never pytorch_model.safetensors, matching the gate that already treats it as a decoy. - Treat a present but unreadable weight index as incomplete: transformers opens and parses any present index, so a malformed one or one without a weight_map fails the load rather than falling back to filename-numbered shards. - Require the CLIP image-processor config (preprocessor_config.json) for a CLIP module: CLIPModel.load builds a CLIPProcessor that needs it, so a tokenizer alone is not enough. - Require a SparseStaticEmbedding config to actually select idf.json (a path ending .json) or ship loadable weights; a bare idf.json the config does not name falls through to load_torch_weights and raises. - Do not use the tag-only recorded-verdict fallback when modules.json is present: with the file present the loader takes the modules.json path, so a present but empty or malformed manifest must not be validated as a plain root Transformer. - Import-hoist linter: only a module-level conditional mutation or a function that declares global __all__ makes the export set opaque; a __all__ bound as a local in a nested function or class no longer masks a genuinely unused hoisted import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope Router-child pickles to their deepest load root and gate the ST offline kwarg The online scan stripped the first matching load-subdir prefix from a flagged file, so a nested Router child pickle (0_Router/query_0_WordEmbeddings/pytorch_model.bin) matched the parent 0_Router root, looked like an unreferenced nested shard, and slipped the gate even though Router.load() deserializes that child directly. Match the deepest (longest) load subdir instead, so the child becomes root-level under its own load root and blocks. pyproject sets no lower bound on sentence-transformers and the local_files_only constructor arg is absent on older releases, so always forwarding it broke every embedder warm on those installs. Pass it only for an offline load; an online warm never forwards it and works as before, while the offline capability still requires a version that supports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject snapshot-escaping shard paths and credit Transformer submodule safetensors The offline pickle enumerator joined a weight-index weight_map value straight to the load root and followed it, so a repo-controlled index mapping "../.." into a sibling snapshot made an offline from_pretrained deserialize an out-of-snapshot pickle, and an online load would then hash and record that external file as the scanned commit's clean content. Reject any shard path that escapes the snapshot root and fail closed, mirroring the canonical-root check the online shard scan already applies. A complete model.safetensors.index.json was credited over a sibling pickle only at the snapshot root, but a Transformer module subdirectory (0_Transformer/) is loaded via AutoModel.from_pretrained, which honors that shard set and never reads the pickle. Credit the sharded index for Transformer-typed modules declared in modules.json so a cached model that ships both a sharded safetensors checkpoint and an unused PyTorch checkpoint is no longer falsely blocked offline. Non-Transformer modules (Dense, WordEmbeddings, StaticEmbedding) read a flat weight with no index and keep their pickle blocked. Limit the import-hoist verifier's global __all__ scan to the declaring function's own scope so a nested inner-scope local __all__ no longer marks the module export set opaque and mask an unused hoisted import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope Router children against the snapshot and mirror the ST alias rewrite Router.load resolves each child at Path(subfolder, model_id) relative to the Router dir, so a nested 1_Router with a "../evil" child points at evil/ inside the snapshot and the loader deserializes evil/pytorch_model.bin. The offline enumerator canonicalized the child against the Router dir alone and dropped anything with "..", so that pickle was never scanned and the gate reported the cache pickle-free. Canonicalize router children against the snapshot, retaining in-snapshot siblings as load roots and failing closed on a child that escapes the snapshot itself, matching the online scan which already joins the prefix before normalizing. The security gate resolved a slashless model id by probing the bare cache dir first, but the SentenceTransformer constructor rewrites a non-basic slashless name to sentence-transformers/ <name> and loads THAT snapshot (only the basic ORIGINAL_TRANSFORMER_MODELS load bare). With both models--<name> and models--sentence-transformers--<name> cached, the gate inspected the bare dir while the loader read the namespaced one, so a pickle there bypassed the local-only gate. Mirror the constructor: try the namespaced candidate first for non-basic slashless names. Add the same not (snapshot / modules.json).is_file() guard to the transient-Hub-failure tag-only fallback that the offline branch already carries, so a cache whose present manifest is empty or malformed is no longer reported as a loadable embedder. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten root shard credit, module-path escapes, and weight-set probe order Credit the sharded safetensors index at the snapshot ROOT only when the root is actually loaded through an AutoModel/from_pretrained path. A modules.json root module of a non-Transformer type (StaticEmbedding / WordEmbeddings / Dense) loads via load_torch_weights, which reads pytorch_model.bin and ignores the index, so crediting a root shard index there suppressed a live root pickle and let the offline gate report the cache pickle-free. Recognize the Transformer subclasses CLIPModel and MLMTransformer as index-honoring load roots (they load via from_pretrained), so a sharded-safetensors CLIP/MLM submodule with a legacy pytorch_model.bin sibling is no longer falsely blocked offline. Mirrors the classifier dispatch. Fail closed on an absolute or snapshot-escaping modules.json module path (or load_subdirs entry) instead of silently dropping it: SentenceTransformer resolves such a path outside the snapshot and would deserialize an external pytorch_model.bin the gate cannot scan. On the classifier side, walk the weight set in the exact from_pretrained probe order (model.safetensors, its index, pytorch_model.bin, its index) so a pickle behind a malformed safetensors index is no longer accepted as complete, and restrict shard names to the loader-probed stem/ext pairs so a decoy model-*.bin / pytorch_model-*.safetensors set is not treated as loadable. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore scripts/verify_import_hoist.py to main The offline embedding cache fix does not depend on the __all__ scope handling that had accumulated in this linter, so revert the file to its main version and keep the PR focused on the feature. The feature modules still pass the existing import hoist check unchanged. * Reuse a shared HF cache skeleton in the offline classification tests Extract _mk_repo and _activate helpers for the repeated snapshot cache setup that every per-type builder duplicated, and fold the two StaticEmbedding missing-asset cases into one parametrized test. Same 125 collected items, all still passing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reclassify embedding models from the cache on every offline call is_embedding_model consulted its process memo before the offline branch, so an online lookup that memoized True from tags (without caching any weights) was returned unchanged once the session went offline -- the studio flips HF_HUB_OFFLINE in-process on a dead DNS, and the ungated check-embedding route can populate the memo. Settings would then accept a repo the offline loader cannot open. Run the offline cache-marker reclassification ahead of the memo and never record it, so an offline verdict always reflects the local cache and a later cache materialization is not masked by a stale negative. Add regression tests. * Tighten comments on the offline embedding path Condense the offline-embedding helper docstrings and inline comments added in this PR to fewer, clearer lines, keeping the non-obvious security and offline rationale. Comments and docstrings only; no code change. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothai@gmail.com> |
||
|
oobabooga
|
1b3bce0530
|
Studio: validate Hugging Face tokens before use (#7261)
* Studio: validate Hugging Face tokens before use * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep token validation failures non-blocking * Studio: harden Hugging Face token preflight * Studio: make token validation effect lint-safe --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> |
||
|
Daniel Han
|
187144d4e7
|
Reduce and tighten code comments and docstrings repo-wide (#6095)
Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison. |
||
|
Daniel Han
|
8292e699e4
|
Studio: make code comments and docstrings more succinct (#6029)
Trim and tighten code comments and docstrings across studio/ Python. Comment-only: every changed file verified code-identical to main via AST/token comparison. |
||
|
Daniel Han
|
3ce187da02
|
Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079)
Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent. |
||
|
Daniel Han
|
8ccdf596aa
|
Studio: stop leaking internal exceptions to API clients; harden sandbox path (#6072)
* Studio: stop leaking internal exceptions to API clients; harden sandbox path
Security hardening for the FastAPI backend.
Error exposure (CodeQL py/stack-trace-exposure): many route handlers returned
raw caught-exception text to clients via HTTPException detail / response bodies,
which can leak internal filesystem paths and stack detail. Add shared helpers in
utils/utils.py (safe_error_detail, log_and_http_error) that log the full
exception server-side and return a generic message, and sweep the route layer
(inference, models, export, training, datasets, chat_history, providers,
mcp_servers, settings, data_recipe/{jobs,seed,validate,mcp}) to use them.
Intentionally user-facing validation messages, the existing _friendly_error SSE
paths, and upstream-service body passthrough (llama-server / OpenAI) are kept;
absolute server paths echoed in models.py browse/read errors are redacted.
Path injection (CodeQL py/path-injection): serve_sandbox_file already does
basename + realpath containment; add a strict filename allowlist
(^[A-Za-z0-9._-]{1,255}$) before the path is built as defense-in-depth and to
give the analyzer a clear sanitizer.
No behavior change beyond error-message text; status codes preserved.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: keep curated error messages, fix remaining load leak
- inference.py /load non-native path: redact str(e) instead of leaking it
(matched the native branch which already redacted).
- llama_extra_args validation: return the curated, path-redacted message
instead of the generic fallback so users see the offending flag.
- sandbox file serving: allowlist now forbids only separators/control chars
via fullmatch, so generated images like 'loss curve.png' render again
while traversal is still blocked by basename + extension + realpath.
- Add safe_curated_detail() for domain/validation exceptions whose message
is intentionally user-facing; apply it to data_recipe job/validate,
chat conflict, provider test, and MCP probe paths (these were collapsing
to 'An internal error occurred', and 'connection' even mis-mapped to an
upstream-service message). Generic Exception paths keep safe_error_detail.
- log_and_http_error: tolerate stdlib loggers (no structlog kwargs).
- delete_openai_container: log transport errors with exc_info like list/create.
- Drop helper/HTTPException imports this change left unused.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* log_and_http_error: log original error traceback on stdlib-logger fallback
* Tidy error-helper and sandbox comments for PR #6072
* Trim redundant comments in studio error-hardening routes for PR #6072
* Re-trigger CI now that unsloth-zoo #727 is merged (Core pulls zoo main)
* Address PR #6072 review feedback
- inference.py: keep the actionable NativePathLeaseError detail (path-redacted)
instead of collapsing it to the generic message, matching the other curated
validation paths in this file.
- utils.py: log via a single formatted log.error(exc_info=error) call that works
for structlog and stdlib loggers; drop the now-unneeded try/except helper.
- models.py: use Path.name instead of os.path.basename(str(current)).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
|
||
Roland Tannous
|
47654cb91c | Final cleanup | ||
|
Roland Tannous
|
a2baf80511 | Update license headers | ||
|
Roland Tannous
|
817f2e8dcc | feat: integrate structlog, configure workers for prod logging, and migrate print statements | ||
|
Roland Tannous
|
d882678fe4 | Add AGPL-3.0 SPDX headers to all source files | ||
|
Roland Tannous
|
63c583c54f | replace torch MPS with MLX | ||
|
Roland Tannous
|
59d5f24eb5 | integrate global hardware detection at lifespan entrypoint | ||
|
Roland Tannous
|
107bd2be4c | feat: add Apple Silicon (MPS) compatibility to backend utils + tests | ||
|
Roland Tannous
|
544d6944d1 | root studio folder |
Renamed from backend/utils/utils.py (Browse further)