mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-24 00:04:14 +00:00
90644db9e8
* Studio: repair duplicate package metadata during updates * Studio: clarify duplicate metadata recovery * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: preserve metadata repair across install paths * Studio: complete duplicate metadata repair * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden duplicate metadata repair * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: reject unreadable package metadata * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: simplify metadata conflict handling * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: protect malformed metadata repair on Windows * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: verify editable foreign environments * Studio: reject versionless foreign metadata * Studio: repair malformed package metadata * Studio: do not read one site-packages twice on a lib64 build purelib hardcodes lib while platlib follows sys.platlibdir, and venv creates lib64 as a symlink to lib. On Fedora and SuSE the two schemes are therefore two names for one directory, and deduplicating the scan roots by string kept both. installed_versions() then returned every record twice, so metadata_conflict() was true for every installed package in a perfectly healthy environment: update would exit 1 telling the user to recreate the venv, and the repair could never converge because the duplication was an illusion. Deduplicate on the real path instead. * Studio: apply the same scan rules to the CLI's own metadata checks Two changes to _studio_deps, both keeping it consistent with install_manifest: _scan_paths and _venv_site_packages deduplicate their site-packages roots by real path, so a lib64 build does not report every installed package as duplicated, locally or when inspecting a foreign managed venv. installed_metadata_conflicts grouped on METADATA's Name and dropped records without one, so a nameless or non-UTF-8 record was invisible to it while installed_versions() and _distributions_in() both called the same directory a conflict. It now applies the dist-info filename fallback the manifest already uses. Such a record is marked unreadable rather than trusted, so damaged_installed_files keeps skipping it: pip cannot parse it either. * Studio: fetch the replacement before removing any metadata record The uninstall loop removes every record it finds, and the reinstall went through pip_install, which routes failure into run() and exits the process. So a failure between the two left the venv with no unsloth at all, took the entry point with it, and never reached the function's own return False. An unreachable index was one way in; --package with a name that is not on an index was certain to hit it, since such a package can never be reinstalled by bare name. Stage the replacement with pip download --no-deps first, install from that directory, and report a failed uninstall or install instead of exiting, so the caller's return 1 runs and the existing install is left alone when the replacement cannot be obtained. * Studio: probe for duplicate metadata in install.sh too install.ps1 and both setup scripts were moved onto installed_version_probe, but install.sh still reported the installed version through importlib.metadata.version(), which answers from whichever record the finder yields first. The same duplicated venv was therefore reported as a conflict on Windows and as an ordinary version on Linux and macOS. Both installers also fall back to the plain version lookup when studio.install_manifest cannot be imported, which is the case when --package installs something that does not ship studio/: that used to print the version and would otherwise have started warning that it could not be determined. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make a failed metadata repair non-destructive in both directions Two gaps in the staging I added, both reproduced. An unreadable record has to be out of the tree before pip runs, since a non-UTF-8 METADATA makes pip list, show and uninstall raise for the whole environment. It was deleted outright, so a package whose ONLY record was unreadable lost it and was left with files and no install, while the message said the existing install had been left in place. It is now moved to a quarantine directory and put back on any outcome short of a completed repair. pip download leaves an sdist when the source has no compatible wheel, and the install that follows runs --no-index, so its isolated build could not fetch setuptools and the package stayed uninstalled. Stage with pip wheel instead, which builds while the index is still reachable, and require a wheel to be present before treating the package as staged. * Studio: apply the repo's kwarg-spacing formatter to the metadata repair changes * Studio: stage overlay sources too, and put the payload back on a partial uninstall Three gaps, each reproduced. --local pulls unsloth-zoo from git, so an overlay is a network fetch just as much as an index install is, yet the overlay path skipped staging entirely. An unreachable GitHub therefore left unsloth-zoo uninstalled after the loop had already removed every record. Stage whichever source will be installed, and let a failed overlay fall back to the staged artifact, which was built from that same source so it never substitutes a release. The uninstall loop returned as soon as one uninstall failed, even though an earlier successful one had already deleted the package tree, leaving a surviving dist-info that claimed an installed core package whose files were gone. It now reinstalls from the staged replacement first. installed_metadata_conflicts treated a record with a name but no version as readable, so a single such record was silently trusted while install_manifest.metadata_conflict called the same directory inconsistent. Readability now requires both, and the message distinguishes an unreadable record from a duplicated one. * Studio: build the foreign venv fixture at the platform's interpreter path The cross-venv tests write bin/python, which _venv_site_packages only looks at on POSIX. On Windows the probe has nothing to run, so the fixture falls through to the glob fallback and the resolution under test never happens: both new cases failed there while passing on Linux and macOS. Found by running these files on a windows-latest runner. The repo's Windows matrix covers three parity files only, so nothing else would have caught it. * Repair duplicate core metadata again before writing the manifest The repair pass runs before the core packages are installed, so an upgrade that itself leaves a superseded .dist-info behind survived it. write_manifest then recorded a null version and the installer exited successfully while every later check rejected the environment. Run the same repair a second time immediately before write_manifest. It is a no-op when nothing is ambiguous. * Keep the metadata repair working under uv index and hash policies Three ways a hardened or privately mirrored machine could defeat the duplicate-metadata repair: - pip wheel obeys require-hashes exactly as pip install does, so staging a replacement was refused before anything was built and the repair aborted on the conflict it exists to remove. Confirmed on pip 26.2. _relaxed_pip_policy_env now covers wheel alongside install and download. - The staged wheel was reinstalled through uv when uv is the package manager, and UV_REQUIRE_HASHES rejects the unpinned name after the uninstall loop has removed every record, leaving the package gone. Both staged reinstalls force pip; the wheel is already built. - uv has no wheel subcommand, so staging runs pip, which reads none of UV_DEFAULT_INDEX, UV_INDEX, UV_INDEX_URL, UV_EXTRA_INDEX_URL or UV_FIND_LINKS. A private build could be uninstalled and replaced by a public PyPI wheel. Those are now translated into the pip equivalents, and only where pip has no setting of its own. * Keep uv index priority and upload cutoff when staging a repair wheel Staging runs pip because uv has no wheel subcommand, and the previous commit handed uv's indexes across as PIP_INDEX_URL plus PIP_EXTRA_INDEX_URL. That restores the reach but not the semantics. uv's default index-strategy is first-index: it stops at the first index carrying the package, which is what stops a public release shadowing a private one. pip pools its indexes and takes the highest version, so a newer public wheel could replace a private build. Offer each index alone instead, in uv's own priority order, and take the first that can build the wheel. An inherited extra index is cleared for those attempts so the candidates stay separated. UV_EXCLUDE_NEWER was ignored the same way. pip's --uploaded-prior-to is the same upload-time filter and takes the same date spellings, so the cutoff is passed through. It only exists from pip 25.3; on an older pip staging refuses rather than install a wheel the cutoff forbids, which leaves the existing installation intact. * Only advance past an index that confirmed the package is absent Staging offers each uv index alone, in uv's priority order. The loop treated every failure as absence, so a private index that was merely unreachable or refused the credentials advanced to the public fallback, which is the dependency confusion first-index exists to prevent. pip cannot be asked directly: measured on pip 26.2, a refused connection and an unresolvable host both print the same two no-match ERROR lines a genuine absence does, above a WARNING about retrying after a broken connection, and never the Could not fetch URL string. So a no-match is only believed when no transport, authentication or TLS marker appears beside it. Anything else stops the loop with the install still intact. Also honour UV_INDEX_STRATEGY: unsafe-best-match really does pool every index and take the best version, which is pip's own default, so those staging attempts are pooled too. first-index and unsafe-first-match both exhaust one index before the next, which is what the loop already does. * Ask uv which release and index to stage instead of reconstructing them Staging must run pip, because uv has no wheel subcommand. Reading uv's index configuration out of the environment and translating it cannot be made correct: uv also discovers uv.toml, pyproject [tool.uv] and a user config, honours UV_CONFIG_FILE, applies an implicit PyPI default, and resolves under an index-strategy pip has no equivalent for. A private index configured any of those ways was invisible, so the repair could uninstall a private build and reinstall the public package. Ask uv instead. uv pip compile --emit-index-annotation reports the exact index each package resolved from, under uv's own discovery, priority, strategy and upload cutoff, and pip is then pointed at that one index with that one version, for this interpreter. An unreachable higher-priority index fails the compile outright, which is the behaviour first-index exists to give, so the pip output markers that approximated it are gone. UV_OFFLINE forbids network access and pip has no offline mode, so the repair refuses rather than reaching the network. Plain pip is unchanged: it already reads its own configuration, so there is nothing to ask. * Stage a direct reference as written and replace pip's candidate sources Two gaps in the uv-resolved staging path. The overlay paths hand staging a git URL or a local checkout rather than a bare name. Such a requirement is its own provenance, since no index chose it, and uv appends the resolved commit to what it emits, so the name comparison could never match and every local duplicate unsloth-zoo repair aborted before it started. A direct reference is now staged as written, without asking uv. Replaying uv's answer also has to replace pip's candidate sources rather than add to them. An inherited PIP_NO_INDEX blocked the index uv picked, and an inherited extra index or find-links directory could satisfy the same version from a source uv never looked at, which is the provenance swap this path exists to stop. pip.conf can carry the same three settings, so it is dropped for that one command exactly as the pinned-index branch already does. * Keep pip's transport config and replay uv's artifact policy when staging Dropping pip.conf to remove its candidate sources took proxy, cert, client-cert and trusted-host with it, and those are how a private index is reached at all, so uv could resolve and pip would then fail to fetch, leaving the repair blocked. The environment overrides cannot do this alone: measured on pip 26.2, an empty PIP_EXTRA_INDEX_URL does NOT suppress an extra-index-url set in pip.conf, and pip contacts that index exactly as it does with the variable unset. So pip's own configuration is asked for with pip config list and written back minus the four source keys, which leaves the transport settings in force. A no-binary or only-binary rule was also ignored, so the repair could download a wheel under a no-binary rule or build an sdist under an only-binary one. --emit-build-options carries the uv.toml form through. Measured on uv 0.10.7, it does not surface the environment-variable spelling, so UV_NO_BINARY and UV_ONLY_BINARY are translated by hand, and only where pip has no setting of its own. * Keep index credentials and let a local checkout stage while offline uv emits every configured index with its credentials, but strips userinfo from the annotation that says which one answered. Taking the annotation at face value handed pip an unauthenticated URL for a private index, which answers 401 and aborts the repair, and reading only --index-url named the wrong index entirely when the credentialed one was configured as an extra. The annotation is now matched back to the emitted URL that carries the credentials, preferring the credentialed form when one index is emitted both ways. The offline refusal also ran before the direct-reference branch, so --local could not repair a duplicate unsloth record even though its replacement is a checkout on disk that needs no network. It now applies only to sources that are not already local. A git reference is still a network fetch and is still refused. * Remove the old payload when every metadata record is unreadable Quarantining the only record left pip nothing to uninstall, so the staged wheel was laid over the existing tree and any module the new release dropped stayed on disk and importable while the repair reported success. Give pip a parseable METADATA beside the intact RECORD instead, so it removes exactly the files that record lists. Confirmed against a real venv: with the METADATA corrupted pip show raises UnicodeDecodeError for the whole environment, and after the rewrite pip uninstalls the package and its entire payload. Without a usable RECORD this fails closed and asks for the environment to be recreated, since nothing then knows which files belong to the package. A find-links origin also no longer displaces the index. uv annotates a flat source with a file:// URL, which belongs in PIP_FIND_LINKS, already set from the emitted find-links lines; an sdist taken from a flat directory still needs the real index for its build backend. * Build a local checkout without the network when uv is offline The offline exemption for a local checkout let staging through, but pip builds it in an isolated environment and fetches the build backend for that, which UV_OFFLINE does not reach. Measured: an isolated build of a local project with no index reachable fails at installing build dependencies, and this repository pins setuptools and setuptools-scm exactly, so they would be fetched unless already cached. Either way the no-network policy was broken or the repair failed for a reason the user could not see. Offline local staging now builds against the interpreter's own backend and forbids the index outright, so no-network means no network. Giving up isolation is how the pinned build requirements stop being honoured, so it happens only in that case; every other staging keeps it. If the backend is not importable the build fails and the installation is left intact. * Pin the staged artifact by hash and carry uv's no-index policy Suppressing pip's other candidate sources turned out not to be achievable. Measured: with PIP_CONFIG_FILE pointed at a sanitised copy, a venv's own pip.conf is still read and its extra-index-url is still contacted, and --isolated does not suppress it either. So pip may always consult a source uv never considered. Ask uv for hashes instead and let pip enforce them. Whatever source pip reaches, the artifact has to be the one uv resolved, which is a stronger guarantee than removing sources and does not depend on suppressing any configuration. Measured: pip verifies the hashes even with PIP_REQUIRE_HASHES=0, and rejects an artifact that does not match. They are only accepted from a requirements file, written into the staging directory so it is removed with it. A flat source resolved with no index in play is also what a configured no-index looks like on the way out: uv emits the find-links entry and no index line. PIP_NO_INDEX is now carried in that case rather than cleared, which had handed pip back the default PyPI. * Apply every usable RECORD and commit each package as it is repaired Quarantining an unreadable record drops its RECORD on the floor. Where one sat beside a readable duplicate, the uninstall loop then removed only what the readable record claimed, so a module existing solely in the older release stayed on disk and importable while the repair reported success. Every unreadable record that has a usable RECORD is now made parseable so pip uninstalls it normally; only what cannot be rewritten is moved aside, and if that leaves nothing to uninstall the repair fails closed as before. The quarantine was also shared across both core packages. If the first was reinstalled and the second then failed, the rollback restored the first package's stale record on top of the install that had already replaced it, reintroducing the conflict and leaving an old RECORD describing a payload that was gone. Each package now has its own, and it is discarded as soon as that package is back in place, so rollback only ever touches the package being repaired. * Back up rewritten metadata and accept a direct-reference pin from uv The rewrite has to happen before staging, and staging can still fail. Overwriting the original METADATA in place left no way back, and what remained parsed, so the next run would see one readable record, decide nothing was wrong, and never attempt the payload repair still owed. The quarantine now keeps a copy of every file it rewrites and puts it back on any unsuccessful exit, byte for byte, so the conflict is still detected. A record whose backup fails is quarantined instead of rewritten. uv also emits a PEP 508 direct reference rather than an == pin when an override redirects a package to a path, repository or URL. The whole line was being compared as the name, so the requirement stayed empty and every repair under that policy aborted. The name is now parsed from either spelling while the reference itself is staged as written. * Drop an import left unused by a replaced test The repo lint blocks on an import the PR adds but does not use. shutil was needed by a staging test that a later round replaced. * Pin the scan boundary so a record outside the venv cannot fail it _installed_version answered from all of sys.path before this branch and now scans purelib/platlib only. Simulated against a real pip --user install: the old lookup reports the version, the new one reports nothing. Unreachable in production, since every caller runs against the managed venv and Studio never creates one with system site-packages, but nothing pinned the intent. * Cover the two ways a duplicate record actually appears pip renames the outgoing distribution to a ~ prefixed sibling during an upgrade and a kill mid-operation keeps both, which is the documented real-world source of this state and had no test. The opposite case had none either: a setuptools editable install resolves to a dist-info in site-packages plus an egg-info in the checkout, and scanning purelib and platlib rather than all of sys.path is what stops that reading as a conflict. * Tighten four comment blocks without dropping any of their reasoning * Make an unusable METADATA repairable whether it is corrupt or absent invalid_metadata_paths returns Path, so _rewrite_minimal_metadata raised AttributeError on rstrip for every record whose METADATA was present but unparseable, which is the case the rewrite exists to handle. The surrounding tests passed str and hid it. An absent METADATA took a second route to the same end: back_up reported failure, the rewrite was skipped and the record was quarantined instead, so the uninstall loop removed only what the readable record claimed and a module shipped solely by the older release stayed on disk and importable while the repair reported success. An absent file is now nothing to back up rather than a failure, and restore removes the synthetic METADATA instead of reinstating one that never existed. * Carry uv's keyring provider into the staging pip command The staging plan already replays uv's chosen index and its userinfo, but an index authenticated through the keyring CLI reached pip as a bare URL, so pip could not fetch what uv had just resolved and every repair on a private index aborted. Verified against the installed tools: uv 0.10.7 exposes UV_KEYRING_PROVIDER with disabled and subprocess, and pip 26.2.1 accepts both as PIP_KEYRING_PROVIDER, so the existing artifact-policy translation covers it unchanged. A uv.toml keyring-provider is not emitted and stays out of reach, same as the artifact policy. * Move pip's abandoned backup aside so the uninstall loop can converge An interrupted pip upgrade leaves the outgoing distribution renamed to a ~ prefixed sibling. Its METADATA still names the project, so installed_versions counts it as a duplicate, but pip calls the directory an invalid distribution and skips it, so uninstalling by package name can never consume it. Measured in a real venv: pip logged Ignoring invalid distribution ~impkg, the loop hit remaining >= record_count, and the repair returned False with both records still present, which would abort every future update on the commonest real cause of this state. The backup is now found by directory name and quarantined before the loop, so a failure to stage still restores it. Reported by review. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fail closed on an unrecorded stale record even beside a readable one _rewrite_minimal_metadata already refuses a record with no RECORD, because nothing then knows which files that release owned. The repair only honoured that when no other record survived. With one surviving, the count stayed nonzero, pip uninstalled just the readable record, and the quarantine was discarded on success, so whatever the older release owned alone stayed importable and the directory that was the evidence was deleted for good. Measured before the change: repair returned True, the stale module was still importable, and the stale dist-info was gone. Now the repair stops with the directory untouched, so a later run can still see the conflict. Four tests built their malformed record without a RECORD and asserted the repair continued, which is the behaviour above. They use an unversioned directory name instead, so the rewrite still fails and the quarantine path they exist to cover is unchanged. Reported by review. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse the repair for any record that cannot be made pip-uninstallable The previous commit only covered records with no RECORD. Any unrewritable record leaks the same way: quarantining it hides it, pip removes just the readable records, the quarantine is discarded once the reinstall succeeds, and whatever that release owned alone stays importable while the repair reports success and deletes the directory that was the evidence. The reachable route is a METADATA that exists but cannot be read, as an elevated install leaves root-owned. Measured in a real venv with the file unreadable: repair returned True, the stale module was still importable and its dist-info was gone. It now stops with the tree as found. An unreadable METADATA that can be rewritten still repairs, as does an ordinary duplicate. Quarantine-and-proceed had no remaining purpose once nothing runs after an unusable record, so it is gone and its three tests collapse into one that pins the invariant. Quarantine itself is still used for pip's ~ leftover, which is what the restore-on-staging-failure test now covers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the rollback's own metadata when the quarantine unwinds When the rewritten record has already been uninstalled and a later uninstall fails, _restore_from_staged reinstalls the package before the finally block runs quarantine.restore(). If the staged wheel recreates the same dist-info path, restore then either overwrote the wheel's valid METADATA with the original corrupt bytes or, where the original was absent, deleted a record pip had just written, leaving the core package malformed after a recovery whose whole purpose was to make it whole. Reproduced as a unit: the repair returned False and reading the resulting METADATA raised UnicodeDecodeError. A successful staged reinstall now drops the backed-up copies, since the wheel's metadata is authoritative. The moved entries are kept, because a record pip cannot consume still has to go back exactly as found. Reported by review. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a sole pip backup as a metadata conflict for PR #8515 * Mark a foreign venv's sole pip backup as a conflict for PR #8515 * Tighten the docstrings this PR adds without dropping their reasoning --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: danielhanchen <unslothshared@gmail.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| conftest.py | ||
| test_bitsandbytes_kernel_readiness.py | ||
| test_change_system_message.py | ||
| test_conftest_bitsandbytes_preimport.py | ||
| test_construct_chat_template_validation.py | ||
| test_cpo_processor_text_tokenizer.py | ||
| test_cross_platform_parity.py | ||
| test_dpo_vision_processor_passthrough.py | ||
| test_e2e_no_torch_sandbox.py | ||
| test_fast_language_model_text_only.py | ||
| test_fast_model_config_passthrough.py | ||
| test_fast_sentence_transformer_embedding_parity.py | ||
| test_fast_sentence_transformer_redirect_lifecycle.py | ||
| test_flash_attn_install_python_stack.py | ||
| test_get_chat_template_escaping.py | ||
| test_get_lora_parameters_bias_fp8_block_size.py | ||
| test_get_lora_parameters_fp8_block_size.py | ||
| test_gpu_init_ldconfig_guard.py | ||
| test_grpo_ddp_model_config.py | ||
| test_import_without_bitsandbytes.py | ||
| test_install_python_stack.py | ||
| test_install_uv_override_space.py | ||
| test_mlx_public_trainer_api.py | ||
| test_module_entry_point.py | ||
| test_no_torch_filtering.py | ||
| test_orpo_processor_text_tokenizer.py | ||
| test_pad_token_fix.py | ||
| test_patch_trl_rl_trainers_defensive.py | ||
| test_remove_special_tokens_no_bos.py | ||
| test_revision_forwarding.py | ||
| test_rl_config_pickling.py | ||
| test_studio_import_no_torch.py | ||
| test_studio_runtime_gate.py | ||
| test_to_sharegpt_optional_none.py | ||
| test_tokenizers_and_torch_constraint.py | ||
| test_torchcodec_torch_compat.py | ||
| test_unsloth_run_tool_policy_resolver.py | ||
| test_v100_fullft_precision.py | ||
| test_virustotal_scan.py | ||
| test_vision_lora_targeting.py | ||
| test_windows_arm64_python_choice.py | ||
| test_windows_git_gate.py | ||
| test_windows_installer_addtype_fallback.py | ||
| test_windows_installer_concurrency_guard.py | ||
| test_windows_no_torch_setup.py | ||
| test_windows_python_313_8_screen.py | ||
| test_windows_python_venv_hardening.py | ||
| test_windows_setup_output_encoding.py | ||
| test_windows_studio_update_launcher.py | ||
| test_windows_vcredist_download_tls.py | ||
| test_windows_xformers_installer.py | ||
| test_windows_xformers_wheel_match.py | ||