mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-24 00:04:14 +00:00
* fix(studio): verify TLS against the OS trust store at runtime On macOS/Windows behind a TLS-inspecting proxy (Cisco Umbrella, Zscaler, ...), every huggingface.co request from the Studio backend fails with CERTIFICATE_VERIFY_FAILED: Python verifies against certifi's bundled Mozilla roots and ignores the OS trust store where the proxy CA lives. A terminal user can export SSL_CERT_FILE, but macOS .app / desktop-shortcut launches never read shell profiles, so GUI-launched Studio has no workaround. Inject truststore into ssl at each network-touching entry point (server, HF download worker, STT download worker, training worker) — injection is per-process and does not survive spawns. Default-on for macOS/Windows, opt-in on Linux, opt-out anywhere via UNSLOTH_STUDIO_NATIVE_TLS=0, mirroring UV_NATIVE_TLS in install.sh (#6671), whose runtime counterpart this is. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(studio): cover inference/export workers and the studio extra Review follow-ups: the inference worker resolves remote LoRA bases and the export worker pushes/loads from the Hub, both in fresh spawned interpreters that missed the injection; and pyproject's studio extra must mirror requirements/studio.txt (test_studio_extra_matches_requirements), so truststore is added there too for `pip install "unsloth[studio]"`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(studio): propagate native TLS to remaining children, gate 3.9 Second review round: the data-recipe jobs worker and the transformers -c config probe are further fresh interpreters with Hub/provider traffic, so activate in the worker and inline the gated injection in the probe script (it cannot import backend modules). uv/pip child installers do their own TLS (rustls ignores in-process injection), so activation now exports UV_SYSTEM_CERTS/UV_NATIVE_TLS, mirroring install.sh. truststore requires Python >=3.10 while the project floor is 3.9, so the dependency gets a python_version marker on both sides of the requirements mirror; activate_native_tls() already falls back to certifi where the import is unavailable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(studio): activate native TLS in diffusion trainer * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix uv TLS opt-out and remaining uncovered spawns for PR #8108 - native_tls: mirror one resolved value across UV_SYSTEM_CERTS/UV_NATIVE_TLS. uv takes either variable as an opt-in, so UV_SYSTEM_CERTS=0 alone was re-enabled by the legacy name defaulting to 1. Matches install.sh. - native_tls: warn instead of debug when injection fails, and document that explicit CA bundles become additive rather than exclusive under truststore. - model_config: activate in the _VISION_CHECK_SCRIPT child, which calls AutoConfig.from_pretrained and fails quiet, so VLM detection silently degraded behind a proxy. - prebuilt_core: activate at import so the llama.cpp / whisper.cpp installers and the resolve-prebuilt probe verify their urllib downloads the same way. - mlx_repair: forward the UV cert variables through the allowlisted child env, the one uv installer that did not inherit them. - diffusion_training_service: activate inside run_without_native_path_secret so truststore imports after the lease secret is scrubbed, and scrub the parent env before the child starts, like the other orchestrators. - tests: pin the uv mirror both ways, stop leaking the UV vars into the pytest process, assert the diffusion ordering, and add static coverage that every spawned entry point still activates. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the vision probe's helper block intact for PR #8108 test_vision_subprocess_unions_sidecar_registry slices the embedded script at its first 'try:' to exec the inlined helpers, so putting the TLS activation above them left that slice empty. Move it below the helpers, still ahead of AutoConfig.from_pretrained. * Cut the native TLS log noise and pin the client-only limitation for PR #8108 Simulation findings on the branch: - A failed injection logged a full traceback at warning level. On Python 3.9, where truststore is excluded by the dependency marker on purpose, that is one stack per worker start. Now one line carrying the reason. - truststore's injection is client-side: an SSLContext built after it verifies a peer chain on every handshake, so it cannot serve TLS. Studio serves plain HTTP on loopback so nothing breaks today, but an in-process HTTPS listener added later would fail at handshake on the default-on platforms. Documented, and test_native_tls_entrypoints.py now fails if one appears. * Tighten native TLS comments * Cap truststore at the next minor truststore is pre-1.0 and 0.x minors are where it has changed verification behaviour (0.9 to 0.10), while fixes land as patches inside a minor (0.10.1, 0.10.3, 0.10.4). Since activation applies process-wide to every TLS connection, a semantic change should get a human look rather than arrive on the next install. 0.10.4 has been the current release for a year, so nothing is held back today. Mirrored in both declarations. * Generate the inline native TLS gate from native_tls.py The python -c probes and prebuilt_core.py cannot import backend modules, so each carried its own hand-written copy of the UNSLOTH_STUDIO_NATIVE_TLS gate. Three copies of the same platform defaults drift silently: a child would keep downloading against certifi while everything else used the OS store. native_tls.inline_gate_source() now renders the gate from the same constants native_tls_enabled() reads. The probes concatenate it, and prebuilt_core.py keeps a paste that test_native_tls_entrypoints.py asserts verbatim against the generated text. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Compare the prebuilt TLS gate by AST, not by text ruff-format restyles the pasted gate in prebuilt_core.py (quote style, line wrapping), so a substring assertion against inline_gate_source() fails the moment the formatter runs, as it did on 668cd9f76. Compare the parsed statements instead: still catches a real edit to the platform defaults, and ignores formatting that cannot change behaviour. * Vendor truststore instead of depending on it truststore was a new third-party dependency for every Studio user, added to fix CERTIFICATE_VERIFY_FAILED for the few behind a TLS-inspecting proxy. Nothing in Unsloth required it before, directly or transitively. It is MIT, six pure-Python files, no dependencies of its own, so vendor it at studio/backend/vendor/ and drop both declarations. pip vendors it for the same reason. utils/third_party_source.py is not an option here: it downloads over urllib at first use, which behind the very proxy this fixes would itself fail to verify. The copy has to exist before the network does. Consumers append the vendor directory to sys.path and import the top-level name, so a truststore the user installed themselves still wins. The directory is deliberately not a package: a dotted import would load the same files under a second module name, and each would wrap ssl again. test_vendored_truststore.py pins the tree against the published 0.10.4 wheel and scripts/sync_vendored_truststore.py re-syncs it, since vendored code is invisible to the dependency scanners. * Tighten native TLS comments * Drop the truststore sync script and treat the copy as static Nothing in Studio ever ran it: it only re-downloaded a wheel for a maintainer bumping the pin, which implies a refresh story this repo does not have. The vendored files are a static copy. truststore_manifest.json and test_vendored_truststore.py stay, since they are what makes them static: they pin the tree to the published 0.10.4 hashes and fail if anything edits it. A version bump now means replacing the tree and updating the manifest in the same commit, deliberately, which the README spells out. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <moonshotaisubstack@gmail.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>
132 lines
5.4 KiB
Python
132 lines
5.4 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""Verify TLS against the OS trust store (corporate TLS-inspection proxies).
|
|
|
|
Python's ``ssl`` trusts only certifi's roots, so behind a TLS-inspecting proxy
|
|
(Cisco Umbrella, Zscaler, Netskope) every huggingface.co request fails with
|
|
``CERTIFICATE_VERIFY_FAILED``: the proxy re-signs traffic with a corporate CA
|
|
that lives only in the OS store. A shell user can export ``SSL_CERT_FILE``, but
|
|
GUI launches (macOS ``.app``, desktop shortcuts) never read shell profiles.
|
|
|
|
``truststore.inject_into_ssl()`` makes ``ssl.SSLContext`` verify against the OS
|
|
store instead, the runtime counterpart of ``UV_NATIVE_TLS`` in install.sh.
|
|
Injection is process-wide but does not survive a spawn, so every
|
|
network-touching entry point calls :func:`activate_native_tls` before its first
|
|
TLS connection; the ``python -c`` probes and the standalone prebuilt installers
|
|
carry an inline copy of the gating because they cannot import backend modules.
|
|
|
|
truststore is vendored at ``backend/vendor/`` rather than depended on, so no
|
|
Studio user gains a package for a proxy they do not have; see the README there.
|
|
Every consumer appends that directory to ``sys.path`` and imports the top-level
|
|
name, which keeps a truststore the user installed themselves in front of ours.
|
|
|
|
Defaults mirror install.sh: on for macOS and Windows, opt-in on Linux via
|
|
``UNSLOTH_STUDIO_NATIVE_TLS=1`` (distro OpenSSL configurations vary), opt-out
|
|
anywhere with ``0``. Explicit ``SSL_CERT_FILE``/``REQUESTS_CA_BUNDLE`` keep
|
|
working, but become additive rather than exclusive, since truststore keeps the
|
|
OS anchors alongside them; ``0`` is the way back to a bundle being the only
|
|
trust root.
|
|
|
|
Client side only: the injected class verifies a peer chain on every handshake,
|
|
so an ``SSLContext`` built after activation cannot serve TLS. Studio serves
|
|
plain HTTP on loopback and ``test_native_tls_entrypoints.py`` keeps it that way;
|
|
a future in-process HTTPS listener needs ``truststore.SSLContext`` for outbound
|
|
connections instead of this process-wide injection.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
import os
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
_NATIVE_TLS_ENV = "UNSLOTH_STUDIO_NATIVE_TLS"
|
|
_DEFAULT_ON_PLATFORMS = ("darwin", "win32")
|
|
_TRUTHY = ("1", "true", "yes")
|
|
_FALSEY = ("0", "false", "no")
|
|
|
|
# Resolved from this file so it is right in a checkout and an installed wheel
|
|
# alike. Never build it from the cwd or a hardcoded "studio/backend".
|
|
_VENDOR_DIR = str(Path(__file__).resolve().parent.parent / "vendor")
|
|
|
|
_logger = logging.getLogger(__name__)
|
|
_activated = False
|
|
|
|
|
|
def native_tls_enabled() -> bool:
|
|
"""Resolve ``UNSLOTH_STUDIO_NATIVE_TLS`` against the platform default."""
|
|
flag = os.environ.get(_NATIVE_TLS_ENV, "").strip().lower()
|
|
if flag in _TRUTHY:
|
|
return True
|
|
if flag in _FALSEY:
|
|
return False
|
|
return sys.platform in _DEFAULT_ON_PLATFORMS
|
|
|
|
|
|
# Children that cannot import this module (the `python -c` probes,
|
|
# prebuilt_core.py) carry the gate as source; generating it from the same
|
|
# constants stops it drifting from native_tls_enabled(). The child supplies os,
|
|
# sys and _TRUSTSTORE_VENDOR itself, which is what keeps the gate identical
|
|
# everywhere despite each child locating the vendor directory differently.
|
|
_INLINE_GATE = """\
|
|
_flag = os.environ.get({env!r}, '').strip().lower()
|
|
if _flag in {truthy!r} or (_flag not in {falsey!r} and sys.platform in {platforms!r}):
|
|
try:
|
|
if _TRUSTSTORE_VENDOR not in sys.path:
|
|
sys.path.append(_TRUSTSTORE_VENDOR)
|
|
import truststore
|
|
truststore.inject_into_ssl()
|
|
except Exception:
|
|
pass
|
|
del _flag
|
|
"""
|
|
|
|
|
|
def vendor_dir() -> str:
|
|
"""Where the vendored truststore lives, for a child that must be told."""
|
|
return _VENDOR_DIR
|
|
|
|
|
|
def inline_gate_source() -> str:
|
|
"""The gate as executable source, for a child that cannot import this module.
|
|
|
|
The child must bind ``_TRUSTSTORE_VENDOR`` to the vendor directory first.
|
|
"""
|
|
return _INLINE_GATE.format(
|
|
env = _NATIVE_TLS_ENV,
|
|
truthy = _TRUTHY,
|
|
falsey = _FALSEY,
|
|
platforms = _DEFAULT_ON_PLATFORMS,
|
|
)
|
|
|
|
|
|
def activate_native_tls() -> bool:
|
|
"""Idempotently patch ``ssl`` to verify against the OS trust store.
|
|
|
|
Returns True when injection is active. Failure is non-fatal: falling back to
|
|
certifi is the pre-existing, strictly less permissive behaviour.
|
|
"""
|
|
global _activated
|
|
if _activated:
|
|
return True
|
|
if not native_tls_enabled():
|
|
return False
|
|
# uv's rustls ignores in-process injection (uv >= 0.11 reads UV_SYSTEM_CERTS,
|
|
# older reads UV_NATIVE_TLS). Mirror one value across both: uv takes either as
|
|
# an opt-in, so an opt-out in one spelling must carry to the other.
|
|
os.environ.setdefault("UV_SYSTEM_CERTS", os.environ.get("UV_NATIVE_TLS", "1"))
|
|
os.environ.setdefault("UV_NATIVE_TLS", os.environ["UV_SYSTEM_CERTS"])
|
|
# append, not insert(0): a user-installed truststore must win over the vendored copy.
|
|
if _VENDOR_DIR not in sys.path:
|
|
sys.path.append(_VENDOR_DIR)
|
|
try:
|
|
import truststore
|
|
truststore.inject_into_ssl()
|
|
except Exception as exc: # noqa: BLE001
|
|
# Warn, no traceback: a silent certifi fallback is what this exists to prevent.
|
|
_logger.warning("native TLS unavailable (%s); TLS keeps certifi defaults", exc)
|
|
return False
|
|
_activated = True
|
|
return True
|