mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-25 08:42:25 +00:00
* fix(studio): verify TLS against the OS trust store at runtime On macOS/Windows behind a TLS-inspecting proxy (Cisco Umbrella, Zscaler, ...), every huggingface.co request from the Studio backend fails with CERTIFICATE_VERIFY_FAILED: Python verifies against certifi's bundled Mozilla roots and ignores the OS trust store where the proxy CA lives. A terminal user can export SSL_CERT_FILE, but macOS .app / desktop-shortcut launches never read shell profiles, so GUI-launched Studio has no workaround. Inject truststore into ssl at each network-touching entry point (server, HF download worker, STT download worker, training worker) — injection is per-process and does not survive spawns. Default-on for macOS/Windows, opt-in on Linux, opt-out anywhere via UNSLOTH_STUDIO_NATIVE_TLS=0, mirroring UV_NATIVE_TLS in install.sh (#6671), whose runtime counterpart this is. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(studio): cover inference/export workers and the studio extra Review follow-ups: the inference worker resolves remote LoRA bases and the export worker pushes/loads from the Hub, both in fresh spawned interpreters that missed the injection; and pyproject's studio extra must mirror requirements/studio.txt (test_studio_extra_matches_requirements), so truststore is added there too for `pip install "unsloth[studio]"`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(studio): propagate native TLS to remaining children, gate 3.9 Second review round: the data-recipe jobs worker and the transformers -c config probe are further fresh interpreters with Hub/provider traffic, so activate in the worker and inline the gated injection in the probe script (it cannot import backend modules). uv/pip child installers do their own TLS (rustls ignores in-process injection), so activation now exports UV_SYSTEM_CERTS/UV_NATIVE_TLS, mirroring install.sh. truststore requires Python >=3.10 while the project floor is 3.9, so the dependency gets a python_version marker on both sides of the requirements mirror; activate_native_tls() already falls back to certifi where the import is unavailable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(studio): activate native TLS in diffusion trainer * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix uv TLS opt-out and remaining uncovered spawns for PR #8108 - native_tls: mirror one resolved value across UV_SYSTEM_CERTS/UV_NATIVE_TLS. uv takes either variable as an opt-in, so UV_SYSTEM_CERTS=0 alone was re-enabled by the legacy name defaulting to 1. Matches install.sh. - native_tls: warn instead of debug when injection fails, and document that explicit CA bundles become additive rather than exclusive under truststore. - model_config: activate in the _VISION_CHECK_SCRIPT child, which calls AutoConfig.from_pretrained and fails quiet, so VLM detection silently degraded behind a proxy. - prebuilt_core: activate at import so the llama.cpp / whisper.cpp installers and the resolve-prebuilt probe verify their urllib downloads the same way. - mlx_repair: forward the UV cert variables through the allowlisted child env, the one uv installer that did not inherit them. - diffusion_training_service: activate inside run_without_native_path_secret so truststore imports after the lease secret is scrubbed, and scrub the parent env before the child starts, like the other orchestrators. - tests: pin the uv mirror both ways, stop leaking the UV vars into the pytest process, assert the diffusion ordering, and add static coverage that every spawned entry point still activates. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the vision probe's helper block intact for PR #8108 test_vision_subprocess_unions_sidecar_registry slices the embedded script at its first 'try:' to exec the inlined helpers, so putting the TLS activation above them left that slice empty. Move it below the helpers, still ahead of AutoConfig.from_pretrained. * Cut the native TLS log noise and pin the client-only limitation for PR #8108 Simulation findings on the branch: - A failed injection logged a full traceback at warning level. On Python 3.9, where truststore is excluded by the dependency marker on purpose, that is one stack per worker start. Now one line carrying the reason. - truststore's injection is client-side: an SSLContext built after it verifies a peer chain on every handshake, so it cannot serve TLS. Studio serves plain HTTP on loopback so nothing breaks today, but an in-process HTTPS listener added later would fail at handshake on the default-on platforms. Documented, and test_native_tls_entrypoints.py now fails if one appears. * Tighten native TLS comments * Cap truststore at the next minor truststore is pre-1.0 and 0.x minors are where it has changed verification behaviour (0.9 to 0.10), while fixes land as patches inside a minor (0.10.1, 0.10.3, 0.10.4). Since activation applies process-wide to every TLS connection, a semantic change should get a human look rather than arrive on the next install. 0.10.4 has been the current release for a year, so nothing is held back today. Mirrored in both declarations. * Generate the inline native TLS gate from native_tls.py The python -c probes and prebuilt_core.py cannot import backend modules, so each carried its own hand-written copy of the UNSLOTH_STUDIO_NATIVE_TLS gate. Three copies of the same platform defaults drift silently: a child would keep downloading against certifi while everything else used the OS store. native_tls.inline_gate_source() now renders the gate from the same constants native_tls_enabled() reads. The probes concatenate it, and prebuilt_core.py keeps a paste that test_native_tls_entrypoints.py asserts verbatim against the generated text. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Compare the prebuilt TLS gate by AST, not by text ruff-format restyles the pasted gate in prebuilt_core.py (quote style, line wrapping), so a substring assertion against inline_gate_source() fails the moment the formatter runs, as it did on 668cd9f76. Compare the parsed statements instead: still catches a real edit to the platform defaults, and ignores formatting that cannot change behaviour. * Vendor truststore instead of depending on it truststore was a new third-party dependency for every Studio user, added to fix CERTIFICATE_VERIFY_FAILED for the few behind a TLS-inspecting proxy. Nothing in Unsloth required it before, directly or transitively. It is MIT, six pure-Python files, no dependencies of its own, so vendor it at studio/backend/vendor/ and drop both declarations. pip vendors it for the same reason. utils/third_party_source.py is not an option here: it downloads over urllib at first use, which behind the very proxy this fixes would itself fail to verify. The copy has to exist before the network does. Consumers append the vendor directory to sys.path and import the top-level name, so a truststore the user installed themselves still wins. The directory is deliberately not a package: a dotted import would load the same files under a second module name, and each would wrap ssl again. test_vendored_truststore.py pins the tree against the published 0.10.4 wheel and scripts/sync_vendored_truststore.py re-syncs it, since vendored code is invisible to the dependency scanners. * Tighten native TLS comments * Drop the truststore sync script and treat the copy as static Nothing in Studio ever ran it: it only re-downloaded a wheel for a maintainer bumping the pin, which implies a refresh story this repo does not have. The vendored files are a static copy. truststore_manifest.json and test_vendored_truststore.py stay, since they are what makes them static: they pin the tree to the published 0.10.4 hashes and fail if anything edits it. A version bump now means replacing the tree and updating the manifest in the same commit, deliberately, which the README spells out. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <moonshotaisubstack@gmail.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>
156 lines
4.9 KiB
Python
156 lines
4.9 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""Tests for the OS-trust-store TLS activation (utils/native_tls.py).
|
|
|
|
truststore is stubbed: these assert only Unsloth's seam -- the platform defaults,
|
|
the UNSLOTH_STUDIO_NATIVE_TLS tri-state, idempotency, and the fail-open-to-certifi
|
|
behaviour when truststore is unavailable. CPU-only, no network.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import sys
|
|
import types as _types
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
_BACKEND_DIR = str(Path(__file__).resolve().parent.parent)
|
|
if _BACKEND_DIR not in sys.path:
|
|
sys.path.insert(0, _BACKEND_DIR)
|
|
|
|
from utils import native_tls
|
|
|
|
|
|
@pytest.fixture(autouse = True)
|
|
def _reset_activation(monkeypatch):
|
|
import os
|
|
|
|
monkeypatch.setattr(native_tls, "_activated", False)
|
|
for key in ("UNSLOTH_STUDIO_NATIVE_TLS", "UV_SYSTEM_CERTS", "UV_NATIVE_TLS"):
|
|
monkeypatch.delenv(key, raising = False)
|
|
yield
|
|
# monkeypatch cannot undo vars that were absent, so drop what setdefault added.
|
|
for key in ("UV_SYSTEM_CERTS", "UV_NATIVE_TLS"):
|
|
os.environ.pop(key, None)
|
|
|
|
|
|
def _fake_truststore(monkeypatch):
|
|
calls = []
|
|
fake = _types.ModuleType("truststore")
|
|
fake.inject_into_ssl = lambda: calls.append("inject")
|
|
monkeypatch.setitem(sys.modules, "truststore", fake)
|
|
return calls
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("platform", "expected"),
|
|
[("darwin", True), ("win32", True), ("linux", False)],
|
|
)
|
|
def test_platform_defaults(monkeypatch, platform, expected):
|
|
monkeypatch.setattr(sys, "platform", platform)
|
|
assert native_tls.native_tls_enabled() is expected
|
|
|
|
|
|
@pytest.mark.parametrize("value", ["0", "false", "NO", " 0 "])
|
|
def test_env_opt_out_wins_on_default_on_platform(monkeypatch, value):
|
|
monkeypatch.setattr(sys, "platform", "darwin")
|
|
monkeypatch.setenv("UNSLOTH_STUDIO_NATIVE_TLS", value)
|
|
assert native_tls.native_tls_enabled() is False
|
|
|
|
|
|
@pytest.mark.parametrize("value", ["1", "true", "YES"])
|
|
def test_env_opt_in_wins_on_default_off_platform(monkeypatch, value):
|
|
monkeypatch.setattr(sys, "platform", "linux")
|
|
monkeypatch.setenv("UNSLOTH_STUDIO_NATIVE_TLS", value)
|
|
assert native_tls.native_tls_enabled() is True
|
|
|
|
|
|
def test_activate_injects_once(monkeypatch):
|
|
monkeypatch.setattr(sys, "platform", "darwin")
|
|
calls = _fake_truststore(monkeypatch)
|
|
|
|
assert native_tls.activate_native_tls() is True
|
|
assert native_tls.activate_native_tls() is True
|
|
assert calls == ["inject"]
|
|
|
|
|
|
def test_activate_exports_uv_native_tls(monkeypatch):
|
|
import os
|
|
|
|
monkeypatch.setattr(sys, "platform", "darwin")
|
|
_fake_truststore(monkeypatch)
|
|
|
|
assert native_tls.activate_native_tls() is True
|
|
assert os.environ["UV_SYSTEM_CERTS"] == "1"
|
|
assert os.environ["UV_NATIVE_TLS"] == "1"
|
|
|
|
|
|
def test_activate_keeps_explicit_uv_override(monkeypatch):
|
|
import os
|
|
|
|
monkeypatch.setattr(sys, "platform", "darwin")
|
|
monkeypatch.setenv("UV_SYSTEM_CERTS", "0")
|
|
_fake_truststore(monkeypatch)
|
|
|
|
assert native_tls.activate_native_tls() is True
|
|
assert os.environ["UV_SYSTEM_CERTS"] == "0"
|
|
# uv takes either var as an opt-in, so the legacy name must mirror the opt-out.
|
|
assert os.environ["UV_NATIVE_TLS"] == "0"
|
|
|
|
|
|
def test_activate_mirrors_legacy_uv_override(monkeypatch):
|
|
import os
|
|
|
|
monkeypatch.setattr(sys, "platform", "darwin")
|
|
monkeypatch.setenv("UV_NATIVE_TLS", "0")
|
|
_fake_truststore(monkeypatch)
|
|
|
|
assert native_tls.activate_native_tls() is True
|
|
assert os.environ["UV_NATIVE_TLS"] == "0"
|
|
assert os.environ["UV_SYSTEM_CERTS"] == "0"
|
|
|
|
|
|
def test_disabled_does_not_touch_uv_env(monkeypatch):
|
|
import os
|
|
|
|
monkeypatch.setattr(sys, "platform", "linux")
|
|
_fake_truststore(monkeypatch)
|
|
|
|
assert native_tls.activate_native_tls() is False
|
|
assert "UV_SYSTEM_CERTS" not in os.environ
|
|
assert "UV_NATIVE_TLS" not in os.environ
|
|
|
|
|
|
def test_activate_noop_when_disabled(monkeypatch):
|
|
monkeypatch.setattr(sys, "platform", "linux")
|
|
calls = _fake_truststore(monkeypatch)
|
|
|
|
assert native_tls.activate_native_tls() is False
|
|
assert calls == []
|
|
|
|
|
|
def test_activate_fails_open_without_truststore(monkeypatch):
|
|
monkeypatch.setattr(sys, "platform", "darwin")
|
|
# None in sys.modules makes `import truststore` raise ImportError.
|
|
monkeypatch.setitem(sys.modules, "truststore", None)
|
|
|
|
assert native_tls.activate_native_tls() is False
|
|
# A later call with truststore available recovers.
|
|
calls = _fake_truststore(monkeypatch)
|
|
assert native_tls.activate_native_tls() is True
|
|
assert calls == ["inject"]
|
|
|
|
|
|
def test_activate_fails_open_when_injection_raises(monkeypatch):
|
|
monkeypatch.setattr(sys, "platform", "win32")
|
|
fake = _types.ModuleType("truststore")
|
|
|
|
def _boom():
|
|
raise OSError("no cert store")
|
|
|
|
fake.inject_into_ssl = _boom
|
|
monkeypatch.setitem(sys.modules, "truststore", fake)
|
|
|
|
assert native_tls.activate_native_tls() is False
|