unsloth/studio/backend/tests/test_native_tls.py
Luca Cesarano 16fac7b1e6
fix(studio): verify TLS against the OS trust store at runtime (corporate TLS-inspection proxies) (#8108)
* fix(studio): verify TLS against the OS trust store at runtime

On macOS/Windows behind a TLS-inspecting proxy (Cisco Umbrella, Zscaler,
...), every huggingface.co request from the Studio backend fails with
CERTIFICATE_VERIFY_FAILED: Python verifies against certifi's bundled
Mozilla roots and ignores the OS trust store where the proxy CA lives. A
terminal user can export SSL_CERT_FILE, but macOS .app / desktop-shortcut
launches never read shell profiles, so GUI-launched Studio has no
workaround.

Inject truststore into ssl at each network-touching entry point (server,
HF download worker, STT download worker, training worker) — injection is
per-process and does not survive spawns. Default-on for macOS/Windows,
opt-in on Linux, opt-out anywhere via UNSLOTH_STUDIO_NATIVE_TLS=0,
mirroring UV_NATIVE_TLS in install.sh (#6671), whose runtime counterpart
this is.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(studio): cover inference/export workers and the studio extra

Review follow-ups: the inference worker resolves remote LoRA bases and the
export worker pushes/loads from the Hub, both in fresh spawned interpreters
that missed the injection; and pyproject's studio extra must mirror
requirements/studio.txt (test_studio_extra_matches_requirements), so
truststore is added there too for `pip install "unsloth[studio]"`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(studio): propagate native TLS to remaining children, gate 3.9

Second review round: the data-recipe jobs worker and the transformers
-c config probe are further fresh interpreters with Hub/provider
traffic, so activate in the worker and inline the gated injection in
the probe script (it cannot import backend modules). uv/pip child
installers do their own TLS (rustls ignores in-process injection), so
activation now exports UV_SYSTEM_CERTS/UV_NATIVE_TLS, mirroring
install.sh. truststore requires Python >=3.10 while the project floor
is 3.9, so the dependency gets a python_version marker on both sides
of the requirements mirror; activate_native_tls() already falls back
to certifi where the import is unavailable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(studio): activate native TLS in diffusion trainer

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix uv TLS opt-out and remaining uncovered spawns for PR #8108

- native_tls: mirror one resolved value across UV_SYSTEM_CERTS/UV_NATIVE_TLS.
  uv takes either variable as an opt-in, so UV_SYSTEM_CERTS=0 alone was
  re-enabled by the legacy name defaulting to 1. Matches install.sh.
- native_tls: warn instead of debug when injection fails, and document that
  explicit CA bundles become additive rather than exclusive under truststore.
- model_config: activate in the _VISION_CHECK_SCRIPT child, which calls
  AutoConfig.from_pretrained and fails quiet, so VLM detection silently
  degraded behind a proxy.
- prebuilt_core: activate at import so the llama.cpp / whisper.cpp installers
  and the resolve-prebuilt probe verify their urllib downloads the same way.
- mlx_repair: forward the UV cert variables through the allowlisted child env,
  the one uv installer that did not inherit them.
- diffusion_training_service: activate inside run_without_native_path_secret
  so truststore imports after the lease secret is scrubbed, and scrub the
  parent env before the child starts, like the other orchestrators.
- tests: pin the uv mirror both ways, stop leaking the UV vars into the pytest
  process, assert the diffusion ordering, and add static coverage that every
  spawned entry point still activates.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Keep the vision probe's helper block intact for PR #8108

test_vision_subprocess_unions_sidecar_registry slices the embedded script at
its first 'try:' to exec the inlined helpers, so putting the TLS activation
above them left that slice empty. Move it below the helpers, still ahead of
AutoConfig.from_pretrained.

* Cut the native TLS log noise and pin the client-only limitation for PR #8108

Simulation findings on the branch:

- A failed injection logged a full traceback at warning level. On Python 3.9,
  where truststore is excluded by the dependency marker on purpose, that is one
  stack per worker start. Now one line carrying the reason.
- truststore's injection is client-side: an SSLContext built after it verifies a
  peer chain on every handshake, so it cannot serve TLS. Studio serves plain
  HTTP on loopback so nothing breaks today, but an in-process HTTPS listener
  added later would fail at handshake on the default-on platforms. Documented,
  and test_native_tls_entrypoints.py now fails if one appears.

* Tighten native TLS comments

* Cap truststore at the next minor

truststore is pre-1.0 and 0.x minors are where it has changed verification
behaviour (0.9 to 0.10), while fixes land as patches inside a minor (0.10.1,
0.10.3, 0.10.4). Since activation applies process-wide to every TLS connection,
a semantic change should get a human look rather than arrive on the next
install. 0.10.4 has been the current release for a year, so nothing is held
back today. Mirrored in both declarations.

* Generate the inline native TLS gate from native_tls.py

The python -c probes and prebuilt_core.py cannot import backend modules, so
each carried its own hand-written copy of the UNSLOTH_STUDIO_NATIVE_TLS gate.
Three copies of the same platform defaults drift silently: a child would keep
downloading against certifi while everything else used the OS store.

native_tls.inline_gate_source() now renders the gate from the same constants
native_tls_enabled() reads. The probes concatenate it, and prebuilt_core.py
keeps a paste that test_native_tls_entrypoints.py asserts verbatim against the
generated text.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Compare the prebuilt TLS gate by AST, not by text

ruff-format restyles the pasted gate in prebuilt_core.py (quote style, line
wrapping), so a substring assertion against inline_gate_source() fails the
moment the formatter runs, as it did on 668cd9f76. Compare the parsed
statements instead: still catches a real edit to the platform defaults, and
ignores formatting that cannot change behaviour.

* Vendor truststore instead of depending on it

truststore was a new third-party dependency for every Studio user, added to fix
CERTIFICATE_VERIFY_FAILED for the few behind a TLS-inspecting proxy. Nothing in
Unsloth required it before, directly or transitively.

It is MIT, six pure-Python files, no dependencies of its own, so vendor it at
studio/backend/vendor/ and drop both declarations. pip vendors it for the same
reason. utils/third_party_source.py is not an option here: it downloads over
urllib at first use, which behind the very proxy this fixes would itself fail to
verify. The copy has to exist before the network does.

Consumers append the vendor directory to sys.path and import the top-level name,
so a truststore the user installed themselves still wins. The directory is
deliberately not a package: a dotted import would load the same files under a
second module name, and each would wrap ssl again.

test_vendored_truststore.py pins the tree against the published 0.10.4 wheel and
scripts/sync_vendored_truststore.py re-syncs it, since vendored code is invisible
to the dependency scanners.

* Tighten native TLS comments

* Drop the truststore sync script and treat the copy as static

Nothing in Studio ever ran it: it only re-downloaded a wheel for a maintainer
bumping the pin, which implies a refresh story this repo does not have.

The vendored files are a static copy. truststore_manifest.json and
test_vendored_truststore.py stay, since they are what makes them static: they
pin the tree to the published 0.10.4 hashes and fail if anything edits it. A
version bump now means replacing the tree and updating the manifest in the same
commit, deliberately, which the README spells out.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <moonshotaisubstack@gmail.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
2026-08-09 01:36:48 -07:00

156 lines
4.9 KiB
Python

# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""Tests for the OS-trust-store TLS activation (utils/native_tls.py).
truststore is stubbed: these assert only Unsloth's seam -- the platform defaults,
the UNSLOTH_STUDIO_NATIVE_TLS tri-state, idempotency, and the fail-open-to-certifi
behaviour when truststore is unavailable. CPU-only, no network.
"""
from __future__ import annotations
import sys
import types as _types
from pathlib import Path
import pytest
_BACKEND_DIR = str(Path(__file__).resolve().parent.parent)
if _BACKEND_DIR not in sys.path:
sys.path.insert(0, _BACKEND_DIR)
from utils import native_tls
@pytest.fixture(autouse = True)
def _reset_activation(monkeypatch):
import os
monkeypatch.setattr(native_tls, "_activated", False)
for key in ("UNSLOTH_STUDIO_NATIVE_TLS", "UV_SYSTEM_CERTS", "UV_NATIVE_TLS"):
monkeypatch.delenv(key, raising = False)
yield
# monkeypatch cannot undo vars that were absent, so drop what setdefault added.
for key in ("UV_SYSTEM_CERTS", "UV_NATIVE_TLS"):
os.environ.pop(key, None)
def _fake_truststore(monkeypatch):
calls = []
fake = _types.ModuleType("truststore")
fake.inject_into_ssl = lambda: calls.append("inject")
monkeypatch.setitem(sys.modules, "truststore", fake)
return calls
@pytest.mark.parametrize(
("platform", "expected"),
[("darwin", True), ("win32", True), ("linux", False)],
)
def test_platform_defaults(monkeypatch, platform, expected):
monkeypatch.setattr(sys, "platform", platform)
assert native_tls.native_tls_enabled() is expected
@pytest.mark.parametrize("value", ["0", "false", "NO", " 0 "])
def test_env_opt_out_wins_on_default_on_platform(monkeypatch, value):
monkeypatch.setattr(sys, "platform", "darwin")
monkeypatch.setenv("UNSLOTH_STUDIO_NATIVE_TLS", value)
assert native_tls.native_tls_enabled() is False
@pytest.mark.parametrize("value", ["1", "true", "YES"])
def test_env_opt_in_wins_on_default_off_platform(monkeypatch, value):
monkeypatch.setattr(sys, "platform", "linux")
monkeypatch.setenv("UNSLOTH_STUDIO_NATIVE_TLS", value)
assert native_tls.native_tls_enabled() is True
def test_activate_injects_once(monkeypatch):
monkeypatch.setattr(sys, "platform", "darwin")
calls = _fake_truststore(monkeypatch)
assert native_tls.activate_native_tls() is True
assert native_tls.activate_native_tls() is True
assert calls == ["inject"]
def test_activate_exports_uv_native_tls(monkeypatch):
import os
monkeypatch.setattr(sys, "platform", "darwin")
_fake_truststore(monkeypatch)
assert native_tls.activate_native_tls() is True
assert os.environ["UV_SYSTEM_CERTS"] == "1"
assert os.environ["UV_NATIVE_TLS"] == "1"
def test_activate_keeps_explicit_uv_override(monkeypatch):
import os
monkeypatch.setattr(sys, "platform", "darwin")
monkeypatch.setenv("UV_SYSTEM_CERTS", "0")
_fake_truststore(monkeypatch)
assert native_tls.activate_native_tls() is True
assert os.environ["UV_SYSTEM_CERTS"] == "0"
# uv takes either var as an opt-in, so the legacy name must mirror the opt-out.
assert os.environ["UV_NATIVE_TLS"] == "0"
def test_activate_mirrors_legacy_uv_override(monkeypatch):
import os
monkeypatch.setattr(sys, "platform", "darwin")
monkeypatch.setenv("UV_NATIVE_TLS", "0")
_fake_truststore(monkeypatch)
assert native_tls.activate_native_tls() is True
assert os.environ["UV_NATIVE_TLS"] == "0"
assert os.environ["UV_SYSTEM_CERTS"] == "0"
def test_disabled_does_not_touch_uv_env(monkeypatch):
import os
monkeypatch.setattr(sys, "platform", "linux")
_fake_truststore(monkeypatch)
assert native_tls.activate_native_tls() is False
assert "UV_SYSTEM_CERTS" not in os.environ
assert "UV_NATIVE_TLS" not in os.environ
def test_activate_noop_when_disabled(monkeypatch):
monkeypatch.setattr(sys, "platform", "linux")
calls = _fake_truststore(monkeypatch)
assert native_tls.activate_native_tls() is False
assert calls == []
def test_activate_fails_open_without_truststore(monkeypatch):
monkeypatch.setattr(sys, "platform", "darwin")
# None in sys.modules makes `import truststore` raise ImportError.
monkeypatch.setitem(sys.modules, "truststore", None)
assert native_tls.activate_native_tls() is False
# A later call with truststore available recovers.
calls = _fake_truststore(monkeypatch)
assert native_tls.activate_native_tls() is True
assert calls == ["inject"]
def test_activate_fails_open_when_injection_raises(monkeypatch):
monkeypatch.setattr(sys, "platform", "win32")
fake = _types.ModuleType("truststore")
def _boom():
raise OSError("no cert store")
fake.inject_into_ssl = _boom
monkeypatch.setitem(sys.modules, "truststore", fake)
assert native_tls.activate_native_tls() is False