unsloth/studio/backend/utils/native_tls.py
Luca Cesarano 16fac7b1e6
fix(studio): verify TLS against the OS trust store at runtime (corporate TLS-inspection proxies) (#8108)
* fix(studio): verify TLS against the OS trust store at runtime

On macOS/Windows behind a TLS-inspecting proxy (Cisco Umbrella, Zscaler,
...), every huggingface.co request from the Studio backend fails with
CERTIFICATE_VERIFY_FAILED: Python verifies against certifi's bundled
Mozilla roots and ignores the OS trust store where the proxy CA lives. A
terminal user can export SSL_CERT_FILE, but macOS .app / desktop-shortcut
launches never read shell profiles, so GUI-launched Studio has no
workaround.

Inject truststore into ssl at each network-touching entry point (server,
HF download worker, STT download worker, training worker) — injection is
per-process and does not survive spawns. Default-on for macOS/Windows,
opt-in on Linux, opt-out anywhere via UNSLOTH_STUDIO_NATIVE_TLS=0,
mirroring UV_NATIVE_TLS in install.sh (#6671), whose runtime counterpart
this is.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(studio): cover inference/export workers and the studio extra

Review follow-ups: the inference worker resolves remote LoRA bases and the
export worker pushes/loads from the Hub, both in fresh spawned interpreters
that missed the injection; and pyproject's studio extra must mirror
requirements/studio.txt (test_studio_extra_matches_requirements), so
truststore is added there too for `pip install "unsloth[studio]"`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(studio): propagate native TLS to remaining children, gate 3.9

Second review round: the data-recipe jobs worker and the transformers
-c config probe are further fresh interpreters with Hub/provider
traffic, so activate in the worker and inline the gated injection in
the probe script (it cannot import backend modules). uv/pip child
installers do their own TLS (rustls ignores in-process injection), so
activation now exports UV_SYSTEM_CERTS/UV_NATIVE_TLS, mirroring
install.sh. truststore requires Python >=3.10 while the project floor
is 3.9, so the dependency gets a python_version marker on both sides
of the requirements mirror; activate_native_tls() already falls back
to certifi where the import is unavailable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(studio): activate native TLS in diffusion trainer

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix uv TLS opt-out and remaining uncovered spawns for PR #8108

- native_tls: mirror one resolved value across UV_SYSTEM_CERTS/UV_NATIVE_TLS.
  uv takes either variable as an opt-in, so UV_SYSTEM_CERTS=0 alone was
  re-enabled by the legacy name defaulting to 1. Matches install.sh.
- native_tls: warn instead of debug when injection fails, and document that
  explicit CA bundles become additive rather than exclusive under truststore.
- model_config: activate in the _VISION_CHECK_SCRIPT child, which calls
  AutoConfig.from_pretrained and fails quiet, so VLM detection silently
  degraded behind a proxy.
- prebuilt_core: activate at import so the llama.cpp / whisper.cpp installers
  and the resolve-prebuilt probe verify their urllib downloads the same way.
- mlx_repair: forward the UV cert variables through the allowlisted child env,
  the one uv installer that did not inherit them.
- diffusion_training_service: activate inside run_without_native_path_secret
  so truststore imports after the lease secret is scrubbed, and scrub the
  parent env before the child starts, like the other orchestrators.
- tests: pin the uv mirror both ways, stop leaking the UV vars into the pytest
  process, assert the diffusion ordering, and add static coverage that every
  spawned entry point still activates.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Keep the vision probe's helper block intact for PR #8108

test_vision_subprocess_unions_sidecar_registry slices the embedded script at
its first 'try:' to exec the inlined helpers, so putting the TLS activation
above them left that slice empty. Move it below the helpers, still ahead of
AutoConfig.from_pretrained.

* Cut the native TLS log noise and pin the client-only limitation for PR #8108

Simulation findings on the branch:

- A failed injection logged a full traceback at warning level. On Python 3.9,
  where truststore is excluded by the dependency marker on purpose, that is one
  stack per worker start. Now one line carrying the reason.
- truststore's injection is client-side: an SSLContext built after it verifies a
  peer chain on every handshake, so it cannot serve TLS. Studio serves plain
  HTTP on loopback so nothing breaks today, but an in-process HTTPS listener
  added later would fail at handshake on the default-on platforms. Documented,
  and test_native_tls_entrypoints.py now fails if one appears.

* Tighten native TLS comments

* Cap truststore at the next minor

truststore is pre-1.0 and 0.x minors are where it has changed verification
behaviour (0.9 to 0.10), while fixes land as patches inside a minor (0.10.1,
0.10.3, 0.10.4). Since activation applies process-wide to every TLS connection,
a semantic change should get a human look rather than arrive on the next
install. 0.10.4 has been the current release for a year, so nothing is held
back today. Mirrored in both declarations.

* Generate the inline native TLS gate from native_tls.py

The python -c probes and prebuilt_core.py cannot import backend modules, so
each carried its own hand-written copy of the UNSLOTH_STUDIO_NATIVE_TLS gate.
Three copies of the same platform defaults drift silently: a child would keep
downloading against certifi while everything else used the OS store.

native_tls.inline_gate_source() now renders the gate from the same constants
native_tls_enabled() reads. The probes concatenate it, and prebuilt_core.py
keeps a paste that test_native_tls_entrypoints.py asserts verbatim against the
generated text.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Compare the prebuilt TLS gate by AST, not by text

ruff-format restyles the pasted gate in prebuilt_core.py (quote style, line
wrapping), so a substring assertion against inline_gate_source() fails the
moment the formatter runs, as it did on 668cd9f76. Compare the parsed
statements instead: still catches a real edit to the platform defaults, and
ignores formatting that cannot change behaviour.

* Vendor truststore instead of depending on it

truststore was a new third-party dependency for every Studio user, added to fix
CERTIFICATE_VERIFY_FAILED for the few behind a TLS-inspecting proxy. Nothing in
Unsloth required it before, directly or transitively.

It is MIT, six pure-Python files, no dependencies of its own, so vendor it at
studio/backend/vendor/ and drop both declarations. pip vendors it for the same
reason. utils/third_party_source.py is not an option here: it downloads over
urllib at first use, which behind the very proxy this fixes would itself fail to
verify. The copy has to exist before the network does.

Consumers append the vendor directory to sys.path and import the top-level name,
so a truststore the user installed themselves still wins. The directory is
deliberately not a package: a dotted import would load the same files under a
second module name, and each would wrap ssl again.

test_vendored_truststore.py pins the tree against the published 0.10.4 wheel and
scripts/sync_vendored_truststore.py re-syncs it, since vendored code is invisible
to the dependency scanners.

* Tighten native TLS comments

* Drop the truststore sync script and treat the copy as static

Nothing in Studio ever ran it: it only re-downloaded a wheel for a maintainer
bumping the pin, which implies a refresh story this repo does not have.

The vendored files are a static copy. truststore_manifest.json and
test_vendored_truststore.py stay, since they are what makes them static: they
pin the tree to the published 0.10.4 hashes and fail if anything edits it. A
version bump now means replacing the tree and updating the manifest in the same
commit, deliberately, which the README spells out.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <moonshotaisubstack@gmail.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
2026-08-09 01:36:48 -07:00

132 lines
5.4 KiB
Python

# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""Verify TLS against the OS trust store (corporate TLS-inspection proxies).
Python's ``ssl`` trusts only certifi's roots, so behind a TLS-inspecting proxy
(Cisco Umbrella, Zscaler, Netskope) every huggingface.co request fails with
``CERTIFICATE_VERIFY_FAILED``: the proxy re-signs traffic with a corporate CA
that lives only in the OS store. A shell user can export ``SSL_CERT_FILE``, but
GUI launches (macOS ``.app``, desktop shortcuts) never read shell profiles.
``truststore.inject_into_ssl()`` makes ``ssl.SSLContext`` verify against the OS
store instead, the runtime counterpart of ``UV_NATIVE_TLS`` in install.sh.
Injection is process-wide but does not survive a spawn, so every
network-touching entry point calls :func:`activate_native_tls` before its first
TLS connection; the ``python -c`` probes and the standalone prebuilt installers
carry an inline copy of the gating because they cannot import backend modules.
truststore is vendored at ``backend/vendor/`` rather than depended on, so no
Studio user gains a package for a proxy they do not have; see the README there.
Every consumer appends that directory to ``sys.path`` and imports the top-level
name, which keeps a truststore the user installed themselves in front of ours.
Defaults mirror install.sh: on for macOS and Windows, opt-in on Linux via
``UNSLOTH_STUDIO_NATIVE_TLS=1`` (distro OpenSSL configurations vary), opt-out
anywhere with ``0``. Explicit ``SSL_CERT_FILE``/``REQUESTS_CA_BUNDLE`` keep
working, but become additive rather than exclusive, since truststore keeps the
OS anchors alongside them; ``0`` is the way back to a bundle being the only
trust root.
Client side only: the injected class verifies a peer chain on every handshake,
so an ``SSLContext`` built after activation cannot serve TLS. Studio serves
plain HTTP on loopback and ``test_native_tls_entrypoints.py`` keeps it that way;
a future in-process HTTPS listener needs ``truststore.SSLContext`` for outbound
connections instead of this process-wide injection.
"""
from __future__ import annotations
import logging
import os
import sys
from pathlib import Path
_NATIVE_TLS_ENV = "UNSLOTH_STUDIO_NATIVE_TLS"
_DEFAULT_ON_PLATFORMS = ("darwin", "win32")
_TRUTHY = ("1", "true", "yes")
_FALSEY = ("0", "false", "no")
# Resolved from this file so it is right in a checkout and an installed wheel
# alike. Never build it from the cwd or a hardcoded "studio/backend".
_VENDOR_DIR = str(Path(__file__).resolve().parent.parent / "vendor")
_logger = logging.getLogger(__name__)
_activated = False
def native_tls_enabled() -> bool:
"""Resolve ``UNSLOTH_STUDIO_NATIVE_TLS`` against the platform default."""
flag = os.environ.get(_NATIVE_TLS_ENV, "").strip().lower()
if flag in _TRUTHY:
return True
if flag in _FALSEY:
return False
return sys.platform in _DEFAULT_ON_PLATFORMS
# Children that cannot import this module (the `python -c` probes,
# prebuilt_core.py) carry the gate as source; generating it from the same
# constants stops it drifting from native_tls_enabled(). The child supplies os,
# sys and _TRUSTSTORE_VENDOR itself, which is what keeps the gate identical
# everywhere despite each child locating the vendor directory differently.
_INLINE_GATE = """\
_flag = os.environ.get({env!r}, '').strip().lower()
if _flag in {truthy!r} or (_flag not in {falsey!r} and sys.platform in {platforms!r}):
try:
if _TRUSTSTORE_VENDOR not in sys.path:
sys.path.append(_TRUSTSTORE_VENDOR)
import truststore
truststore.inject_into_ssl()
except Exception:
pass
del _flag
"""
def vendor_dir() -> str:
"""Where the vendored truststore lives, for a child that must be told."""
return _VENDOR_DIR
def inline_gate_source() -> str:
"""The gate as executable source, for a child that cannot import this module.
The child must bind ``_TRUSTSTORE_VENDOR`` to the vendor directory first.
"""
return _INLINE_GATE.format(
env = _NATIVE_TLS_ENV,
truthy = _TRUTHY,
falsey = _FALSEY,
platforms = _DEFAULT_ON_PLATFORMS,
)
def activate_native_tls() -> bool:
"""Idempotently patch ``ssl`` to verify against the OS trust store.
Returns True when injection is active. Failure is non-fatal: falling back to
certifi is the pre-existing, strictly less permissive behaviour.
"""
global _activated
if _activated:
return True
if not native_tls_enabled():
return False
# uv's rustls ignores in-process injection (uv >= 0.11 reads UV_SYSTEM_CERTS,
# older reads UV_NATIVE_TLS). Mirror one value across both: uv takes either as
# an opt-in, so an opt-out in one spelling must carry to the other.
os.environ.setdefault("UV_SYSTEM_CERTS", os.environ.get("UV_NATIVE_TLS", "1"))
os.environ.setdefault("UV_NATIVE_TLS", os.environ["UV_SYSTEM_CERTS"])
# append, not insert(0): a user-installed truststore must win over the vendored copy.
if _VENDOR_DIR not in sys.path:
sys.path.append(_VENDOR_DIR)
try:
import truststore
truststore.inject_into_ssl()
except Exception as exc: # noqa: BLE001
# Warn, no traceback: a silent certifi fallback is what this exists to prevent.
_logger.warning("native TLS unavailable (%s); TLS keeps certifi defaults", exc)
return False
_activated = True
return True