mirror of
https://github.com/unslothai/unsloth.git
synced 2026-07-10 00:08:58 +00:00
7ecbf5a770
* Use UTF-8 for Python code-execution subprocess I/O Studio's code-execution tool already tells the child to emit UTF-8 (PYTHONIOENCODING=utf-8 in _build_safe_env), but _python_exec writes the temp script and decodes the subprocess pipe with the OS default codec. On Windows (cp1252), non-ASCII in model-written code or its output -- arrows, CJK, emoji -- raises UnicodeEncodeError / UnicodeDecodeError and breaks execution. Complete the UTF-8 wiring in core/inference/tools.py: - write the temp script with encoding="utf-8" - decode _python_exec stdout as utf-8, errors="replace" - set PYTHONIOENCODING=utf-8 in _build_bypass_env too (matches _build_safe_env, so the bypass path's child also emits utf-8) The child is python with PYTHONIOENCODING=utf-8, so it emits UTF-8 regardless of the console code page and the decode is always correct. Shell execution via cmd.exe has a separate console-code-page story and is left to a follow-up. Refs unslothai/unsloth#6489 * Scope Python exec UTF-8 env to Python tool * Make bash bypass test robust to a host-set PYTHONIOENCODING for PR #6548 Bypass mode preserves benign host env vars, so a host-set PYTHONIOENCODING was inherited into the bash bypass env and tripped the new assertion even though _bash_exec never adds it. Clear it in the test so the assertion checks _bash_exec, not the runner environment. --------- Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>
33 lines
1.2 KiB
Python
33 lines
1.2 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
|
|
|
|
"""_python_exec must round-trip non-ASCII output end to end.
|
|
|
|
Model-written code routinely contains non-ASCII (arrows, CJK, emoji). The temp
|
|
script and the child's stdout pipe both have to be UTF-8 or it crashes/garbles
|
|
on Windows, whose default codec is cp1252. Mirrors the report in
|
|
unslothai/unsloth#6489. The child is ``python`` with PYTHONIOENCODING=utf-8, so
|
|
it emits UTF-8 on every OS; this proves the round-trip on a UTF-8 host and
|
|
guards against a regression to the OS default codec.
|
|
"""
|
|
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
_BACKEND_ROOT = Path(__file__).resolve().parents[1]
|
|
if str(_BACKEND_ROOT) not in sys.path:
|
|
sys.path.insert(0, str(_BACKEND_ROOT))
|
|
|
|
from core.inference.tools import _python_exec
|
|
|
|
# Arrow, em-dash, accent, CJK, check mark, astral-plane emoji -- none encodable
|
|
# in cp1252, so the OS default codec would raise on write or read.
|
|
_UNICODE = "café — 数字 → ✓ 😀"
|
|
|
|
|
|
@pytest.mark.parametrize("disable_sandbox", [False, True])
|
|
def test_python_exec_round_trips_non_ascii(disable_sandbox):
|
|
out = _python_exec(f"print({_UNICODE!r})", disable_sandbox = disable_sandbox)
|
|
assert _UNICODE in out, repr(out)
|