unsloth/tests/python/test_unsloth_run_tool_policy_resolver.py
Daniel Han 098a6a0957
Studio: honor a request's enable_tools: false instead of overriding it (#8547)
* Studio: honor a request's enable_tools: false instead of overriding it

The process-wide tool policy was an override, not a default. unsloth studio run
installed set_tool_policy(True) at startup, and _effective_enable_tools returned
that value whenever it was non-None, so the request's own enable_tools field was
never read.

The Studio UI sends its tool pills as an explicit request field, and expresses
'every pill off' by omitting enable_tools entirely. Against a True override that
omission read as 'tools on', and with enabled_tools also absent the route
selected ALL_TOOLS, so a chat with every tool switched off still advertised
web_search, python, terminal and render_html. Thread-title generation, which
posts to /v1/chat/completions with no tool fields, picked them up the same way.
Only unsloth studio run installed the policy, so unsloth studio, the desktop app
and Colab behaved correctly and the two commands disagreed on the same UI.

Split the policy into two slots. The override still comes from an explicit
--enable-tools/--disable-tools and still beats the request. The new default is
what an omitted enable_tools falls back to, installed as True by every launcher,
so tools stay on for every bind including --secure. A request that says
enable_tools: false now turns them off.

Frontend sends the off state explicitly rather than by omission, in the local
chat, external provider and token-count paths, and pins enable_tools: false on
title generation so a 24-token summarisation never carries tool schemas.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: do not let the tools-on default answer a request that stated its intent

The safetensors/MLX path resolves _sf_tools_on straight from
_effective_enable_tools, without the two withdrawals the GGUF router applies. So
the launcher default introduced here reached requests that had already expressed
their own tool intent:

- tool_choice: "none" with enable_tools omitted resolved to tools on, and with
  no enabled_tools allowlist that selected every built-in, python and terminal
  included, turning a standard opt-out into server-side execution.
- A client tools catalog with enable_tools omitted made _sf_client_tools false,
  so the request left the client-tool passthrough for Unsloth's own loop and the
  caller got built-ins instead of calls for its own functions.

The GGUF router avoids both with _client_disabled_tool_calls and
_explicit_studio_tool_loop_requested. Draw the same line on the safetensors gate:
the default only answers a request that said nothing, so tool_choice: "none",
a client catalog, or tool-result history withdraws it. An explicit
enable_tools/mcp_enabled ask, and a CLI --enable-tools or --disable-tools, are
unchanged.

Also read the resolved _sf_tools_on in the _sf_client_tools gate rather than
recomputing _effective_enable_tools, which would have hidden the withdrawal.

* Studio: let a response_format contract withdraw the tools-on default too

_takes_tool_passthrough already ends with _extract_response_format(payload) is
not None, so on the GGUF router a structured-output request keeps the passthrough
and never enters the server tool loop. The safetensors withdrawal missed it, so a
request supplying response_format while omitting enable_tools still resolved
_sf_tools_on to true and could select and run every built-in.

response_format is not a declared field on ChatCompletionRequest; the model is
extra=allow and OpenAI-SDK clients spread extra_body at the top level, so read it
through _extract_response_format rather than an attribute.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: resolve the tool policy before protocol selection, and scope the default to unsloth studio run

Two fixes for the same root cause: the tools-on default reaching code built
around an omitted enable_tools meaning no tools.

_sf_server_tool_intent read the raw policy while the withdrawal ran ~40 lines
later, so a tool_choice: "none" or response_format request classified the
response protocol on the template's tool_use branch and then generated on the
plain one. On a model whose reasoning markers live only in the tool template the
extractor starts in the wrong mode and can return the answer as
reasoning_content. Resolve _sf_cli_policy / _sf_tools_on / _sf_mcp_allowed once,
above the classification, and derive the intent from the resolved value.

The default is also no longer installed by run_server. It belongs to
unsloth studio run, the launcher that has always forced tools on, and which
installs it itself. Installing it in _apply_cli_tool_policy extended it to
unsloth studio, the desktop app and Colab, where paths that assume an omitted
enable_tools means no tools started seeing it: n > 1 is rejected by the tool
loop though the plain path implements it, max_tool_calls_per_message: 0 still
advertises schemas and the nudge, and the pre-switch passthrough guard does not
recognise tool-result history so it 400s a non-streaming continuation. Those
paths predate this PR and are unchanged on unsloth studio run; scoping the
default keeps them that way everywhere else.

unsloth studio run still defaults tools on for every bind, --secure included,
and a request's enable_tools: false is still honored.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: correct the tool-policy help and docstrings for the scoped default

Scoping the tools-on default to unsloth studio run left three places claiming it
applies everywhere. The --help for plain unsloth studio and for a direct
run.py launch both said 'Default: on for every bind', which is now the opposite
of what those launchers do, and the tool_policy module said 'Launchers install
True'. A --help line about a tool-execution default is worth keeping exact.

unsloth studio run's own help is unchanged, since it is the launcher that does
default them on.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-08-12 06:26:58 -07:00

170 lines
4.8 KiB
Python

# Copyright 2025-present the Unsloth AI Inc. team. All rights reserved.
"""Truth-table tests for `resolve_tool_policy`: no flag installs no process-wide
OVERRIDE on any bind (loopback, --secure tunnel, raw network), so a request's own
`enable_tools: false` is honored -- tools still default on for a request that
omits the field, via the backend's separate tool-policy default. Explicit on/off
wins, and the resolver never prompts (yes/silent/prompt kept for compatibility)."""
import pytest
from unsloth_cli._tool_policy import is_external_host, resolve_tool_policy
def _never_prompt(_msg: str) -> bool:
raise AssertionError("resolve_tool_policy must not prompt")
class TestLocalhostHost:
@pytest.mark.parametrize("flag", [None, True, False])
def test_no_prompt(self, flag):
# localhost never prompts regardless of flag
result = resolve_tool_policy(
host = "127.0.0.1",
flag = flag,
yes = False,
silent = False,
prompt = _never_prompt,
)
assert result is flag
def test_default_is_unset(self):
assert (
resolve_tool_policy(
host = "127.0.0.1",
flag = None,
yes = False,
silent = False,
prompt = _never_prompt,
)
is None
)
def test_explicit_off(self):
assert (
resolve_tool_policy(
host = "127.0.0.1",
flag = False,
yes = False,
silent = False,
prompt = _never_prompt,
)
is False
)
class TestZeroHost:
def test_default_is_unset(self):
# A network bind installs no override, so the Studio UI's tool pills (which
# send enable_tools: false when all off) are honored rather than overridden.
assert (
resolve_tool_policy(
host = "0.0.0.0",
flag = None,
yes = False,
silent = False,
prompt = _never_prompt,
)
is None
)
def test_explicit_off_no_prompt(self):
assert (
resolve_tool_policy(
host = "0.0.0.0",
flag = False,
yes = False,
silent = False,
prompt = _never_prompt,
)
is False
)
def test_explicit_on_no_prompt(self):
assert (
resolve_tool_policy(
host = "0.0.0.0",
flag = True,
yes = False,
silent = False,
prompt = _never_prompt,
)
is True
)
def test_yes_and_silent_accepted_but_do_not_change_result(self):
# Retained for backward compatibility; they no longer gate the result.
assert (
resolve_tool_policy(
host = "0.0.0.0",
flag = None,
yes = True,
silent = True,
prompt = _never_prompt,
)
is None
)
class TestIsExternalHost:
@pytest.mark.parametrize("host", ["127.0.0.1", "localhost", "::1", "LOCALHOST", "Localhost"])
def test_loopback_aliases_are_local(self, host):
assert is_external_host(host) is False
@pytest.mark.parametrize(
"host", ["0.0.0.0", "::", "127.0.0.2", "192.168.1.5", "10.0.0.1", "example.com"]
)
def test_non_loopback_is_external(self, host):
assert is_external_host(host) is True
class TestSpecificNetworkIP:
"""Binding to a specific LAN IP follows the same rules as 0.0.0.0."""
def test_default_is_unset(self):
assert (
resolve_tool_policy(
host = "192.168.1.5",
flag = None,
yes = False,
silent = False,
prompt = _never_prompt,
)
is None
)
def test_explicit_on_no_prompt(self):
assert (
resolve_tool_policy(
host = "192.168.1.5",
flag = True,
yes = False,
silent = False,
prompt = _never_prompt,
)
is True
)
def test_explicit_off(self):
assert (
resolve_tool_policy(
host = "192.168.1.5",
flag = False,
yes = False,
silent = False,
prompt = _never_prompt,
)
is False
)
def test_localhost_alias_does_not_prompt(self):
assert (
resolve_tool_policy(
host = "localhost",
flag = True,
yes = False,
silent = False,
prompt = _never_prompt,
)
is True
)