mirror of
https://github.com/unslothai/unsloth.git
synced 2026-07-10 00:08:58 +00:00
bc85ecd145
* Studio: report the real llama-server context window and add an opt-in overflow policy for OpenAI-compatible serving A community report showed OpenCode failing tool calls every few minutes against Studio's OpenAI-compatible API while the same GGUF was stable on LM Studio. Root cause: Studio advertises the requested context length, but llama-server can allocate less (memory-fit step on small GPUs, --parallel slot split), so clients budget against a window that does not exist. Their generations truncate mid tool call at the real wall (finish_reason=length with cut JSON arguments) and eventually the prompt itself exceeds the real window, returning a 400 that agentic clients treat as non-retryable. Changes: - After llama-server health, read default_generation_settings.n_ctx from /props and adopt it whenever it is below Studio's computed context, with a warning. The load response, status route, UI value, and the passthrough max_tokens ceiling all become honest automatically. - Expose context_length and max_context_length on /v1/models so clients can budget against the enforced window. - Accept empty role=tool content (commands with no output are routine in agentic loops; OpenAI and llama-server both accept it) instead of a 400. - Add context_overflow=truncate_middle (per request, or server-wide via UNSLOTH_CONTEXT_OVERFLOW=truncate_middle): on exceed_context_size_error the passthrough drops whole middle turn-groups (system prompt, first turn, and recent turns kept; tool calls stay paired with their results), clips oversized contents middle-out when group-dropping is not enough, clamps max_tokens to the generation headroom, and retries. Default stays 'error' with code=context_length_exceeded so clients running their own compaction keep full control. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: allocate the requested context for real (kv-unified, fit-ctx floor) Two launch-flag gaps caused the advertised vs allocated divergence at the source: - llama-server enables --kv-unified only when the slot count is auto; Studio always passes --parallel N, which silently splits -c into per-slot windows of -c/N. Pass --kv-unified when N > 1 so a single request can use the full advertised window (same total KV memory, shared pool). - with --fit on the fit step may set ctx as low as 4096; pass --fit-ctx <requested> for explicit requests so fit offloads or fails into the existing --fit off retry instead of silently shrinking the window. Both flags are gated on --help capability probing so older builds keep the current behavior, where the /props readback remains the backstop. Verified live: -c 98304 --parallel 4 now serves per-slot n_ctx 98304 (was 24576), 48k-token requests pass through the passthrough, and the readback warning no longer fires. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
53 lines
1.7 KiB
Python
53 lines
1.7 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""Empty ``role="tool"`` content must be accepted on the OpenAI-compat surface.
|
|
|
|
Agentic clients send ``content: ""`` when a command produced no output;
|
|
OpenAI and llama-server both accept it. Studio used to 400, which standard
|
|
clients treat as non-retryable and kill the session. The validator must
|
|
normalize empty/missing tool content to ``""`` instead of raising.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
_BACKEND_DIR = str(Path(__file__).resolve().parent.parent)
|
|
if _BACKEND_DIR not in sys.path:
|
|
sys.path.insert(0, _BACKEND_DIR)
|
|
|
|
from models.inference import ChatMessage
|
|
|
|
|
|
def test_tool_message_empty_string_content_is_accepted():
|
|
msg = ChatMessage(role = "tool", content = "", tool_call_id = "call_1")
|
|
assert msg.content == ""
|
|
|
|
|
|
def test_tool_message_none_content_normalizes_to_empty_string():
|
|
msg = ChatMessage(role = "tool", content = None, tool_call_id = "call_1")
|
|
assert msg.content == ""
|
|
|
|
|
|
def test_tool_message_empty_list_content_normalizes_to_empty_string():
|
|
msg = ChatMessage(role = "tool", content = [], tool_call_id = "call_1")
|
|
assert msg.content == ""
|
|
|
|
|
|
def test_tool_message_real_content_is_preserved():
|
|
msg = ChatMessage(role = "tool", content = "ok", tool_call_id = "call_1")
|
|
assert msg.content == "ok"
|
|
|
|
|
|
def test_user_message_still_requires_content():
|
|
with pytest.raises(ValueError):
|
|
ChatMessage(role = "user", content = None)
|
|
|
|
|
|
def test_assistant_empty_content_still_collapses_to_none():
|
|
msg = ChatMessage(role = "assistant", content = "")
|
|
assert msg.content is None
|