Studio: tighten Llama-3.2 bare-JSON guard for PR #5811

A fuzz pass turned up that ``_parse_llama3_bare_json`` accepted
``parameters`` as a string, contradicting the docstring's
"parameters or arguments is a dict" guard. Prose like
``{"name":"foo","parameters":"a sentence"}`` would wrongly fire
the parser, which the agentic loop would then heal into a real
``foo(query="a sentence")`` call.

Tightened guard:

  - ``parameters`` must be a dict (Llama-3 spec).
  - ``arguments`` may be a dict, or a JSON-encoded string that
    decodes to a dict (OpenAI shape, e.g.
    ``"arguments":"{\"q\":\"x\"}"``). Plain non-JSON strings,
    JSON-strings of lists / scalars / null no longer pass.

Added 4 regression tests under TestParserMultiFormat:

  - test_llama3_2_bare_json_string_parameters_does_not_fire
  - test_llama3_2_bare_json_string_arguments_not_json_does_not_fire
  - test_llama3_2_bare_json_string_arguments_json_dict_fires
  - test_llama3_2_bare_json_string_arguments_json_non_dict_does_not_fire

Existing tests stay green (104 -> 108 passing) and a 50-case
cross-version fuzz suite passes on Python 3.10 / 3.11 / 3.12 / 3.13.
This commit is contained in:
Daniel Han 2026-05-27 14:28:08 +00:00
parent 518d0a5570
commit 615b860803
2 changed files with 49 additions and 6 deletions

View file

@ -472,16 +472,28 @@ def _parse_llama3_bare_json(content: str, *, id_offset: int) -> list[dict]:
name = obj.get("name") or obj.get("function") or ""
if not isinstance(name, str) or not name:
break
# ``parameters`` must be a dict (Llama-3 spec).
# ``arguments`` may be a dict or a JSON-string of a dict (OpenAI shape).
# Anything looser would fire on prose like ``{"name":"x","parameters":"sentence"}``.
if "parameters" in obj:
args = obj.get("parameters")
if not isinstance(args, dict):
break
args_str = json.dumps(args)
elif "arguments" in obj:
args = obj.get("arguments")
else:
break
if isinstance(args, dict):
args_str = json.dumps(args)
elif isinstance(args, str):
args_str = args
if isinstance(args, dict):
args_str = json.dumps(args)
elif isinstance(args, str):
try:
parsed = json.loads(args)
except (json.JSONDecodeError, ValueError):
break
if not isinstance(parsed, dict):
break
args_str = args
else:
break
else:
break
out.append(

View file

@ -260,6 +260,37 @@ class TestParserMultiFormat:
text = '{"name":"x","parameters":42}'
assert parse_tool_calls_from_text(text) == []
def test_llama3_2_bare_json_string_parameters_does_not_fire(self):
# Llama-3 spec: parameters must be a dict. Prose like
# ``{"name":"foo","parameters":"a sentence"}`` must NOT trigger.
text = '{"name":"foo","parameters":"this is a sentence"}'
assert parse_tool_calls_from_text(text) == []
def test_llama3_2_bare_json_string_arguments_not_json_does_not_fire(self):
# OpenAI ``arguments`` may be a JSON-string of a dict, but a
# plain non-JSON string must not pass the guard.
text = '{"name":"foo","arguments":"not json"}'
assert parse_tool_calls_from_text(text) == []
def test_llama3_2_bare_json_string_arguments_json_dict_fires(self):
# OpenAI shape: arguments is a JSON-encoded string of a dict.
text = '{"name":"foo","arguments":"{\\"q\\":\\"x\\"}"}'
result = parse_tool_calls_from_text(text)
assert len(result) == 1
assert result[0]["function"]["name"] == "foo"
# arguments stays as the original JSON-string.
assert result[0]["function"]["arguments"] == '{"q":"x"}'
def test_llama3_2_bare_json_string_arguments_json_non_dict_does_not_fire(self):
# JSON-string that parses to a list / scalar / null must NOT fire.
for bad in (
'{"name":"foo","arguments":"[1,2,3]"}',
'{"name":"foo","arguments":"\\"plain\\""}',
'{"name":"foo","arguments":"null"}',
'{"name":"foo","arguments":"42"}',
):
assert parse_tool_calls_from_text(bad) == [], bad
# ── Mistral pre-v11 ───────────────────────────────────────────
def test_mistral_pre_v11_array(self):