feat(skills): add agent reproduction workflows (#4118)

* chore(skills): add codex reproduce workflows

* feat(agent-reproduce): implement agent reproduction workflow and supporting scripts

* feat(skills): capture reference agent state diffs
This commit is contained in:
Dragon 2026-06-02 12:10:54 +08:00 committed by GitHub
parent 5d05cded3e
commit bec97f445d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
12 changed files with 1944 additions and 1 deletions

3
.gitignore vendored
View file

@ -67,6 +67,7 @@ packages/web-templates/src/generated/
packages/vscode-ide-companion/*.vsix
logs/
.repro-runs/
# GHA credentials
gha-creds-*.json
@ -96,4 +97,4 @@ tmp/
# code graph skills
.venv
.codegraph
.codegraph

View file

@ -0,0 +1,98 @@
---
name: agent-reproduce-align
description: Use after a Codex or Claude Code feature has been implemented in Qwen Code to run the selected reference agent and Qwen Code under the same scenario, capture HTTP and terminal traces, compare request bodies, tool/function schemas, outputs, and iterate until the reproduced behavior is close enough.
---
# Agent Reproduce Align
## Purpose
Use this skill when Qwen Code already has a candidate implementation and needs evidence-based parity with a selected reference agent: `codex` or `claude-code`. The goal is not byte-for-byte equality; it is matching the observable contract that matters for the feature.
Default target repo: the current working directory. Use a user-specified path only when the user explicitly provides one.
## Reference Agent Selection
Use the same reference agent selected during `$agent-reproduce-feature`. If the earlier choice is unavailable, ask once and record the answer in the scenario or run notes.
## Workflow
1. Re-state the parity target:
- feature name and trigger
- selected reference agent
- one baseline prompt or interaction script
- acceptable differences
- must-match fields
2. Run the reference agent and Qwen Code in separate capture directories with the same scenario.
3. Capture the selected reference agent's local state before and after the
reference run when state may affect parity.
4. Normalize traces with `scripts/normalize_trace.py`.
5. Compare normalized traces with `scripts/compare_traces.py`.
6. Inspect differences in this order:
- reference-agent state changes that explain behavior
- missing tool/function names
- schema shape and required fields
- model settings and response mode
- prompt role/order differences that affect behavior
- terminal-visible output and exit status
7. Patch Qwen Code, rerun the smallest failing scenario, and repeat.
8. Preserve only redacted minimal fixtures in the repo.
Read `references/alignment-workflow.md` before the first comparison pass.
## Common Commands
Normalize:
```sh
.qwen/skills/agent-reproduce-align/scripts/normalize_trace.py \
.repro-runs/reference/http.jsonl \
> .repro-runs/reference/normalized.json
```
Compare:
```sh
.qwen/skills/agent-reproduce-align/scripts/compare_traces.py \
.repro-runs/reference/normalized.json \
.repro-runs/qwen/normalized.json
```
Run a paired shell scenario:
```sh
REPRO_REFERENCE_AGENT=codex \
.qwen/skills/agent-reproduce-align/scripts/run_pair_capture.sh \
.repro-runs/slash-help \
"codex exec '/help'" \
"npm test -- --runInBand"
```
For Claude Code, set `REPRO_REFERENCE_AGENT=claude-code` and replace the first
command with the discovered Claude Code command. When `REPRO_REFERENCE_AGENT`
is set, the paired runner writes `reference/state-before`,
`reference/state-after`, and `reference/state-diff`. Use the paired runner only
when shell quoting is simple. For interactive slash commands, run the two
captures manually with tmux so each side can receive the same keystrokes. Use
`REPRO_REFERENCE_STATE_ROOT=/tmp/some-root` only for tests or custom state
directories.
## Comparison Rules
- Compare contracts before wording. Exact prompt text is usually implementation detail.
- Treat absent schemas, wrong required fields, or wrong argument names as high-signal failures.
- Treat output ordering as significant only when the user-visible workflow depends on it.
- Do not chase provider-specific endpoints, model names, IDs, timestamps, token counts, or ephemeral headers unless the feature depends on them.
- Do not chase every local state write. Treat state diffs as explanatory
evidence unless the feature contract requires a particular config, memory, or
permission side effect.
- Stop when Qwen Code passes the user-visible scenario and the remaining trace differences are documented as intentional.
## Done Criteria
- Reference-agent and Qwen Code traces for the same scenario exist locally.
- Reference-agent state diff exists or state capture is documented as
irrelevant for the scenario.
- The normalized comparison has no unexplained must-match differences.
- Qwen Code tests or smoke commands cover the fixed behavior.
- Any remaining mismatch is written down in the task notes or Qwen Code docs when it affects users.

View file

@ -0,0 +1,84 @@
# Alignment Workflow Reference
The alignment phase starts after Qwen Code has a candidate implementation. Use it to create a tight loop: run the selected reference agent and Qwen Code, compare traces, patch the target, and rerun only the failing scenario.
## Trace Inputs
Expected raw capture layout:
```text
.repro-runs/<scenario>/
reference/
http.jsonl
command.stdout
command.stderr
command.exit
state-before/state-manifest.json
state-after/state-manifest.json
state-diff/state-diff.md
qwen/
http.jsonl
command.stdout
command.stderr
command.exit
```
Use capture scripts from `$agent-reproduce-feature` for raw capture, or use
`run_pair_capture.sh` for simple non-interactive shell scenarios. Set
`REPRO_REFERENCE_AGENT=codex` or `REPRO_REFERENCE_AGENT=claude-code` with the
paired runner to capture reference-agent state automatically.
## Normalization
`normalize_trace.py` reads mitm JSONL output and emits stable JSON:
- request method and URL path
- JSON request body summary
- message role order and brief content hashes
- tool/function names
- schema required fields
- response status code
It intentionally drops:
- timestamps
- authorization and cookie headers
- provider request IDs
- full message text unless needed for a hash
## Diff Triage
High priority:
- missing request entirely
- wrong endpoint family
- missing tool/function schema
- incompatible required fields or enum values
- slash command not routed to the same behavior class
- state changes that prove the feature writes config, memory, permissions, or
another user-visible local store
Medium priority:
- prompt role ordering differences
- terminal output phrasing differences
- streaming versus non-streaming if users can observe it
- unexplained state changes that plausibly affect future runs
Low priority:
- timestamps, IDs, token counts
- harmless wording differences
- extra target-side metadata ignored by the provider
## Iteration Loop
1. Pick the highest-priority unexplained mismatch.
2. Patch only the likely owner module in Qwen Code.
3. Run the focused test/smoke path.
4. Capture only the affected scenario again.
5. Refresh the reference state diff if the suspected mismatch involves local
state.
6. Normalize and compare again.
Stop when the target behavior is compatible and remaining differences are either irrelevant or explicitly documented.

View file

@ -0,0 +1,156 @@
#!/usr/bin/env python3
"""Compare normalized reproduction traces and print actionable differences."""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
from typing import Any
def load(path: Path) -> dict[str, Any]:
return json.loads(path.read_text(encoding="utf-8"))
def tool_index(request: dict[str, Any]) -> dict[str, dict[str, Any]]:
return {
tool.get("name") or f"<unnamed-{idx}>": tool
for idx, tool in enumerate(request.get("tools") or [])
}
def tool_name_counts(request: dict[str, Any]) -> dict[str, int]:
counts: dict[str, int] = {}
for idx, tool in enumerate(request.get("tools") or []):
name = tool.get("name") or f"<unnamed-{idx}>"
counts[name] = counts.get(name, 0) + 1
return counts
def compare_request(idx: int, left: dict[str, Any], right: dict[str, Any]) -> list[str]:
diffs: list[str] = []
prefix = f"request[{idx}]"
for key in (
"method",
"url_path",
"body_keys",
"body_values",
"model",
"stream",
"response_status",
):
if left.get(key) != right.get(key):
diffs.append(f"{prefix}.{key}: {left.get(key)!r} != {right.get(key)!r}")
left_messages = left.get("messages") or []
right_messages = right.get("messages") or []
left_roles = [item.get("role") for item in left_messages]
right_roles = [item.get("role") for item in right_messages]
if left_roles != right_roles:
diffs.append(f"{prefix}.message_roles: {left_roles!r} != {right_roles!r}")
# Surface count mismatches explicitly. zip() below silently truncates to the
# shorter list, so without this diagnostic an extra trailing message
# carrying the feature-relevant prompt / tool result would never be
# reported (the message_roles diff alone hides which side is longer and by
# how much, and only fires when the *prefix* roles differ at some index).
if len(left_messages) != len(right_messages):
diffs.append(
f"{prefix}.message_count: {len(left_messages)} != {len(right_messages)}"
)
for msg_idx, (left_msg, right_msg) in enumerate(zip(left_messages, right_messages)):
if left_msg.get("content_hash") != right_msg.get("content_hash"):
diffs.append(
f"{prefix}.messages[{msg_idx}].content_hash: "
f"{left_msg.get('content_hash')!r} != "
f"{right_msg.get('content_hash')!r}"
)
# Mirror the request-level missing/extra handling so the user sees the
# actual content of trailing messages that fell off the zip().
if len(left_messages) > len(right_messages):
for msg_idx, message in enumerate(
left_messages[len(right_messages) :], len(right_messages)
):
diffs.append(f"{prefix}.messages[{msg_idx}].missing_in_right: {message!r}")
elif len(right_messages) > len(left_messages):
for msg_idx, message in enumerate(
right_messages[len(left_messages) :], len(left_messages)
):
diffs.append(f"{prefix}.messages[{msg_idx}].extra_in_right: {message!r}")
left_tool_list = left.get("tools") or []
right_tool_list = right.get("tools") or []
if len(left_tool_list) != len(right_tool_list):
diffs.append(
f"{prefix}.tools_count: {len(left_tool_list)} != {len(right_tool_list)}"
)
if tool_name_counts(left) != tool_name_counts(right):
diffs.append(
f"{prefix}.tool_name_counts: "
f"{tool_name_counts(left)!r} != {tool_name_counts(right)!r}"
)
left_tools = tool_index(left)
right_tools = tool_index(right)
missing = sorted(set(left_tools) - set(right_tools))
extra = sorted(set(right_tools) - set(left_tools))
if missing:
diffs.append(f"{prefix}.tools_missing_in_right: {missing}")
if extra:
diffs.append(f"{prefix}.tools_extra_in_right: {extra}")
for name in sorted(set(left_tools) & set(right_tools)):
for key in ("type", "description_hash", "required", "properties", "schema"):
if left_tools[name].get(key) != right_tools[name].get(key):
diffs.append(
f"{prefix}.tool[{name}].{key}: "
f"{left_tools[name].get(key)!r} != {right_tools[name].get(key)!r}"
)
return diffs
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("left", type=Path, help="Reference normalized trace")
parser.add_argument("right", type=Path, help="Target normalized trace, usually Qwen Code")
args = parser.parse_args()
try:
left = load(args.left)
right = load(args.right)
except (OSError, json.JSONDecodeError) as exc:
print(f"Failed to load normalized trace: {exc}", file=sys.stderr)
return 2
diffs: list[str] = []
if left.get("request_count") != right.get("request_count"):
diffs.append(
f"request_count: {left.get('request_count')!r} != {right.get('request_count')!r}"
)
for idx, (left_req, right_req) in enumerate(
zip(left.get("requests") or [], right.get("requests") or [])
):
diffs.extend(compare_request(idx, left_req, right_req))
left_requests = left.get("requests") or []
right_requests = right.get("requests") or []
if len(left_requests) > len(right_requests):
for idx, request in enumerate(left_requests[len(right_requests) :], len(right_requests)):
diffs.append(f"request[{idx}].missing_in_right: {request!r}")
elif len(right_requests) > len(left_requests):
for idx, request in enumerate(right_requests[len(left_requests) :], len(left_requests)):
diffs.append(f"request[{idx}].extra_in_right: {request!r}")
if not diffs:
print("No normalized trace differences found.")
return 0
print("Normalized trace differences:")
for diff in diffs:
print(f"- {diff}")
return 1
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -0,0 +1,244 @@
#!/usr/bin/env python3
"""Normalize mitm JSONL traces into a stable comparison format."""
from __future__ import annotations
import argparse
import hashlib
import json
import sys
from pathlib import Path
from typing import Any
from urllib.parse import urlparse
def content_hash(value: str) -> str:
return hashlib.sha256(value.encode("utf-8")).hexdigest()[:16]
def json_body(record: dict[str, Any]) -> Any:
body = record.get("body") or {}
if body.get("json") is not None:
return body["json"]
text = body.get("text")
if not text:
return None
try:
return json.loads(text)
except json.JSONDecodeError:
return {"text_hash": content_hash(text), "text_len": len(text)}
SCHEMA_KEYS = (
"type",
"enum",
"const",
"items",
"properties",
"required",
"anyOf",
"allOf",
"oneOf",
"additionalProperties",
"description",
"default",
"examples",
"format",
"minimum",
"maximum",
"minLength",
"maxLength",
"pattern",
"$ref",
"minItems",
"maxItems",
"uniqueItems",
"nullable",
)
PARITY_BODY_VALUE_KEYS = (
"model",
"stream",
"temperature",
"max_tokens",
"max_completion_tokens",
"tool_choice",
"top_p",
"top_k",
"n",
"stop",
"response_format",
"seed",
"reasoning_effort",
"parallel_tool_calls",
)
def normalize_schema(value: Any) -> Any:
if isinstance(value, dict):
normalized: dict[str, Any] = {}
for key in SCHEMA_KEYS:
if key not in value:
continue
child = value[key]
if key == "required" and isinstance(child, list):
normalized[key] = sorted(str(item) for item in child)
elif key == "properties" and isinstance(child, dict):
normalized[key] = {
str(name): normalize_schema(schema)
for name, schema in sorted(child.items())
}
elif key in {"anyOf", "allOf", "oneOf"} and isinstance(child, list):
normalized[key] = [normalize_schema(item) for item in child]
else:
normalized[key] = normalize_schema(child)
return normalized
if isinstance(value, list):
return [normalize_schema(item) for item in value]
return value
def walk_tools(value: Any) -> list[dict[str, Any]]:
tools: list[dict[str, Any]] = []
if isinstance(value, dict):
if "tools" in value and isinstance(value["tools"], list):
for tool in value["tools"]:
tools.append(summarize_tool(tool))
if "functions" in value and isinstance(value["functions"], list):
for fn in value["functions"]:
tools.append(summarize_tool({"type": "function", "function": fn}))
return tools
def summarize_tool(tool: Any) -> dict[str, Any]:
if not isinstance(tool, dict):
return {"raw_type": type(tool).__name__}
fn = tool.get("function") if isinstance(tool.get("function"), dict) else tool
params = None
if isinstance(fn, dict):
params = fn.get("parameters") or fn.get("input_schema")
schema = normalize_schema(params) if isinstance(params, dict) else {}
return {
"type": tool.get("type"),
"name": fn.get("name") if isinstance(fn, dict) else None,
"description_hash": content_hash(fn.get("description", ""))
if isinstance(fn, dict) and isinstance(fn.get("description"), str)
else None,
"required": sorted(params.get("required", []))
if isinstance(params, dict) and isinstance(params.get("required"), list)
else [],
"properties": sorted(params.get("properties", {}).keys())
if isinstance(params, dict) and isinstance(params.get("properties"), dict)
else [],
"schema": schema,
}
def summarize_messages(value: Any) -> list[dict[str, Any]]:
messages = None
system_messages: list[Any] = []
if isinstance(value, dict):
# Provider conventions for the system prompt:
# - Anthropic Messages API: top-level "system"
# - OpenAI Responses API: top-level "instructions"
# - Gemini / Qwen Code: top-level "systemInstruction" (camelCase)
for key in ("system", "instructions", "systemInstruction"):
if key in value:
system_messages.append(value[key])
if isinstance(value.get("messages"), list):
messages = value["messages"]
elif isinstance(value.get("input"), list):
messages = value["input"]
if messages is None:
messages = []
summary = []
for system in system_messages:
content = (
system
if isinstance(system, str)
else json.dumps(system, ensure_ascii=False, sort_keys=True)
)
summary.append(
{
"role": "system",
"content_hash": content_hash(content),
"content_len": len(content),
}
)
for item in messages:
if not isinstance(item, dict):
continue
content = item.get("content", "")
if not isinstance(content, str):
content = json.dumps(content, ensure_ascii=False, sort_keys=True)
summary.append(
{
"role": item.get("role"),
"content_hash": content_hash(content),
"content_len": len(content),
}
)
return summary
def summarize_body_values(body: Any) -> dict[str, Any]:
if not isinstance(body, dict):
return {}
return {key: body[key] for key in PARITY_BODY_VALUE_KEYS if key in body}
def normalize(path: Path) -> dict[str, Any]:
requests = []
for line_num, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
if not line.strip():
continue
try:
raw = json.loads(line)
except json.JSONDecodeError as exc:
print(
f"Warning: skipping malformed line {line_num} in {path}: {exc}",
file=sys.stderr,
)
continue
# Valid JSONL lines may decode to non-objects (`[]`, `"hello"`, `42`,
# `null`); those do not have `.get()` and would crash the entire
# normalization with an AttributeError. Skip with a warning instead.
if not isinstance(raw, dict):
print(
f"Warning: skipping non-object line {line_num} in {path}",
file=sys.stderr,
)
continue
req = raw.get("request") or {}
resp = raw.get("response") or {}
parsed = urlparse(req.get("url", ""))
url_path = parsed.path
if parsed.query:
url_path = f"{url_path}?{parsed.query}"
body = json_body(req)
requests.append(
{
"method": req.get("method"),
"url_path": url_path,
"body_keys": sorted(body.keys()) if isinstance(body, dict) else [],
"body_values": summarize_body_values(body),
"model": body.get("model") if isinstance(body, dict) else None,
"stream": body.get("stream") if isinstance(body, dict) else None,
"messages": summarize_messages(body),
"tools": sorted(walk_tools(body), key=lambda item: (item.get("name") or "")),
"response_status": resp.get("status_code") if isinstance(resp, dict) else None,
}
)
return {"source": str(path), "request_count": len(requests), "requests": requests}
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("trace", type=Path)
args = parser.parse_args()
print(json.dumps(normalize(args.trace), ensure_ascii=False, indent=2, sort_keys=True))
return 0
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -0,0 +1,118 @@
#!/usr/bin/env bash
set -euo pipefail
if [[ $# -ne 3 ]]; then
echo "Usage: $0 OUT_DIR REFERENCE_SHELL_COMMAND QWEN_SHELL_COMMAND" >&2
exit 2
fi
out_dir="$1"
reference_command="$2"
qwen_command="$3"
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
feature_run="${script_dir}/../../agent-reproduce-feature/scripts/run_with_mitm.sh"
state_capture="${script_dir}/../../agent-reproduce-feature/scripts/capture_state.py"
reference_agent="${REPRO_REFERENCE_AGENT:-}"
reference_state_root="${REPRO_REFERENCE_STATE_ROOT:-}"
mkdir -p "${out_dir}/reference" "${out_dir}/qwen"
if [[ -n "${reference_agent}" ]]; then
state_args=(--agent "${reference_agent}")
if [[ -n "${reference_state_root}" ]]; then
state_args+=(--root "${reference_state_root}")
fi
"${state_capture}" snapshot \
"${out_dir}/reference/state-before" \
"${state_args[@]}"
fi
set +e
"${feature_run}" "${out_dir}/reference" -- bash -lc "${reference_command}"
reference_status=$?
set -e
if [[ -n "${reference_agent}" ]]; then
"${state_capture}" snapshot \
"${out_dir}/reference/state-after" \
"${state_args[@]}"
"${state_capture}" diff \
"${out_dir}/reference/state-before" \
"${out_dir}/reference/state-after" \
--out-dir "${out_dir}/reference/state-diff"
fi
set +e
"${feature_run}" "${out_dir}/qwen" -- bash -lc "${qwen_command}"
qwen_status=$?
set -e
set +e
"${script_dir}/normalize_trace.py" "${out_dir}/reference/http.jsonl" \
> "${out_dir}/reference/normalized.json" \
2> "${out_dir}/reference/normalize.err"
normalize_ref_status=$?
"${script_dir}/normalize_trace.py" "${out_dir}/qwen/http.jsonl" \
> "${out_dir}/qwen/normalized.json" \
2> "${out_dir}/qwen/normalize.err"
normalize_qwen_status=$?
set -e
compare_status=0
if [[ "${normalize_ref_status}" -ne 0 || "${normalize_qwen_status}" -ne 0 ]]; then
{
echo "Trace normalization failed."
echo "reference_normalize_status=${normalize_ref_status}"
echo "qwen_normalize_status=${normalize_qwen_status}"
echo "reference_normalize_err=${out_dir}/reference/normalize.err"
echo "qwen_normalize_err=${out_dir}/qwen/normalize.err"
} > "${out_dir}/trace.diff"
compare_status=2
else
request_counts="$(
python3 - "${out_dir}/reference/normalized.json" "${out_dir}/qwen/normalized.json" <<'PY'
import json
import sys
for path in sys.argv[1:]:
with open(path, encoding="utf-8") as handle:
print(json.load(handle).get("request_count", 0))
PY
)"
reference_count="$(printf '%s\n' "${request_counts}" | sed -n '1p')"
qwen_count="$(printf '%s\n' "${request_counts}" | sed -n '2p')"
if [[ "${reference_count}" == "0" && "${qwen_count}" == "0" ]]; then
{
echo "Both captures produced empty traces."
echo "reference_http=${out_dir}/reference/http.jsonl"
echo "qwen_http=${out_dir}/qwen/http.jsonl"
} > "${out_dir}/trace.diff"
compare_status=1
else
set +e
"${script_dir}/compare_traces.py" \
"${out_dir}/reference/normalized.json" \
"${out_dir}/qwen/normalized.json" \
> "${out_dir}/trace.diff"
compare_status=$?
set -e
fi
fi
echo "reference_status=${reference_status}"
echo "qwen_status=${qwen_status}"
echo "normalize_reference_status=${normalize_ref_status}"
echo "normalize_qwen_status=${normalize_qwen_status}"
echo "compare_status=${compare_status}"
echo "diff=${out_dir}/trace.diff"
echo "reference_stdout=${out_dir}/reference/command.stdout"
echo "reference_stderr=${out_dir}/reference/command.stderr"
echo "qwen_stdout=${out_dir}/qwen/command.stdout"
echo "qwen_stderr=${out_dir}/qwen/command.stderr"
if [[ "${reference_status}" -ne 0 || "${qwen_status}" -ne 0 || "${normalize_ref_status}" -ne 0 || "${normalize_qwen_status}" -ne 0 || "${compare_status}" -ne 0 ]]; then
exit 1
fi

View file

@ -0,0 +1,132 @@
---
name: agent-reproduce-feature
description: Use when reproducing an existing Codex or Claude Code feature in Qwen Code or another agent CLI by choosing a reference agent, capturing HTTP request bodies, prompts, tool/function schemas, terminal output, and then implementing the matching behavior in the target repo.
---
# Agent Reproduce Feature
## Purpose
Use this skill to turn an observed feature from a reference agent into an implementation task for Qwen Code. The workflow treats the current session as the outer harness and runs a nested reference agent process as the program under test.
Default target repo: the current working directory. Use a user-specified path only when the user explicitly provides one.
## Reference Agent Selection
Start by selecting exactly one reference agent:
- `codex`: use nested Codex as the reference implementation.
- `claude-code`: use nested Claude Code as the reference implementation.
If the user did not choose one, ask once before capture. Then discover the local commands instead of assuming them:
```sh
command -v codex || true
command -v claude || command -v claude-code || true
```
Record the selected adapter in the run notes or scenario:
```json
{
"reference_agent": "codex",
"reference_interactive_command": "codex",
"reference_headless_command": "codex exec",
"target_agent": "qwen-code",
"target_repo": "."
}
```
## Workflow
1. Define the feature surface in one sentence: command, trigger, expected UI/output, and a minimal prompt that exercises it.
2. Select `codex` or `claude-code` as the reference agent and discover its local launch command.
3. Inspect the target repo enough to identify the likely module boundaries and Qwen Code launch command before changing code.
4. Run the nested reference agent against the feature with capture enabled:
- Local state capture via `scripts/capture_state.py` before and after the
scenario.
- HTTP/body capture via `scripts/run_with_mitm.sh`.
- Terminal capture via `scripts/run_tmux_capture.sh` when the feature is interactive or TUI-visible.
- Headless/non-interactive execution when the feature has a stable command-line path.
5. Extract behavioral facts from the trace:
- system/developer prompt deltas relevant to the feature
- request body shape, including `messages`, `tools`, `functions`, schemas, tool choice, model settings
- visible terminal states and command output
- local agent state changes, file edits, exit status, and error paths
6. Implement the smallest compatible behavior in Qwen Code using its existing patterns.
7. Add focused tests or a reproducible smoke command.
8. Hand off to `$agent-reproduce-align` when implementation exists and parity needs iteration.
Read `references/capture-workflow.md` before running capture for the first time in a session.
## Capture Defaults
Prefer a fresh output directory per run:
```sh
mkdir -p .repro-runs/slash-command-baseline
.qwen/skills/agent-reproduce-feature/scripts/run_with_mitm.sh \
.repro-runs/slash-command-baseline \
-- codex exec "exercise the Codex feature here"
```
For Claude Code, use the discovered headless command if available; otherwise use tmux:
```sh
.qwen/skills/agent-reproduce-feature/scripts/run_tmux_capture.sh \
.repro-runs/slash-command-claude \
claude
```
For interactive slash commands or terminal rendering, use tmux:
```sh
.qwen/skills/agent-reproduce-feature/scripts/run_tmux_capture.sh \
.repro-runs/slash-command-tui \
codex
```
The mitm script sets common proxy and CA variables for Node, Python, and curl-based CLIs. If TLS fails, read the certificate notes in `references/capture-workflow.md` and fix trust before interpreting missing traffic as product behavior.
Capture reference-agent state before and after a run:
```sh
.qwen/skills/agent-reproduce-feature/scripts/capture_state.py \
snapshot .repro-runs/slash-command-baseline/state-before \
--agent codex
# Run the reference scenario here.
.qwen/skills/agent-reproduce-feature/scripts/capture_state.py \
snapshot .repro-runs/slash-command-baseline/state-after \
--agent codex
.qwen/skills/agent-reproduce-feature/scripts/capture_state.py \
diff \
.repro-runs/slash-command-baseline/state-before \
.repro-runs/slash-command-baseline/state-after \
--out-dir .repro-runs/slash-command-baseline/state-diff
```
Use `--agent claude-code` to snapshot `~/.claude` instead of `~/.codex`.
Use `--root PATH` only for a custom state directory or tests.
## Implementation Rules
- Do not copy all captured prompt text into Qwen Code. Convert it into the minimum behavior, schema, or test needed.
- Treat captured request bodies as sensitive local artifacts. Redact tokens before saving examples into docs, commits, issues, or PRs.
- Treat state diffs as sensitive local artifacts too. The state tool redacts
common token shapes and omits content for sensitive paths, but review
`state-diff.md` before copying any excerpt into a tracked file.
- Keep the first implementation narrow: one feature, one trigger path, one observable parity target.
- Prefer compatibility tests that assert behavior over brittle tests that assert exact prompt wording.
- If a captured schema reveals a stable public contract, encode that contract as a typed structure or fixture in Qwen Code.
## Done Criteria
- A baseline reference-agent trace exists under `.repro-runs/` or an equivalent ignored/local path.
- Reference-agent state changes are captured or explicitly marked as not
relevant for the scenario.
- Qwen Code contains a focused implementation and at least one verification path.
- Any user-visible command behavior is documented in Qwen Code if that repo already documents similar features.
- The next parity step can be run by `$agent-reproduce-align` without re-discovering the setup.

View file

@ -0,0 +1,160 @@
# Capture Workflow Reference
This skill follows the nested-agent pattern described in "解决问题的原始冲动": run the original tool under a harness, capture the real request bodies and tool schemas, implement the substitute, then compare traces.
## Local Roles
- Outer harness: the current agent session.
- Reference program: a nested `codex`, `claude`, or `claude-code` command that demonstrates the feature.
- Target program: Qwen Code in the current working directory unless the user explicitly provides another path.
- Capture layer: local state snapshots, `mitmdump`, and terminal transcript
capture.
## Reference Adapters
Select one reference adapter before capture:
| Adapter | Interactive command | Headless command |
| ------------- | ------------------------- | ------------------------------------------------------ |
| `codex` | `codex` | `codex exec "<prompt>"` |
| `claude-code` | `claude` or `claude-code` | Discover locally; if unavailable, use tmux interaction |
Do not assume Claude Code's exact non-interactive flags. Check `claude --help` or `claude-code --help` in the user's environment and record the command used.
## Choosing Execution Mode
Use non-interactive/headless mode when:
- the feature has a stable CLI entrypoint
- output can be asserted from stdout/stderr/files
- request bodies are the primary evidence
Use tmux when:
- the feature depends on slash-command input, readline behavior, or a TUI state
- screen output matters
- you need to send multiple keystroke batches
Use both when a feature has model calls and visible terminal state.
## State Capture
Run a state snapshot before and after the reference scenario:
```sh
.qwen/skills/agent-reproduce-feature/scripts/capture_state.py \
snapshot OUT_DIR/state-before --agent codex
.qwen/skills/agent-reproduce-feature/scripts/capture_state.py \
snapshot OUT_DIR/state-after --agent codex
.qwen/skills/agent-reproduce-feature/scripts/capture_state.py \
diff OUT_DIR/state-before OUT_DIR/state-after \
--out-dir OUT_DIR/state-diff
```
Default state roots:
| Adapter | State root |
| ------------- | ----------- |
| `codex` | `~/.codex` |
| `claude-code` | `~/.claude` |
Generated files:
- `state-manifest.json`: file metadata plus redacted text for safe small text
files.
- `state-diff.md`: model-readable summary of added, removed, and modified
files.
- `state-diff.json`: machine-readable equivalent.
The snapshot tool records symlinks but does not follow them. It emits only
metadata, without content hashes, for paths that look like auth, token, session,
history, cache, log, or credential files. Review the Markdown before putting
any state diff into a tracked artifact.
## HTTP Capture
Install mitmproxy if needed:
```sh
python -m pip install --user mitmproxy
```
Run a command under capture:
```sh
.qwen/skills/agent-reproduce-feature/scripts/run_with_mitm.sh OUT_DIR -- COMMAND ARG...
```
Generated files:
- `mitm.log`: mitmdump process log
- `http.jsonl`: redacted request/response records
- `command.stdout`, `command.stderr`, `command.exit`: child process result
- `env.txt`: non-secret capture metadata
The script sets:
- `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`
- `NODE_EXTRA_CA_CERTS`
- `SSL_CERT_FILE`, `REQUESTS_CA_BUNDLE`
- `REPRO_CAPTURE_OUT`
The default CA path is `~/.mitmproxy/mitmproxy-ca-cert.pem`. Some CLIs ignore one or more of these variables; if `http.jsonl` is empty, verify proxy support before changing product code.
## Terminal Capture
Run:
```sh
.qwen/skills/agent-reproduce-feature/scripts/run_tmux_capture.sh OUT_DIR COMMAND ARG...
```
Generated files:
- `tmux-pane.txt`: captured pane contents
- `tmux-session.txt`: session metadata and attach instructions
- `command.txt`: the launched command
The tmux session stays alive so the outer agent can send keys, inspect output, and capture again. Kill it after use:
```sh
tmux kill-session -t SESSION_NAME
```
## What To Extract
From HTTP records:
- model name and model settings
- system/developer message fragments that explain the feature
- user-visible command mapping
- tool/function schema names, descriptions, and JSON schemas
- response format or streaming protocol details
From terminal records:
- exact slash command syntax and completion behavior
- visible state transitions
- error text and recoverable failure paths
- whether the feature is synchronous, streaming, or backgrounded
From state diffs:
- added or modified config files
- permission, MCP, memory, or preference stores touched by the scenario
- state changes that explain later behavior but were not visible in HTTP or
terminal output
## Redaction
Never commit raw traces. Before moving examples into docs or tests, remove:
- authorization headers and API keys
- user-specific paths
- unrelated prompt content
- private repository names and issue content
- full request bodies that are not needed for the feature contract
- state diff content that could expose account, prompt, session, or credential
data

View file

@ -0,0 +1,594 @@
#!/usr/bin/env python3
"""Capture and diff redacted local state for reference agent reproduction."""
from __future__ import annotations
import argparse
import difflib
import hashlib
import json
import os
import re
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
AGENT_ROOTS = {
"codex": ".codex",
"claude-code": ".claude",
}
TEXT_EXTENSIONS = {
".cfg",
".conf",
".ini",
".json",
".jsonc",
".lock",
".md",
".mjs",
".py",
".sh",
".toml",
".txt",
".yaml",
".yml",
}
TEXT_NAMES = {
"config",
"settings",
"preferences",
}
SENSITIVE_PATH_PARTS = {
"access_token",
"auth",
"cache",
"cert",
"certificate",
"conversation",
"conversations",
"cookie",
"cookies",
"credential",
"credentials",
"docker",
"env",
"gcloud",
"gh",
"gnupg",
"history",
"id_ed25519",
"id_rsa",
"identity",
"key",
"keys",
"kube",
"log",
"logs",
"netrc",
"npmrc",
"oauth",
"pgp",
"private_key",
"pypirc",
"refresh_token",
"secret",
"secrets",
"session",
"sessions",
"ssh",
"token",
"tokens",
"transcript",
"transcripts",
}
SENSITIVE_KEY_PATTERN = (
r"[A-Za-z0-9_.-]*(?:api[_-]?key|authorization|cookie|password|secret|"
r"token|credential|access[_-]?token|refresh[_-]?token|"
r"client[_-]?secret)[A-Za-z0-9_.-]*"
)
QUOTED_KEY_QUOTED_VALUE_RE = re.compile(
rf"(?i)([\"'])({SENSITIVE_KEY_PATTERN})\1(\s*:\s*)([\"'])(.*?)\4"
)
UNQUOTED_KEY_QUOTED_VALUE_RE = re.compile(
rf"(?i)(\b(?:{SENSITIVE_KEY_PATTERN})\b)(\s*[=:]\s*)([\"'])(.*?)\3"
)
QUOTED_KEY_BARE_VALUE_RE = re.compile(
rf"(?i)([\"'])({SENSITIVE_KEY_PATTERN})\1(\s*:\s*)([^\"'\s,}}]+)"
)
UNQUOTED_KEY_BARE_VALUE_RE = re.compile(
rf"(?i)(\b(?:{SENSITIVE_KEY_PATTERN})\b)(\s*[=:]\s*)([^\"'\s,}}]+)"
)
BEARER_RE = re.compile(r"(?i)\bbearer\s+[a-z0-9._~+/=-]+")
OPENAI_STYLE_KEY_RE = re.compile(r"\bsk-[A-Za-z0-9_-]{12,}\b")
GITHUB_TOKEN_RE = re.compile(r"\b(?:ghp|gho|ghu|ghs)_[A-Za-z0-9_]{20,}\b")
GITHUB_PAT_RE = re.compile(r"\bgithub_pat_[A-Za-z0-9_]{20,}\b")
AWS_KEY_RE = re.compile(r"\bAKIA[0-9A-Z]{16}\b")
GOOGLE_API_KEY_RE = re.compile(r"\bAIza[0-9A-Za-z_-]{20,}\b")
GENERIC_AUTH_RE = re.compile(r"(?i)\b(?:token|basic)\s+[a-z0-9._~+/=-]{8,}")
PEM_KEY_RE = re.compile(
r"-----BEGIN\s+\w+(?:\s+\w+)*\s+PRIVATE\s+KEY-----.*?"
r"-----END\s+\w+(?:\s+\w+)*\s+PRIVATE\s+KEY-----",
re.DOTALL,
)
def now_iso() -> str:
return datetime.now(timezone.utc).isoformat()
def resolve_root(agent: str, root: Path | None) -> Path:
if root is not None:
return root.expanduser().resolve()
return (Path.home() / AGENT_ROOTS[agent]).resolve()
def sha256_file(path: Path, max_bytes: int) -> str | None:
size = path.stat().st_size
if size > max_bytes:
return None
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def is_sensitive_path(rel_path: str) -> bool:
# Match whole path segments (split on `/`) and check the full basename so
# composite filenames keep their identity. The previous regex split on
# `[/._ -]+`, which produced both false negatives (`id_rsa` -> `["id",
# "rsa"]` missed `id_rsa`) and false positives (`tokenizer.json` ->
# `["token", "izer", "json"]` matched `token`). Hidden directories like
# `.ssh` / `.gnupg` are still matched via their non-dot equivalent, and
# basenames are also checked with their suffix stripped so files like
# `credentials.json` continue to match `credentials`.
lower = rel_path.lower()
parts = lower.split("/")
basename = parts[-1] if parts else lower
if basename in SENSITIVE_PATH_PARTS:
return True
stem = basename.rsplit(".", 1)[0] if "." in basename else basename
if stem and stem in SENSITIVE_PATH_PARTS:
return True
for part in parts:
if part in SENSITIVE_PATH_PARTS:
return True
if part.startswith(".") and part[1:] in SENSITIVE_PATH_PARTS:
return True
return False
def looks_like_text_path(path: Path) -> bool:
if path.suffix.lower() in TEXT_EXTENSIONS:
return True
return path.name.lower() in TEXT_NAMES
def redact_text(text: str) -> str:
home = str(Path.home())
text = re.sub(re.escape(home) + r"(?=[/\s\"',;]|$)", "~", text)
text = BEARER_RE.sub("Bearer <redacted>", text)
text = OPENAI_STYLE_KEY_RE.sub("sk-<redacted>", text)
text = GITHUB_TOKEN_RE.sub("gh_<redacted>", text)
text = GITHUB_PAT_RE.sub("github_pat_<redacted>", text)
text = AWS_KEY_RE.sub("AKIA<redacted>", text)
text = GOOGLE_API_KEY_RE.sub("AIza<redacted>", text)
text = GENERIC_AUTH_RE.sub(lambda m: m.group(0).split()[0] + " <redacted>", text)
text = PEM_KEY_RE.sub(
"-----BEGIN PRIVATE KEY-----<redacted>-----END PRIVATE KEY-----",
text,
)
def replace_quoted_key_quoted_value(match: re.Match[str]) -> str:
return (
f"{match.group(1)}{match.group(2)}{match.group(1)}"
f"{match.group(3)}{match.group(4)}<redacted>{match.group(4)}"
)
text = QUOTED_KEY_QUOTED_VALUE_RE.sub(
replace_quoted_key_quoted_value,
text,
)
text = UNQUOTED_KEY_QUOTED_VALUE_RE.sub(r"\1\2\3<redacted>\3", text)
text = QUOTED_KEY_BARE_VALUE_RE.sub(r"\1\2\1\3<redacted>", text)
return UNQUOTED_KEY_BARE_VALUE_RE.sub(r"\1\2<redacted>", text)
def capture_text(
path: Path,
rel_path: str,
max_text_bytes: int,
) -> tuple[str, str | None]:
if is_sensitive_path(rel_path):
return "sensitive_path", None
if path.stat().st_size > max_text_bytes:
return "too_large", None
if not looks_like_text_path(path):
return "not_text_path", None
raw = path.read_bytes()
if b"\0" in raw:
return "binary", None
try:
text = raw.decode("utf-8")
except UnicodeDecodeError:
return "decode_error", None
return "captured", redact_text(text)
def entry_for_file(
path: Path,
rel_path: str,
max_hash_bytes: int,
max_text_bytes: int,
) -> dict[str, Any]:
stat = path.lstat()
sensitive = is_sensitive_path(rel_path)
digest = None if sensitive else sha256_file(path, max_hash_bytes)
entry: dict[str, Any] = {
"kind": "file",
"size": stat.st_size,
"mtime_ns": stat.st_mtime_ns,
"mode": oct(stat.st_mode & 0o777),
"sha256": digest,
"hash_status": hash_status(sensitive, digest),
}
text_status, redacted_text = capture_text(path, rel_path, max_text_bytes)
entry["text_status"] = text_status
if redacted_text is not None:
entry["redacted_text"] = redacted_text
return entry
def entry_for_symlink(path: Path) -> dict[str, Any]:
try:
target = os.readlink(path)
except OSError:
target = None
return {"kind": "symlink", "target": target}
def collect_entries(
root: Path,
max_hash_bytes: int,
max_text_bytes: int,
) -> dict[str, dict[str, Any]]:
entries: dict[str, dict[str, Any]] = {}
for dirpath, dirnames, filenames in os.walk(root, followlinks=False):
walkable_dirnames = []
for dirname in sorted(dirnames):
path = Path(dirpath) / dirname
rel_path = path.relative_to(root).as_posix()
try:
if path.is_symlink():
entries[rel_path] = entry_for_symlink(path)
else:
walkable_dirnames.append(dirname)
except OSError as exc:
entries[rel_path] = {"kind": "error", "error": str(exc)}
dirnames[:] = walkable_dirnames
for filename in sorted(filenames):
path = Path(dirpath) / filename
rel_path = path.relative_to(root).as_posix()
try:
if path.is_symlink():
entries[rel_path] = entry_for_symlink(path)
elif path.is_file():
entries[rel_path] = entry_for_file(
path,
rel_path,
max_hash_bytes,
max_text_bytes,
)
else:
entries[rel_path] = {"kind": "other"}
except OSError as exc:
entries[rel_path] = {"kind": "error", "error": str(exc)}
return entries
def hash_status(sensitive: bool, digest: str | None) -> str:
if sensitive:
return "sensitive_path"
if digest is None:
return "too_large"
return "captured"
def write_snapshot(args: argparse.Namespace) -> int:
root = resolve_root(args.agent, args.root)
out_dir = args.out_dir
out_dir.mkdir(parents=True, exist_ok=True)
manifest: dict[str, Any] = {
"schema_version": 1,
"created_at": now_iso(),
"agent": args.agent,
"root": str(root),
"root_exists": root.exists(),
"max_hash_bytes": args.max_hash_bytes,
"max_text_bytes": args.max_text_bytes,
"entries": {},
}
if root.exists():
manifest["entries"] = collect_entries(
root,
args.max_hash_bytes,
args.max_text_bytes,
)
manifest_path = out_dir / "state-manifest.json"
manifest_path.write_text(
json.dumps(manifest, ensure_ascii=False, indent=2, sort_keys=True),
encoding="utf-8",
)
os.chmod(manifest_path, 0o600)
print(manifest_path)
return 0
def load_manifest(path: Path) -> dict[str, Any]:
manifest_path = path / "state-manifest.json" if path.is_dir() else path
return json.loads(manifest_path.read_text(encoding="utf-8"))
def changed_fields(before: dict[str, Any], after: dict[str, Any]) -> list[str]:
fields = []
for field in (
"kind",
"size",
"mtime_ns",
"mode",
"sha256",
"hash_status",
"text_status",
"target",
):
if before.get(field) != after.get(field):
fields.append(field)
return fields
def compact_entry(entry: dict[str, Any]) -> dict[str, Any]:
return {key: value for key, value in entry.items() if key != "redacted_text"}
def redacted_text_lines(
entry: dict[str, Any],
max_lines: int,
) -> tuple[list[str], bool]:
text = entry.get("redacted_text")
if not isinstance(text, str):
return [], False
lines = text.splitlines()
truncated = len(lines) > max_lines
return lines[:max_lines], truncated
def added_or_removed_item(
path: str,
entry: dict[str, Any],
max_lines: int,
) -> dict[str, Any]:
lines, truncated = redacted_text_lines(entry, max_lines)
return {
"path": path,
"entry": compact_entry(entry),
"redacted_text": lines,
"redacted_text_truncated": truncated,
}
def text_diff(
path: str,
before: dict[str, Any],
after: dict[str, Any],
max_lines: int,
) -> tuple[list[str], bool]:
before_text = before.get("redacted_text")
after_text = after.get("redacted_text")
if not isinstance(before_text, str) or not isinstance(after_text, str):
return [], False
lines = list(
difflib.unified_diff(
before_text.splitlines(),
after_text.splitlines(),
fromfile=f"before/{path}",
tofile=f"after/{path}",
lineterm="",
)
)
truncated = len(lines) > max_lines
return lines[:max_lines], truncated
def build_diff(
before_manifest: dict[str, Any],
after_manifest: dict[str, Any],
max_diff_lines: int,
) -> dict[str, Any]:
before_entries = before_manifest.get("entries") or {}
after_entries = after_manifest.get("entries") or {}
before_paths = set(before_entries)
after_paths = set(after_entries)
added = sorted(after_paths - before_paths)
removed = sorted(before_paths - after_paths)
common = sorted(before_paths & after_paths)
modified = []
unchanged_count = 0
for path in common:
before = before_entries[path]
after = after_entries[path]
fields = changed_fields(before, after)
if not fields:
unchanged_count += 1
continue
diff_lines, truncated = text_diff(path, before, after, max_diff_lines)
modified.append(
{
"path": path,
"changed_fields": fields,
"before": compact_entry(before),
"after": compact_entry(after),
"text_diff": diff_lines,
"text_diff_truncated": truncated,
}
)
return {
"schema_version": 1,
"created_at": now_iso(),
"agent": after_manifest.get("agent") or before_manifest.get("agent"),
"before_root": before_manifest.get("root"),
"after_root": after_manifest.get("root"),
"root_exists_before": before_manifest.get("root_exists"),
"root_exists_after": after_manifest.get("root_exists"),
"summary": {
"added": len(added),
"removed": len(removed),
"modified": len(modified),
"unchanged": unchanged_count,
},
"added": [
added_or_removed_item(path, after_entries[path], max_diff_lines)
for path in added
],
"removed": [
added_or_removed_item(path, before_entries[path], max_diff_lines)
for path in removed
],
"modified": modified,
}
def metadata_line(entry: dict[str, Any]) -> str:
parts = [f"kind={entry.get('kind')}"]
for key in ("size", "mode", "sha256", "hash_status", "text_status", "target"):
value = entry.get(key)
if value is not None:
parts.append(f"{key}={value}")
return ", ".join(parts)
def markdown_for_diff(diff: dict[str, Any]) -> str:
summary = diff["summary"]
lines = [
"# Agent State Diff",
"",
f"- agent: `{diff.get('agent')}`",
f"- before_root: `{diff.get('before_root')}`",
f"- after_root: `{diff.get('after_root')}`",
(
f"- summary: added={summary['added']}, removed={summary['removed']}, "
f"modified={summary['modified']}, unchanged={summary['unchanged']}"
),
"",
]
if diff["added"]:
lines.extend(["## Added", ""])
for item in diff["added"]:
lines.append(f"- `{item['path']}` ({metadata_line(item['entry'])})")
if item["redacted_text"]:
lines.extend(["", "```"])
lines.extend(item["redacted_text"])
if item["redacted_text_truncated"]:
lines.append("... <content truncated>")
lines.extend(["```", ""])
lines.append("")
if diff["removed"]:
lines.extend(["## Removed", ""])
for item in diff["removed"]:
lines.append(f"- `{item['path']}` ({metadata_line(item['entry'])})")
if item["redacted_text"]:
lines.extend(["", "```"])
lines.extend(item["redacted_text"])
if item["redacted_text_truncated"]:
lines.append("... <content truncated>")
lines.extend(["```", ""])
lines.append("")
if diff["modified"]:
lines.extend(["## Modified", ""])
for item in diff["modified"]:
lines.append(f"### `{item['path']}`")
lines.append("")
lines.append(f"- changed_fields: {', '.join(item['changed_fields'])}")
lines.append(f"- before: {metadata_line(item['before'])}")
lines.append(f"- after: {metadata_line(item['after'])}")
if item["text_diff"]:
lines.extend(["", "```diff"])
lines.extend(item["text_diff"])
if item["text_diff_truncated"]:
lines.append("... <diff truncated>")
lines.append("```")
else:
before_status = item["before"].get("text_status")
after_status = item["after"].get("text_status")
lines.append(
f"- content_diff: omitted ({before_status} -> {after_status})"
)
lines.append("")
if not diff["added"] and not diff["removed"] and not diff["modified"]:
lines.append("No state differences found.")
lines.append("")
return "\n".join(lines)
def write_diff(args: argparse.Namespace) -> int:
before = load_manifest(args.before)
after = load_manifest(args.after)
diff = build_diff(before, after, args.max_diff_lines)
args.out_dir.mkdir(parents=True, exist_ok=True)
json_path = args.out_dir / "state-diff.json"
md_path = args.out_dir / "state-diff.md"
json_path.write_text(
json.dumps(diff, ensure_ascii=False, indent=2, sort_keys=True),
encoding="utf-8",
)
md_path.write_text(
markdown_for_diff(diff),
encoding="utf-8",
)
os.chmod(json_path, 0o600)
os.chmod(md_path, 0o600)
print(md_path)
return 0
def main() -> int:
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest="command", required=True)
snapshot = subparsers.add_parser("snapshot")
snapshot.add_argument("out_dir", type=Path)
snapshot.add_argument("--agent", choices=sorted(AGENT_ROOTS), required=True)
snapshot.add_argument("--root", type=Path)
snapshot.add_argument("--max-hash-bytes", type=int, default=10 * 1024 * 1024)
snapshot.add_argument("--max-text-bytes", type=int, default=200 * 1024)
snapshot.set_defaults(func=write_snapshot)
diff = subparsers.add_parser("diff")
diff.add_argument("before", type=Path)
diff.add_argument("after", type=Path)
diff.add_argument("--out-dir", type=Path, required=True)
diff.add_argument("--max-diff-lines", type=int, default=400)
diff.set_defaults(func=write_diff)
args = parser.parse_args()
return args.func(args)
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -0,0 +1,182 @@
"""mitmproxy addon for local agent reproduction traces.
Writes JSONL records to REPRO_CAPTURE_OUT. Headers are redacted and bodies are
decoded when they look textual. Keep raw outputs local unless manually redacted.
"""
from __future__ import annotations
import base64
import json
import os
import re
import sys
import time
from typing import Any
from urllib.parse import parse_qsl, urlencode, urlparse, urlunparse
from mitmproxy import http
OUT = os.environ.get("REPRO_CAPTURE_OUT", "http.jsonl")
MAX_BODY = int(os.environ.get("REPRO_CAPTURE_MAX_BODY", "500000"))
CAPTURE_ALL = os.environ.get("REPRO_CAPTURE_ALL", "0") == "1"
SENSITIVE_HEADERS = {
"authorization",
"cookie",
"set-cookie",
"x-api-key",
"proxy-authorization",
"api-key",
"x-auth-token",
"x-session-token",
"x-refresh-token",
"openai-organization",
"openai-project",
}
SENSITIVE_KEY_RE = re.compile(
r"(?i)(api[-_]?key|authorization|cookie|password|secret|token|credential|"
r"access[-_]?token|refresh[-_]?token|client[-_]?secret|session)"
)
TOKEN_PATTERNS = (
(re.compile(r"(?i)\bbearer\s+[a-z0-9._~+/=-]+"), "Bearer [REDACTED]"),
(re.compile(r"(?i)\bbasic\s+[a-z0-9._~+/=-]+"), "Basic [REDACTED]"),
(re.compile(r"(?i)\btoken\s+[a-z0-9._~+/=-]+"), "Token [REDACTED]"),
(re.compile(r"\bsk-[A-Za-z0-9_-]{12,}\b"), "sk-[REDACTED]"),
(re.compile(r"\bAKIA[0-9A-Z]{16}\b"), "AKIA[REDACTED]"),
(re.compile(r"\bAIza[0-9A-Za-z_-]{20,}\b"), "AIza[REDACTED]"),
(re.compile(r"\b(?:ghp|gho|ghu|ghs)_[A-Za-z0-9_]{20,}\b"), "gh_[REDACTED]"),
(re.compile(r"\bgithub_pat_[A-Za-z0-9_]{20,}\b"), "github_pat_[REDACTED]"),
(
re.compile(
r"-----BEGIN\s+[\w\s]+PRIVATE\s+KEY-----.*?-----END\s+[\w\s]+PRIVATE\s+KEY-----",
re.DOTALL,
),
"-----BEGIN PRIVATE KEY-----[REDACTED]-----END PRIVATE KEY-----",
),
)
INTERESTING_PATH_HINTS = (
"/chat/completions",
"/responses",
"/v1/messages",
"/v1beta/",
"/generate",
"/completions",
)
def _headers(headers: http.Headers) -> dict[str, str]:
redacted: dict[str, str] = {}
for key, value in headers.items():
key_lower = key.lower()
redacted[key] = (
"[REDACTED]"
if key_lower in SENSITIVE_HEADERS or SENSITIVE_KEY_RE.search(key_lower)
else _redact_text(value)
)
return redacted
def _redact_text(text: str) -> str:
for pattern, replacement in TOKEN_PATTERNS:
text = pattern.sub(replacement, text)
return text
def _redact_json(value: Any, key: str | None = None) -> Any:
if key is not None and SENSITIVE_KEY_RE.search(key):
return "[REDACTED]"
if isinstance(value, dict):
return {str(k): _redact_json(v, str(k)) for k, v in value.items()}
if isinstance(value, list):
return [_redact_json(item) for item in value]
if isinstance(value, str):
return _redact_text(value)
return value
def _redact_url(url: str) -> str:
parsed = urlparse(url)
query = []
for key, value in parse_qsl(parsed.query, keep_blank_values=True):
query.append((key, "[REDACTED]" if SENSITIVE_KEY_RE.search(key) else value))
return urlunparse(parsed._replace(query=urlencode(query, doseq=True)))
def _decode(content: bytes | None) -> dict[str, Any]:
if not content:
return {"kind": "empty", "text": ""}
truncated = len(content) > MAX_BODY
content_sample = content[:MAX_BODY]
try:
text = content_sample.decode("utf-8")
except UnicodeDecodeError:
if truncated:
text = content_sample.decode("utf-8", errors="ignore")
else:
return {
"kind": "base64",
"base64": base64.b64encode(content_sample).decode("ascii"),
"truncated": truncated,
}
parsed: Any = None
try:
parsed = _redact_json(json.loads(text))
redacted_text = json.dumps(parsed, ensure_ascii=False, sort_keys=True)
except json.JSONDecodeError:
redacted_text = _redact_text(text)
return {
"kind": "text",
"text": redacted_text,
"json": parsed,
"truncated": truncated,
}
def _write_record(record: dict[str, Any]) -> None:
try:
os.makedirs(os.path.dirname(os.path.abspath(OUT)), exist_ok=True)
with open(OUT, "a", encoding="utf-8") as handle:
handle.write(json.dumps(record, ensure_ascii=False, sort_keys=True) + "\n")
os.chmod(os.path.abspath(OUT), 0o600)
except Exception as exc:
print(f"[llm_dump] FAILED to write record: {exc}", file=sys.stderr)
def _interesting(flow: http.HTTPFlow) -> bool:
if CAPTURE_ALL:
return True
url = flow.request.pretty_url.lower()
request_ctype = flow.request.headers.get("content-type", "").lower()
response_ctype = ""
if flow.response is not None:
response_ctype = flow.response.headers.get("content-type", "").lower()
return (
any(hint in url for hint in INTERESTING_PATH_HINTS)
or "application/json" in request_ctype
or "application/json" in response_ctype
or "text/event-stream" in request_ctype
or "text/event-stream" in response_ctype
)
def response(flow: http.HTTPFlow) -> None:
if not _interesting(flow):
return
record = {
"ts": time.time(),
"request": {
"method": flow.request.method,
"url": _redact_url(flow.request.pretty_url),
"headers": _headers(flow.request.headers),
"body": _decode(flow.request.content),
},
"response": None,
}
if flow.response is not None:
record["response"] = {
"status_code": flow.response.status_code,
"headers": _headers(flow.response.headers),
"body": _decode(flow.response.content),
}
_write_record(record)

View file

@ -0,0 +1,44 @@
#!/usr/bin/env bash
set -euo pipefail
if [[ $# -lt 2 ]]; then
echo "Usage: $0 OUT_DIR COMMAND [ARG...]" >&2
exit 2
fi
out_dir="$1"
shift
if ! command -v tmux >/dev/null 2>&1; then
echo "tmux not found." >&2
exit 127
fi
mkdir -p "${out_dir}"
out_dir="$(cd "${out_dir}" && pwd)"
session="repro-$(date +%Y%m%d-%H%M%S)-$$"
printf '%q ' "$@" > "${out_dir}/command.txt"
echo >> "${out_dir}/command.txt"
tmux new-session -d -s "${session}" "$@"
cleanup() {
if [[ "${REPRO_TMUX_KEEP_SESSION:-0}" != "1" ]]; then
tmux kill-session -t "${session}" >/dev/null 2>&1 || true
fi
}
trap cleanup EXIT
sleep "${REPRO_TMUX_SETTLE_SECONDS:-2}"
tmux capture-pane -t "${session}" -p -S - > "${out_dir}/tmux-pane.txt"
{
echo "session=${session}"
echo "attach=tmux attach -t ${session}"
echo "capture=tmux capture-pane -t ${session} -p -S - > ${out_dir}/tmux-pane.txt"
echo "kill=tmux kill-session -t ${session}"
echo "keep_session=REPRO_TMUX_KEEP_SESSION=1"
} > "${out_dir}/tmux-session.txt"
cat "${out_dir}/tmux-session.txt"

View file

@ -0,0 +1,130 @@
#!/usr/bin/env bash
set -euo pipefail
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
if [[ $# -lt 3 || "${2:-}" != "--" ]]; then
echo "Usage: $0 OUT_DIR -- COMMAND [ARG...]" >&2
exit 2
fi
out_dir="$1"
shift 2
mkdir -p "${out_dir}"
out_dir="$(cd "${out_dir}" && pwd)"
port="${REPRO_PROXY_PORT:-18080}"
ca_file="${MITMPROXY_CA_FILE:-${HOME}/.mitmproxy/mitmproxy-ca-cert.pem}"
http_out="${out_dir}/http.jsonl"
mitm_log="${out_dir}/mitm.log"
if ! command -v mitmdump >/dev/null 2>&1; then
echo "mitmdump not found. Install mitmproxy first." >&2
exit 127
fi
if [[ ! -f "${ca_file}" ]]; then
echo "WARNING: CA cert not found at ${ca_file}." >&2
echo "Run mitmproxy once to generate it, or set MITMPROXY_CA_FILE." >&2
fi
: > "${http_out}"
: > "${mitm_log}"
# --set ssl_insecure=true disables upstream TLS verification so mitmproxy
# can intercept HTTPS calls from the wrapped command. Intended for local
# dev only; do NOT run this script on shared or untrusted networks.
REPRO_CAPTURE_OUT="${http_out}" \
mitmdump \
--listen-host 127.0.0.1 \
--listen-port "${port}" \
--set block_global=false \
--set ssl_insecure=true \
-s "${script_dir}/llm_dump.py" \
>"${mitm_log}" 2>&1 &
mitm_pid="$!"
cleanup() {
kill "${mitm_pid}" >/dev/null 2>&1 || true
wait "${mitm_pid}" >/dev/null 2>&1 || true
}
trap cleanup EXIT
proxy_ready=0
for _attempt in {1..50}; do
if ! kill -0 "${mitm_pid}" >/dev/null 2>&1; then
echo "mitmdump exited before the wrapped command started." >&2
cat "${mitm_log}" >&2
exit 1
fi
if python3 - "${port}" <<'PY' >/dev/null 2>&1
import socket
import sys
with socket.create_connection(("127.0.0.1", int(sys.argv[1])), timeout=0.2):
pass
PY
then
proxy_ready=1
break
fi
sleep 0.1
done
if [[ "${proxy_ready}" != "1" ]]; then
echo "mitmdump did not start listening on 127.0.0.1:${port}." >&2
cat "${mitm_log}" >&2
exit 1
fi
redacted_command="$(
# Note: avoid the GNU-only /I (case-insensitive) sed flag — BSD sed
# (macOS pre-Sequoia) silently fails to match with /I, so previously
# `API_KEY=…`, `Secret=…`, etc. would not be redacted on macOS. Use
# explicit per-letter character classes for the case-insensitive
# token-name matches; both BSD and GNU sed accept them.
printf '%q ' "$@" |
sed -E \
-e 's/sk-[A-Za-z0-9_-]{12,}/sk-<redacted>/g' \
-e 's/AKIA[0-9A-Z]{16}/AKIA<redacted>/g' \
-e 's/AIza[0-9A-Za-z_-]{20,}/AIza<redacted>/g' \
-e 's/(ghp|gho|ghu|ghs)_[A-Za-z0-9_]{20,}/gh_<redacted>/g' \
-e 's/github_pat_[A-Za-z0-9_]{20,}/github_pat_<redacted>/g' \
-e 's/([A-Za-z0-9_.-]*([Aa][Pp][Ii][-_]?[Kk][Ee][Yy]|[Tt][Oo][Kk][Ee][Nn]|[Ss][Ee][Cc][Rr][Ee][Tt]|[Cc][Rr][Ee][Dd][Ee][Nn][Tt][Ii][Aa][Ll])[A-Za-z0-9_.-]*=)[^[:space:]]+/\1<redacted>/g'
)"
{
echo "out_dir=${out_dir}"
echo "proxy=http://127.0.0.1:${port}"
echo "ca_file=${ca_file}"
echo "command=${redacted_command}"
} > "${out_dir}/env.txt"
set +e
HTTP_PROXY="http://127.0.0.1:${port}" \
HTTPS_PROXY="http://127.0.0.1:${port}" \
ALL_PROXY="http://127.0.0.1:${port}" \
http_proxy="http://127.0.0.1:${port}" \
https_proxy="http://127.0.0.1:${port}" \
all_proxy="http://127.0.0.1:${port}" \
NO_PROXY="localhost,127.0.0.1" \
no_proxy="localhost,127.0.0.1" \
NODE_EXTRA_CA_CERTS="${ca_file}" \
SSL_CERT_FILE="${ca_file}" \
REQUESTS_CA_BUNDLE="${ca_file}" \
REPRO_CAPTURE_OUT="${http_out}" \
"$@" >"${out_dir}/command.stdout" 2>"${out_dir}/command.stderr"
status=$?
set -e
sleep "${REPRO_MITM_DRAIN_SECONDS:-1}"
echo "${status}" > "${out_dir}/command.exit"
if [[ "${status}" -ne 0 ]]; then
echo "command_failed: exit=${status}" >&2
echo "stdout=${out_dir}/command.stdout" >&2
echo "stderr=${out_dir}/command.stderr" >&2
fi
exit "${status}"