open-code-review/cmd/opencodereview/shared.go
Gongyl01 0ce730a3c8
feat(manifest): run manifest coverage contract for review (#367) (#520)
* feat(session): add run manifest coverage data model and builder

First slice of issue #367 (run manifest coverage contract): the data
model and state machine only. Not yet wired into the agent or CLI, so
existing review/scan output is unchanged.

Introduce the versioned, immutable RunManifest (schema ocr.run-manifest/v1)
and a concurrency-safe ManifestBuilder that tracks per-file coverage
(selected/completed/reused/failed/waived) and freezes into a terminal
state.

- terminal state derived solely from coverage sets, never comments/warnings
  (complete/partial/failed/skipped)
- Finalize sweeps any undecided selected item to failed/unknown so no item
  is silently dropped
- single-mutex builder: first terminal state wins, frozen after Finalize,
  nil-receiver safe
- fixed failure classification enum with an unknown catch-all
- redaction floor on failure/waive reasons (strip secrets, cap length) as a
  single write entry so callers cannot bypass it
- 22 unit tests, race-clean

Refs: issue #367
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(session): harden run manifest per adversarial review

Address findings from the concurrency / JSON-contract / PR#306-coupling
adversarial review of the manifest data model (still slice 1; not wired to
agent or CLI).

- SetSweepClass: Finalize can classify undispatched items as cancelled/budget
  instead of a blanket unknown (the one real model gap the review found)
- ItemID(fingerprint)=SHA-256 canonical mint helper; an item_id is never a raw
  fingerprint, keeping the resume cross-reference explicit and mix-ups caught
- sanitizeReason: strip control/ANSI chars, coerce valid UTF-8, redact quoted
  secret values, guarantee single line
- Finalize returns deep-copied coverage slices so the frozen snapshot is never
  aliased across the two outlets
- RegisterSelected: nil-safe (lazy-init map) + documents that only the
  post-deletion/post-filter dispatchable set may be registered

+7 unit tests (29 total), race-clean.

Refs: issue #367
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(manifest): wire input identity, config hashes and run-level failure (shard ②d)

- Freeze per-mode input identity (mode + resolved_base/head + exact_range +
  source_artifact_sha256) via diff.ResolveInput/commitParents, and repository
  identity via RemoteIdentity/canonicalRemote (credential-free).
- Add rule_config_sha256 and runtime_config_sha256 over an allowlist of
  non-secret fields using a length-prefixed SHA-256 framework (no tokens/URLs).
- Replace SetRunLevelFailure(bool) with structured SetRunFailure(class, reason)
  and set ManifestInput.mode; fill execution.* (ocr version, provider, model,
  concurrency, config hashes).
- Thread error returns through Finalize/WriteSessionEnd (main review path
  surfaces them; skip/all-failed/scan paths hardened in follow-up).
- Tests: manifest_hash, canonical_config, git_resolve.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(manifest): propagate persistence errors and harden remote/error classification

Merged review themes A/B/E from the 07-22 consolidated assessment.

Theme A — Finalize / session_end delivery errors no longer swallowed:
- agent.go no-files path returns the Finalize error instead of nil (A1)
- agent.go loadDiffs failure joins the Finalize error via errors.Join (A2)
- session.Finalize uses sync.Once + cached finalizeErr: written exactly
  once, concurrency-safe, and every caller replays the same result so a
  retry cannot falsely report success (A3)
- scan/agent.go wires both Finalize call sites to surface the error (A4)

Theme B — canonicalRemote rewritten (internal/diff/git.go):
- keep the port (u.Host, not u.Hostname) so endpoints differing only by
  port stay distinct (B1)
- split scp syntax on the first ':' so an '@' inside the path survives (B2)
- recognize local/file/Windows/UNC remotes and omit identity rather than
  misparsing a path as a host (B3; local-remote policy still open)

Theme E — main_task-empty is now a sentinel (errMainTaskEmpty) classified
via errors.Is instead of matching error text.

Theme D (TOCTOU) deferred to shard 4 per issue #367 open-issues OI-12.

Tests: go build ./... + go vet + go test ./... all green (23 pkgs).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(manifest): report both dispatch and persistence errors on the normal path

The success-path Finalize wiring used `ferr != nil && err == nil`, so when the
review (or scan) failed AND session_end also failed to persist, the persistence
error was dropped and only the dispatch error surfaced — the caller never
learned the session/manifest was not saved.

Join both with errors.Join when both occur (matching the loadDiffs path), so a
persistence failure is always reported even alongside a dispatch failure. This
closes the last gap in the OI-10 contract.

- internal/agent/agent.go: review normal path
- internal/scan/agent.go: scan normal path (+ errors import)

Tests: go build ./... + go vet + go test ./... all green (23 pkgs).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(manifest): 接入 CLI 与 viewer 并补齐验收用例

- 使用冻结 manifest 统一 review JSON、文本与退出状态\n- session CLI 和 viewer 展示五集合覆盖并兼容 legacy/aborted\n- 补充本地 mock、跨出口一致性及安全验收用例

* test(manifest): 补齐验收矩阵缺口并修复审核发现的缺陷

验收用例:configuration 分类(run 级 sweep + item 级映射)、budget/timeout/panic 混合 partial 隔离、跨出口一致性改为规范化原始字节比对、flag 校验失败无产物断言。

代码修复:sanitizeReason 先剥控制字符再脱敏(堵控制字节绕过)、失败项异分类二次标记报冲突错误、source_artifact_sha256 按 item_id 去重并稳定排序、sortItems 改 SliceStable 对齐设计用词。

全仓 go test 23 包通过。

* test(manifest): 补充 provider transition resume 测试用例

覆盖 issue #367 验收标准 provider transition:resume 时 provider/model 改变后,子 manifest 记录当前值而非继承父运行,并经 parent_run_id 链接父会话以支持审计。用 mock client,不依赖真实 provider key。

* fix(manifest): 对齐预算终态与持久化语义

统一聚合预算停止时的 coverage、status 与退出码。传播 session writer 初始化错误,并补齐 merge first-parent 输入身份及回归测试。移除代码注释中的外部设计文档引用。

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: kite <254839944+lizhengfeng101@users.noreply.github.com>
2026-08-01 16:50:21 +08:00

364 lines
13 KiB
Go

package main
import (
"context"
"fmt"
"net/url"
"os"
"path/filepath"
"strings"
"time"
"github.com/alibaba/open-code-review/internal/agent"
"github.com/alibaba/open-code-review/internal/config/rules"
"github.com/alibaba/open-code-review/internal/config/template"
"github.com/alibaba/open-code-review/internal/config/toolsconfig"
"github.com/alibaba/open-code-review/internal/diff"
"github.com/alibaba/open-code-review/internal/gitcmd"
"github.com/alibaba/open-code-review/internal/llm"
"github.com/alibaba/open-code-review/internal/model"
"github.com/alibaba/open-code-review/internal/session"
"github.com/alibaba/open-code-review/internal/stdout"
"github.com/alibaba/open-code-review/internal/telemetry"
"github.com/alibaba/open-code-review/internal/tool"
)
// commonContext bundles the state that both `ocr review` and `ocr scan`
// need to load *before* deciding whether to dispatch a preview or a real
// LLM session: a validated template, the resolved repo path, review rules,
// and a shared git subprocess limiter.
type commonContext struct {
Template *template.Template
RepoDir string
Resolver rules.Resolver
FileFilter *rules.FileFilter
GitRunner *gitcmd.Runner
// IsGitRepo reports whether RepoDir is inside a git repository. Always
// true when requireGit was set; may be false when scan accepts non-git
// directories.
IsGitRepo bool
}
// loadCommonContext validates the working directory, loads the embedded
// template, raises MaxToolRequestTimes when maxTools exceeds the default,
// resolves the absolute repo path, loads system review rules, and creates
// the global git subprocess limiter. Both review and scan callers go
// through this so the startup sequence stays consistent.
//
// requireGit=true fails fast when the directory is not a git repo (review
// path: diff concept requires git). requireGit=false allows non-git
// directories (scan path: provider falls back to filepath.Walk).
func loadCommonContext(repoDirInput, rulePath string, maxTools, maxGitProcs int, requireGit bool) (*commonContext, error) {
tpl, err := template.LoadDefault()
if err != nil {
return nil, fmt.Errorf("load default template: %w", err)
}
if maxTools > tpl.MaxToolRequestTimes {
tpl.MaxToolRequestTimes = maxTools
}
if err := tpl.Validate(); err != nil {
return nil, fmt.Errorf("invalid config: %w", err)
}
repoDir, isGit, err := resolveWorkingDir(repoDirInput, requireGit)
if err != nil {
return nil, err
}
resolver, fileFilter, err := rules.NewResolver(repoDir, rulePath)
if err != nil {
return nil, fmt.Errorf("load rules: %w", err)
}
return &commonContext{
Template: tpl,
RepoDir: repoDir,
Resolver: resolver,
FileFilter: fileFilter,
GitRunner: gitcmd.New(maxGitProcs),
IsGitRepo: isGit,
}, nil
}
// resolveWorkingDir returns (absPath, isGitRepo, err). When requireGit is
// true, returns an error if the directory is not a git repo. When false,
// returns IsGitRepo=false instead of erroring (scan path uses this).
func resolveWorkingDir(input string, requireGit bool) (string, bool, error) {
if input == "" {
wd, err := os.Getwd()
if err != nil {
return "", false, fmt.Errorf("get working directory: %w", err)
}
input = wd
}
absPath, err := filepath.Abs(input)
if err != nil {
return "", false, fmt.Errorf("resolve absolute path: %w", err)
}
if _, statErr := os.Stat(absPath); statErr != nil {
return "", false, fmt.Errorf("stat %s: %w", absPath, statErr)
}
out, err := runGitCmd(absPath, "rev-parse", "--git-dir")
isGit := err == nil && len(out) > 0
if !isGit && requireGit {
return "", false, fmt.Errorf("%s is not a git repository", absPath)
}
// #287: git reports diff and `git show HEAD:<path>` paths relative to the
// repository root, not the current directory. When `ocr review` runs from a
// subdirectory of a monorepo, anchor RepoDir at the git top-level so those
// root-relative paths resolve for both disk reads and git-show reads.
// requireGit is true only for the review path; scan (requireGit=false) keeps
// the CWD so its `git ls-files` walk stays scoped to the subdirectory.
if isGit && requireGit {
// runGitCmdStdout captures stdout only so git stderr notices can't
// pollute the resolved path. --show-toplevel fails (or is empty) when
// there is no work tree — e.g. a bare repo, where --git-dir succeeds so
// isGit is true. Fail loudly there instead of silently reusing the
// subdir, which would reproduce the #287 root-relative-path bug.
top, topErr := runGitCmdStdout(absPath, "rev-parse", "--show-toplevel")
t := strings.TrimSpace(string(top))
if topErr != nil || t == "" {
return "", false, fmt.Errorf("%s is a git repository without a work tree (bare repo?); cannot resolve its top level for review", absPath)
}
absPath = t
}
return absPath, isGit, nil
}
// llmRuntime bundles the LLM-side state both subcommands need once they've
// decided to actually run a session: tool definitions, an app-language
// adjusted template (mutated in place via ApplyLanguage), the LLM client,
// the resolved model name, and a fresh comment collector.
type llmRuntime struct {
Client llm.LLMClient
Model string
Provider string // configured provider name (non-secret label; empty for env-resolved endpoints)
PlanToolDefs []llm.ToolDef
MainToolDefs []llm.ToolDef
Collector *tool.CommentCollector
AppCfg *Config
// RuntimeConfig holds the allowlisted, non-secret runtime settings (protocol,
// sanitized endpoint host, language, timeout) derived from the resolved
// endpoint and app config, for the run manifest's runtime_config_sha256. It
// never carries the token or full URL.
RuntimeConfig agent.RuntimeConfig
}
// loadLLMRuntime loads tool defs from toolConfigPath, reads the app config
// from the user's default config path (applying the configured language to
// tpl — defaulting when the config file is absent), resolves the LLM
// endpoint (honoring modelOverride from --model when non-empty), and
// returns the runtime bundle. tpl is mutated in place.
func loadLLMRuntime(tpl *template.Template, toolConfigPath, modelOverride string) (*llmRuntime, error) {
toolEntries, err := toolsconfig.Load(toolConfigPath)
if err != nil {
return nil, fmt.Errorf("load tools: %w", err)
}
planToolDefs := agent.BuildToolDefs(toolEntries, true)
mainToolDefs := agent.BuildToolDefs(toolEntries, false)
cfgPath, err := defaultConfigPath()
if err != nil {
return nil, err
}
appCfg, err := LoadAppConfig(cfgPath)
if err != nil {
return nil, fmt.Errorf("load app config: %w", err)
}
// Apply the language directive even when the config file is missing
// (upstream #fix: ApplyLanguage with empty lang falls back to default).
var lang, provider string
if appCfg != nil {
lang = appCfg.Language
provider = appCfg.Provider
}
tpl.ApplyLanguage(lang)
ep, err := llm.ResolveEndpointWithModelOverride(cfgPath, modelOverride)
if err != nil {
return nil, fmt.Errorf("resolve LLM endpoint: %w", err)
}
return &llmRuntime{
Client: llm.NewLLMClient(ep),
Model: ep.Model,
Provider: provider,
PlanToolDefs: planToolDefs,
MainToolDefs: mainToolDefs,
Collector: tool.NewCommentCollector(),
AppCfg: appCfg,
RuntimeConfig: agent.RuntimeConfig{
Protocol: ep.Protocol,
EndpointHost: sanitizeEndpointHost(ep.URL),
Language: lang,
Timeout: ep.Timeout,
},
}, nil
}
// sanitizeEndpointHost extracts the credential-free host[:port] from a full LLM
// endpoint URL, dropping scheme, any embedded userinfo, path, query and fragment
// so no secret material survives into the manifest's runtime_config hash. The
// host is lowercased for a stable identity (DNS is case-insensitive). An empty
// or unparseable URL, or one without a host, yields "".
func sanitizeEndpointHost(rawURL string) string {
if strings.TrimSpace(rawURL) == "" {
return ""
}
u, err := url.Parse(rawURL)
if err != nil || u.Host == "" {
return ""
}
return strings.ToLower(u.Host) // u.Host is host[:port]; userinfo lives in u.User
}
// applyCLIExcludes appends user-supplied --exclude patterns (already split
// into a []string) onto cc.FileFilter.Exclude. Creates the FileFilter if
// none was returned by rule.json layers. Idempotent on empty input.
func applyCLIExcludes(cc *commonContext, patterns []string) {
if len(patterns) == 0 {
return
}
if cc.FileFilter == nil {
cc.FileFilter = &rules.FileFilter{}
}
cc.FileFilter.Exclude = append(cc.FileFilter.Exclude, patterns...)
}
// excludeToolDef returns a copy of defs with any entries whose function name
// matches name removed. Used by `ocr scan` to hide tools that don't make
// sense in full-scan mode (e.g. file_read_diff).
func excludeToolDef(defs []llm.ToolDef, name string) []llm.ToolDef {
out := make([]llm.ToolDef, 0, len(defs))
for _, d := range defs {
if d.Function.Name == name {
continue
}
out = append(out, d)
}
return out
}
// quietHandle wraps a stdout.Quiet() restorer so callers can `defer
// q.Restore()` for safety while emitRunResult restores it early when the
// agent-text audience needs the trace summary on the user's terminal.
// Restore is idempotent.
type quietHandle struct {
fn func()
}
// newQuietHandle silences stdout when outputFormat=="json" or
// audience=="agent"; otherwise the returned handle is a no-op restorer.
func newQuietHandle(outputFormat, audience string) *quietHandle {
h := &quietHandle{}
if outputFormat == "json" || audience == "agent" {
h.fn = stdout.Quiet()
}
return h
}
// Restore re-enables stdout. Safe to call multiple times.
func (h *quietHandle) Restore() {
if h == nil || h.fn == nil {
return
}
h.fn()
h.fn = nil
}
// ResultProvider abstracts the metadata both internal/agent.Agent and
// internal/scan.Agent expose post-run, so emitRunResult can finalize
// either without knowing which kind it has.
type ResultProvider interface {
Diffs() []model.Diff
FilesReviewed() int64
TotalInputTokens() int64
TotalOutputTokens() int64
TotalTokensUsed() int64
TotalCacheReadTokens() int64
TotalCacheWriteTokens() int64
Warnings() []agent.AgentWarning
// ProjectSummary is the markdown project-level summary produced by
// scan's PROJECT_SUMMARY_TASK. Empty for review mode and for scans
// that skipped / failed the summary phase.
ProjectSummary() string
ToolCalls() map[string]int64
// SessionID returns the persisted session identifier so callers can show it
// in JSON output or failure diagnostics. Returns "" when no session was
// created.
SessionID() string
// BudgetExceeded reports whether the aggregate token budget gate stopped the
// run before all files were reviewed. It is a diagnostic signal only — it
// feeds summary.budget_exceeded and the failure usage record, and never
// decides the run's terminal state. The terminal state comes solely from the
// manifest's coverage: the stop marks the undispatched items
// failed(budget) without recording a run_failure, so it reads as partial
// whenever anything was covered.
BudgetExceeded() bool
// RunManifest returns the frozen v1 coverage result for review runs. Scan
// remains legacy and returns nil.
RunManifest() *session.RunManifest
}
type resumeInfoProvider interface {
ResumeInfo() *agent.ResumeInfo
}
// emitRunResult is the post-LLM-run finalization shared by `ocr review` and
// `ocr scan`: resolves comment line numbers, records telemetry, restores
// stdout early for agent-text audiences so the summary is visible, prints
// the trace summary, and writes the result in the requested format.
//
// q is the silencing handle returned by newQuietHandle; pass nil if no
// silencing was set up (in which case the early restore is a no-op).
func emitRunResult(
ctx context.Context,
ag ResultProvider,
comments []model.LlmComment,
startTime time.Time,
outputFormat, audience string,
q *quietHandle,
) error {
comments = diff.ResolveLineNumbers(comments, ag.Diffs())
duration := time.Since(startTime)
telemetry.RecordReviewDuration(ctx, duration)
if len(comments) > 0 {
telemetry.RecordCommentsGenerated(ctx, int64(len(comments)))
}
traceID := telemetry.TraceIDFromContext(ctx)
manifest := ag.RunManifest()
if outputFormat == "json" && manifest == nil && len(comments) == 0 && ag.FilesReviewed() == 0 {
return outputJSONNoFiles(traceID)
}
// Agent-text audiences need stdout back before PrintTraceSummary so the
// summary line lands on their terminal.
if audience == "agent" && outputFormat != "json" {
q.Restore()
}
if outputFormat != "json" {
telemetry.PrintTraceSummary(ag.FilesReviewed(), int64(len(comments)),
ag.TotalInputTokens(), ag.TotalOutputTokens(), ag.TotalTokensUsed(),
ag.TotalCacheReadTokens(), ag.TotalCacheWriteTokens(), duration)
}
if outputFormat == "json" {
var resumeInfo *agent.ResumeInfo
if p, ok := ag.(resumeInfoProvider); ok {
resumeInfo = p.ResumeInfo()
}
return outputJSONWithWarnings(comments, ag.Warnings(), ag.FilesReviewed(),
ag.TotalInputTokens(), ag.TotalOutputTokens(), ag.TotalTokensUsed(),
ag.TotalCacheReadTokens(), ag.TotalCacheWriteTokens(), duration,
ag.ProjectSummary(), ag.ToolCalls(), traceID, resumeInfo, ag.SessionID(), manifest, ag.BudgetExceeded())
}
outputTextWithWarnings(comments, ag.Warnings(), manifest)
if summary := ag.ProjectSummary(); summary != "" {
fmt.Printf("\n\n──────── Project Summary ────────\n\n%s\n", summary)
}
return nil
}