openclaw/security
Jesse Merhi d7627d6f4c
refactor(prompt): use plain inbound context labels and drop system-tag sanitizer (#112000)
* refactor(prompt): plain inbound context labels with a provenance marker

Replaces trust-worded inbound context labels ("(untrusted metadata)",
"(untrusted, for context)") with plain labels plus a fixed provenance
marker suffix appended to every OpenClaw-injected context header.

Detection keys on the marker, not label text, so strippers stay correct
across UI, TUI, replay, /trace segmentation, memory recall, and the Swift
chat preprocessor. Drops sanitizeInboundSystemTags in favor of the marker
boundary plus trusted system-prompt narration.

Renames the untrusted-named plugin SDK context identifiers to
channel-provenance names, keeping deprecated aliases registered for
removal after 2026-09-08.

Adds `openclaw doctor --fix` migrations that rewrite legacy inbound
labels in stored SQLite transcripts and purge legacy envelope-
contaminated LanceDB recall rows.

* fix(ci): resolve gate failures for plain inbound context labels

- doctor sqlite readers: open read-only connections via openNodeSqliteDatabase
  so the Kysely connection-boundary guardrail holds; unexport the now-internal
  transcript snapshot type (Knip unused-export gate).
- compat registry: split the record table into registry-records.ts and
  plugin-sdk-subpath-records.ts. The new compat record pushed registry.ts past
  the 700-line oxlint cap; suppressions are disallowed, so follow the existing
  sibling record-module pattern. Public exports and PluginCompatCode literals
  unchanged.
- acp-runtime test: assert current finalization behavior (newline normalization
  only). The bracket de-fang and System: rewrite it expected were removed with
  sanitizeInboundSystemTags; forged system lines are neutralized at the
  system-event queue, the single chokepoint feeding the System:-per-line render.
- regenerate docs_map and the plugin SDK API baseline manifest.

* fix(prompt): harden inbound context label migration and drop in-band sanitizer

Review follow-ups on the plain-label + provenance-marker change:

- Remove src/security/system-tags.ts. Rewriting inbound text to neutralize
  look-alike `System:`/`[System]` markers corrupted legitimate user text and is
  not a real injection boundary; role separation plus external-content wrapping
  is. Explicit product decision, recorded at the system-event queue.
- Narrow the LanceDB legacy-row purge so it cannot delete benign memories. It
  now requires a complete known legacy sentinel line, a legacy label followed by
  a fenced JSON body, or the complete legacy external-content header. The prior
  predicates matched ordinary prose such as `Notes (untrusted metadata):`, and
  deletion is irreversible.
- Make explicit-empty canonical ChannelStructuredContext win over the deprecated
  alias via a present/absent result instead of collapsing `[]` to undefined.
- Keep `\r?` in the active-memory doctor rule. It is the only rule spanning the
  header's line break, migrated assistant rows skip newline normalization, and
  without it the marked-header replace wins and the body strips to empty. Added
  a CRLF regression test.
- Fix stale comments that described removed behavior, and cover the Swift
  prose-block strip path.

Claude-Session: https://claude.ai/code/session_01WNzsPddQmxy9Y7jKD4wAxH
2026-07-27 11:27:53 +10:00
..
opengrep refactor(prompt): use plain inbound context labels and drop system-tag sanitizer (#112000) 2026-07-27 11:27:53 +10:00
README.md

Security tooling

This directory holds OpenClaw's shipped OpenGrep security rulepack and the supporting tooling that validates and runs it. Maintainer-only advisory triage and detector-generation prompts live outside the public repo; this repo keeps the durable artifacts needed to block regressions in PRs and support local rule validation.

Layout

security/
├── README.md                              <- this file
└── opengrep/
    ├── README.md                          <- precise rulepack details + compile recipe
    └── precise.yml                        <- compiled super-config: precise rules

The related scripts are:

  • security/opengrep/compile-rules.mjs — gathers source OpenGrep rule YAMLs from a folder and appends new compiled rule IDs to security/opengrep/precise.yml.
  • security/opengrep/check-rule-metadata.mjs — enforces that every committed rule carries durable source/provenance metadata.
  • scripts/run-opengrep.sh — runs the compiled precise rulepack locally or in CI with consistent paths and exclusions.

Rule lifecycle

Maintainers investigate advisories and generate candidate rules outside the public repo. Once a candidate rule has been validated and reviewed, put the shippable source rule YAML in any local folder and compile it into this repo:

node security/opengrep/compile-rules.mjs \
  --rules-dir <folder-with-source-rule-yaml>

Commit the resulting security/opengrep/precise.yml diff. Durable rule provenance lives in each compiled rule's metadata and is checked by pnpm check:opengrep-rule-metadata.

Rule quality contract: precise rules must catch the vulnerable behavior they were written for, should be silent on corresponding fixed behavior when a fix exists, and should keep current findings limited to verified regressions or variants.

Writing precise OpenGrep rules

A rule is appropriate for security/opengrep/precise.yml only when the dangerous shape is stable enough to block PRs. Prefer, in order:

  1. Variant detector — source-to-sink or missing-guard detection across the same bug family.
  2. Scoped behavioral regression — a narrow subsystem-specific rule anchored on the affected API or trust boundary.
  3. Exact regression canary — a labelled canary for the original vulnerable shape when broader variants would be noisy.
  4. No OpenGrep rule — if runtime state, product policy, or external data is required to distinguish vulnerable and safe behavior.

Before compiling a rule, validate it against vulnerable/fixed/current code when those surfaces exist. Every current finding must be classified as a true original issue or true variant, or the rule must be tightened/dropped before it ships.

Running the rules locally

The wrapper script handles paths, exclusions, and output formatting so local scans match CI exactly.

scripts/run-opengrep.sh                 # precise rules, human output
scripts/run-opengrep.sh --json          # write .opengrep-out/precise.json
scripts/run-opengrep.sh --sarif         # write .opengrep-out/precise.sarif
scripts/run-opengrep.sh --changed       # scan changed first-party paths
scripts/run-opengrep.sh -- src/agents/  # scan a single dir

If you'd rather invoke opengrep directly, the equivalent is:

opengrep scan --no-strict --no-git-ignore \
  --config security/opengrep/precise.yml \
  src/ extensions/ apps/ packages/ scripts/

Both forms read .semgrepignore at the repo root automatically — that's the single source of truth for which paths are skipped (test files, fixtures, mocks, QA-tooling extensions, test-orchestration scripts, …). Add a glob there if a new test naming convention shows up.

Running the rules in CI

There are two OpenGrep workflows:

  • OpenGrep — PR Diff (.github/workflows/opengrep-precise.yml) runs on pull requests and executes scripts/run-opengrep.sh --changed --sarif --error so findings stay scoped to changed first-party paths.
  • OpenGrep — Full (.github/workflows/opengrep-precise-full.yml) is manual dispatch only and executes scripts/run-opengrep.sh --sarif --error across the full first-party source set for maintainers who want a repository-wide audit.

Both workflows:

  • Inherit the same .semgrepignore exclusions used by the local wrapper
  • Upload SARIF to GitHub Code Scanning under stable OpenGrep categories
  • Fail on precise findings so the rulepack acts as a regression firewall
  • Enforce committed rule provenance with pnpm check:opengrep-rule-metadata

Editing, silencing, or removing rules

precise.yml is the checked-in compiled rulepack. Prefer editing source rule YAML and recompiling instead of hand-editing compiled rules, because the compiler normalizes rule IDs, metadata, duplicates, and OpenGrep validation. The compiler appends new rule IDs by default; use --replace-precise only when intentionally rebuilding the rulepack from a complete source folder.

To drop a noisy rule:

  1. Delete the offending source rule from the local source-rule folder.
  2. Re-run node security/opengrep/compile-rules.mjs --rules-dir <folder-with-source-rule-yaml>.
  3. Commit the resulting security/opengrep/precise.yml diff.

To narrow a rule's path scope, edit the source rule's paths.include / paths.exclude fields in the same local artifact location and recompile.

Tracing a finding back to its source

Every compiled rule's id is <source-id>.<original-id>. For GHSA-backed rules, <source-id> is the lower-case GHSA ID. For other source-backed rules, use a stable source identifier without dots such as a CVE, OSV ID, internal advisory ID, or other review identifier. Rule metadata must include advisory-url, detector-bucket, and source-rule-id, plus either ghsa or advisory-id. New compilations also add source-file when available. pnpm check:opengrep-rule-metadata enforces these durable source fields so each committed rule is traceable without a separate committed manifest.