mirror of
https://github.com/openclaw/openclaw.git
synced 2026-10-03 01:29:56 +00:00
feat(typesafe): support local System One decision models (#154059)
* fix(typesafe): distribute as an official external plugin Exclude TypeSafe runtime from the core package and enroll @openclaw/typesafe in npm and ClawHub publication. Preserve the plugin ID, protected credentials, and decision-model configuration. Require the post-2026.9.5 decision API and document the pending supporting release. * fix(typesafe): register the official external install catalog Keep official package discovery and trust aligned with the TypeSafe distribution cutover. Existing catalog completeness and ClawHub-counterpart checks pass. * feat(typesafe): support local System One decision models Add an explicit loopback endpoint for Kev, without forwarding hosted credentials or changing managed proxy authority. Preserve strict answer validation while adapting local instructions, Score legends, and timing metadata. Keep the declared local model out of hosted inference. Document endpoint scope and runtime ownership. * fix(typesafe): validate local payload types and document Kev Preserve discriminated question types without assertions and regenerate config documentation for the requested baseUrl field. Add the pinned Kev setup and API-test commands exercised in live inference. Core and channel config budgets remain unchanged; the plugin budget grows by one intentional field. * docs(agents): allow exact synthetic scanner fixture qualification
This commit is contained in:
Peter Steinberger
GitHub
parent
894fb140a2
commit
5acfb83d9f
24 changed files with 727 additions and 68 deletions
|
|
@ -112,7 +112,7 @@ not authorize local execution or a broader test plan.
|
|||
- Bug fixes within the authorized task do not need renewed approval, including compatible SDK changes needed to restore intended behavior. Ask again for new configuration options, breaking public contracts, intentional changes to schemas, durability, retention, or permissions beyond the bug fix, paid services, or destructive actions. Preserve FIFO ordering, live-authority and integrity checks, and settlement of write-capable work.
|
||||
- Protocol/version bumps, dependency patches/overrides/vendor changes, paid services, releases, and publishing need explicit approval; fix/ship authority does not imply release authority. Advisory workflows require an explicit request for that security action.
|
||||
- Extended-stable is one line: the trailing completed month relative to `main`'s version. Older `.33+` lines retire when `main` advances another month; publishing a retired line needs an explicit maintainer decision, not a routine guard bypass.
|
||||
- Baseline, snapshot, ignore, and expected-failure exceptions need approval; exact shrink-only ratchet updates are maintenance.
|
||||
- Baseline, snapshot, ignore, and expected-failure exceptions need approval, except narrow scanner qualifications for verified synthetic test fixtures. Agents may add these within the authorized task without asking when matching is bound to exact fixture bytes and source location, scanning and verification stay enabled, and proof rejects changed inputs. Broad suppressions and uncertain or real credentials still require approval. Exact shrink-only ratchet updates are maintenance.
|
||||
- `CODEOWNERS` routes review; check live GitHub enforcement. Restricted/security paths and material product, behavior, security, or ownership changes need listed-owner involvement. For ownership/review governance, verified active organization-admin direction also qualifies; repository admin/bypass alone does not. Neither route waives enforced reviews.
|
||||
- Complete the authorized workflow's review/merge gates; resolve substantive findings or explain rejections. Address failures under the best-effort test-failure policy above; distinguish proven unrelated failures from unexplained ones and cite an owning fix when known. Verify remote outcomes before success or cleanup; uncertain writes require reconciliation, not blind retries.
|
||||
- Stage only intended files and use concise Conventional Commits with verified author/writer identities. Preserve contributor credit; team-session credit requires consented, verified humans and its canonical backlink. A bare URL grants no public mutation authority. Keep PR bodies current with problem, solution, impact, and evidence; use body files/heredocs for shell-sensitive text.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
{
|
||||
"core": 2449,
|
||||
"channel": 3740,
|
||||
"plugin": 4330
|
||||
"plugin": 4331
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
8b9f9600edac3554646c57753462f902706062466a6fd5176778232a52c46cd8 config-baseline.json
|
||||
b852dd7bc95d5248478373132f942dfa1f93a1ce4f8ab4441da9e7d94e95e008 config-baseline.json
|
||||
332b2736b83f706e63961b6ac5cd958931da506b1c83f48450341d5425d3fc26 config-baseline.core.json
|
||||
51c84b118b136dfff84993e3cee0ff76ee06b614842b125d0fde110635fc10b8 config-baseline.channel.json
|
||||
ccb4e05cc4d2d5d39458f1fd1718044382c4561327a5adf85e89a57911dd022d config-baseline.plugin.json
|
||||
ab4f44747880f8979d8bbf1ad5d35d9569c44543410f1233d63973449b4fe4f5 config-baseline.plugin.json
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ meets a condition.
|
|||
model architectures and inference backends. Sharing the interface does not make
|
||||
their reasoning ability or probabilities interchangeable.
|
||||
|
||||
The role and bundled TypeSafe AI adapter were added after released OpenClaw
|
||||
The role and TypeSafe AI adapter were added after released OpenClaw
|
||||
`2026.9.5`. These instructions apply to development checkouts containing those
|
||||
features and to later releases that include them. See each provider's setup
|
||||
page for its host requirements.
|
||||
|
|
@ -41,9 +41,10 @@ Configure the provider plugin before selecting its model:
|
|||
- [ONNX](/plugins/onnx) runs local CPU classifiers in a persistent subprocess.
|
||||
Follow its development-checkout or compatible-package setup, then explicitly
|
||||
download the model or prepare a local export. Inference needs no hosted API credential.
|
||||
- [TypeSafe AI](/plugins/typesafe) connects to hosted Jev inference. Enable the
|
||||
bundled plugin and configure its protected credential. Evaluations send the
|
||||
selected evidence to TypeSafe and incur its normal usage charges.
|
||||
- [TypeSafe AI](/plugins/typesafe) connects to hosted Jev or a local System One
|
||||
server such as Kev. Install and enable the external plugin, then configure a
|
||||
protected hosted credential or an explicit loopback `baseUrl`. Hosted
|
||||
evaluations send the selected evidence to TypeSafe and incur its normal usage charges.
|
||||
|
||||
The current plugins declare these model references:
|
||||
|
||||
|
|
@ -58,8 +59,9 @@ The current plugins declare these model references:
|
|||
| `onnx/gliner2.5-small-v1` | GLiNER 2.5 Small | Download pinned ONNX artifacts |
|
||||
| `typesafe/jev-1.13.0` | Jev 1.13.0 | TypeSafe credential |
|
||||
| `typesafe/jev-latest` | Jev | TypeSafe credential; follows the vendor's latest model |
|
||||
| `typesafe/kev-latest` | Kev (local server) | Running System One server and explicit loopback URL |
|
||||
|
||||
ONNX support is currently an unpublished candidate. Its plugin page explains
|
||||
Both plugins are currently unpublished candidates. Their setup pages explain
|
||||
source-checkout use and the packaged host floor. The table describes the plugins' declared models,
|
||||
not which artifacts or credentials are ready on your machine.
|
||||
|
||||
|
|
|
|||
|
|
@ -50,7 +50,7 @@ Each entry lists the package, distribution route, and description.
|
|||
|
||||
## Core npm package
|
||||
|
||||
63 plugins
|
||||
62 plugins
|
||||
|
||||
- **[a2a](/plugins/reference/a2a)** (`@openclaw/a2a`) - included in OpenClaw. A2A v1.0 Agent-to-Agent protocol channel plugin.
|
||||
|
||||
|
|
@ -164,8 +164,6 @@ Each entry lists the package, distribution route, and description.
|
|||
|
||||
- **[tts-local-cli](/plugins/reference/tts-local-cli)** (`@openclaw/tts-local-cli`) - included in OpenClaw. Adds text-to-speech provider support.
|
||||
|
||||
- **[typesafe](/plugins/reference/typesafe)** (`@openclaw/typesafe`) - included in OpenClaw. Optional typed evaluations, not a conversational model provider.
|
||||
|
||||
- **[vault](/plugins/reference/vault)** (`@openclaw/vault`) - included in OpenClaw. HashiCorp Vault SecretRef provider integration.
|
||||
|
||||
- **[vllm](/plugins/reference/vllm)** (`@openclaw/vllm-provider`) - included in OpenClaw. Adds vLLM model provider support to OpenClaw.
|
||||
|
|
@ -180,7 +178,7 @@ Each entry lists the package, distribution route, and description.
|
|||
|
||||
## Official external packages
|
||||
|
||||
94 plugins
|
||||
95 plugins
|
||||
|
||||
- **[acpx](/plugins/reference/acpx)** (`@openclaw/acpx`) - npm or ClawHub: `clawhub:@openclaw/acpx`. OpenClaw ACP runtime backend with plugin-owned session and transport management.
|
||||
|
||||
|
|
@ -346,6 +344,8 @@ Each entry lists the package, distribution route, and description.
|
|||
|
||||
- **[twitch](/plugins/reference/twitch)** (`@openclaw/twitch`) - npm or ClawHub: `clawhub:@openclaw/twitch`. OpenClaw Twitch channel plugin for chat and moderation workflows.
|
||||
|
||||
- **[typesafe](/plugins/reference/typesafe)** (`@openclaw/typesafe`) - npm or ClawHub: `clawhub:@openclaw/typesafe`. Optional typed evaluations, not a conversational model provider.
|
||||
|
||||
- **[venice](/plugins/reference/venice)** (`@openclaw/venice-provider`) - npm or ClawHub: `clawhub:@openclaw/venice-provider`. Adds Venice model provider support to OpenClaw.
|
||||
|
||||
- **[vercel-ai-gateway](/plugins/reference/vercel-ai-gateway)** (`@openclaw/vercel-ai-gateway-provider`) - npm or ClawHub: `clawhub:@openclaw/vercel-ai-gateway-provider`. Adds Vercel AI Gateway model provider support to OpenClaw.
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ Optional typed evaluations, not a conversational model provider.
|
|||
## Distribution
|
||||
|
||||
- Package: `@openclaw/typesafe`
|
||||
- Install route: included in OpenClaw
|
||||
- Install route: npm or ClawHub: `clawhub:@openclaw/typesafe`
|
||||
|
||||
## Surface
|
||||
|
||||
|
|
|
|||
|
|
@ -153,9 +153,10 @@ sent to the selected provider may incur its normal usage charges. Plugin disable
|
|||
wins; installing a tool or credential alone does not select a provider. Vendor adapters
|
||||
own transport and model-specific translation; no vendor is a core dependency.
|
||||
|
||||
The [ONNX plugin](/plugins/onnx) supplies local classifiers; the bundled
|
||||
[TypeSafe AI plugin](/plugins/typesafe) supplies a Jev adapter. Both require
|
||||
explicit setup and role selection.
|
||||
The [ONNX plugin](/plugins/onnx) supplies local classifiers; the
|
||||
[TypeSafe AI plugin](/plugins/typesafe) supplies hosted Jev and local System One
|
||||
adapters, including Kev. Both plugins require
|
||||
separate installation, explicit setup, and role selection.
|
||||
|
||||
### Calling from a third-party plugin
|
||||
|
||||
|
|
|
|||
|
|
@ -1,31 +1,54 @@
|
|||
---
|
||||
summary: "Use TypeSafe AI's Jev model for optional typed decisions"
|
||||
summary: "Use hosted Jev or a local System One server for typed decisions"
|
||||
title: "TypeSafe AI"
|
||||
read_when:
|
||||
- Configuring a typed decision model
|
||||
- Using the TypeSafe evaluation tool
|
||||
- Running Kev locally through the System One API
|
||||
---
|
||||
|
||||
# TypeSafe AI
|
||||
|
||||
The bundled `typesafe` plugin connects OpenClaw's optional decision model role to
|
||||
TypeSafe AI's Jev models. Its models appear in the separate **Decision** picker,
|
||||
never in the conversational model picker.
|
||||
The official external `typesafe` plugin connects OpenClaw's optional decision
|
||||
model role to TypeSafe AI's hosted Jev models or an explicitly configured local
|
||||
System One server such as [Kev](https://github.com/jaredpalmer/kev). Its models appear in the separate
|
||||
**Decision** picker, never in the conversational model picker.
|
||||
|
||||
The bundled adapter and decision model role were added after released OpenClaw
|
||||
`2026.9.5`. Use a development checkout containing these features, or a later
|
||||
release that includes them; enabling this configuration on `2026.9.5` does not
|
||||
add the missing API or bundled plugin.
|
||||
The adapter and decision model role were added after released OpenClaw
|
||||
`2026.9.5`. Packaged installs require a host and plugin API of at least
|
||||
`2026.9.6`; the installer rejects older hosts before loading the plugin.
|
||||
|
||||
See [Decision models](/concepts/decision-models) for the model role, available
|
||||
backends, rubric examples, and provider-neutral plugin API.
|
||||
|
||||
The plugin is disabled by default. Bundling or enabling it does not select a
|
||||
The plugin is disabled by default. Installing or enabling it does not select a
|
||||
decision model or schedule background work.
|
||||
|
||||
## Install
|
||||
|
||||
TypeSafe AI is packaged separately from core for publication to npm and
|
||||
ClawHub. Its first publication is pending a supporting release. Once published,
|
||||
install it from npm on a compatible host:
|
||||
|
||||
```sh
|
||||
openclaw plugins install @openclaw/typesafe
|
||||
```
|
||||
|
||||
To select ClawHub explicitly:
|
||||
|
||||
```sh
|
||||
openclaw plugins install clawhub:@openclaw/typesafe
|
||||
```
|
||||
|
||||
Until a supporting release is available, use a source checkout containing the
|
||||
decision-provider API and `extensions/typesafe`. Build it with
|
||||
`pnpm install --frozen-lockfile` and `pnpm build`, then apply the configuration
|
||||
below. Source-checkout plugins use the host's co-versioned development API;
|
||||
that does not make the packaged plugin compatible with OpenClaw `2026.9.5`.
|
||||
|
||||
## Enable and configure
|
||||
|
||||
Create a protected credential in Settings → Secrets, then reference it from the
|
||||
For hosted Jev, create a protected credential in Settings → Secrets, then reference it from the
|
||||
plugin configuration. Merge this example into your existing configuration; keep
|
||||
any other entries in `plugins.allow`.
|
||||
|
||||
|
|
@ -57,16 +80,116 @@ any other entries in `plugins.allow`.
|
|||
`agents.defaults.decisionModel`; an empty override disables decisions for that
|
||||
agent. An unset or empty global role leaves decisions off by default.
|
||||
|
||||
The plugin reads the host's prepared SecretRef value for each request. It does
|
||||
Hosted mode reads the host's prepared SecretRef value for each request. It does
|
||||
not independently read environment credentials or cache a previous credential.
|
||||
A missing or unavailable credential makes decisions unavailable. Use the normal
|
||||
[secret refresh flow](/gateway/secrets) after changing a credential.
|
||||
|
||||
Selecting a decision model authorizes supported, otherwise-enabled consumers to
|
||||
send their selected evidence to TypeSafe and incur its normal usage charges.
|
||||
send their selected evidence to the configured endpoint. Hosted Jev requests
|
||||
incur TypeSafe's normal usage charges.
|
||||
Consumer scheduling and publication permissions remain unchanged. Clearing the
|
||||
role or explicitly disabling the plugin prevents its use by those consumers.
|
||||
|
||||
## Local System One server
|
||||
|
||||
### Run Kev
|
||||
|
||||
[Kev](https://github.com/jaredpalmer/kev) is an Apache-2.0 family of decision
|
||||
models that serves the System One API through a persistent Python process.
|
||||
It supports Apple Silicon and CUDA. The Qwen3-based Kev-0.6B, Kev-4B, and
|
||||
Kev-8B checkpoints have been tested with this adapter.
|
||||
|
||||
For a Mac, start with the Qwen3-based Kev-4B checkpoint. This example requires
|
||||
Python 3.12+ and [uv](https://docs.astral.sh/uv/), and pins the tested adapter
|
||||
revision in a local directory. The first server start also downloads its base
|
||||
model weights:
|
||||
|
||||
```sh
|
||||
git clone https://github.com/jaredpalmer/kev.git
|
||||
cd kev
|
||||
git checkout 5f78968927069eaacc3b2bdb688586989b3933ac
|
||||
uv sync --frozen --extra serve
|
||||
uv run python - <<'PY'
|
||||
from huggingface_hub import snapshot_download
|
||||
|
||||
snapshot_download(
|
||||
"jaredpalmer/kev-4b",
|
||||
revision="c4bfa11b0dc07691884f2d97f1c4c4c05c92e416",
|
||||
local_dir="models/kev-4b-qwen3",
|
||||
)
|
||||
PY
|
||||
KEV_DTYPE=bf16 uv run --extra serve python -m kev.serve \
|
||||
--run models/kev-4b-qwen3 --port 8009
|
||||
```
|
||||
|
||||
The newer default Kev-4B checkpoint uses Qwen3.5; its Mac performance differs
|
||||
from the Qwen3 checkpoint above. Follow the upstream model cards when choosing
|
||||
another checkpoint. Kev-0.6B uses less memory; Kev-8B trades more memory and
|
||||
latency for decision quality. All of them use the same OpenClaw model label
|
||||
for the server you configure below.
|
||||
|
||||
From the same Kev checkout in a second terminal, verify the loaded checkpoint
|
||||
and run Kev's API tests:
|
||||
|
||||
```sh
|
||||
curl --fail http://127.0.0.1:8009/v1/models
|
||||
KEV_BASE_URL=http://127.0.0.1:8009 \
|
||||
uv run --extra serve python -m pytest tests/test_api.py -q
|
||||
```
|
||||
|
||||
### Connect OpenClaw
|
||||
|
||||
Start your System One server separately, then set `baseUrl` to its loopback
|
||||
origin and select `typesafe/kev-latest`:
|
||||
|
||||
```json5
|
||||
{
|
||||
plugins: {
|
||||
allow: ["typesafe"],
|
||||
entries: {
|
||||
typesafe: {
|
||||
enabled: true,
|
||||
config: { baseUrl: "http://127.0.0.1:8009" },
|
||||
},
|
||||
},
|
||||
},
|
||||
agents: {
|
||||
defaults: { decisionModel: "typesafe/kev-latest" },
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
Merge the example with existing settings, preserving other allowed plugins.
|
||||
Omit `apiKey` for local inference. The plugin does not read or send the hosted
|
||||
credential on this path; remove a retained SecretRef if the host should also
|
||||
stop preparing it.
|
||||
|
||||
The endpoint applies to every request from this plugin, including requests
|
||||
whose model label names Jev. Model selection does not choose between hosted and
|
||||
local endpoints. The `kev-latest` label requires `baseUrl` and is never sent to
|
||||
the hosted TypeSafe endpoint.
|
||||
|
||||
`baseUrl` accepts HTTP or HTTPS on `localhost`, `127.0.0.1`, or `[::1]`, with an
|
||||
optional port and trailing slash. Supply the origin, without `/v1`, credentials,
|
||||
query, or fragment; the plugin appends `/v1/systemone`. LAN and remote hosts are
|
||||
not accepted. Ordinary ambient HTTP proxy variables are not used for these
|
||||
requests; explicitly enabled managed proxy policy still applies.
|
||||
|
||||
Kev runs one checkpoint per server process. Its request model label does not
|
||||
load or switch weights. Choose the checkpoint when starting the server and
|
||||
inspect `GET /v1/models` to verify it. See Kev's [serving instructions](https://github.com/jaredpalmer/kev#quick-start)
|
||||
for installation, model selection, and hardware requirements. OpenClaw does not
|
||||
download weights or start that process. An unavailable server produces an
|
||||
unavailable decision, without automatically switching to hosted Jev.
|
||||
|
||||
For local compatibility, omitted question instructions are sent as `null`.
|
||||
Structured Score rubric levels are encoded as text; returned legends must match
|
||||
that transmitted rubric before the original level descriptions are restored in
|
||||
tool results. Kev's optional nonnegative `latency_ms` field is validated and
|
||||
removed; all answer types, labels, probabilities, and rubric bounds retain the
|
||||
same validation as hosted results.
|
||||
|
||||
## Decision contract
|
||||
|
||||
Consumers call the provider-neutral
|
||||
|
|
@ -94,7 +217,8 @@ not demonstrated accuracy guarantees or permission to act.
|
|||
The host owns concurrency, circuit health, deadlines, cancellation, and provider
|
||||
lifecycle. Native decisions have a ten-second maximum; shorter consumer or
|
||||
plugin timeouts still apply. The adapter shares transport and response validation
|
||||
with the tool below. Requests use the fixed TypeSafe HTTPS endpoint, reject
|
||||
with the tool below. Requests use the fixed TypeSafe HTTPS endpoint unless
|
||||
`baseUrl` selects a local server. Both paths reject
|
||||
redirects, and do not retry automatically. Consumers decide what to do with
|
||||
unavailable decisions; caller cancellation must not start fallback work.
|
||||
|
||||
|
|
@ -106,7 +230,8 @@ It accepts shared `state`, a map of `questions`, and an optional vendor `model`
|
|||
override. Its TypeSafe-facing question names are `choice`, `score`, and `noul`.
|
||||
|
||||
For this tool only, `plugins.entries.typesafe.config.model` supplies the default
|
||||
vendor model, initially `jev-latest`. It does not override the native
|
||||
model, initially `jev-latest` for hosted inference or `kev-latest` for local
|
||||
inference. It does not override the native
|
||||
`decisionModel` role or select a provider. Pin a model version for reproducible
|
||||
tool evaluations. `timeoutMs` limits tool requests and caps native requests at
|
||||
the shorter of this setting and the host's remaining deadline.
|
||||
|
|
@ -118,9 +243,11 @@ publish, send messages, or change durable state.
|
|||
|
||||
## Existing external installation
|
||||
|
||||
This bundled plugin uses the same `typesafe` plugin ID as the external prototype.
|
||||
Do not configure two installations as independent providers. Inspect plugin
|
||||
resolution before switching, preserve existing configuration and credentials,
|
||||
and use the supported [plugin management flow](/plugins/manage-plugins) to remove
|
||||
an external override if you want the bundled copy to own the ID. Installing this
|
||||
change does not delete external plugin files or credentials.
|
||||
The official package keeps the `typesafe` plugin ID used by the prototype and
|
||||
earlier development checkouts. Preserve `plugins.entries.typesafe`, its
|
||||
protected credential, and agent `decisionModel` selections when switching.
|
||||
Use the supported [plugin management flow](/plugins/manage-plugins) to replace
|
||||
the old installation, and remove an explicit prototype path from
|
||||
`plugins.load.paths` if it would override the installed package. Do not configure
|
||||
two copies as independent providers. Installing the package does not delete
|
||||
prototype files or credentials.
|
||||
|
|
|
|||
28
extensions/typesafe/README.md
Normal file
28
extensions/typesafe/README.md
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
# TypeSafe AI for OpenClaw
|
||||
|
||||
Official external plugin for typed decisions with hosted TypeSafe AI Jev models
|
||||
or a local Kev System One server.
|
||||
It provides Choice, Score, and Boolean judgments through OpenClaw's shared
|
||||
decision-model API, plus the optional `typesafe_evaluate` tool.
|
||||
|
||||
Requires OpenClaw and plugin API **2026.9.6 or later**. Released OpenClaw
|
||||
2026.9.5 does not include the decision API.
|
||||
|
||||
```sh
|
||||
openclaw plugins install @openclaw/typesafe
|
||||
```
|
||||
|
||||
The ClawHub install spec is `clawhub:@openclaw/typesafe`. First publication is
|
||||
pending a supporting release. Enable the plugin, configure a protected TypeSafe credential, and select
|
||||
`typesafe/jev-latest` as your agent's `decisionModel`. Evaluations send the
|
||||
supplied evidence to TypeSafe AI and incur its normal usage charges.
|
||||
|
||||
For local Kev, configure `plugins.entries.typesafe.config.baseUrl` with the
|
||||
server's loopback origin, such as `http://127.0.0.1:8009`, omit `apiKey`, and select
|
||||
`typesafe/kev-latest`. The plugin calls `/v1/systemone` without a hosted credential.
|
||||
Start the server separately with your chosen checkpoint; the decision-model
|
||||
selection labels requests and does not download or load a model.
|
||||
|
||||
See the [TypeSafe AI setup guide](https://docs.openclaw.ai/plugins/typesafe) and
|
||||
[decision-model documentation](https://docs.openclaw.ai/concepts/decision-models)
|
||||
for configuration, rubrics, and API semantics.
|
||||
|
|
@ -19,14 +19,14 @@ export default definePluginEntry({
|
|||
name: "typesafe_evaluate",
|
||||
label: "TypeSafe typed decisions",
|
||||
description:
|
||||
"Ask Jev for typed decisions over explicit shared state: classify/select with Choice (2–255 options), rate with Score (2–10 ordered levels; fractional zero-based result), or estimate probability of yes with Noul (optional true/false criteria). Batch independent questions; they cannot see each other’s answers. Instructions and descriptions accept text, JSON objects/arrays, or null. Returns distributions, confidence for Choice/Score, model, and usage—not generated explanations or authorization. Requires credentials; sends supplied data to TypeSafe and may incur API charges.",
|
||||
"Ask the configured System One model for typed decisions over explicit shared state: classify/select with Choice (2–255 options), rate with Score (2–10 ordered levels; fractional zero-based result), or estimate probability of yes with Noul (optional true/false criteria). Batch independent questions; they cannot see each other’s answers. Instructions and descriptions accept text, JSON objects/arrays, or null. Returns distributions, confidence for Choice/Score, model, and usage—not generated explanations or authorization. Hosted Jev requires credentials and may incur API charges; a configured local Kev endpoint receives supplied data without hosted credentials.",
|
||||
parameters: EvaluateInput,
|
||||
outputSchema: EvaluateOutput,
|
||||
resultContentSource: "network",
|
||||
async execute(_id, params, signal) {
|
||||
signal?.throwIfAborted();
|
||||
const config = resolveRuntimeConfig(api.runtime.config.current());
|
||||
if (!config.apiKey) {
|
||||
if (!config.baseUrl && !config.apiKey) {
|
||||
throw new Error(
|
||||
"TypeSafe API key is missing. Configure a SecretRef in plugin Settings.",
|
||||
);
|
||||
|
|
|
|||
|
|
@ -37,16 +37,20 @@
|
|||
"uiHints": {
|
||||
"apiKey": {
|
||||
"label": "TypeSafe API credential",
|
||||
"help": "Select a SecretRef. Create protected credentials in Settings \u2192 Secrets. Never paste credentials in chat.",
|
||||
"help": "Hosted Jev only. Select a SecretRef in Settings \u2192 Secrets. Ignored for a local baseUrl. Never paste credentials in chat.",
|
||||
"sensitive": true
|
||||
},
|
||||
"model": {
|
||||
"label": "Evaluation tool default model",
|
||||
"help": "Default only for the optional typesafe_evaluate tool. Native decisions use the agent decisionModel selection."
|
||||
"help": "Default only for typesafe_evaluate: jev-latest for hosted Jev, kev-latest for local System One. Local model labels do not load checkpoints. Native decisions use the agent decisionModel selection."
|
||||
},
|
||||
"timeoutMs": {
|
||||
"label": "Request timeout (ms)",
|
||||
"advanced": true
|
||||
},
|
||||
"baseUrl": {
|
||||
"label": "Local System One origin",
|
||||
"help": "Optional loopback origin, such as http://127.0.0.1:8009, for a running Kev server. Omit for hosted Jev. Local requests do not use the TypeSafe credential; the server chooses the loaded checkpoint."
|
||||
}
|
||||
},
|
||||
"configSchema": {
|
||||
|
|
@ -98,13 +102,18 @@
|
|||
"minLength": 1,
|
||||
"maxLength": 128,
|
||||
"pattern": "^[a-zA-Z0-9._/-]+$",
|
||||
"default": "jev-latest"
|
||||
"description": "Tool default: jev-latest for hosted Jev, kev-latest for a local endpoint."
|
||||
},
|
||||
"timeoutMs": {
|
||||
"type": "integer",
|
||||
"minimum": 1000,
|
||||
"maximum": 60000,
|
||||
"default": 10000
|
||||
},
|
||||
"baseUrl": {
|
||||
"type": "string",
|
||||
"maxLength": 128,
|
||||
"pattern": "^https?://(?:localhost|127\\.0\\.0\\.1|\\[::1\\])(?::[0-9]{1,5})?/?$"
|
||||
}
|
||||
},
|
||||
"additionalProperties": false
|
||||
|
|
@ -122,6 +131,11 @@
|
|||
"provider": "typesafe",
|
||||
"id": "jev-1.13.0",
|
||||
"name": "Jev 1.13.0"
|
||||
},
|
||||
{
|
||||
"provider": "typesafe",
|
||||
"id": "kev-latest",
|
||||
"name": "Kev (local server)"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,8 +1,11 @@
|
|||
{
|
||||
"name": "@openclaw/typesafe",
|
||||
"version": "2026.9.4",
|
||||
"private": true,
|
||||
"version": "2026.9.5",
|
||||
"description": "OpenClaw TypeSafe typed decisions and optional evaluation tool",
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "https://github.com/openclaw/openclaw"
|
||||
},
|
||||
"type": "module",
|
||||
"dependencies": {
|
||||
"typebox": "1.3.30"
|
||||
|
|
@ -13,6 +16,23 @@
|
|||
"openclaw": {
|
||||
"extensions": [
|
||||
"./index.ts"
|
||||
]
|
||||
],
|
||||
"install": {
|
||||
"clawhubSpec": "clawhub:@openclaw/typesafe",
|
||||
"npmSpec": "@openclaw/typesafe",
|
||||
"defaultChoice": "npm",
|
||||
"minHostVersion": ">=2026.9.6"
|
||||
},
|
||||
"compat": {
|
||||
"pluginApi": ">=2026.9.6"
|
||||
},
|
||||
"build": {
|
||||
"bundledDist": false,
|
||||
"openclawVersion": "2026.9.5"
|
||||
},
|
||||
"release": {
|
||||
"publishToClawHub": true,
|
||||
"publishToNpm": true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
name: typesafe-evaluate
|
||||
description: Make explicit typed TypeSafe/Jev decisions with the typesafe_evaluate tool.
|
||||
description: Make explicit typed hosted Jev or local Kev decisions with the typesafe_evaluate tool.
|
||||
---
|
||||
|
||||
# TypeSafe evaluations
|
||||
|
|
@ -8,7 +8,9 @@ description: Make explicit typed TypeSafe/Jev decisions with the typesafe_evalua
|
|||
Use `typesafe_evaluate` for semantic decisions over explicit supplied state. It
|
||||
returns typed decisions, not generated explanations or permission to act. If the
|
||||
tool is unavailable, report that; do not substitute shell/HTTP calls or ask for a
|
||||
credential in chat. Calls send the supplied data to TypeSafe and may incur charges.
|
||||
credential in chat. Hosted calls send the supplied data to TypeSafe and may incur
|
||||
charges. With a configured local System One origin, calls go to that server
|
||||
without a hosted credential.
|
||||
|
||||
Supply `{state, questions}`; see [mixed request example](references/request-example.json).
|
||||
State and descriptions accept text, JSON objects/arrays, or null. Each question
|
||||
|
|
@ -26,6 +28,8 @@ Only send necessary evidence authorized for sharing. Never include credentials.
|
|||
Do not silently truncate evidence or split competing Choice options to fit a
|
||||
request. The plugin bounds JSON to 4 MiB, 262144 nodes, and depth 64; vendor token
|
||||
limits are separate. A model override is optional; pin a version when comparing runs.
|
||||
For local Kev, the server's loaded checkpoint determines the model; an override
|
||||
only labels the request and does not change the checkpoint.
|
||||
|
||||
Reported probabilities may be rounded and need not sum exactly to one. Preserve the
|
||||
vendor-selected label and score; normalization or selecting the largest reported
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
import type { RuntimeConfig } from "./config.js";
|
||||
import { evaluationError } from "./errors.js";
|
||||
import { EvaluationError, evaluationError } from "./errors.js";
|
||||
import { localInput, parseLocalResult } from "./local.js";
|
||||
import { parseInput, parseResult } from "./schema.js";
|
||||
import { requestEvaluation } from "./transport.js";
|
||||
|
||||
|
|
@ -9,19 +10,30 @@ export async function evaluate(input: unknown, config: RuntimeConfig, signal?: A
|
|||
throw evaluationError(undefined, true);
|
||||
}
|
||||
const parsed = parseInput(input);
|
||||
if (!config.apiKey) {
|
||||
const model = parsed.model ?? config.model;
|
||||
if (model === "kev-latest" && !config.baseUrl) {
|
||||
throw new EvaluationError(
|
||||
"Kev requires a local System One server. Configure baseUrl in TypeSafe plugin Settings.",
|
||||
"unsupported-input",
|
||||
);
|
||||
}
|
||||
if (!config.baseUrl && !config.apiKey) {
|
||||
throw new Error("TypeSafe API key is missing. Configure a SecretRef in plugin Settings.");
|
||||
}
|
||||
try {
|
||||
const wireInput = config.baseUrl ? localInput(parsed) : parsed;
|
||||
const response = await requestEvaluation({
|
||||
body: { ...parsed, model: parsed.model ?? config.model },
|
||||
apiKey: config.apiKey,
|
||||
body: { ...wireInput, model },
|
||||
apiKey: config.baseUrl ? undefined : config.apiKey,
|
||||
baseUrl: config.baseUrl,
|
||||
timeoutMs: config.timeoutMs,
|
||||
signal,
|
||||
});
|
||||
signal?.throwIfAborted();
|
||||
const evaluation = parseResult(response, parsed);
|
||||
if (JSON.stringify(evaluation).includes(config.apiKey)) {
|
||||
const evaluation = config.baseUrl
|
||||
? parseLocalResult(response, wireInput, parsed)
|
||||
: parseResult(response, parsed);
|
||||
if (!config.baseUrl && config.apiKey && JSON.stringify(evaluation).includes(config.apiKey)) {
|
||||
throw new Error("Invalid TypeSafe response.");
|
||||
}
|
||||
return { evaluation };
|
||||
|
|
|
|||
|
|
@ -1,8 +1,12 @@
|
|||
import { Type } from "typebox";
|
||||
|
||||
const DEFAULT_MODEL = "jev-latest";
|
||||
const LOCAL_BASE_URL_PATTERN =
|
||||
"^https?://(?:localhost|127\\.0\\.0\\.1|\\[::1\\])(?::[0-9]{1,5})?/?$";
|
||||
const localBaseUrlPattern = new RegExp(LOCAL_BASE_URL_PATTERN);
|
||||
export const ConfigSchema = Type.Object(
|
||||
{
|
||||
baseUrl: Type.Optional(Type.String({ maxLength: 128, pattern: LOCAL_BASE_URL_PATTERN })),
|
||||
apiKey: Type.Optional(
|
||||
Type.Object(
|
||||
{
|
||||
|
|
@ -23,7 +27,7 @@ export const ConfigSchema = Type.Object(
|
|||
minLength: 1,
|
||||
maxLength: 128,
|
||||
pattern: "^[a-zA-Z0-9._/-]+$",
|
||||
default: DEFAULT_MODEL,
|
||||
description: "Tool default: jev-latest for hosted Jev, kev-latest for a local endpoint.",
|
||||
}),
|
||||
),
|
||||
timeoutMs: Type.Optional(Type.Integer({ minimum: 1000, maximum: 60000, default: 10000 })),
|
||||
|
|
@ -31,12 +35,29 @@ export const ConfigSchema = Type.Object(
|
|||
{ additionalProperties: false },
|
||||
);
|
||||
|
||||
export type RuntimeConfig = { apiKey?: string; model: string; timeoutMs: number };
|
||||
export type RuntimeConfig = { apiKey?: string; baseUrl?: string; model: string; timeoutMs: number };
|
||||
|
||||
/** A configured endpoint grants access to one loopback origin, never arbitrary private hosts. */
|
||||
export function localBaseUrl(value: unknown): string | undefined {
|
||||
if (value === undefined) {
|
||||
return undefined;
|
||||
}
|
||||
try {
|
||||
if (typeof value !== "string" || value !== value.trim() || !localBaseUrlPattern.test(value)) {
|
||||
throw new Error();
|
||||
}
|
||||
return new URL(value).origin;
|
||||
} catch {
|
||||
throw new Error(
|
||||
"Invalid TypeSafe baseUrl; use an http(s) loopback origin without a path, credentials, query, or fragment.",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/** Validate runtime settings and recognize materialized credentials without resolving inputs. */
|
||||
export function runtimeConfig(config: Record<string, unknown> | undefined): RuntimeConfig {
|
||||
const key = config?.apiKey;
|
||||
const model = config?.model ?? DEFAULT_MODEL;
|
||||
const baseUrl = localBaseUrl(config?.baseUrl);
|
||||
const model = config?.model ?? (baseUrl ? "kev-latest" : DEFAULT_MODEL);
|
||||
const timeoutMs = config?.timeoutMs ?? 10000;
|
||||
if (
|
||||
typeof model !== "string" ||
|
||||
|
|
@ -48,5 +69,9 @@ export function runtimeConfig(config: Record<string, unknown> | undefined): Runt
|
|||
) {
|
||||
throw new Error("Invalid TypeSafe configuration; check plugin Settings.");
|
||||
}
|
||||
if (baseUrl) {
|
||||
return { baseUrl, model, timeoutMs };
|
||||
}
|
||||
const key = config?.apiKey;
|
||||
return { apiKey: typeof key === "string" && key.trim() ? key : undefined, model, timeoutMs };
|
||||
}
|
||||
|
|
|
|||
|
|
@ -8,6 +8,9 @@ export function resolveRuntimeConfig(
|
|||
): RuntimeConfig {
|
||||
const configured = snapshot.plugins?.entries?.typesafe?.config;
|
||||
const validated = runtimeConfig(configured);
|
||||
if (validated.baseUrl) {
|
||||
return validated;
|
||||
}
|
||||
const prepared = getPreparedPluginSecretInput("typesafe", "apiKey");
|
||||
return { ...validated, apiKey: prepared.value };
|
||||
}
|
||||
|
|
|
|||
|
|
@ -13,11 +13,14 @@ export function createDecisionProvider(getConfig: () => RuntimeConfig): Decision
|
|||
return {
|
||||
id: "typesafe",
|
||||
contractVersion: 1,
|
||||
isReady: () => Boolean(getConfig().apiKey),
|
||||
isReady: () => {
|
||||
const config = getConfig();
|
||||
return Boolean(config.baseUrl || config.apiKey);
|
||||
},
|
||||
async evaluate(batch: DecisionBatch, context) {
|
||||
context.signal.throwIfAborted();
|
||||
const config = getConfig();
|
||||
if (!config.apiKey) {
|
||||
if (!config.baseUrl && !config.apiKey) {
|
||||
return { status: "unavailable", reason: "credentials-unavailable" };
|
||||
}
|
||||
const remaining = context.deadlineMonotonicMs - performance.now();
|
||||
|
|
|
|||
337
extensions/typesafe/src/local.transport.test.ts
Normal file
337
extensions/typesafe/src/local.transport.test.ts
Normal file
|
|
@ -0,0 +1,337 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { lookup } from "node:dns/promises";
|
||||
import { createServer } from "node:http";
|
||||
import type { DecisionProviderV1 } from "openclaw/plugin-sdk/decisions";
|
||||
import type { AnyAgentTool, OpenClawPluginApi } from "openclaw/plugin-sdk/plugin-entry";
|
||||
import { getPreparedPluginSecretInput } from "openclaw/plugin-sdk/secret-input-runtime";
|
||||
import { afterEach, beforeEach, expect, it, vi } from "vitest";
|
||||
import plugin from "../index.js";
|
||||
import { evaluate } from "./client.js";
|
||||
import { runtimeConfig } from "./config.js";
|
||||
import type { EvaluationInput } from "./schema.js";
|
||||
|
||||
vi.mock("openclaw/plugin-sdk/secret-input-runtime", () => ({
|
||||
getPreparedPluginSecretInput: vi.fn(),
|
||||
}));
|
||||
vi.mock("node:dns/promises", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("node:dns/promises")>();
|
||||
return { ...actual, lookup: vi.fn(actual.lookup) };
|
||||
});
|
||||
|
||||
const baseUrl = "http://127.0.0.1:8009";
|
||||
const input = {
|
||||
state: { message: "Synthetic outage" },
|
||||
questions: {
|
||||
c: { type: "choice", criteria: { support: "Service outages", sales: null } },
|
||||
s: { type: "score", criteria: [null, { impact: ["widespread", true, 2] }] },
|
||||
b: { type: "noul", instructions: { question: "Escalate?" } },
|
||||
},
|
||||
} satisfies EvaluationInput;
|
||||
const localAnswer = {
|
||||
model: "kev-latest",
|
||||
answers: {
|
||||
c: {
|
||||
type: "choice",
|
||||
choice: "support",
|
||||
confidence: 0.8,
|
||||
probabilities: { support: 0.9, sales: 0.1 },
|
||||
},
|
||||
s: {
|
||||
type: "score",
|
||||
score: 0.8,
|
||||
confidence: 0.8,
|
||||
probabilities: { 0: 0.2, 1: 0.8 },
|
||||
legend: { 0: "", 1: '{"impact":["widespread",true,2]}' },
|
||||
},
|
||||
b: { type: "noul", noul: 0.75 },
|
||||
},
|
||||
usage: { input_tokens: 30, output_tokens: 12 },
|
||||
latency_ms: 15.7,
|
||||
};
|
||||
|
||||
beforeEach(() => {
|
||||
vi.mocked(getPreparedPluginSecretInput).mockReset();
|
||||
vi.mocked(getPreparedPluginSecretInput).mockReturnValue({ revision: 1, value: "hosted-secret" });
|
||||
});
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
vi.unstubAllEnvs();
|
||||
vi.mocked(lookup).mockReset();
|
||||
});
|
||||
|
||||
it("runs the registered tool and decision provider locally without reading hosted credentials", async () => {
|
||||
const fetch = vi.fn(
|
||||
async (_url: RequestInfo | URL, _init?: RequestInit) =>
|
||||
new Response(JSON.stringify(localAnswer)),
|
||||
);
|
||||
vi.stubGlobal("fetch", fetch);
|
||||
vi.stubEnv("HTTP_PROXY", "http://proxy.invalid:3128");
|
||||
vi.stubEnv("HTTPS_PROXY", "http://proxy.invalid:3128");
|
||||
vi.stubEnv("ALL_PROXY", "http://proxy.invalid:3128");
|
||||
vi.stubEnv("NO_PROXY", "");
|
||||
const registerTool = vi.fn<OpenClawPluginApi["registerTool"]>();
|
||||
const registerDecisionProvider = vi.fn<OpenClawPluginApi["registerDecisionProvider"]>();
|
||||
plugin.register({
|
||||
runtime: {
|
||||
config: {
|
||||
current: () => ({
|
||||
plugins: {
|
||||
entries: { typesafe: { config: { baseUrl, apiKey: "hosted-materialized" } } },
|
||||
},
|
||||
}),
|
||||
},
|
||||
},
|
||||
registerTool,
|
||||
registerDecisionProvider,
|
||||
} as unknown as OpenClawPluginApi);
|
||||
const tool = registerTool.mock.calls[0]?.[0] as AnyAgentTool;
|
||||
const provider = registerDecisionProvider.mock.calls[0]?.[0] as DecisionProviderV1;
|
||||
expect(provider.isReady?.()).toBe(true);
|
||||
const result = await tool.execute("local-test", input);
|
||||
expect(result.details).toEqual({
|
||||
evaluation: {
|
||||
model: "kev-latest",
|
||||
usage: localAnswer.usage,
|
||||
answers: {
|
||||
...localAnswer.answers,
|
||||
s: {
|
||||
...localAnswer.answers.s,
|
||||
legend: { 0: null, 1: input.questions.s.criteria[1] },
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
await expect(
|
||||
provider.evaluate(
|
||||
{
|
||||
...input,
|
||||
questions: { ...input.questions, b: { ...input.questions.b, type: "boolean" } },
|
||||
},
|
||||
{
|
||||
model: "kev-latest",
|
||||
signal: new AbortController().signal,
|
||||
deadlineMonotonicMs: performance.now() + 1000,
|
||||
},
|
||||
),
|
||||
).resolves.toMatchObject({
|
||||
status: "ok",
|
||||
result: {
|
||||
answers: {
|
||||
c: localAnswer.answers.c,
|
||||
s: { type: "score", score: 0.8, probabilities: [0.2, 0.8] },
|
||||
b: { type: "boolean", probabilityTrue: 0.75 },
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(getPreparedPluginSecretInput).not.toHaveBeenCalled();
|
||||
expect(fetch).toHaveBeenCalledTimes(2);
|
||||
for (const [url, init] of fetch.mock.calls) {
|
||||
expect(url).toBe(`${baseUrl}/v1/systemone`);
|
||||
expect(new Headers(init?.headers).has("authorization")).toBe(false);
|
||||
assert(typeof init?.body === "string");
|
||||
expect(JSON.parse(init.body)).toEqual({
|
||||
...input,
|
||||
model: "kev-latest",
|
||||
questions: {
|
||||
c: { ...input.questions.c, instructions: null },
|
||||
s: {
|
||||
...input.questions.s,
|
||||
instructions: null,
|
||||
criteria: ["", '{"impact":["widespread",true,2]}'],
|
||||
},
|
||||
b: input.questions.b,
|
||||
},
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
it("refuses to send the local Kev selection to hosted inference through registered handlers", async () => {
|
||||
const fetch = vi.fn(
|
||||
async () =>
|
||||
new Response(
|
||||
JSON.stringify({
|
||||
model: "kev-latest",
|
||||
answers: { q: { type: "noul", noul: 0.75 } },
|
||||
usage: localAnswer.usage,
|
||||
}),
|
||||
),
|
||||
);
|
||||
vi.stubGlobal("fetch", fetch);
|
||||
const registerTool = vi.fn<OpenClawPluginApi["registerTool"]>();
|
||||
const registerDecisionProvider = vi.fn<OpenClawPluginApi["registerDecisionProvider"]>();
|
||||
plugin.register({
|
||||
runtime: { config: { current: () => ({}) } },
|
||||
registerTool,
|
||||
registerDecisionProvider,
|
||||
} as unknown as OpenClawPluginApi);
|
||||
const provider = registerDecisionProvider.mock.calls[0]?.[0];
|
||||
assert(provider);
|
||||
await expect(
|
||||
provider.evaluate(
|
||||
{ state: "local-only evidence", questions: { q: { type: "boolean" } } },
|
||||
{
|
||||
model: "kev-latest",
|
||||
signal: new AbortController().signal,
|
||||
deadlineMonotonicMs: performance.now() + 10000,
|
||||
},
|
||||
),
|
||||
).resolves.toEqual({ status: "unavailable", reason: "unsupported-input" });
|
||||
const tool = registerTool.mock.calls[0]?.[0] as AnyAgentTool;
|
||||
await expect(
|
||||
tool.execute("local-test", {
|
||||
state: "local-only evidence",
|
||||
questions: { q: { type: "noul" } },
|
||||
model: "kev-latest",
|
||||
}),
|
||||
).rejects.toThrow("baseUrl");
|
||||
expect(fetch).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([
|
||||
"http://localhost:8009",
|
||||
"http://127.0.0.1:8009/",
|
||||
"http://[::1]:8009",
|
||||
"https://localhost",
|
||||
])("accepts an explicit loopback origin %s", (url) => {
|
||||
expect(runtimeConfig({ baseUrl: url, apiKey: "ignored-secret" })).toEqual({
|
||||
baseUrl: new URL(url).origin,
|
||||
model: "kev-latest",
|
||||
timeoutMs: 10000,
|
||||
});
|
||||
});
|
||||
|
||||
it.each([
|
||||
"",
|
||||
"http://192.168.1.2:8009",
|
||||
"https://remote.example",
|
||||
"http://localhost.example",
|
||||
"http://localhost:8009/v1",
|
||||
"http://localhost:8009?x=1",
|
||||
"http://localhost:8009#x",
|
||||
"http://user:password@localhost:8009",
|
||||
"file:///localhost",
|
||||
"http://localhost:65536",
|
||||
"http://127.1:8009",
|
||||
"http://2130706433:8009",
|
||||
"http://localhost.:8009",
|
||||
null,
|
||||
8009,
|
||||
])("rejects non-origin, non-loopback, or ambiguous endpoint %s", (url) => {
|
||||
expect(() => runtimeConfig({ baseUrl: url })).toThrow("baseUrl");
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ ...localAnswer, latency_ms: -1 },
|
||||
{ ...localAnswer, latency_ms: "15" },
|
||||
{ ...localAnswer, debug: "extra metadata" },
|
||||
{
|
||||
...localAnswer,
|
||||
answers: {
|
||||
...localAnswer.answers,
|
||||
s: {
|
||||
...localAnswer.answers.s,
|
||||
legend: { 0: "", 1: "wrong rubric" },
|
||||
},
|
||||
},
|
||||
},
|
||||
{ ...localAnswer, answers: { ...localAnswer.answers, b: { type: "noul", noul: 2 } } },
|
||||
])("keeps answer and metadata validation strict for local responses", async (response) => {
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async () => new Response(JSON.stringify(response))),
|
||||
);
|
||||
await expect(evaluate(input, runtimeConfig({ baseUrl }))).rejects.toThrow("invalid response");
|
||||
});
|
||||
|
||||
it("keeps hosted response validation strict and ignores stale keys even on direct local calls", async () => {
|
||||
const fetch = vi.fn(
|
||||
async (_url: RequestInfo | URL, _init?: RequestInit) =>
|
||||
new Response(JSON.stringify(localAnswer)),
|
||||
);
|
||||
vi.stubGlobal("fetch", fetch);
|
||||
await expect(
|
||||
evaluate(input, {
|
||||
...runtimeConfig({ baseUrl }),
|
||||
apiKey: "hosted-secret",
|
||||
}),
|
||||
).resolves.toHaveProperty("evaluation.model", "kev-latest");
|
||||
expect(new Headers(fetch.mock.calls[0]?.[1]?.headers).has("authorization")).toBe(false);
|
||||
const hostedInput = {
|
||||
...input,
|
||||
questions: {
|
||||
...input.questions,
|
||||
s: {
|
||||
...input.questions.s,
|
||||
criteria: ["", '{"impact":["widespread",true,2]}'],
|
||||
},
|
||||
},
|
||||
};
|
||||
await expect(evaluate(hostedInput, runtimeConfig({ apiKey: "hosted-secret" }))).rejects.toThrow(
|
||||
"invalid response",
|
||||
);
|
||||
expect(fetch.mock.calls[1]?.[0]).toBe("https://api.typesafe.ai/v1/systemone");
|
||||
expect(new Headers(fetch.mock.calls[1]?.[1]?.headers).get("authorization")).toBe(
|
||||
"Bearer hosted-secret",
|
||||
);
|
||||
});
|
||||
|
||||
it.each(["127.0.0.1", "localhost", "[::1]"])(
|
||||
"reaches a real %s server without ambient proxies or localhost DNS",
|
||||
async (hostname) => {
|
||||
const received: { url?: string; authorization?: string; host?: string; body: string }[] = [];
|
||||
const server = createServer((request, response) => {
|
||||
const chunks: Buffer[] = [];
|
||||
request.on("data", (chunk: Buffer) => chunks.push(chunk));
|
||||
request.on("end", () => {
|
||||
received.push({
|
||||
url: request.url,
|
||||
authorization: request.headers.authorization,
|
||||
host: request.headers.host,
|
||||
body: Buffer.concat(chunks).toString("utf8"),
|
||||
});
|
||||
response.setHeader("Content-Type", "application/json");
|
||||
response.end(JSON.stringify(localAnswer));
|
||||
});
|
||||
});
|
||||
await new Promise<void>((resolve) => {
|
||||
server.listen(0, hostname === "[::1]" ? "::1" : "127.0.0.1", resolve);
|
||||
});
|
||||
try {
|
||||
const address = server.address();
|
||||
assert(address && typeof address === "object");
|
||||
for (const name of [
|
||||
"HTTP_PROXY",
|
||||
"HTTPS_PROXY",
|
||||
"ALL_PROXY",
|
||||
"http_proxy",
|
||||
"https_proxy",
|
||||
"all_proxy",
|
||||
]) {
|
||||
vi.stubEnv(name, "http://127.0.0.1:1");
|
||||
}
|
||||
vi.stubEnv("NO_PROXY", "");
|
||||
vi.stubEnv("no_proxy", "");
|
||||
vi.stubEnv("OPENCLAW_DEBUG_PROXY_ENABLED", "false");
|
||||
if (hostname === "localhost") {
|
||||
vi.mocked(lookup).mockRejectedValue(new Error("Synthetic untrusted localhost resolver"));
|
||||
}
|
||||
await expect(
|
||||
evaluate(input, runtimeConfig({ baseUrl: `http://${hostname}:${address.port}` })),
|
||||
).resolves.toHaveProperty("evaluation.answers.b.noul", 0.75);
|
||||
expect(received).toHaveLength(1);
|
||||
expect(received[0]).toMatchObject({
|
||||
url: "/v1/systemone",
|
||||
authorization: undefined,
|
||||
host: `${hostname}:${address.port}`,
|
||||
});
|
||||
expect(JSON.parse(received[0]!.body).questions.c.instructions).toBeNull();
|
||||
if (hostname === "localhost") {
|
||||
expect(lookup).not.toHaveBeenCalled();
|
||||
}
|
||||
} finally {
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
server.close((error) => (error ? reject(error) : resolve()));
|
||||
});
|
||||
}
|
||||
},
|
||||
);
|
||||
49
extensions/typesafe/src/local.ts
Normal file
49
extensions/typesafe/src/local.ts
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
import { parseResult, type Evaluation, type EvaluationInput } from "./schema.js";
|
||||
|
||||
export function localInput(input: EvaluationInput): EvaluationInput {
|
||||
const questions: EvaluationInput["questions"] = {};
|
||||
for (const [id, question] of Object.entries(input.questions)) {
|
||||
const instructions = question.instructions ?? null;
|
||||
if (question.type === "score") {
|
||||
questions[id] = {
|
||||
...question,
|
||||
instructions,
|
||||
// Send explicit text so the returned legend can be verified without reproducing Kev's renderer.
|
||||
criteria: question.criteria.map((level) =>
|
||||
typeof level === "string" ? level : level === null ? "" : JSON.stringify(level),
|
||||
),
|
||||
};
|
||||
} else {
|
||||
questions[id] = { ...question, instructions };
|
||||
}
|
||||
}
|
||||
return { ...input, questions };
|
||||
}
|
||||
|
||||
export function parseLocalResult(
|
||||
value: unknown,
|
||||
wireInput: EvaluationInput,
|
||||
originalInput: EvaluationInput,
|
||||
): Evaluation {
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) {
|
||||
throw new Error("Invalid local System One response.");
|
||||
}
|
||||
let result = value;
|
||||
if ("latency_ms" in value) {
|
||||
const { latency_ms, ...payload } = value;
|
||||
if (typeof latency_ms !== "number" || !Number.isFinite(latency_ms) || latency_ms < 0) {
|
||||
throw new Error("Invalid local System One latency metadata.");
|
||||
}
|
||||
result = payload;
|
||||
}
|
||||
const evaluation = parseResult(result, wireInput);
|
||||
for (const [id, question] of Object.entries(originalInput.questions)) {
|
||||
const answer = evaluation.answers[id];
|
||||
if (question.type === "score" && answer?.type === "score") {
|
||||
answer.legend = Object.fromEntries(
|
||||
question.criteria.map((level, index) => [String(index), level]),
|
||||
);
|
||||
}
|
||||
}
|
||||
return evaluation;
|
||||
}
|
||||
|
|
@ -68,6 +68,7 @@ describe("plugin ownership and configuration", () => {
|
|||
expect(manifest.decisionModels).toEqual([
|
||||
{ provider: "typesafe", id: "jev-latest", name: "Jev" },
|
||||
{ provider: "typesafe", id: "jev-1.13.0", name: "Jev 1.13.0" },
|
||||
{ provider: "typesafe", id: "kev-latest", name: "Kev (local server)" },
|
||||
]);
|
||||
expect(manifest.providers).toBeUndefined();
|
||||
expect(manifest.modelCatalog).toBeUndefined();
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ import { Type, type Static, type TSchema } from "typebox";
|
|||
import { Compile } from "typebox/compile";
|
||||
import { Check } from "typebox/value";
|
||||
|
||||
// Transport/CPU guards, not Jev token limits. Jev enforces its own context budget.
|
||||
// Transport/CPU guards; each server enforces its model's context budget.
|
||||
export const MAX_JSON_BYTES = 4 * 1024 * 1024;
|
||||
const MAX_JSON_NODES = 262144;
|
||||
const MAX_JSON_DEPTH = 64;
|
||||
|
|
@ -31,7 +31,7 @@ const entry = Type.Union(
|
|||
const instructions = Type.Optional(
|
||||
Type.Union(entry.anyOf, {
|
||||
description:
|
||||
"The complete judgment to make; question IDs are not read by Jev. Text, structured object/array, or null. May be omitted when criteria express the judgment.",
|
||||
"The complete judgment to make; question IDs are not read by the model. Text, structured object/array, or null. May be omitted when criteria express the judgment.",
|
||||
}),
|
||||
);
|
||||
const model = Type.String({ minLength: 1, maxLength: 128, pattern: "^[a-zA-Z0-9._/-]+$" });
|
||||
|
|
@ -102,13 +102,13 @@ export const EvaluateInput = Type.Object(
|
|||
questions: map(question, {
|
||||
minProperties: 1,
|
||||
description:
|
||||
"Nonempty map of question IDs to Choice, Score, or Noul questions. Mix types in one call. IDs only match answers; put all meaning in instructions/criteria. Questions are independent. Jev enforces token limits; plugin JSON guard is 4 MiB.",
|
||||
"Nonempty map of question IDs to Choice, Score, or Noul questions. Mix types in one call. IDs only match answers; put all meaning in instructions/criteria. Questions are independent. The server enforces token limits; plugin JSON guard is 4 MiB.",
|
||||
}),
|
||||
model: Type.Optional(
|
||||
Type.String({
|
||||
...model,
|
||||
description:
|
||||
"Optional Jev model ID or alias for this explicit tool call; defaults to the plugin’s evaluation-tool model. Native decisions use the host-selected model.",
|
||||
"Optional System One model ID or alias for this explicit tool call; defaults to the plugin’s evaluation-tool model. Native decisions use the host-selected model. A local Kev server uses its loaded checkpoint regardless of this label.",
|
||||
}),
|
||||
),
|
||||
},
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import {
|
|||
} from "openclaw/plugin-sdk/fetch-runtime";
|
||||
import { parseRetryAfterHeaderSeconds } from "openclaw/plugin-sdk/retry-runtime";
|
||||
import { fetchWithSsrFGuard } from "openclaw/plugin-sdk/ssrf-runtime";
|
||||
import { localBaseUrl } from "./config.js";
|
||||
import { EvaluationError } from "./errors.js";
|
||||
import { MAX_JSON_BYTES, type EvaluationInput } from "./schema.js";
|
||||
|
||||
|
|
@ -78,10 +79,13 @@ async function readBody(response: Response, signal?: AbortSignal): Promise<Buffe
|
|||
|
||||
export async function requestEvaluation(params: {
|
||||
body: EvaluationInput & { model: string };
|
||||
apiKey: string;
|
||||
apiKey?: string;
|
||||
baseUrl?: string;
|
||||
timeoutMs: number;
|
||||
signal?: AbortSignal;
|
||||
}): Promise<unknown> {
|
||||
const baseUrl = localBaseUrl(params.baseUrl);
|
||||
const endpoint = baseUrl ? `${baseUrl}/v1/systemone` : ENDPOINT;
|
||||
const body = JSON.stringify(params.body);
|
||||
if (Buffer.byteLength(body) > MAX_JSON_BYTES) {
|
||||
throw new EvaluationError("TypeSafe request exceeds its limit.", "unsupported-input");
|
||||
|
|
@ -94,16 +98,26 @@ export async function requestEvaluation(params: {
|
|||
try {
|
||||
signal?.throwIfAborted();
|
||||
const request = {
|
||||
url: ENDPOINT,
|
||||
url: endpoint,
|
||||
fetchImpl: globalThis.fetch,
|
||||
requireHttps: true,
|
||||
requireHttps: !baseUrl,
|
||||
...(baseUrl ? { policy: { allowedOrigins: [baseUrl] } } : {}),
|
||||
...(baseUrl && new URL(baseUrl).hostname === "localhost"
|
||||
? {
|
||||
// Keep localhost local even when system DNS or hosts entries override its meaning.
|
||||
lookupFn: async () => [
|
||||
{ address: "127.0.0.1", family: 4 },
|
||||
{ address: "::1", family: 6 },
|
||||
],
|
||||
}
|
||||
: {}),
|
||||
maxRedirects: 0,
|
||||
signal,
|
||||
beforeRequest: () => signal?.throwIfAborted(),
|
||||
init: {
|
||||
method: "POST",
|
||||
headers: {
|
||||
Authorization: `Bearer ${params.apiKey}`,
|
||||
...(!baseUrl ? { Authorization: `Bearer ${params.apiKey}` } : {}),
|
||||
Accept: "application/json",
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
|
|
@ -111,7 +125,7 @@ export async function requestEvaluation(params: {
|
|||
},
|
||||
};
|
||||
const guarded = await fetchWithSsrFGuard(
|
||||
shouldUseEnvHttpProxyForUrl(ENDPOINT)
|
||||
!baseUrl && shouldUseEnvHttpProxyForUrl(endpoint)
|
||||
? withTrustedEnvProxyGuardedFetchMode(request)
|
||||
: request,
|
||||
);
|
||||
|
|
|
|||
|
|
@ -361,6 +361,7 @@
|
|||
"!dist/extensions/tokenjuice/**",
|
||||
"!dist/extensions/tlon/**",
|
||||
"!dist/extensions/twitch/**",
|
||||
"!dist/extensions/typesafe/**",
|
||||
"!dist/extensions/venice/**",
|
||||
"!dist/extensions/vercel-ai-gateway/**",
|
||||
"!dist/extensions/visitor-access/**",
|
||||
|
|
|
|||
|
|
@ -825,6 +825,24 @@
|
|||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "@openclaw/typesafe",
|
||||
"description": "OpenClaw TypeSafe typed decisions and optional evaluation tool",
|
||||
"source": "official",
|
||||
"kind": "plugin",
|
||||
"openclaw": {
|
||||
"plugin": {
|
||||
"id": "typesafe",
|
||||
"label": "TypeSafe AI"
|
||||
},
|
||||
"install": {
|
||||
"clawhubSpec": "clawhub:@openclaw/typesafe",
|
||||
"npmSpec": "@openclaw/typesafe",
|
||||
"defaultChoice": "npm",
|
||||
"minHostVersion": ">=2026.9.6"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "@openclaw/voice-call",
|
||||
"description": "OpenClaw voice-call plugin",
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue