OmniRoute/docs/frameworks/EMBEDDED-SERVICES.md

title	description
Embedded Services	Reference for 9Router, CLIProxyAPI, Mux, and Bifrost

Embedded Services

Version: v3.8.44 Last updated: 2026-07-03 Audience: Engineers adding, maintaining, or debugging embedded services (9Router, CLIProxyAPI, Mux, Bifrost).

Embedded services are locally-installed process sidecar tools that OmniRoute installs, supervises, and exposes as first-class routing targets. Unlike external providers (which are reached over the internet via API keys), embedded services run on the same machine as OmniRoute and communicate over loopback.

---

Table of Contents

1. Overview
2. Architecture — 4 layers
3. Lifecycle state machine
4. API reference
5. Security
6. Adding a new embedded service
7. Troubleshooting
8. FAQ

---

1. Overview

Why embedded services?

Four services are embedded as of v3.8.44:

Service	npm package	Default port	Purpose
9Router	9router	20130	AI router that OmniRoute can use as a sub-provider. Models exposed as 9router/{sub}/{model}
CLIProxyAPI	@anthropic/cli-proxy (via cliproxy binary)	auto	Local proxy adapter for Anthropic CLI auth flows. Provides fallback routing when OAuth tokens expire
Mux	mux (headless mux server)	8322	Local agent-orchestration daemon (coder/mux). Lifecycle-managed only — not a routing target (no LLM proxying).
Bifrost	@maximhq/bifrost	8080	Go AI-gateway relay backend. When running, auto-selected by the relay route (/v1/relay/)

All four follow the same supervisory model:

• OmniRoute installs them under DATA_DIR/services/{name}/ (isolated from OmniRoute's own package.json)
• OmniRoute spawns and monitors them as child processes
• OmniRoute injects an ephemeral API key into the child's environment and rotates it without downtime (where applicable)
• All management routes (/api/services/*) are LOCAL_ONLY — accessible only from loopback (hard rule #17)

Key decisions (from design plan)

Decision	Value
Dashboard access to 9Router native UI	Reverse proxy at /dashboard/providers/services/9router/embed/*
Installation mechanism	npm install {package} via execFile (no shell interpolation)
Consumption mode	Provider registered as 9router/{sub}/{model} in routing engine
API key management	OmniRoute generates, encrypts at-rest (AES-256-GCM), and injects via env
Dashboard location	/dashboard/providers/services (three tabs)
Auto-start	Toggle per service, default OFF

---

2. Architecture — 4 layers

┌────────────────────────────────────────────────────────────────────┐
│  Layer 1 — UI                                                      │
│  /dashboard/providers/services  (tabs: CLIProxyAPI | 9Router | Mux)│
│  Logs live (SSE), Start/Stop/Restart/Update, Settings, Install     │
│                                                                    │
│  src/app/(dashboard)/dashboard/providers/services/                 │
│    ├── page.tsx               Shell + tab routing by ?tab=         │
│    ├── tabs/                  CliproxyServiceTab, NinerouterServiceTab,│
│    │                          MuxServiceTab                        │
│    └── components/            ServiceStatusCard, ServiceLifecycleButtons,│
│                               ServiceLogsPanel, ApiKeyCard, ...    │
└──────────────────────┬─────────────────────────────────────────────┘
                       │ HTTP (Next.js fetch)
┌──────────────────────▼─────────────────────────────────────────────┐
│  Layer 2 — API (LOCAL_ONLY — loopback only)                        │
│                                                                    │
│  /api/services/9router/{install|start|stop|restart|update|         │
│                          rotate-key|status|auto-start|logs}        │
│  /api/services/cliproxy/{install|start|stop|restart|update|        │
│                           status|auto-start|logs}                  │
│  /api/services/mux/{install|start|stop|restart|update|             │
│                      status|auto-start|logs}                       │
│  /dashboard/providers/services/9router/embed/[...path]             │
│    (reverse HTTP + WebSocket proxy → 9Router upstream)             │
│                                                                    │
│  Gate: LOCAL_ONLY_API_PREFIXES includes "/api/services/" and       │
│        "/dashboard/providers/services/*/embed/"                    │
└──────────────────────┬─────────────────────────────────────────────┘
                       │ in-process calls
┌──────────────────────▼─────────────────────────────────────────────┐
│  Layer 3 — ServiceSupervisor (src/lib/services/)                   │
│                                                                    │
│  ServiceSupervisor.ts   Generic supervisor (child_process.spawn)   │
│    ├── install:    execFile('npm', ['install', pkg, '--prefix'])    │
│    ├── start:      spawn(node, [entrypoint], {env, cwd})           │
│    ├── api_key:    crypto.randomBytes(32) → env NINEROUTER_API_KEY  │
│    ├── port:       20130 for 9Router (configurable)                │
│    ├── logs:       stdio ring buffer 5 MB → SSE events             │
│    ├── health:     HTTP GET /health every 2–5 s, lazy recovery     │
│    └── lifecycle:  SIGTERM 15 s → SIGKILL                          │
│                                                                    │
│  registry.ts        getSupervisor(name) / registerSupervisor()     │
│  bootstrap.ts       Bootstraps all SERVICES[] at process start     │
│  apiKey.ts          getOrCreateApiKey(), generateServiceApiKey()   │
│  modelSync.ts       Periodic GET /v1/models → service_models table │
│  ringBuffer.ts      Circular log buffer (5 MB per service)         │
│  healthCheck.ts     Polling HTTP health probe                      │
│  installers/        ninerouter.ts, cliproxy.ts, mux.ts             │
│                      (installer adapters)                          │
└──────────────────────┬─────────────────────────────────────────────┘
                       │ OpenAI-compatible HTTP (loopback)
┌──────────────────────▼─────────────────────────────────────────────┐
│  Layer 4 — Provider / Routing                                      │
│                                                                    │
│  open-sse/executors/ninerouter.ts                                  │
│    Re-looks up port and API key per-request (no caching).          │
│    Strips "9router/" prefix from model id before proxying.         │
│    Returns 503 service_not_running if supervisor not in "running". │
│                                                                    │
│  src/shared/constants/providers.ts                                 │
│    Entry for "9router": isEmbeddedService: true                    │
│                                                                    │
│  open-sse/config/providerRegistry.ts                               │
│    Models stored as "9router/{sub}/{model}" (prefixed).            │
│    Synced every 5 min by modelSync.ts.                             │
│                                                                    │
│  Mux is lifecycle-managed ONLY (Layers 1-3) — it is an agent-       │
│  orchestration daemon, not an LLM proxy, so it has no Layer 4      │
│  executor/provider entry and is never a routing target.            │
└────────────────────────────────────────────────────────────────────┘

Key source files

File	Role
src/lib/services/ServiceSupervisor.ts	Core class: lifecycle, lock, health, ring buffer
src/lib/services/bootstrap.ts	Process-level registration and auto-start
src/lib/services/registry.ts	Singleton map tool → supervisor
src/lib/services/apiKey.ts	Key generation, AES-256-GCM encryption at-rest
src/lib/services/modelSync.ts	Periodic model sync (5 min) + on-demand
src/lib/services/ringBuffer.ts	5 MB circular log buffer with SSE subscribe
src/lib/services/healthCheck.ts	HTTP health probe (configurable interval)
src/lib/services/installers/ninerouter.ts	npm install/update/uninstall for 9Router
src/lib/services/installers/cliproxy.ts	npm install/update/uninstall for CLIProxyAPI
src/lib/services/installers/mux.ts	npm install/update/uninstall for Mux
src/app/api/services/9router/_lib.ts	getOrInitSupervisor() helper
src/app/api/services/[name]/logs/route.ts	Shared SSE logs endpoint
open-sse/executors/ninerouter.ts	Provider executor (Layer 4)

---

3. Lifecycle state machine

                    install()
  ┌─────────────┐ ──────────► ┌─────────────┐
  │ not_installed│             │   stopped   │◄──────────────────┐
  └─────────────┘             └──────┬──────┘                   │
                                     │ start()                   │
                                     ▼                           │ stop()
                               ┌──────────┐                      │
                               │ starting │                      │
                               └────┬─────┘                     │
                  health probe ok   │         crash / SIGTERM    │
                               ┌────▼─────┐  (exit within 5s)   │
                               │ running  │──── crash ──────────►┤
                               └────┬─────┘                   ┌─▼────┐
                             stop() │                          │error │
                                    ▼                          └──────┘
                               ┌──────────┐
                               │ stopping │
                               └──────────┘

States stored in the version_manager DB table (status column) and mirrored in ServiceSupervisor in-memory state. The in-memory state is authoritative for a running process; the DB state is the durable fallback at boot.

State transitions

From	Event	To
not_installed	install() succeeds	stopped
stopped	start() called	starting
starting	health probe returns 200	running
starting	process exits before healthy	error
running	stop() called	stopping → stopped
running	process exits unexpectedly (< 5 s)	error (fast crash)
running	process exits unexpectedly (> 5 s)	error
error	start() called	starting
any	stop() while stopping	no-op

Operation lock

ServiceSupervisor serializes lifecycle operations through an async operation lock (withLock()). Concurrent start() calls on the same supervisor result in exactly one spawn; the second caller waits and returns the existing status. This prevents race conditions when, for example, auto-start and a UI button fire simultaneously.

---

4. API reference

All routes under /api/services/ are LOCAL_ONLY (loopback only, hard rule #17). Non-loopback requests receive 403 LOCAL_ONLY regardless of auth token.

4.1 9Router endpoints (8 routes)

POST /api/services/9router/install

Install 9Router from npm. Creates DATA_DIR/services/9router/ with its own package.json and node_modules/. Does not conflict with OmniRoute's own deps.

Request body (all optional):

{ "version": "latest" }

Field	Type	Default	Description
version	string	"latest"	npm version tag or semver to install

Responses:

Status	Description
200	{ ok: true, installedVersion: "x.y.z", path: "..." }
400	Invalid request body (Zod validation failure)
409	Already installing (lock held)
500	npm install failed — see message for friendly error

Notes: Uses execFile('npm', [...]) — no shell, no interpolation (hard rule #13). EACCES errors are surfaced as friendly messages.

---

POST /api/services/9router/start

Start 9Router. Registers a supervisor if not already registered, then calls supervisor.start(). Idempotent when already running.

Request body: none

Responses:

Status	Description
200	ServiceStatus object (see schema below)
409	9Router is not installed (status: "not_installed")
503	Start failed (process error — see lastError)

ServiceStatus schema:

{
  "tool": "9router",
  "state": "running",
  "pid": 12345,
  "port": 20130,
  "health": "healthy",
  "startedAt": "2026-05-25T10:00:00.000Z",
  "lastError": null
}

---

POST /api/services/9router/stop

Gracefully stop 9Router. Sends SIGTERM, waits 15 s, then SIGKILL if still alive. Idempotent when already stopped.

Request body: none

Responses:

Status	Description
200	ServiceStatus (state: "stopped")
503	Stop failed unexpectedly

---

POST /api/services/9router/restart

Equivalent to stop() then start() under the operation lock.

Request body: none

Responses: same as start (returns final ServiceStatus).

---

POST /api/services/9router/update

Updates 9Router to a newer npm version. If the service is running, it is stopped first, npm install is run (installing the newer version in-place), and then the service is restarted.

Request body (all optional):

{ "version": "latest" }

Responses:

Status	Description
200	{ ok: true, previousVersion: "...", installedVersion: "..." }
400	Invalid body
500	npm update failed

---

POST /api/services/9router/rotate-key

Generates a new API key for 9Router, encrypts it at-rest, and restarts the service (if running) so it picks up the new key from its environment. The old key is invalidated immediately.

Request body: none

Responses:

Status	Description
200	{ keyRotated: true, restarted: boolean }
500	Rotation failed

Security: The new key is never returned in the response (no credential leak). It is stored encrypted (AES-256-GCM) in the version_manager table.

---

GET /api/services/9router/status

Returns combined live + DB status including version metadata and API key preview.

Responses:

Status	Description
200	See schema below
500	Status read failed

Response schema:

{
  "tool": "9router",
  "state": "running",
  "pid": 12345,
  "port": 20130,
  "health": "healthy",
  "startedAt": "2026-05-25T10:00:00.000Z",
  "lastError": null,
  "installedVersion": "1.2.3",
  "latestVersion": "1.2.4",
  "updateAvailable": true,
  "apiKeyMasked": "nr_****abcd",
  "autoStart": false,
  "providerExpose": false
}

---

POST /api/services/9router/auto-start

Toggle the auto-start flag. When enabled: true, the service starts automatically the next time OmniRoute boots (if the service is installed).

Request body:

{ "enabled": true }

Responses:

Status	Description
200	{ autoStart: true }
400	Invalid body

---

GET /api/services/9router/logs

SSE stream of live logs from 9Router's stdout/stderr ring buffer.

Query parameters:

Param	Type	Default	Description
tail	integer	200	How many historical lines to send first (max 1000)
filter	string	none	Case-insensitive substring filter (no regex — ReDoS-safe)

SSE events:

Event	Data	Description
snapshot	LogLine[]	Initial historical tail
log	LogLine	Live log line
heartbeat	{}	Keep-alive every 15 s

LogLine schema:

{ "ts": 1716633600000, "stream": "stdout", "line": "[9router] Listening on :20130" }

Responses:

Status	Description
200	text/event-stream
400	filter parameter too long (> 200 chars)
404	Service not found (supervisor not registered)

---

4.2 CLIProxyAPI endpoints (7 routes)

CLIProxyAPI has the same endpoint shape as 9Router minus rotate-key (CLIProxyAPI does not require an injected API key; it authenticates via the host's existing CLI config) and status includes fewer fields.

Method	Path	Description
POST	/api/services/cliproxy/install	Install CLIProxyAPI from npm
POST	/api/services/cliproxy/start	Start CLIProxyAPI
POST	/api/services/cliproxy/stop	Stop CLIProxyAPI
POST	/api/services/cliproxy/restart	Restart CLIProxyAPI
POST	/api/services/cliproxy/update	Update to newer version
GET	/api/services/cliproxy/status	Live + DB status (no apiKeyMasked)
POST	/api/services/cliproxy/auto-start	Toggle auto-start

The shared GET /api/services/{name}/logs endpoint (see §4.1) works for all four services using the [name] dynamic segment.

---

4.3 Mux endpoints (7 routes)

Mux has the same endpoint shape as CLIProxyAPI — no rotate-key route in the API surface (the bearer token is generated the same way as 9Router's via getOrCreateApiKey("mux") and injected via the MUX_SERVER_AUTH_TOKEN env var, but there is no dedicated rotation endpoint yet). Mux is lifecycle-managed only: unlike 9Router, it has no Layer 4 executor and is never registered as a routing provider.

Method	Path	Description
POST	/api/services/mux/install	Install Mux from npm (npm i mux)
POST	/api/services/mux/start	Start Mux (mux server)
POST	/api/services/mux/stop	Stop Mux
POST	/api/services/mux/restart	Restart Mux
POST	/api/services/mux/update	Update to newer npm version
GET	/api/services/mux/status	Live + DB status
POST	/api/services/mux/auto-start	Toggle auto-start

---

4.4 Bifrost endpoints (7 routes)

Bifrost is a Go AI-gateway relay backend (@maximhq/bifrost). It uses the same endpoint shape as CLIProxyAPI (no rotate-key — Bifrost manages its own provider keys in config.json under its -app-dir).

Method	Path	Description
POST	/api/services/bifrost/install	Install Bifrost from npm (@maximhq/bifrost)
POST	/api/services/bifrost/start	Start Bifrost on port 8080 (default)
POST	/api/services/bifrost/stop	Stop Bifrost
POST	/api/services/bifrost/restart	Restart Bifrost
POST	/api/services/bifrost/update	Update to newer version
GET	/api/services/bifrost/status	Live + DB status
POST	/api/services/bifrost/auto-start	Toggle auto-start
GET	/api/services/bifrost/logs	SSE log tail (via shared [name]/logs dynamic route)

Routing wiring: When BIFROST_BASE_URL is unset and the supervised Bifrost instance is running, getBifrostRoutingConfig() (in routingBackend.ts) automatically uses http://127.0.0.1:{port} as the relay base URL. Explicit BIFROST_BASE_URL env always takes precedence.

---

4.4 Reverse proxy (9Router dashboard embed)

The dashboard embeds the 9Router web UI inside an iframe via an internal reverse proxy at:

GET|POST|... /dashboard/providers/services/9router/embed/[...path]

This proxy:

• Forwards the request to http://127.0.0.1:{port}/{path} (loopback only)
• Strips incoming cookie and authorization headers (no leakage of OmniRoute session)
• Injects Authorization: Bearer {apiKey} for 9Router authentication
• Strips set-cookie, content-security-policy, x-frame-options, cross-origin-* from the response
• Rewrites HTML responses to inject <base href> and normalize absolute paths (/foo → /dashboard/.../embed/foo)

WebSocket upgrades for the embedded dashboard are handled by a companion server on a dedicated port (see src/lib/services/embedWsProxy.ts).

Security: The embed proxy routes are classified under LOCAL_ONLY_API_PREFIXES and can only be reached from loopback. An attacker who obtains a JWT via a Cloudflare/Ngrok tunnel cannot proxy into embedded services.

---

5. Security

LOCAL_ONLY enforcement (hard rule #17)

All routes under /api/services/ and /dashboard/providers/services/*/embed/ are classified as LOCAL_ONLY in src/server/authz/routeGuard.ts. The loopback check runs unconditionally before any auth branch:

request arrives
  → isLocalOnlyPath(path)?
      → non-loopback → 403 LOCAL_ONLY (always, before auth check)
      → loopback    → fall through to normal auth

This prevents a leaked JWT (e.g., via a tunnel) from triggering npm install or process spawning. See docs/security/ROUTE_GUARD_TIERS.md for the full tier matrix.

API key injection

9Router and Mux require an API key/bearer token for their own HTTP endpoints. OmniRoute:

1. Generates a key via crypto.randomBytes(32).toString("base64url") with a service-specific prefix (nr_ for 9Router, mx_ for Mux).
2. Encrypts it at-rest using AES-256-GCM (same cipher used for provider credentials).
3. Decrypts and injects it as an environment variable at spawn time — NINEROUTER_API_KEY for 9Router, MUX_SERVER_AUTH_TOKEN for Mux (never a CLI flag, so the token never appears in ps/process listings).
4. Never returns the plaintext key in any HTTP response.

CLIProxyAPI does not require an injected key (it authenticates via the host's existing CLI config).

SSRF defense

The reverse HTTP proxy (/dashboard/.../embed/[...path]) is hardcoded to forward only to http://127.0.0.1:{port}. It never follows redirects to non-loopback destinations. The ssrf-req-filter library is used to reject any upstream URL that resolves outside the loopback range.

Shell safety (hard rule #13)

npm install is invoked via execFile('npm', ['install', pkg, '--prefix', dir]) — no template literals, no shell, no interpolation of external paths into the command string. Runtime values (ports, API keys) are passed via the child's env object.

Error sanitization (hard rule #12)

All error responses from /api/services/* go through buildErrorBody() or sanitizeErrorMessage(). Raw err.stack and err.message are never returned verbatim to the caller.

---

6. Adding a new embedded service

Follow these 8 steps. Read the existing implementations in src/lib/services/installers/ and src/app/api/services/ as the canonical reference.

Step 1 — Create the installer

Create src/lib/services/installers/{name}.ts modeled on ninerouter.ts:

export const NAME_PACKAGE = "your-npm-package";
export const NAME_DEFAULT_PORT = 20132; // pick a free port

export async function install(version = "latest"): Promise<InstallResult> { ... }
export async function update(version = "latest"): Promise<InstallResult> { ... }
export async function uninstall(): Promise<void> { ... }
export function resolveSpawnArgs(apiKey: string, port: number): SpawnArgs { ... }
export async function getInstalledVersion(): Promise<string | null> { ... }
export async function getLatestVersion(): Promise<string | null> { ... }

Use runNpm(['install', NAME_PACKAGE, '--prefix', dir]) from installers/utils.ts — never execSync or shell interpolation.

Step 2 — Register in bootstrap

Add a ServiceEntry to the SERVICES array in src/lib/services/bootstrap.ts:

{
  tool: "myservice",
  port: NAME_DEFAULT_PORT,
  healthPath: "/health",
  healthIntervalMs: 5_000,
  stopTimeoutMs: 15_000,
  logsBufferBytes: 5_242_880,
  needsApiKey: true, // false if no API key needed
}

Extend buildSpawnArgsFactory() to handle cfg.tool === "myservice".

Step 3 — Add migration and DB seed

Ensure the service has a row in version_manager via a migration in src/lib/db/migrations/. The row should have:

INSERT OR IGNORE INTO version_manager (tool, status, auto_start, provider_expose)
VALUES ('myservice', 'not_installed', 0, 0);

Step 4 — Create the 7 API endpoints

Under src/app/api/services/{name}/:

_lib.ts            getOrInitSupervisor() helper
install/route.ts   POST — calls installer.install()
start/route.ts     POST — calls supervisor.start()
stop/route.ts      POST — calls supervisor.stop()
restart/route.ts   POST — calls supervisor.restart()
update/route.ts    POST — calls installer.update()
status/route.ts    GET  — merges live + DB status
auto-start/route.ts POST — toggles auto_start flag

The shared GET /api/services/[name]/logs route is already wired — no changes needed there.

Delegate all error responses through createErrorResponse() / buildErrorBody().

Step 5 — Add to LOCAL_ONLY_API_PREFIXES

In src/server/authz/routeGuard.ts, verify that /api/services/ is already listed. If you introduce a new prefix (e.g., /api/tools/), add it to both LOCAL_ONLY_API_PREFIXES and, if it spawns processes, to SPAWN_CAPABLE_PREFIXES. Add a test in tests/unit/authz/routeGuard.test.ts.

Step 6 — Add the UI tab

Create src/app/(dashboard)/dashboard/providers/services/tabs/{Name}ServiceTab.tsx. Reuse shared components:

• ServiceStatusCard — live state + health badge
• ServiceLifecycleButtons — Start / Stop / Restart / Update
• ServiceLogsPanel — SSE log tail (connects to /api/services/{name}/logs)
• ApiKeyCard — key reveal + rotate (if needsApiKey: true)

Register the tab in ServicesPageShell.tsx.

Step 7 — Add the provider entry (if the service is a routing target)

If the embedded service exposes an OpenAI-compatible /v1/chat/completions endpoint:

1. Add a provider entry in src/shared/constants/providers.ts with isEmbeddedService: true.
2. Create open-sse/executors/{name}.ts extending BaseExecutor. Re-lookup port and API key per-request (never cache in the constructor). Return a 503 service_not_running response when the supervisor state is not "running".
3. Register models in open-sse/config/providerRegistry.ts with the service prefix (e.g., myservice/sub/model). modelSync.ts will keep them updated.

Step 8 — Document and test

1. Update docs/frameworks/EMBEDDED-SERVICES.md (this file) — add the service to the table in §1 and any new endpoints to §4.
2. Add unit tests in tests/unit/services/ (lifecycle, installer, API shape).
3. Add integration test in tests/integration/services/ (behind RUN_SERVICES_INT=1).
4. Update docs/openapi.yaml with the new endpoints.

---

7. Troubleshooting

Service does not start

Symptoms: Start button returns 503, state stays "error" or "starting".

Checklist:

1. Check GET /api/services/{name}/logs (or the Logs panel in the dashboard). Look for lines like Error: ENOENT, address already in use, or Cannot find module.
2. Verify npm is in PATH: which npm from the same user account that runs OmniRoute.
3. Verify the service is installed: check GET /api/services/{name}/status for installedVersion. If null, run install first.
4. Check DATA_DIR/services/{name}/node_modules/ exists and is not empty.
5. Check the lastError field in the status response for the sanitized exit reason.

---

Cold start is slow (> 10 s to reach running)

Symptoms: State stays "starting" for a long time before going to "running" or "error".

Explanation: 9Router's cold start includes importing large dependency trees (DNS, tunnel, MITM modules). Default health interval is 2 s with 3 attempts before the supervisor declares a timeout (but continues polling).

Fix: The healthIntervalMs and the waitForHealthy timeout (healthIntervalMs * 3) are configurable in bootstrap.ts. For services with longer startup times, increase healthIntervalMs to 5000 and stopTimeoutMs to 30 000.

---

Port collision (EADDRINUSE)

Symptoms: Logs show address already in use :::20130.

Causes:

• Another process is already using port 20130.
• A previous 9Router process was not fully stopped (zombie PID).

Fix:

1. Change the default port via NINEROUTER_PORT environment variable in .env.
2. Find and kill the conflicting process: lsof -ti :20130 | xargs kill -9.
3. The port is configurable per service in bootstrap.ts via the port field.

Note: 9Router defaults to port 20130 specifically to avoid colliding with OmniRoute's default port 20128.

---

Permission denied (EACCES) on install

Symptoms: Install returns 500, logs show EACCES or permission denied.

Causes:

• DATA_DIR or its parent is not writable by the OmniRoute process.
• Running inside Docker rootless without write access to the mapped volume.

Fix:

1. Check DATA_DIR (default: ~/.omniroute/): ls -la ~/.omniroute/
2. Ensure the OmniRoute process user owns the directory: chown -R $USER ~/.omniroute/
3. In Docker, ensure the volume mount has the correct permissions for the container user.

---

Update fails (npm install timeout or network error)

Symptoms: Update returns 500 with InstallError, logs show network timeout.

Checklist:

1. Confirm npm registry is reachable: npm ping.
2. Check for corporate proxy: npm config get proxy, npm config get https-proxy.
3. Try the install manually: npm install {package}@latest --prefix ~/.omniroute/services/{name}/.
4. If behind an air-gap, pre-download the tarball and use npm install /path/to/tarball.tgz.

---

Service shows "error" state immediately after start (fast crash)

Symptoms: State transitions from "starting" to "error" in under 5 seconds. lastError shows "Fast crash (exited with code 1)".

Checklist:

1. Read the full log tail: GET /api/services/{name}/logs?tail=500.
2. Common cause: missing environment variables expected by the service.
3. For 9Router: verify NINEROUTER_DISABLE_MITM=true and NINEROUTER_DISABLE_TUNNEL=true are in the env passed at spawn (see installers/ninerouter.ts resolveSpawnArgs).

---

8. FAQ

Q: Can I expose the embedded services endpoints to non-loopback clients?

No. The LOCAL_ONLY tier is intentional (hard rule #17). Routes that can invoke npm install or spawn node processes must not be reachable from non-loopback traffic, because a leaked JWT via a tunnel (Cloudflare, Ngrok, Tailscale) would otherwise allow arbitrary process spawning. There is no opt-out carve-out for /api/services/ — unlike /api/mcp/, it is excluded from the manage-scope bypass list. See docs/security/ROUTE_GUARD_TIERS.md.

---

Q: Will 9Router and CLIProxyAPI be available in production/cloud deployments?

Yes. Both services follow the same local-first model as OmniRoute itself. They run on the same machine and communicate over loopback. "Production" here means the VPS or local server where OmniRoute is deployed, not a remote cloud provider.

---

Q: How do I debug the supervisor?

1. Tail the SSE log stream: curl -N http://localhost:20128/api/services/9router/logs.
2. Check structured logs in OmniRoute's pino output filtered by service:supervisor namespace.
3. Inspect the DB row: sqlite3 ~/.omniroute/omniroute.db "SELECT * FROM version_manager WHERE tool='9router'".
4. Use GET /api/services/9router/status to see the current live state, PID, health, and lastError in one call.

---

Q: The supervisor shows health: "degraded" or health: "unknown" but state is "running". Is that a problem?

"degraded" means the health probe returned a non-200 response. "unknown" means no probe has completed yet (race with first poll). Both are transient during startup. If health stays "degraded" for more than healthIntervalMs * 3 ms after "running", the embedded service is running but its HTTP API is not responding. Check whether the port is correct in the status response and whether the service is actually listening on that port.

---

Q: Can I change the 9Router API key without a full restart?

No. The API key is passed to 9Router via an environment variable at spawn time. Environment variables cannot be changed in a running process. POST .../rotate-key automatically stops and restarts the service to apply the new key. The key rotation takes effect within the service's stopTimeoutMs (default 15 s) plus its startup time.

---

Q: What is the ring buffer limit and what happens when it fills?

Each service has a dedicated 5 MB ring buffer. When the buffer is full, the oldest log lines are evicted to make room for new ones. The SSE snapshot event returns the most recent lines within the tail limit. Logs are not persisted to disk unless logsBufferPath is set in the DB row.

---

See also

• docs/security/ROUTE_GUARD_TIERS.md — LOCAL_ONLY tier details
• docs/architecture/CODEBASE_DOCUMENTATION.md — §3.2 Embedded Services module mapping
• docs/architecture/ARCHITECTURE.md — system-level context
• docs/openapi.yaml — machine-readable endpoint definitions
• CLAUDE.md §"Adding a New Embedded Service" — quick-reference checklist