diff --git a/.gitignore b/.gitignore index ffa8d7166..ea570457f 100644 --- a/.gitignore +++ b/.gitignore @@ -26,9 +26,6 @@ lerna-debug.log* *.local # e.g. Claude Code settings.local.json *.local.* -.env -.env.* -!.env.example # Audio binaries *.pcm diff --git a/apps/server/docs/ai-context/metrics-ownership.md b/apps/server/docs/ai-context/metrics-ownership.md index 5f455b653..9a0229a4d 100644 --- a/apps/server/docs/ai-context/metrics-ownership.md +++ b/apps/server/docs/ai-context/metrics-ownership.md @@ -112,17 +112,17 @@ ### PostHog（前端 / 外部数据源，产品侧） 已接入： -- 前端 `posthog-js` 通过 `packages/stage-ui/src/stores/analytics/posthog.ts` 初始化；web / desktop / pocket / auth / docs 共用一个 project key，以 `app_surface` 区分运行端。 -- Server 先把产品事实写入 `product_events`，再异步 best-effort 转发注册、支付、订阅等白名单业务事实到 PostHog；LLM / TTS per-request 事件不转发，PostHog 失败也不能影响请求主链路。 +- 前端 `posthog-js` 通过 `packages/stage-ui/src/stores/analytics/posthog.ts` 初始化，三个 app（web / desktop / pocket）按 `isStageTamagotchi()` 等选 project key +- Server 不接入 `posthog-node`。后端产品事件写 `product_events`，Grafana 展示低基数聚合；需要 PostHog revenue/person analytics 时优先用 PostHog Stripe source connector 或离线导入，不允许在 API 请求路径同步发 PostHog。 - 前端 identity：`useSharedAnalyticsStore.initialize()` watch `authStore.isAuthenticated` 自动调 `posthog.identify(user.id)` / `reset()` -- 平台统一写入 `app_surface`；`entry_surface` 只表示 `settings_flux` 这类业务入口，避免同名字段混用或覆盖 PostHog super property。 +- Conversation controls 事件的 `surface` 由 `packages/stage-ui/src/composables/use-analytics.ts` 统一按 runtime 推断，UI 调用点只传业务字段。 已埋点： | 域 | 事件 | 来源 | 落点 | Truth | |---|---|---|---|---| | 付费漏斗 | `pricing_page_viewed` / `plan_selected` / `checkout_started` | 前端 | `packages/stage-pages/src/pages/settings/flux.vue` | PostHog | -| 付费漏斗终点 | `payment_completed` | 后端 webhook | `product_events` + Grafana，并 best-effort 转发 PostHog | Postgres | +| 付费漏斗终点 | `payment_completed` | 后端 webhook | `product_events` + Grafana；PostHog 走 Stripe source connector/离线导入 | Postgres | | Activation / Retention | `first_model_selected` / `model_switched` | 前端（consciousness store watcher） | `packages/stage-ui/src/stores/analytics/index.ts` | PostHog | | Retention | `character_created` | 前端 | `apps/stage-web/src/pages/settings/characters/components/CharacterDialog.vue` | PostHog | | Retention | `chat_session_started` | 前端 | `packages/stage-ui/src/components/scenarios/chat/components/sessions-drawer.vue` | PostHog | @@ -130,16 +130,13 @@ | Conversation controls | `chat_message_deleted` / `chat_messages_cleared` / `chat_message_retried` | 前端 | `packages/stage-layouts/src/components/Layouts/*InteractiveArea.vue` / `packages/stage-layouts/src/components/Widgets/ChatActionButtons.vue` / `apps/stage-tamagotchi/src/renderer/components/InteractiveArea.vue` | PostHog | | Conversation controls | `tts_stop_clicked` | 前端 | `packages/stage-layouts/src/composables/useStopSpeakingButton.ts` | PostHog | | Churn | `subscription_cancelled`（带 cancellation_reason） | 外部 Stripe 数据源 | PostHog Stripe source connector | Stripe/Postgres | -| 老事件 | `provider_card_clicked` | 前端 | `packages/stage-ui/src/composables/use-analytics.ts` | PostHog | -| 兼容指标 | `first_message_sent` | 前端 | 仍有生产者，仅供历史 dashboard；新激活口径使用 `chat_activation_succeeded` | PostHog | -| 聊天轮次 | `message_send_started` / `message_sent` / `llm_*` / `message_round` / `message_round_failed` | 前端 core runtime | 以 `conversation_id` / `round_id` / `turn_index` 关联；成功和失败各有唯一终点事件 | PostHog | -| 注册 UI | `signup_form_completed` | auth SPA | 匿名表单完成信号，不计作注册事实 | PostHog | -| 注册事实 | `signup_completed` | Better Auth user create hook | 仅服务端生产，以 Better Auth user id 识别 | Postgres + PostHog | +| 老事件 | `provider_card_clicked` / `first_message_sent` | 前端 | `packages/stage-ui/src/composables/use-analytics.ts` | PostHog | 待埋点（API 已在 `use-analytics.ts` 暴露但调用点未接入）： | 域 | 事件 | 状态 | |---|---|---| +| Activation | `user_signed_up` | 等接到 auth callback 完成事件（Better Auth 的 signUp 成功 hook） | | Retention | `voice_mode_activated` | 需要先在 hearing store 加显式 `enableVoiceMode` action — 当前 hearing 没有单一"用户主动启用"那一刻的 trigger，被动监听 + 录音 action 不构成 user intent 信号 | | Feature adoption | `flux_image_generated` | 等图片生成 feature 上线 | @@ -153,21 +150,22 @@ ## PostHog 接入路线图 -落地分两步，**不要一次性埋全部事件**，否则 schema 漂移会很快出现。PostHog 采集以前端为主；服务端只经 product-events 白名单转发业务事实（注册、支付、订阅），per-request 路径仍只写 Postgres/Grafana。 +落地分两步，**不要一次性埋全部事件**，否则 schema 漂移会很快出现。PostHog 采集只发生在前端或外部 source connector；server 请求路径只写 Postgres/Grafana。 ### 阶段 1（P0 — 付费漏斗 + activation） -所有运行端共用根目录 `posthog.config.ts`（单一 project key，`app_surface` super property 区分端）。初始化实况： +`apps/stage-web`： ```ts -import { DEFAULT_POSTHOG_CONFIG, POSTHOG_PROJECT_KEY } from '../posthog.config' +import posthog from 'posthog-js' -// DEFAULT_POSTHOG_CONFIG 内含 defaults: '2025-05-24'： -// SPA 路由切换自动发 $pageview / $pageleave，页面浏览不再手动埋。 -posthog.init(POSTHOG_PROJECT_KEY, { ...DEFAULT_POSTHOG_CONFIG }) -// 登录后（stage 端在 analytics store，auth 端在 profile.vue） +posthog.init(import.meta.env.VITE_POSTHOG_KEY, { + api_host: 'https://us.i.posthog.com', + capture_pageview: false, // 手动 capture 控制语义 +}) +// 登录后 posthog.identify(user.id) -// 在 flux.vue +// 在 pricing.vue posthog.capture('pricing_page_viewed', { plan_period, source }) ``` @@ -179,20 +177,20 @@ posthog.capture('pricing_page_viewed', { plan_period, source }) import posthog from 'posthog-js/dist/module.full.no-external.js' posthog.init(import.meta.env.VITE_POSTHOG_KEY, { - api_host: 'https://t.airi.build', + api_host: 'https://us.i.posthog.com', autocapture: false, // 桌面应用没有传统 URL 路由，手动控制 }) ``` 埋点事件清单（P0）： -- 前端：`pricing_page_viewed`、`plan_selected`、`checkout_started`、`signup_form_completed`（ui-server-auth 匿名邮箱表单里程碑）、`first_model_selected` -- 后端：`product_events` 是事实账本；其中业务事实白名单（`signup_completed`、`payment_completed`、`subscription_started/renewed/cancelled`）由 product-events 服务经 posthog-node 转发一份到 PostHog（`apps/server/src/services/domain/product-events.ts`，distinctId = Better Auth user id）。LLM / TTS 等 per-request 事件不转发 +- 前端：`pricing_page_viewed`、`plan_selected`、`checkout_started`、`user_signed_up`、`first_message_sent`、`first_model_selected` +- 后端：不直接发 PostHog；`payment_completed` 写入 `product_events`，PostHog 侧通过 Stripe source connector 或离线导入展示 PostHog UI 配两个 funnel： -- **付费漏斗** (7d 窗口)：`pricing_page_viewed → plan_selected → checkout_started → payment_completed`（最后一步来自服务端转发） -- **激活漏斗** (14d 窗口)：`signup_completed → onboarding_started → chat_activation_succeeded → payment_completed` +- **付费漏斗** (7d 窗口)：`pricing_page_viewed → plan_selected → checkout_started → payment_completed`（最后一步来自 Stripe source connector / 离线导入） +- **激活漏斗** (14d 窗口)：`user_signed_up → first_message_sent → first_model_selected → payment_completed`（最后一步同上） ### 阶段 2（P1 — retention / feature adoption / churn） diff --git a/apps/server/docs/ai-context/product-analytics-dashboard-setup.md b/apps/server/docs/ai-context/product-analytics-dashboard-setup.md index 530dae32e..dadc86833 100644 --- a/apps/server/docs/ai-context/product-analytics-dashboard-setup.md +++ b/apps/server/docs/ai-context/product-analytics-dashboard-setup.md @@ -60,7 +60,7 @@ Steps: Breakdowns: - `provider_mode` -- `app_surface` +- `surface` Filters: @@ -116,26 +116,6 @@ Watch for: - `failure_stage = model_list` - `failure_stage = llm_response` -### Insight 2a: All Message Round Failures - -Type: Trends - -Event: - -- `message_round_failed` - -Breakdowns: - -- `failure_stage` -- `error_code` -- `provider_id` -- `app_surface` - -Watch for: - -- Failures where `turn_index > 1`, which are intentionally outside the activation-failure series. -- Repeated failures for the same `conversation_id` with different `round_id` values. - ### Insight 3: Provider Configuration Health Type: Funnel @@ -233,7 +213,7 @@ Breakdowns: - `stt_provider_id` - `error_code` -- `app_surface` +- `surface` ### Insight 7: Feedback And Bug Reports @@ -249,7 +229,7 @@ Breakdowns: - `category` - `severity` - `entrypoint` -- `app_surface` +- `surface` Watch for: diff --git a/apps/server/docs/ai-context/product-analytics-instrumentation.md b/apps/server/docs/ai-context/product-analytics-instrumentation.md index 0dec0f760..53555391f 100644 --- a/apps/server/docs/ai-context/product-analytics-instrumentation.md +++ b/apps/server/docs/ai-context/product-analytics-instrumentation.md @@ -127,8 +127,7 @@ Grafana 当前不能回答： | Field | Values | Notes | |---|---|---| -| `app_surface` | `web` / `electron` / `mobile` / `auth` / `docs` / `server` | 所有关键事件的平台 / 运行端 | -| `entry_surface` | `settings_flux` / `onboarding` / `chat_toolbar` 等受控枚举 | 业务入口；不得用于表示运行端 | +| `surface` | `web` / `desktop` / `mobile` | 所有关键前端事件必带 | | `provider_mode` | `official` / `custom` / `unknown` | 官方开箱即用 vs 用户自配置 | | `provider_id` | 白名单 ID | 不传 raw URL / raw key / 用户输入 | | `model_id` | 白名单或归一化后的 ID | 自定义模型用 `is_custom_model = true` | @@ -148,7 +147,6 @@ Grafana 当前不能回答： | `chat_activation_started` | frontend | PostHog | 用户进入首次聊天路径或点击发送第一条消息前 | | `chat_activation_succeeded` | frontend | PostHog | 首次消息完成并看到 assistant response | | `chat_activation_failed` | frontend | PostHog | 首次消息未完成，包含配置、网络、鉴权、余额、模型等失败 | -| `message_round_failed` | frontend | PostHog | 任意用户轮次在 assistant response 完成前失败；成功轮次的唯一终点仍为 `message_round` | | `official_provider_selected` | frontend | PostHog | 官方 Provider 被默认落地或在设置页被手动选择，记录 provider id 与是否自动选择 | | `second_turn_started` | frontend | PostHog | 同一会话开始第二轮对话 | @@ -159,13 +157,11 @@ Grafana 当前不能回答： | `provider_mode` | yes | `official` / `custom` | | `provider_id` | yes | 归一化 ID | | `model_id` | yes | 归一化 ID | -| `app_surface` | yes | web / electron / mobile | -| `conversation_id` | yes | 应用会话 ID；同一 conversation 的聊天事件保持一致 | -| `round_id` | yes | 单轮关联 ID，复用该轮 user message id；同一轮所有聊天主链路事件保持一致 | -| `turn_index` | yes | conversation 内从 `1` 开始的用户轮次；`second_turn_started` 固定为 `2` | +| `surface` | yes | web / desktop / mobile | | `time_to_first_message_ms` | success only | 从 app start 或 onboarding complete 到首次成功 | | `error_code` | failed only | 稳定错误码 | | `failure_stage` | failed only | `provider_config` / `model_list` / `message_send` / `llm_response` / `tts` | +| `turn_index` | second turn only | 固定为 `2`，用于首轮成功后的二轮启动 | | `auto_selected` | official provider only | 官方默认 Provider 自动落地时为 `true` | 推荐看板： @@ -173,42 +169,12 @@ Grafana 当前不能回答： - 新用户 `chat_activation_started -> chat_activation_succeeded` 漏斗。 - 按 `provider_mode` 拆分 activation conversion。 - `chat_activation_failed` 按 `failure_stage` / `provider_id` 排名。 -- `message_round_failed` 按 `turn_index` / `failure_stage` / `provider_id` 排名，用于分析激活后的聊天失败。 异常提醒： - 新用户 activation conversion 24h 环比下降超过 15%。 - `provider_mode = official` 的 activation failure 上升，优先排查官方 Provider。 -### AI generation 用量事实 - -`$ai_generation` 是 token / usage / 成本覆盖率事实来源；不要把 Prompt、回复正文、API Key、raw endpoint 发到 PostHog。 - -| Field | Required | Notes | -|---|---|---| -| `$ai_trace_id` | yes | 优先使用真实 `conversation_id`；无客户端会话 header 时用 server request id 兜底 | -| `$ai_session_id` | yes | 与 `conversation_id` 保持一致 | -| `$ai_span_id` | yes | 与 `round_id` / generation id 保持一致 | -| `$ai_model` | yes | 原始生成模型；不要跨 Provider 合并 | -| `$ai_provider` | yes | 实际生成 Provider | -| `airi_user_id` | server yes | Better Auth user id，便于和 Pro / 付费事件关联 | -| `conversation_id` | yes | 始终存在；结合 `conversation_id_source` 判断是否真实应用会话 | -| `conversation_id_source` | yes | `client_header` / `client_runtime` / `server_request` | -| `round_id` | yes | 单轮或 request-level generation id | -| `app_surface` | when known | 只表示用户产品端：`web` / `electron` / `mobile`；不要用 `server` 兜底 | -| `capture_surface` | yes | 事件采集端：`client` / `server` | -| `usage_source` | yes | `reported` / `estimated` / `unavailable` | -| `token_usage_available` | yes | token 是否可用于聚合；日报 token 分析先过滤 `true` | -| `cost_usd_source` | yes | `reported` / `estimated` / `unavailable` | -| `cost_usd_known` | yes | `false` 不能按零成本处理，只能计入未知成本覆盖率 | - -日报里的 Pro token / cost： - -- Token 总量、P50、P90：只统计 `token_usage_available = true`。 -- AIRI USD 成本：只统计 `cost_usd_known = true`；`cost_usd_known = false` 单独报 unknown generation count / coverage。 -- 服务端 request fallback：`conversation_id_source = server_request` 只能做 request 级 usage，不能和 `message_round` 当成同一应用会话 join。 -- 模型分析按 `$ai_provider + $ai_model` 看；不要新增 `canonical_model` / `model_family` 把不同供应链揉在一起。 - ### Provider And Model Configuration 目标：定位配置复杂和失败劝退。 @@ -316,14 +282,14 @@ Grafana 当前不能回答： | Field | Required | Notes | |---|---|---| | `stt_provider_id` | yes | 归一化 ID | -| `app_surface` | yes | web / electron / mobile | +| `surface` | yes | web / desktop / mobile | | `duration_ms` | no | 用户按住或录音时长 | | `error_code` | failed only | `permission_denied` / `device_unavailable` / `provider_error` / `timeout` | 推荐看板： - Voice input start -> STT success funnel。 -- Permission denied rate by browser / app surface。 +- Permission denied rate by browser / surface。 - STT failure rate by Provider。 ### Feedback And Bug Reports @@ -346,7 +312,7 @@ Grafana 当前不能回答： | `severity` | yes | `blocker` / `major` / `minor` / `suggestion` | | `user_type` | yes | `new_user` / `paid_user` / `overseas_user` / `developer_user` / `role_chat_user` / `unknown` | | `entrypoint` | yes | `about_update_error` / `community_manual_tag` 等低基数入口 | -| `app_surface` | in-app | web / electron / mobile | +| `surface` | in-app | web / desktop / mobile | | `provider_mode` | no | 可从最近一次配置状态补齐 | | `description_length_bucket` | bug report | `empty` / `short` / `medium` / `long`，不要上传正文 | | `include_triage_context` | bug report | 是否附带页面上下文 | @@ -429,7 +395,7 @@ Grafana 当前不能回答： - `has_voice` - `latency_ms` - `error_code` -- `app_surface` +- `surface` 看板： @@ -454,7 +420,7 @@ Grafana 当前不能回答： - `character_type`: `built_in` / `imported` / `custom` - `has_voice` - `voice_type` -- `app_surface` +- `surface` 看板： @@ -472,7 +438,7 @@ Grafana 当前不能回答： 字段： -- `entry_surface`（付费入口，例如 `settings_flux`；运行端使用 `app_surface`） +- `surface` - `balance_state` - `plan_id` - `currency` @@ -491,8 +457,6 @@ Grafana 当前不能回答： |---|---|---| | `first_message_sent` | 保留历史指标 | 继续用于老 dashboard；新激活口径用 `chat_activation_succeeded` | | `chat_activation_started` / `chat_activation_succeeded` / `chat_activation_failed` | 新核心 activation 口径 | 用来回答“用户能不能正常开始聊天” | -| `message_send_started` / `message_sent` / `llm_*` / `message_round` / `message_round_failed` | 聊天主链路 | 共享 `conversation_id` / `round_id` / `turn_index`；`message_round` 和 `message_round_failed` 分别是单轮成功 / 失败的唯一终点；不再重复发送 `chat_started`、`assistant_response_completed`、`chat_failed` 或 chat 的通用 `feature_used` 别名 | -| `signup_form_completed` / `signup_completed` | UI 里程碑 / 注册事实 | 前者可匿名；后者只由服务端按 Better Auth user id 发送，禁止复用同名客户端事件 | | `provider_card_clicked` | 保留入口点击 | 不等于配置成功；成功 / 失败看 `provider_config_succeeded` / `provider_config_failed` | | `first_model_selected` / `model_switched` | 保留模型选择行为 | 配置链路和模型列表健康看 `model_list_loaded` / `model_list_failed` | | `stt_started` / `stt_succeeded` / `stt_failed` | 保留 STT Provider 结果 | 权限和设备问题用新增 `microphone_*` / `audio_device_unavailable` 拆开 | @@ -509,9 +473,9 @@ PostHog 线上已经能看到 `model` / `model_id` 存在自由文本风险。 - Provider、model、voice 使用稳定 ID。 - 自定义值不要直接 group-by。 -- 不跨 Provider 合并模型名：`deepseek-chat` 和 `deepseek/deepseek-chat` 可能代表不同供应链 / 成本口径，日报按原始 Provider + model 组合看。 - 自定义模型传： - `provider_id = custom` + - `model_family = custom` - `is_custom_model = true` - `custom_model_hash` 可选，必须单向 hash，不能还原原文。 - 自定义 voice 传： @@ -531,8 +495,7 @@ PostHog 线上已经能看到 `model` / `model_id` 存在自由文本风险。 | `voice_pack_id` | medium | 只进 PostHog / Postgres，不进 Prometheus label | | `error_code` | low | enum | | `source` | low | enum | -| `app_surface` | low | runtime enum | -| `entry_surface` | low | 低基数业务入口 enum；不得复用为 runtime | +| `surface` | low | enum | Prometheus label 不放 `user_id`、`session_id`、`voice_pack_id`、自定义模型名、自定义音色名。 @@ -710,7 +673,7 @@ Prometheus label 不放 `user_id`、`session_id`、`voice_pack_id`、自定义 - Top failing providers: - Top failing error codes: -- Rage-click pages / app surfaces: +- Rage-click pages / surfaces: - Discord / QQ feedback categories: - performance: - config: @@ -746,7 +709,7 @@ Prometheus label 不放 `user_id`、`session_id`、`voice_pack_id`、自定义 给负责上手体验的人看： - Funnel：`app_loaded -> chat_activation_started -> provider_config_succeeded -> model_list_loaded -> chat_activation_succeeded` -- Breakdown：`provider_mode`、`app_surface`、`region` +- Breakdown：`provider_mode`、`surface`、`region` - Table：Top `provider_config_failed` by `provider_id` / `error_code` - Timeseries：`time_to_first_message_ms` p50 / p95 @@ -766,7 +729,7 @@ Prometheus label 不放 `user_id`、`session_id`、`voice_pack_id`、自定义 - Grafana：5xx、LLM latency、provider failure、TTS blocked - PostHog：rage-click trend、failed frontend events -- Table：Top error_code by app surface / provider +- Table：Top error_code by surface / provider - Community tags：Discord / QQ 反馈分类趋势 ### 分析方法 @@ -800,7 +763,7 @@ Prometheus label 不放 `user_id`、`session_id`、`voice_pack_id`、自定义 - `chat_activation_failed` by `failure_stage` - `provider_config_failed` by `error_code` - `model_list_failed` by `provider_id` -- `$rageclick` by page / `app_surface` +- `$rageclick` by page / surface - Discord / QQ `category = bug` 的高频词 日报只报异常；周报把异常和社区反馈合并成“优先修复建议”。 @@ -900,14 +863,11 @@ Prometheus label 不放 `user_id`、`session_id`、`voice_pack_id`、自定义 10. 建异常检查：activation、provider config、model list、STT、TTS blocked、bug report。 11. 建周报模板：自动填指标，社区负责人补充 Discord / QQ 反馈解释和下周建议。 -### 当前接入状态（2026-07-10） +### 当前接入状态（2026-06-30） 已接入代码： -- Chat activation：`chat_activation_started`、`chat_activation_succeeded`、`chat_activation_failed` 只覆盖每个 conversation 首次 assistant response 之前的尝试；后续轮次继续发 message / latency events，第二轮单独发 `second_turn_started`。 -- Chat correlation：每次发送以 user message id 作为 `round_id`；activation、message、LLM latency、render、`message_round` 和 `message_round_failed` 事件共享 `conversation_id`、`round_id`、`turn_index`。 -- Identity：匿名 auth SPA 发 `signup_form_completed`；只有服务端 Better Auth user create hook 发 identified `signup_completed`。平台统一使用 `app_surface`，业务入口统一使用 `entry_surface`。 -- Chat event reuse：主链路使用 `message_send_started`、`message_sent`、`llm_*`、`message_round`、`message_round_failed`；每轮成功 / 失败各自只有一个终点事件，不再发送 `chat_started`、`assistant_response_completed`、`chat_failed` 和 chat 的通用 `feature_used` 别名。 +- Chat activation：`chat_activation_started`、`chat_activation_succeeded`、`chat_activation_failed`、`second_turn_started`；官方 Provider 选择事件为 `official_provider_selected`，实际聊天使用口径看 activation events 的 `provider_mode = official`。 - Model list：`model_list_loaded`、`model_list_failed`。 - Provider config：`provider_config_started`、`provider_config_succeeded`、`provider_config_failed`。 - TTS voice：`tts_provider_selected`、`voice_selected`、`voice_preview_played`、`voice_pack_bound`、`official_tts_exposed`、`official_tts_preview_started`、`official_tts_preview_succeeded`、`official_tts_auto_enabled`。 diff --git a/apps/server/docs/ai-context/verifications/posthog-forwarding-and-pageview.md b/apps/server/docs/ai-context/verifications/posthog-forwarding-and-pageview.md deleted file mode 100644 index e1687bca3..000000000 --- a/apps/server/docs/ai-context/verifications/posthog-forwarding-and-pageview.md +++ /dev/null @@ -1,26 +0,0 @@ -# Verification: 服务端 PostHog 转发 + SPA 路由 pageview - -Status: **transport 与 pageview 已真实验证；线上 Stripe webhook 端到端待部署后确认** -Owner: Product Analytics -Last updated: 2026-07-08 -Environment: commit 689f02ac4 + 本次工作区改动；posthog-node 5.39.4；posthog-js 1.306.1；Node 26.3.0 - -## 用户路径 - -- **场景 1**：用户完成 Stripe 支付 → webhook 写 `product_events` → 服务端把 `payment_completed` 转发到 PostHog（distinctId = Better Auth user id）→ PostHog 付费漏斗在 `checkout_started` 后闭环。 -- **场景 2**：用户在 stage-web 内切换路由（如进入 `/settings/flux`）→ PostHog 收到 `$pageview`，带 `$pathname`、上一页路径与停留时长。 - -## 已验证证据 - -| 验证项 | 命令 / 方式 | 实际输出 | -|---|---|---| -| 转发白名单与映射 | `pnpm exec vitest run src/services/domain/product-events.test.ts`（apps/server） | 6 passed：`payment_completed` 原名转发、`user_signed_up→signup_completed` 映射、per-request 动作不转发、sink 抛错时 DB 行仍落库且 track 不抛 | -| posthog-node 真实传输 | `node posthog-smoke.mjs`（captureImmediate → us.i.posthog.com，生产 project key，事件名 `server_forwarding_smoke_test`） | `captureImmediate resolved in 1380ms` + `shutdown clean` | -| SPA 路由 pageview | `VITE_ENABLE_POSTHOG=true pnpm -F @proj-airi/stage-web dev` + agent-browser 两次 `history.pushState` | 两条 `$pageview`，`$pathname` 分别为 `/settings/flux`、`/settings/airi-card`，`navigation_type: pushState`，携带 `$prev_pageview_duration` 与 `app_surface: web` super property；批量 POST `us.i.posthog.com/e/` 返回 200 | -| 服务端 typecheck / lint | `pnpm -F @proj-airi/server typecheck`、eslint 改动文件 | 均通过 | - -## 注意事项 - -- 自动化浏览器（`navigator.webdriver = true`）会被 posthog-js 默认 bot 过滤静默丢弃事件；本次浏览器验证通过会话内 `set_config({ opt_out_useragent_filter: true })` 绕过，仅影响该验证会话，生产配置未改。人工复测时用普通浏览器即可，无需任何绕过。 -- 服务端转发默认开启：`POSTHOG_PROJECT_KEY` 的默认值就是前端共用的 phc_* project key，置空字符串可关闭。部署后在 PostHog 里确认 `payment_completed` 事件出现在真实支付后，即完成端到端收尾。 -- PostHog 项目里会留有一条 `server_forwarding_smoke_test`（distinctId `verification-smoke`）测试事件，分析时按事件名过滤。 diff --git a/apps/server/docs/ai-context/verifications/product-analytics-smoke.md b/apps/server/docs/ai-context/verifications/product-analytics-smoke.md index 5db13df56..35948a7b3 100644 --- a/apps/server/docs/ai-context/verifications/product-analytics-smoke.md +++ b/apps/server/docs/ai-context/verifications/product-analytics-smoke.md @@ -2,7 +2,7 @@ Status: **code-level instrumentation verified; live PostHog dashboard updated; Grafana dashboard updated; alert setup pending** Owner: Community / Product Analytics -Last updated: 2026-07-10 +Last updated: 2026-07-01 Related: - [`product-analytics-instrumentation.md`](../product-analytics-instrumentation.md) - [`product-analytics-dashboard-setup.md`](../product-analytics-dashboard-setup.md) @@ -20,7 +20,6 @@ Related: |---|---| | Frontend analytics API | `packages/stage-ui/src/composables/use-analytics.test.ts` 覆盖 activation、model list、provider config、voice selection、voice input、feedback event API | | Chat activation hooks | `packages/core-agent/src/runtime/chat-orchestrator-runtime.test.ts` 覆盖 activation started / succeeded / failed hook | -| Chat round failures | core runtime 与 stage contract tests 覆盖激活前和激活后的 `message_round_failed`，并验证 `conversation_id` / `round_id` / `turn_index` | | Voice input failures | `packages/stage-ui/src/composables/audio/audio-device.test.ts` 与 `packages/stage-ui/src/stores/modules/hearing.analytics.test.ts` 覆盖 permission / device / cancel / STT failed | | Server TTS metadata | `apps/server/src/routes/openai/v1/route.test.ts` 与 `apps/server/src/routes/audio-speech-ws/route.test.ts` 覆盖 REST / WS TTS `voice_id`、`voice_type`、`voice_pack_id` metadata | | Grafana product row | `apps/server/otel/grafana/dashboards/build.test.ts` 覆盖 Product Analytics panels、layout references、PromQL 不包含 high-cardinality voice / user fields | @@ -52,10 +51,8 @@ Required properties: provider_mode = official provider_id = model_id = -app_surface = web | mobile | electron -conversation_id = -round_id = -turn_index = 1 for the first round; 2 for second_turn_started and the second round +surface = web | mobile | electron +turn_index = 2 ``` Fail if: @@ -63,52 +60,9 @@ Fail if: - `chat_activation_started` appears but `chat_activation_succeeded` never appears for a successful chat. - The second message is sent but `second_turn_started` does not appear. - `provider_mode` is missing or always `unknown`. -- `app_surface` is missing. -- Any chat-chain event is missing `conversation_id`, `round_id`, or `turn_index`. -- Events from one round disagree on `round_id`, or two different rounds reuse the same `round_id`. +- `surface` is missing. -### 1a. PostHog: message round failure - -Action: - -1. Complete a successful first chat round. -2. Force the second round to fail before the assistant response completes. - -Expected PostHog events: - -```text -message_round_failed -``` - -Required properties: - -```text -conversation_id = -round_id = -turn_index = 2 -provider_id = -model_id = -failure_stage = llm_response -error_code = llm_response_failed -app_surface = web | mobile | electron -``` - -Fail if: - -- The failed second round has no `message_round_failed` event. -- The failed round emits `message_round`, `chat_failed`, or `assistant_response_completed` as an alias. -- A new `chat_activation_failed` appears after the conversation already completed its first assistant response. -- Correlation keys disagree with the failed round's preceding message / LLM events. - -### 1b. Signup identity ownership - -1. Complete an email signup in the auth SPA. -2. Confirm the auth SPA emits `signup_form_completed` with `app_surface = auth`. -3. Confirm the Better Auth user-create hook emits exactly one `signup_completed` with `app_surface = server` and the Better Auth user id as `distinctId`. - -Fail if the auth SPA emits `signup_completed`, or if the server event lands on a different PostHog person from later identified onboarding events. - -### 1c. PostHog: official provider selection +### 1b. PostHog: official provider selection Action: @@ -265,7 +219,7 @@ paywall_seen Required properties: ```text -entry_surface = settings_flux +surface = settings_flux reason = manual_topup flux_balance_bucket = zero | 1_100 | 101_1000 | 1001_10000 | 10000_plus | unknown ``` diff --git a/apps/server/otel/grafana/dashboards/airi-server-overview-cloud.json b/apps/server/otel/grafana/dashboards/airi-server-overview-cloud.json index 2ba18149b..9d1aaf0b4 100644 --- a/apps/server/otel/grafana/dashboards/airi-server-overview-cloud.json +++ b/apps/server/otel/grafana/dashboards/airi-server-overview-cloud.json @@ -41,10 +41,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "max(user_total{service_name=~\"$service\", deployment_environment=~\"$env\"})", - "legendFormat": "total users", - "range": true + "legendFormat": "total users" }, "version": "v0" }, @@ -62,10 +60,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum(increase(user_registered_total{service_name=~\"$service\", deployment_environment=~\"$env\"}[24h]))", - "legendFormat": "new today", - "range": true + "legendFormat": "new today" }, "version": "v0" }, @@ -123,7 +119,7 @@ "wideLayout": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -145,10 +141,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "avg(user_active_sessions{service_name=~\"$service\", deployment_environment=~\"$env\"})", - "legendFormat": "sessions", - "range": true + "legendFormat": "sessions" }, "version": "v0" }, @@ -206,7 +200,7 @@ "wideLayout": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -228,10 +222,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum(rate(http_server_request_duration_seconds_count{service_name=~\"$service\", deployment_environment=~\"$env\", http_request_method!=\"OPTIONS\"}[5m]))", - "legendFormat": "req/s", - "range": true + "legendFormat": "req/s" }, "version": "v0" }, @@ -296,7 +288,7 @@ "wideLayout": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -318,10 +310,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "100 * sum(rate(http_server_request_duration_seconds_count{service_name=~\"$service\", deployment_environment=~\"$env\", http_request_method!=\"OPTIONS\", http_response_status_code=~\"5..\"}[5m])) / clamp_min(sum(rate(http_server_request_duration_seconds_count{service_name=~\"$service\", deployment_environment=~\"$env\", http_request_method!=\"OPTIONS\"}[5m])), 1)", - "legendFormat": "fail %", - "range": true + "legendFormat": "fail %" }, "version": "v0" }, @@ -386,7 +376,7 @@ "sizing": "auto" } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -408,10 +398,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum(rate(gen_ai_client_operation_count_total{service_name=~\"$service\", deployment_environment=~\"$env\"}[5m]))", - "legendFormat": "req/s", - "range": true + "legendFormat": "req/s" }, "version": "v0" }, @@ -468,7 +456,7 @@ "wideLayout": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -490,10 +478,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "max(user_active_rolling{service_name=~\"$service\", deployment_environment=~\"$env\", window=\"24h\"})", - "legendFormat": "DAU", - "range": true + "legendFormat": "DAU" }, "version": "v0" }, @@ -552,7 +538,7 @@ "wideLayout": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -574,10 +560,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "max(user_active_rolling{service_name=~\"$service\", deployment_environment=~\"$env\", window=\"7d\"})", - "legendFormat": "WAU", - "range": true + "legendFormat": "WAU" }, "version": "v0" }, @@ -636,7 +620,7 @@ "wideLayout": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -658,10 +642,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "max(user_active_rolling{service_name=~\"$service\", deployment_environment=~\"$env\", window=\"30d\"})", - "legendFormat": "MAU", - "range": true + "legendFormat": "MAU" }, "version": "v0" }, @@ -720,7 +702,7 @@ "wideLayout": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -742,10 +724,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum(ws_connections_active{service_name=~\"$service\", deployment_environment=~\"$env\"})", - "legendFormat": "online", - "range": true + "legendFormat": "online" }, "version": "v0" }, @@ -804,7 +784,7 @@ "wideLayout": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -826,10 +806,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum(ws_connections_active{service_name=~\"$service\", deployment_environment=~\"$env\"})", - "legendFormat": "connections", - "range": true + "legendFormat": "connections" }, "version": "v0" }, @@ -912,8 +890,6 @@ "max" ], "displayMode": "table", - "enableFacetedFilter": false, - "overflow": "ellipsis", "placement": "right", "showLegend": true }, @@ -924,7 +900,7 @@ } } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -946,10 +922,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum(increase(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\"}[$__range]))", - "legendFormat": "events", - "range": true + "legendFormat": "events" }, "version": "v0" }, @@ -1008,7 +982,7 @@ "wideLayout": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -1030,10 +1004,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "100 * sum(increase(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\", status=\"failed\"}[$__range])) / clamp_min(sum(increase(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\"}[$__range])), 1)", - "legendFormat": "failed %", - "range": true + "legendFormat": "failed %" }, "version": "v0" }, @@ -1099,7 +1071,7 @@ "sizing": "auto" } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -1121,7 +1093,6 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "topk(12, sum by (feature, action, status) (increase(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\"}[$__range])))", "legendFormat": "{{feature}} · {{action}} · {{status}}", "instant": true, @@ -1183,7 +1154,7 @@ "valueMode": "color" } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -1205,10 +1176,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum by (feature, action, status) (rate(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\"}[$__rate_interval]))", - "legendFormat": "{{feature}} · {{action}} · {{status}}", - "range": true + "legendFormat": "{{feature}} · {{action}} · {{status}}" }, "version": "v0" }, @@ -1291,8 +1260,6 @@ "max" ], "displayMode": "table", - "enableFacetedFilter": false, - "overflow": "ellipsis", "placement": "right", "showLegend": true }, @@ -1303,7 +1270,7 @@ } } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -1325,10 +1292,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", - "expr": "max (user_active_rolling{service_name=~\"$service\", deployment_environment=~\"$env\", window=\"24h\"})", - "legendFormat": "__auto", - "range": true + "expr": "100 * sum(increase(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\", feature=\"tts\", action=\"speech_succeeded\", status=\"succeeded\"}[$__range])) / clamp_min(sum(increase(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\", feature=\"tts\", action=\"speech_requested\", status=\"started\"}[$__range])), 1)", + "legendFormat": "success %" }, "version": "v0" }, @@ -1340,10 +1305,182 @@ "transformations": [] } }, - "description": "", + "description": "Server-side TTS successes divided by TTS requests over the dashboard range. Includes REST and WS TTS product events. Drops here mean users are asking for speech but not receiving audio; inspect failed/blocked panels next.", "id": 99, "links": [], - "title": "日活", + "title": "TTS Success %", + "vizConfig": { + "group": "gauge", + "kind": "VizConfig", + "spec": { + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": 0 + }, + { + "color": "yellow", + "value": 90 + }, + { + "color": "green", + "value": 98 + } + ] + }, + "unit": "percent", + "decimals": 2, + "noValue": "0", + "min": 0, + "max": 100 + }, + "overrides": [] + }, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto" + } + }, + "version": "13.0.0-23630096546" + } + } + }, + "panel-100": { + "kind": "Panel", + "spec": { + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "hidden": false, + "query": { + "datasource": { + "name": "grafanacloud-projairi-prom" + }, + "group": "prometheus", + "kind": "DataQuery", + "spec": { + "expr": "topk(12, sum by (action, status, source) (increase(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\", feature=\"tts\", action=~\"speech_failed|speech_blocked\"}[$__range])))", + "legendFormat": "{{action}} · {{status}} · {{source}}", + "instant": true, + "range": false + }, + "version": "v0" + }, + "refId": "A" + } + } + ], + "queryOptions": {}, + "transformations": [] + } + }, + "description": "TTS user-impacting failures over the dashboard range, split by action/status/source. `speech_failed` usually means upstream/runtime failure; `speech_blocked` usually means balance/preflight blocked. Keep voice/model drilldown in Postgres metadata, not Prometheus labels.", + "id": 100, + "links": [], + "title": "TTS Failed / Blocked (range)", + "vizConfig": { + "group": "bargauge", + "kind": "VizConfig", + "spec": { + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "short", + "noValue": "0" + }, + "overrides": [] + }, + "options": { + "displayMode": "gradient", + "maxVizHeight": 300, + "minVizHeight": 12, + "minVizWidth": 8, + "namePlacement": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showUnfilled": true, + "sizing": "auto", + "valueMode": "color" + } + }, + "version": "13.0.0-23630096546" + } + } + }, + "panel-101": { + "kind": "Panel", + "spec": { + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "hidden": false, + "query": { + "datasource": { + "name": "grafanacloud-projairi-prom" + }, + "group": "prometheus", + "kind": "DataQuery", + "spec": { + "expr": "sum by (source, action, status) (rate(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\", feature=\"tts\"}[$__rate_interval]))", + "legendFormat": "{{source}} · {{action}} · {{status}}" + }, + "version": "v0" + }, + "refId": "A" + } + } + ], + "queryOptions": {}, + "transformations": [] + } + }, + "description": "TTS product event rate by source and action. Use this to distinguish chat auto-TTS, manual previews/settings tests, and API audio.speech traffic when speech health changes.", + "id": 101, + "links": [], + "title": "TTS Event Rate by Source", "vizConfig": { "group": "timeseries", "kind": "VizConfig", @@ -1351,8 +1488,7 @@ "fieldConfig": { "defaults": { "color": { - "mode": "continuous-GrYlRd", - "seriesBy": "last" + "mode": "palette-classic" }, "custom": { "axisBorderShow": false, @@ -1363,20 +1499,17 @@ "barAlignment": 0, "barWidthFactor": 0.6, "drawStyle": "line", - "fillOpacity": 17, - "gradientMode": "scheme", + "fillOpacity": 15, + "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, - "lineInterpolation": "linear", - "lineStyle": { - "fill": "solid" - }, - "lineWidth": 2, - "pointSize": 3, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, "scaleDistribution": { "type": "linear" }, @@ -1397,13 +1530,10 @@ { "color": "green", "value": 0 - }, - { - "color": "red", - "value": 80 } ] - } + }, + "unit": "eps" }, "overrides": [] }, @@ -1413,21 +1543,188 @@ "multiLane": false }, "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "right", "showLegend": true }, "tooltip": { "hideZeros": false, - "mode": "single", - "sort": "none" + "mode": "multi", + "sort": "desc" } } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" + } + } + }, + "panel-102": { + "kind": "Panel", + "spec": { + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "hidden": false, + "query": { + "datasource": { + "name": "grafanacloud-projairi-prom" + }, + "group": "prometheus", + "kind": "DataQuery", + "spec": { + "expr": "topk(12, sum by (reason, source) (increase(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\", feature=\"tts\", action=\"speech_blocked\", status=\"blocked\"}[$__range])))", + "legendFormat": "{{reason}} · {{source}}", + "instant": true, + "range": false + }, + "version": "v0" + }, + "refId": "A" + } + } + ], + "queryOptions": {}, + "transformations": [] + } + }, + "description": "Blocked TTS events over the dashboard range, grouped by bounded product reason and source. Today this mostly shows insufficient balance; new policy/provider/preflight buckets can be added without exposing user, voice, or model labels.", + "id": 102, + "links": [], + "title": "TTS Blocked by Reason", + "vizConfig": { + "group": "bargauge", + "kind": "VizConfig", + "spec": { + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "short", + "noValue": "0" + }, + "overrides": [] + }, + "options": { + "displayMode": "gradient", + "maxVizHeight": 300, + "minVizHeight": 12, + "minVizWidth": 8, + "namePlacement": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showUnfilled": true, + "sizing": "auto", + "valueMode": "color" + } + }, + "version": "13.0.0-23630096546" + } + } + }, + "panel-103": { + "kind": "Panel", + "spec": { + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "hidden": false, + "query": { + "datasource": { + "name": "grafanacloud-projairi-prom" + }, + "group": "prometheus", + "kind": "DataQuery", + "spec": { + "expr": "topk(8, sum by (flux_balance_bucket, source) (increase(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\", feature=\"tts\", action=\"speech_blocked\", status=\"blocked\", flux_balance_bucket!=\"\"}[$__range])))", + "legendFormat": "{{flux_balance_bucket}} · {{source}}", + "instant": true, + "range": false + }, + "version": "v0" + }, + "refId": "A" + } + } + ], + "queryOptions": {}, + "transformations": [] + } + }, + "description": "Blocked TTS events over the dashboard range, grouped by coarse Flux balance bucket. This helps separate truly empty accounts from low-balance accounts without exposing exact user balances.", + "id": 103, + "links": [], + "title": "TTS Blocked by Flux Bucket", + "vizConfig": { + "group": "bargauge", + "kind": "VizConfig", + "spec": { + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "short", + "noValue": "0" + }, + "overrides": [] + }, + "options": { + "displayMode": "gradient", + "maxVizHeight": 300, + "minVizHeight": 12, + "minVizWidth": 8, + "namePlacement": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showUnfilled": true, + "sizing": "auto", + "valueMode": "color" + } + }, + "version": "13.0.0-23630096546" } } }, @@ -1449,8 +1746,7 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", - "expr": "topk(50, sum by (http_route) (increase(http_server_request_duration_seconds_count{service_name=~\"$service\", deployment_environment=~\"$env\", http_request_method!=\"OPTIONS\", http_route!=\"\"}[$__range])))", + "expr": "topk(10, sum by (http_route) (increase(http_server_request_duration_seconds_count{service_name=~\"$service\", deployment_environment=~\"$env\", http_request_method!=\"OPTIONS\", http_route!=\"\"}[$__range])))", "legendFormat": "{{http_route}}", "instant": true, "range": false @@ -1510,7 +1806,7 @@ "valueMode": "color" } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -1532,10 +1828,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum by (http_response_status_code) (rate(http_server_request_duration_seconds_count{service_name=~\"$service\", deployment_environment=~\"$env\", http_request_method!=\"OPTIONS\"}[$__rate_interval]))", - "legendFormat": "{{http_response_status_code}}", - "range": true + "legendFormat": "{{http_response_status_code}}" }, "version": "v0" }, @@ -1550,7 +1844,7 @@ "description": "HTTP status-code mix over time, one row per status code, colour = request rate in each time bucket. The 200 row dominates in steady state; a 5xx / 4xx row suddenly lighting up flags an incident at a glance. Non-OPTIONS traffic only.", "id": 40, "links": [], - "title": "Status Distribution", + "title": "Error Rate %", "vizConfig": { "group": "heatmap", "kind": "VizConfig", @@ -1610,7 +1904,7 @@ } } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -1632,10 +1926,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", - "expr": "sum by (http_route) (\n rate(http_server_request_duration_seconds_bucket{service_name=~\"$service\", deployment_environment=~\"$env\", http_request_method!=\"OPTIONS\", http_route!~\"/api/v1/openai/.*\", http_response_status_code!=\"404\"}[$__rate_interval])\n)", - "legendFormat": "{{http_route}}", - "range": true + "expr": "histogram_quantile(0.95, sum by (le, http_route) (rate(http_server_request_duration_seconds_bucket{service_name=~\"$service\", deployment_environment=~\"$env\", http_request_method!=\"OPTIONS\", http_route!~\"/api/v1/openai/.*\", http_response_status_code!=\"404\"}[$__rate_interval])))", + "legendFormat": "{{http_route}}" }, "version": "v0" }, @@ -1647,10 +1939,10 @@ "transformations": [] } }, - "description": "", + "description": "P95 latency per Hono-matched route, excluding /api/v1/openai/* (LLM gateway latency lives in the LLM Gateway row). 404s excluded so missing-route noise does not skew the curve.", "id": 20, "links": [], - "title": "Request Latency (by Route)", + "title": "HTTP P95 by Route", "vizConfig": { "group": "timeseries", "kind": "VizConfig", @@ -1718,9 +2010,7 @@ "max" ], "displayMode": "table", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", + "placement": "right", "showLegend": true }, "tooltip": { @@ -1730,7 +2020,7 @@ } } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -1752,10 +2042,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum by (http_route, http_response_status_code) (increase(http_server_request_duration_seconds_count{service_name=~\"$service\", deployment_environment=~\"$env\", http_request_method!=\"OPTIONS\", http_response_status_code!~\"2..|3..|401|402|404\"}[$__rate_interval]))", - "legendFormat": "{{http_response_status_code}} {{http_route}}", - "range": true + "legendFormat": "{{http_response_status_code}} {{http_route}}" }, "version": "v0" }, @@ -1838,8 +2126,6 @@ "max" ], "displayMode": "table", - "enableFacetedFilter": false, - "overflow": "ellipsis", "placement": "right", "showLegend": true }, @@ -1850,7 +2136,7 @@ } } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -1872,10 +2158,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum by (gen_ai_request_model) (rate(gen_ai_client_operation_count_total{service_name=~\"$service\", deployment_environment=~\"$env\", gen_ai_request_model!=\"\"}[$__rate_interval]))", - "legendFormat": "{{gen_ai_request_model}}", - "range": true + "legendFormat": "{{gen_ai_request_model}}" }, "version": "v0" }, @@ -1892,53 +2176,83 @@ "links": [], "title": "LLM Request Rate by Model", "vizConfig": { - "group": "piechart", + "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { - "fixedColor": "#73BF69", "mode": "palette-classic" }, "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" } }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, "unit": "reqps" }, "overrides": [] }, "options": { - "displayLabels": [ - "percent" - ], + "annotations": { + "clustering": -1, + "multiLane": false + }, "legend": { + "calcs": [ + "lastNotNull", + "max" + ], "displayMode": "table", - "overflow": "ellipsis", - "placement": "bottom", + "placement": "right", "showLegend": true }, - "pieType": "pie", - "reduceOptions": { - "calcs": [ - "sum" - ], - "fields": "", - "values": false - }, - "sort": "desc", "tooltip": { "hideZeros": false, - "mode": "single", - "sort": "none" + "mode": "multi", + "sort": "desc" } } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -1960,10 +2274,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", - "expr": "sum by () (rate(gen_ai_client_first_token_duration_seconds_bucket{service_name=~\"$service\", deployment_environment=~\"$env\"}[$__rate_interval]))", - "legendFormat": "TTFB p95", - "range": true + "expr": "histogram_quantile(0.95, sum by (le) (rate(gen_ai_client_first_token_duration_seconds_bucket{service_name=~\"$service\", deployment_environment=~\"$env\"}[$__rate_interval])))", + "legendFormat": "TTFB p95" }, "version": "v0" }, @@ -1981,10 +2293,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "histogram_quantile(0.95, sum by (le) (rate(gen_ai_client_operation_duration_seconds_bucket{service_name=~\"$service\", deployment_environment=~\"$env\"}[$__rate_interval])))", - "legendFormat": "end-to-end p95", - "range": true + "legendFormat": "end-to-end p95" }, "version": "v0" }, @@ -2067,8 +2377,6 @@ "max" ], "displayMode": "table", - "enableFacetedFilter": false, - "overflow": "ellipsis", "placement": "right", "showLegend": true }, @@ -2079,7 +2387,7 @@ } } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -2101,10 +2409,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum by (provider) (rate(gen_ai_client_operation_count_total{service_name=~\"$service\", deployment_environment=~\"$env\", provider!=\"\"}[$__rate_interval]))", - "legendFormat": "{{provider}}", - "range": true + "legendFormat": "{{provider}}" }, "version": "v0" }, @@ -2187,8 +2493,6 @@ "max" ], "displayMode": "table", - "enableFacetedFilter": false, - "overflow": "ellipsis", "placement": "right", "showLegend": true }, @@ -2199,7 +2503,7 @@ } } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -2221,10 +2525,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "histogram_quantile(0.95, sum by (le, provider) (rate(gen_ai_client_operation_duration_seconds_bucket{service_name=~\"$service\", deployment_environment=~\"$env\", provider!=\"\"}[$__rate_interval])))", - "legendFormat": "{{provider}}", - "range": true + "legendFormat": "{{provider}}" }, "version": "v0" }, @@ -2307,8 +2609,6 @@ "max" ], "displayMode": "table", - "enableFacetedFilter": false, - "overflow": "ellipsis", "placement": "right", "showLegend": true }, @@ -2319,7 +2619,7 @@ } } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -2341,10 +2641,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "100 * sum by (provider) (rate(gen_ai_client_operation_count_total{service_name=~\"$service\", deployment_environment=~\"$env\", provider!=\"\", http_response_status_code=~\"4..|5..\"}[$__rate_interval])) / clamp_min(sum by (provider) (rate(gen_ai_client_operation_count_total{service_name=~\"$service\", deployment_environment=~\"$env\", provider!=\"\"}[$__rate_interval])), 1)", - "legendFormat": "{{provider}}", - "range": true + "legendFormat": "{{provider}}" }, "version": "v0" }, @@ -2427,8 +2725,6 @@ "max" ], "displayMode": "table", - "enableFacetedFilter": false, - "overflow": "ellipsis", "placement": "right", "showLegend": true }, @@ -2439,7 +2735,7 @@ } } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -2461,10 +2757,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum by (model) (rate(airi_billing_tts_chars_total{service_name=~\"$service\", deployment_environment=~\"$env\"}[$__rate_interval]))", - "legendFormat": "{{model}}", - "range": true + "legendFormat": "{{model}}" }, "version": "v0" }, @@ -2547,8 +2841,6 @@ "max" ], "displayMode": "table", - "enableFacetedFilter": false, - "overflow": "ellipsis", "placement": "right", "showLegend": true }, @@ -2559,7 +2851,7 @@ } } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -2581,10 +2873,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum(increase(gen_ai_client_token_usage_input_total{service_name=~\"$service\", deployment_environment=~\"$env\"}[$__range]))", - "legendFormat": "input", - "range": true + "legendFormat": "input" }, "version": "v0" }, @@ -2602,10 +2892,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum(increase(gen_ai_client_token_usage_output_total{service_name=~\"$service\", deployment_environment=~\"$env\"}[$__range]))", - "legendFormat": "output", - "range": true + "legendFormat": "output" }, "version": "v0" }, @@ -2664,7 +2952,7 @@ "wideLayout": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -2686,10 +2974,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum(rate(gen_ai_client_token_usage_input_total{service_name=~\"$service\", deployment_environment=~\"$env\"}[$__rate_interval]))", - "legendFormat": "input tokens/s", - "range": true + "legendFormat": "input tokens/s" }, "version": "v0" }, @@ -2707,10 +2993,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum(rate(gen_ai_client_token_usage_output_total{service_name=~\"$service\", deployment_environment=~\"$env\"}[$__rate_interval]))", - "legendFormat": "output tokens/s", - "range": true + "legendFormat": "output tokens/s" }, "version": "v0" }, @@ -2793,8 +3077,6 @@ "max" ], "displayMode": "table", - "enableFacetedFilter": false, - "overflow": "ellipsis", "placement": "right", "showLegend": true }, @@ -2805,7 +3087,7 @@ } } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -2827,10 +3109,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum(increase(airi_billing_flux_unbilled_total{service_name=~\"$service\", deployment_environment=~\"$env\", reason!=\"partial_debit_drained\"}[$__range]))", - "legendFormat": "flux", - "range": true + "legendFormat": "flux" }, "version": "v0" }, @@ -2891,7 +3171,7 @@ "wideLayout": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -2913,10 +3193,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum(increase(airi_gen_ai_stream_interrupted_total{service_name=~\"$service\", deployment_environment=~\"$env\"}[$__range]))", - "legendFormat": "interruptions", - "range": true + "legendFormat": "interruptions" }, "version": "v0" }, @@ -2981,7 +3259,7 @@ "wideLayout": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -3003,10 +3281,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum(increase(airi_gen_ai_gateway_key_exhausted_total{service_name=~\"$service\", deployment_environment=~\"$env\"}[5m]))", - "legendFormat": "events", - "range": true + "legendFormat": "events" }, "version": "v0" }, @@ -3067,7 +3343,7 @@ "wideLayout": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -3089,10 +3365,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum(increase(airi_gen_ai_gateway_decrypt_failures_total{service_name=~\"$service\", deployment_environment=~\"$env\"}[5m]))", - "legendFormat": "events", - "range": true + "legendFormat": "events" }, "version": "v0" }, @@ -3153,7 +3427,7 @@ "wideLayout": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -3175,10 +3449,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "100 * sum(rate(airi_gen_ai_gateway_fallback_count_total{service_name=~\"$service\", deployment_environment=~\"$env\"}[5m])) / clamp_min(sum(rate(gen_ai_client_operation_count_total{service_name=~\"$service\", deployment_environment=~\"$env\"}[5m])), 1)", - "legendFormat": "fallback %", - "range": true + "legendFormat": "fallback %" }, "version": "v0" }, @@ -3244,7 +3516,7 @@ "sizing": "auto" } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -3266,10 +3538,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum by (provider, status_code) (rate(airi_gen_ai_gateway_upstream_errors_total{service_name=~\"$service\", deployment_environment=~\"$env\"}[$__rate_interval]))", - "legendFormat": "{{provider}} · {{status_code}}", - "range": true + "legendFormat": "{{provider}} · {{status_code}}" }, "version": "v0" }, @@ -3352,8 +3622,6 @@ "max" ], "displayMode": "table", - "enableFacetedFilter": false, - "overflow": "ellipsis", "placement": "right", "showLegend": true }, @@ -3364,7 +3632,7 @@ } } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -3386,10 +3654,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum by (currency) (increase(airi_stripe_revenue_minor_unit_total{service_name=~\"$service\", deployment_environment=~\"$env\", currency!=\"\"}[$__range])) / 100", - "legendFormat": "{{currency}}", - "range": true + "legendFormat": "{{currency}}" }, "version": "v0" }, @@ -3449,7 +3715,7 @@ "wideLayout": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -3471,10 +3737,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "100 * sum(increase(stripe_checkout_completed_total{service_name=~\"$service\", deployment_environment=~\"$env\"}[$__range])) / clamp_min(sum(increase(stripe_checkout_created_total{service_name=~\"$service\", deployment_environment=~\"$env\"}[$__range])), 1)", - "legendFormat": "completed %", - "range": true + "legendFormat": "completed %" }, "version": "v0" }, @@ -3540,7 +3804,7 @@ "sizing": "auto" } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -3562,10 +3826,8 @@ "group": "prometheus", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "sum by (event_type) (increase(stripe_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", event_type!=\"\"}[$__range]))", - "legendFormat": "{{event_type}}", - "range": true + "legendFormat": "{{event_type}}" }, "version": "v0" }, @@ -3624,7 +3886,443 @@ "wideLayout": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" + } + } + }, + "panel-50": { + "kind": "Panel", + "spec": { + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "hidden": false, + "query": { + "datasource": { + "name": "grafanacloud-projairi-prom" + }, + "group": "prometheus", + "kind": "DataQuery", + "spec": { + "expr": "histogram_quantile(0.95, sum by (le) (rate(db_client_operation_duration_seconds_bucket{service_name=~\"$service\", deployment_environment=~\"$env\"}[5m])))", + "legendFormat": "p95" + }, + "version": "v0" + }, + "refId": "A" + } + } + ], + "queryOptions": {}, + "transformations": [] + } + }, + "description": "PostgreSQL query duration P95 from PgInstrumentation. Fixed 5m window. Spikes correlate with index misses, connection exhaustion, or backend lock contention.", + "id": 50, + "links": [], + "title": "DB Query P95 (5m)", + "vizConfig": { + "group": "stat", + "kind": "VizConfig", + "spec": { + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "yellow", + "value": 0.05 + }, + { + "color": "red", + "value": 0.5 + } + ] + }, + "unit": "s", + "decimals": 3 + }, + "overrides": [] + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + } + }, + "version": "13.0.0-23630096546" + } + } + }, + "panel-51": { + "kind": "Panel", + "spec": { + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "hidden": false, + "query": { + "datasource": { + "name": "grafanacloud-projairi-prom" + }, + "group": "prometheus", + "kind": "DataQuery", + "spec": { + "expr": "sum by (service_instance_id) (db_client_connection_count{service_name=~\"$service\", deployment_environment=~\"$env\"})", + "legendFormat": "{{service_instance_id}}" + }, + "version": "v0" + }, + "refId": "A" + } + } + ], + "queryOptions": {}, + "transformations": [] + } + }, + "description": "Open PostgreSQL connections, per replica (`service_instance_id`). Each instance has its own pool sized by env `DB_POOL_MAX`. One instance with a permanently-high count = pool leak on that pod.", + "id": 51, + "links": [], + "title": "DB Pool Connections by Instance", + "vizConfig": { + "group": "timeseries", + "kind": "VizConfig", + "spec": { + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + } + }, + "version": "13.0.0-23630096546" + } + } + }, + "panel-52": { + "kind": "Panel", + "spec": { + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "hidden": false, + "query": { + "datasource": { + "name": "grafanacloud-projairi-prom" + }, + "group": "prometheus", + "kind": "DataQuery", + "spec": { + "expr": "100 * sum by (service_instance_id) (v8js_memory_heap_used_bytes{service_name=~\"$service\", deployment_environment=~\"$env\"}) / clamp_min(sum by (service_instance_id) (v8js_memory_heap_limit_bytes{service_name=~\"$service\", deployment_environment=~\"$env\"}), 1)", + "legendFormat": "{{service_instance_id}}" + }, + "version": "v0" + }, + "refId": "A" + } + } + ], + "queryOptions": {}, + "transformations": [] + } + }, + "description": "V8 heap used ÷ heap limit, per replica (`service_instance_id`). A single replica trending up while others stay flat = leak on that pod.", + "id": 52, + "links": [], + "title": "Heap Used % by Instance", + "vizConfig": { + "group": "timeseries", + "kind": "VizConfig", + "spec": { + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "percent" + }, + "overrides": [] + }, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + } + }, + "version": "13.0.0-23630096546" + } + } + }, + "panel-53": { + "kind": "Panel", + "spec": { + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "hidden": false, + "query": { + "datasource": { + "name": "grafanacloud-projairi-prom" + }, + "group": "prometheus", + "kind": "DataQuery", + "spec": { + "expr": "max by (service_instance_id) (nodejs_eventloop_delay_p99_seconds{service_name=~\"$service\", deployment_environment=~\"$env\"})", + "legendFormat": "{{service_instance_id}}" + }, + "version": "v0" + }, + "refId": "A" + } + } + ], + "queryOptions": {}, + "transformations": [] + } + }, + "description": "P99 event-loop delay per replica. One replica climbing while others stay flat = CPU-bound work pinning that pod. >50ms sustained is bad anywhere.", + "id": 53, + "links": [], + "title": "Event Loop Delay P99 by Instance", + "vizConfig": { + "group": "timeseries", + "kind": "VizConfig", + "spec": { + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + } + }, + "version": "13.0.0-23630096546" } } }, @@ -3646,10 +4344,8 @@ "group": "loki", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "{service_name=~\"$service\", deployment_environment=~\"$env\"} | json | level=~\"warn|error\"", - "legendFormat": "", - "range": true + "legendFormat": "" }, "version": "v0" }, @@ -3691,7 +4387,7 @@ "wrapLogMessage": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } }, @@ -3713,10 +4409,8 @@ "group": "loki", "kind": "DataQuery", "spec": { - "editorMode": "code", "expr": "{service_name=~\"$service\", deployment_environment=~\"$env\"} |= ``", - "legendFormat": "", - "range": true + "legendFormat": "" }, "version": "v0" }, @@ -3758,7 +4452,7 @@ "wrapLogMessage": true } }, - "version": "13.2.0-28666480772" + "version": "13.0.0-23630096546" } } } @@ -3780,10 +4474,10 @@ "spec": { "element": { "kind": "ElementReference", - "name": "panel-3" + "name": "panel-1" }, - "height": 5, - "width": 4, + "height": 4, + "width": 6, "x": 0, "y": 0 } @@ -3793,11 +4487,24 @@ "spec": { "element": { "kind": "ElementReference", - "name": "panel-5" + "name": "panel-3" }, - "height": 5, - "width": 4, - "x": 4, + "height": 4, + "width": 6, + "x": 6, + "y": 0 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-4" + }, + "height": 4, + "width": 6, + "x": 12, "y": 0 } }, @@ -3808,25 +4515,38 @@ "kind": "ElementReference", "name": "panel-40" }, - "height": 6, - "width": 10, - "x": 8, - "y": 0 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-90" - }, - "height": 38, + "height": 8, "width": 6, "x": 18, "y": 0 } }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-15" + }, + "height": 4, + "width": 6, + "x": 0, + "y": 4 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-5" + }, + "height": 4, + "width": 6, + "x": 6, + "y": 4 + } + }, { "kind": "GridLayoutItem", "spec": { @@ -3834,36 +4554,10 @@ "kind": "ElementReference", "name": "panel-93" }, - "height": 5, - "width": 4, - "x": 0, - "y": 5 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-4" - }, - "height": 5, - "width": 4, - "x": 4, - "y": 5 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-20" - }, - "height": 15, - "width": 10, - "x": 8, - "y": 6 + "height": 4, + "width": 6, + "x": 12, + "y": 4 } }, { @@ -3873,127 +4567,10 @@ "kind": "ElementReference", "name": "panel-92" }, - "height": 10, - "width": 4, - "x": 0, - "y": 10 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-73" - }, - "height": 10, - "width": 2, - "x": 4, - "y": 10 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-11" - }, - "height": 10, - "width": 2, - "x": 6, - "y": 10 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-16" - }, - "height": 18, - "width": 8, - "x": 0, - "y": 20 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-71" - }, "height": 5, - "width": 5, - "x": 8, - "y": 21 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-21" - }, - "height": 5, - "width": 5, - "x": 13, - "y": 21 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-69" - }, - "height": 6, - "width": 5, - "x": 8, - "y": 26 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-67" - }, - "height": 6, - "width": 5, - "x": 13, - "y": 26 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-66" - }, - "height": 6, - "width": 5, - "x": 8, - "y": 32 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-68" - }, - "height": 6, - "width": 5, - "x": 13, - "y": 32 + "width": 24, + "x": 0, + "y": 8 } } ] @@ -4017,8 +4594,8 @@ "kind": "ElementReference", "name": "panel-80" }, - "height": 5, - "width": 3, + "height": 4, + "width": 8, "x": 0, "y": 0 } @@ -4028,37 +4605,11 @@ "spec": { "element": { "kind": "ElementReference", - "name": "panel-1" + "name": "panel-81" }, - "height": 5, - "width": 3, - "x": 3, - "y": 0 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-99" - }, - "height": 11, - "width": 12, - "x": 6, - "y": 0 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-98" - }, - "height": 11, - "width": 6, - "x": 18, + "height": 4, + "width": 8, + "x": 8, "y": 0 } }, @@ -4069,75 +4620,10 @@ "kind": "ElementReference", "name": "panel-82" }, - "height": 3, - "width": 3, - "x": 0, - "y": 5 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-15" - }, - "height": 6, - "width": 3, - "x": 3, - "y": 5 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-81" - }, - "height": 3, - "width": 3, - "x": 0, - "y": 8 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-95" - }, - "height": 9, - "width": 6, - "x": 0, - "y": 11 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-96" - }, - "height": 9, - "width": 6, - "x": 6, - "y": 11 - } - }, - { - "kind": "GridLayoutItem", - "spec": { - "element": { - "kind": "ElementReference", - "name": "panel-97" - }, - "height": 9, - "width": 12, - "x": 12, - "y": 11 + "height": 4, + "width": 8, + "x": 16, + "y": 0 } } ] @@ -4153,7 +4639,125 @@ "layout": { "kind": "GridLayout", "spec": { - "items": [] + "items": [ + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-95" + }, + "height": 5, + "width": 6, + "x": 0, + "y": 0 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-96" + }, + "height": 5, + "width": 6, + "x": 6, + "y": 0 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-99" + }, + "height": 5, + "width": 6, + "x": 12, + "y": 0 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-100" + }, + "height": 5, + "width": 6, + "x": 18, + "y": 0 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-97" + }, + "height": 8, + "width": 12, + "x": 0, + "y": 5 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-98" + }, + "height": 4, + "width": 12, + "x": 12, + "y": 5 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-101" + }, + "height": 4, + "width": 12, + "x": 12, + "y": 9 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-102" + }, + "height": 5, + "width": 12, + "x": 0, + "y": 13 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-103" + }, + "height": 5, + "width": 12, + "x": 12, + "y": 13 + } + } + ] } }, "title": "Product Analytics" @@ -4179,6 +4783,32 @@ "x": 0, "y": 0 } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-16" + }, + "height": 11, + "width": 7, + "x": 0, + "y": 8 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-20" + }, + "height": 11, + "width": 17, + "x": 7, + "y": 8 + } } ] } @@ -4199,7 +4829,47 @@ "spec": { "element": { "kind": "ElementReference", - "name": "panel-43" + "name": "panel-11" + }, + "height": 8, + "width": 12, + "x": 0, + "y": 0 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-21" + }, + "height": 8, + "width": 12, + "x": 12, + "y": 0 + } + } + ] + } + }, + "title": "LLM Gateway" + } + }, + { + "kind": "RowsLayoutRow", + "spec": { + "collapse": false, + "layout": { + "kind": "GridLayout", + "spec": { + "items": [ + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-66" }, "height": 7, "width": 6, @@ -4212,13 +4882,105 @@ "spec": { "element": { "kind": "ElementReference", - "name": "panel-41" + "name": "panel-67" }, "height": 7, "width": 6, "x": 6, "y": 0 } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-68" + }, + "height": 7, + "width": 6, + "x": 12, + "y": 0 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-69" + }, + "height": 7, + "width": 6, + "x": 18, + "y": 0 + } + } + ] + } + }, + "title": "Provider Upstreams" + } + }, + { + "kind": "RowsLayoutRow", + "spec": { + "collapse": false, + "layout": { + "kind": "GridLayout", + "spec": { + "items": [ + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-73" + }, + "height": 7, + "width": 6, + "x": 0, + "y": 0 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-71" + }, + "height": 7, + "width": 10, + "x": 6, + "y": 0 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-43" + }, + "height": 7, + "width": 4, + "x": 16, + "y": 0 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-41" + }, + "height": 7, + "width": 4, + "x": 20, + "y": 0 + } } ] } @@ -4345,6 +5107,72 @@ "title": "Business" } }, + { + "kind": "RowsLayoutRow", + "spec": { + "collapse": true, + "layout": { + "kind": "GridLayout", + "spec": { + "items": [ + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-50" + }, + "height": 6, + "width": 6, + "x": 0, + "y": 0 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-51" + }, + "height": 6, + "width": 6, + "x": 6, + "y": 0 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-52" + }, + "height": 6, + "width": 6, + "x": 12, + "y": 0 + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-53" + }, + "height": 6, + "width": 6, + "x": 18, + "y": 0 + } + } + ] + } + }, + "title": "Infrastructure" + } + }, { "kind": "RowsLayoutRow", "spec": { @@ -4360,11 +5188,24 @@ "kind": "ElementReference", "name": "panel-91" }, - "height": 8, + "height": 10, "width": 24, "x": 0, "y": 0 } + }, + { + "kind": "GridLayoutItem", + "spec": { + "element": { + "kind": "ElementReference", + "name": "panel-90" + }, + "height": 10, + "width": 24, + "x": 0, + "y": 10 + } } ] } @@ -4384,7 +5225,7 @@ "grafana-cloud" ], "timeSettings": { - "autoRefresh": "", + "autoRefresh": "30s", "autoRefreshIntervals": [ "5s", "10s", @@ -4398,7 +5239,7 @@ "1d" ], "fiscalYearStartMonth": 0, - "from": "now-6h", + "from": "now-1h", "hideTimepicker": false, "timezone": "browser", "to": "now" diff --git a/apps/server/otel/grafana/dashboards/build.test.ts b/apps/server/otel/grafana/dashboards/build.test.ts index f9dcd328e..0b82e47ee 100644 --- a/apps/server/otel/grafana/dashboards/build.test.ts +++ b/apps/server/otel/grafana/dashboards/build.test.ts @@ -58,12 +58,14 @@ describe('grafana dashboard builder', () => { * @example * expect(panelTitle('panel-99')).toBe('TTS Success %') */ - it('keeps the product analytics row focused on Prometheus-safe engagement signals', () => { + it('keeps the product analytics row focused on server-side TTS health', () => { expect(panelTitle('panel-95')).toBe('Product Events (range)') expect(panelTitle('panel-96')).toBe('Product Failure %') - expect(panelTitle('panel-97')).toBe('Top Product Actions (range)') - expect(panelTitle('panel-98')).toBe('Product Event Rate') - expect(panelTitle('panel-99')).toBe('日活') + expect(panelTitle('panel-99')).toBe('TTS Success %') + expect(panelTitle('panel-100')).toBe('TTS Failed / Blocked (range)') + expect(panelTitle('panel-101')).toBe('TTS Event Rate by Source') + expect(panelTitle('panel-102')).toBe('TTS Blocked by Reason') + expect(panelTitle('panel-103')).toBe('TTS Blocked by Flux Bucket') }) /** @@ -78,6 +80,10 @@ describe('grafana dashboard builder', () => { dashboard.elements['panel-97'], dashboard.elements['panel-98'], dashboard.elements['panel-99'], + dashboard.elements['panel-100'], + dashboard.elements['panel-101'], + dashboard.elements['panel-102'], + dashboard.elements['panel-103'], ]).join('\n') expect(productPanelExpressions).not.toContain('voice_id') diff --git a/apps/server/otel/grafana/dashboards/build.ts b/apps/server/otel/grafana/dashboards/build.ts index bb438ce12..0b7fc543c 100644 --- a/apps/server/otel/grafana/dashboards/build.ts +++ b/apps/server/otel/grafana/dashboards/build.ts @@ -34,7 +34,7 @@ import { fileURLToPath, pathToFileURL } from 'node:url' const PROM = { name: 'grafanacloud-projairi-prom' } const LOKI = { name: 'grafanacloud-projairi-logs' } -const SCHEMA_VERSION = '13.2.0-28666480772' +const SCHEMA_VERSION = '13.0.0-23630096546' // Service / env filter applied to every Prom query. Pulled into a helper so // the variable name only appears once. @@ -62,10 +62,9 @@ function query(expr: string, legend: string, refId = 'A', datasource: DataSource group: datasource === LOKI ? 'loki' : 'prometheus', kind: 'DataQuery', spec: { - editorMode: 'code', expr, legendFormat: legend, - ...(opts.instant ? { instant: true, range: false } : { range: true }), + ...(opts.instant && { instant: true, range: false }), }, version: 'v0', }, @@ -129,8 +128,6 @@ interface TimeseriesPanelOpts { stack?: boolean fillOpacity?: number legendCalcs?: LegendCalc[] - legendPlacement?: 'bottom' | 'right' - legendDisplayMode?: 'list' | 'table' } // `noValue` shows a friendly placeholder instead of "No data" red text when @@ -272,14 +269,7 @@ function barGaugePanel(id: number, title: string, description: string, queries: } function timeseriesPanel(id: number, title: string, description: string, queries: PanelQuery[], opts: TimeseriesPanelOpts = {}) { - const { - unit = 'short', - stack = false, - fillOpacity = 20, - legendCalcs = ['lastNotNull', 'max'], - legendPlacement = 'right', - legendDisplayMode = 'table', - } = opts + const { unit = 'short', stack = false, fillOpacity = 20, legendCalcs = ['lastNotNull', 'max'] } = opts return { kind: 'Panel', spec: { @@ -328,14 +318,7 @@ function timeseriesPanel(id: number, title: string, description: string, queries // Show last + max in the legend table so viewers don't have to // click each line to see numbers — same trick as Keycloak's // "Login Errors" panel. - legend: { - calcs: legendCalcs, - displayMode: legendDisplayMode, - enableFacetedFilter: false, - overflow: 'ellipsis', - placement: legendPlacement, - showLegend: true, - }, + legend: { calcs: legendCalcs, displayMode: 'table', placement: 'right', showLegend: true }, tooltip: { hideZeros: false, mode: 'multi', sort: 'desc' }, }, }, @@ -345,119 +328,6 @@ function timeseriesPanel(id: number, title: string, description: string, queries } } -function pieChartPanel(id: number, title: string, description: string, queries: PanelQuery[], unit = 'short') { - return { - kind: 'Panel', - spec: { - data: { kind: 'QueryGroup', spec: { queries, queryOptions: {}, transformations: [] } }, - description, - id, - links: [], - title, - vizConfig: { - group: 'piechart', - kind: 'VizConfig', - spec: { - fieldConfig: { - defaults: { - color: { fixedColor: '#73BF69', mode: 'palette-classic' }, - custom: { hideFrom: { legend: false, tooltip: false, viz: false } }, - unit, - }, - overrides: [], - }, - options: { - displayLabels: ['percent'], - legend: { displayMode: 'table', overflow: 'ellipsis', placement: 'bottom', showLegend: true }, - pieType: 'pie', - reduceOptions: { calcs: ['sum'], fields: '', values: false }, - sort: 'desc', - tooltip: { hideZeros: false, mode: 'single', sort: 'none' }, - }, - }, - version: SCHEMA_VERSION, - }, - }, - } -} - -// Custom Grafana UI panel retained as code so the generated dashboard matches -// the latest hand-tuned cloud version without checking in cloud metadata. -function dailyActiveUsersTrendPanel() { - return { - kind: 'Panel', - spec: { - data: { - kind: 'QueryGroup', - spec: { - queries: [ - query( - `max (user_active_rolling{${SERVICE_FILTER}, window="24h"})`, - '__auto', - ), - ], - queryOptions: {}, - transformations: [], - }, - }, - description: '', - id: 99, - links: [], - title: '日活', - vizConfig: { - group: 'timeseries', - kind: 'VizConfig', - spec: { - fieldConfig: { - defaults: { - color: { mode: 'continuous-GrYlRd', seriesBy: 'last' }, - custom: { - axisBorderShow: false, - axisCenteredZero: false, - axisColorMode: 'text', - axisLabel: '', - axisPlacement: 'auto', - barAlignment: 0, - barWidthFactor: 0.6, - drawStyle: 'line', - fillOpacity: 17, - gradientMode: 'scheme', - hideFrom: { legend: false, tooltip: false, viz: false }, - insertNulls: false, - lineInterpolation: 'linear', - lineStyle: { fill: 'solid' }, - lineWidth: 2, - pointSize: 3, - scaleDistribution: { type: 'linear' }, - showPoints: 'auto', - showValues: false, - spanNulls: false, - stacking: { group: 'A', mode: 'none' }, - thresholdsStyle: { mode: 'off' }, - }, - thresholds: thresholds([{ color: 'green', value: 0 }, { color: 'red', value: 80 }]), - }, - overrides: [], - }, - options: { - annotations: { clustering: -1, multiLane: false }, - legend: { - calcs: [], - displayMode: 'list', - enableFacetedFilter: false, - overflow: 'ellipsis', - placement: 'bottom', - showLegend: true, - }, - tooltip: { hideZeros: false, mode: 'single', sort: 'none' }, - }, - }, - version: SCHEMA_VERSION, - }, - }, - } -} - interface HeatmapPanelOpts { unit?: string } @@ -707,7 +577,69 @@ elements['panel-98'] = timeseriesPanel( { unit: 'eps', fillOpacity: 15 }, ) -elements['panel-99'] = dailyActiveUsersTrendPanel() +elements['panel-99'] = gaugePanel( + 99, + 'TTS Success %', + 'Server-side TTS successes divided by TTS requests over the dashboard range. Includes REST and WS TTS product events. Drops here mean users are asking for speech but not receiving audio; inspect failed/blocked panels next.', + [query( + `100 * sum(increase(airi_product_events_total{${PRODUCT_EVENT_FILTER}, feature="tts", action="speech_succeeded", status="succeeded"}[$__range])) / clamp_min(sum(increase(airi_product_events_total{${PRODUCT_EVENT_FILTER}, feature="tts", action="speech_requested", status="started"}[$__range])), 1)`, + 'success %', + )], + { steps: [{ color: 'red', value: 0 }, { color: 'yellow', value: 90 }, { color: 'green', value: 98 }], max: 100, decimals: 2, noValue: '0' }, +) + +elements['panel-100'] = barGaugePanel( + 100, + 'TTS Failed / Blocked (range)', + 'TTS user-impacting failures over the dashboard range, split by action/status/source. `speech_failed` usually means upstream/runtime failure; `speech_blocked` usually means balance/preflight blocked. Keep voice/model drilldown in Postgres metadata, not Prometheus labels.', + [query( + `topk(12, sum by (action, status, source) (increase(airi_product_events_total{${PRODUCT_EVENT_FILTER}, feature="tts", action=~"speech_failed|speech_blocked"}[$__range])))`, + '{{action}} · {{status}} · {{source}}', + 'A', + PROM, + { instant: true }, + )], + { unit: 'short', noValue: '0' }, +) + +elements['panel-101'] = timeseriesPanel( + 101, + 'TTS Event Rate by Source', + 'TTS product event rate by source and action. Use this to distinguish chat auto-TTS, manual previews/settings tests, and API audio.speech traffic when speech health changes.', + [query( + `sum by (source, action, status) (rate(airi_product_events_total{${PRODUCT_EVENT_FILTER}, feature="tts"}[$__rate_interval]))`, + '{{source}} · {{action}} · {{status}}', + )], + { unit: 'eps', fillOpacity: 15 }, +) + +elements['panel-102'] = barGaugePanel( + 102, + 'TTS Blocked by Reason', + 'Blocked TTS events over the dashboard range, grouped by bounded product reason and source. Today this mostly shows insufficient balance; new policy/provider/preflight buckets can be added without exposing user, voice, or model labels.', + [query( + `topk(12, sum by (reason, source) (increase(airi_product_events_total{${PRODUCT_EVENT_FILTER}, feature="tts", action="speech_blocked", status="blocked"}[$__range])))`, + '{{reason}} · {{source}}', + 'A', + PROM, + { instant: true }, + )], + { unit: 'short', noValue: '0' }, +) + +elements['panel-103'] = barGaugePanel( + 103, + 'TTS Blocked by Flux Bucket', + 'Blocked TTS events over the dashboard range, grouped by coarse Flux balance bucket. This helps separate truly empty accounts from low-balance accounts without exposing exact user balances.', + [query( + `topk(8, sum by (flux_balance_bucket, source) (increase(airi_product_events_total{${PRODUCT_EVENT_FILTER}, feature="tts", action="speech_blocked", status="blocked", flux_balance_bucket!=""}[$__range])))`, + '{{flux_balance_bucket}} · {{source}}', + 'A', + PROM, + { instant: true }, + )], + { unit: 'short', noValue: '0' }, +) // --- Row 2: HTTP — traffic ranking, error trend, latency trend ------------- elements['panel-16'] = barGaugePanel( @@ -715,7 +647,7 @@ elements['panel-16'] = barGaugePanel( 'Top Routes by Requests (range)', 'Top Hono-matched routes by request count over the dashboard range. The main traffic list: which API surfaces are hottest. Wildcard patterns like `/api/v1/openai/*` are requests that did not reach a concrete handler (404 / auth-rejected); concrete paths are successful routes.', [query( - `topk(50, sum by (http_route) (increase(http_server_request_duration_seconds_count{${SERVICE_FILTER}, http_request_method!="OPTIONS", http_route!=""}[$__range])))`, + `topk(10, sum by (http_route) (increase(http_server_request_duration_seconds_count{${SERVICE_FILTER}, http_request_method!="OPTIONS", http_route!=""}[$__range])))`, '{{http_route}}', 'A', PROM, @@ -726,7 +658,7 @@ elements['panel-16'] = barGaugePanel( elements['panel-40'] = heatmapPanel( 40, - 'Status Distribution', + 'Error Rate %', 'HTTP status-code mix over time, one row per status code, colour = request rate in each time bucket. The 200 row dominates in steady state; a 5xx / 4xx row suddenly lighting up flags an incident at a glance. Non-OPTIONS traffic only.', [query( `sum by (http_response_status_code) (rate(http_server_request_duration_seconds_count{${SERVICE_FILTER}, http_request_method!="OPTIONS"}[$__rate_interval]))`, @@ -737,15 +669,13 @@ elements['panel-40'] = heatmapPanel( elements['panel-20'] = timeseriesPanel( 20, - 'Request Latency (by Route)', - '', + 'HTTP P95 by Route', + 'P95 latency per Hono-matched route, excluding /api/v1/openai/* (LLM gateway latency lives in the LLM Gateway row). 404s excluded so missing-route noise does not skew the curve.', [query( - `sum by (http_route) ( - rate(http_server_request_duration_seconds_bucket{${SERVICE_FILTER}, http_request_method!="OPTIONS", http_route!~"/api/v1/openai/.*", http_response_status_code!="404"}[$__rate_interval]) -)`, + `histogram_quantile(0.95, sum by (le, http_route) (rate(http_server_request_duration_seconds_bucket{${SERVICE_FILTER}, http_request_method!="OPTIONS", http_route!~"/api/v1/openai/.*", http_response_status_code!="404"}[$__rate_interval])))`, '{{http_route}}', )], - { unit: 's', legendPlacement: 'bottom' }, + { unit: 's' }, ) elements['panel-94'] = timeseriesPanel( @@ -760,7 +690,7 @@ elements['panel-94'] = timeseriesPanel( ) // --- Row 3: LLM Gateway — request mix + latency ---------------------------- -elements['panel-11'] = pieChartPanel( +elements['panel-11'] = timeseriesPanel( 11, 'LLM Request Rate by Model', 'Per-model request rate (chat + tts). Useful for capacity planning and spotting model-routing regressions.', @@ -768,7 +698,7 @@ elements['panel-11'] = pieChartPanel( `sum by (gen_ai_request_model) (rate(gen_ai_client_operation_count_total{${SERVICE_FILTER}, gen_ai_request_model!=""}[$__rate_interval]))`, '{{gen_ai_request_model}}', )], - 'reqps', + { unit: 'reqps' }, ) elements['panel-21'] = timeseriesPanel( @@ -776,7 +706,7 @@ elements['panel-21'] = timeseriesPanel( 'LLM Latency P95', 'Two P95 latency signals for the LLM gateway, aggregated across models. TTFB = time to first streamed token (streaming chat UX). End-to-end = full operation duration — the only latency signal for non-streaming chat and TTS, which have no first-token event.', [ - query(`sum by () (rate(gen_ai_client_first_token_duration_seconds_bucket{${SERVICE_FILTER}}[$__rate_interval]))`, 'TTFB p95', 'A'), + query(`histogram_quantile(0.95, sum by (le) (rate(gen_ai_client_first_token_duration_seconds_bucket{${SERVICE_FILTER}}[$__rate_interval])))`, 'TTFB p95', 'A'), query(`histogram_quantile(0.95, sum by (le) (rate(gen_ai_client_operation_duration_seconds_bucket{${SERVICE_FILTER}}[$__rate_interval])))`, 'end-to-end p95', 'B'), ], { unit: 's' }, @@ -955,6 +885,51 @@ elements['panel-32'] = statPanel( { unit: 'short', variant: 'count', noValue: '—', graphMode: 'none' }, ) +// --- Row 7: Infrastructure (collapsed) — process / DB health --------------- +elements['panel-50'] = statPanel( + 50, + 'DB Query P95 (5m)', + 'PostgreSQL query duration P95 from PgInstrumentation. Fixed 5m window. Spikes correlate with index misses, connection exhaustion, or backend lock contention.', + [query( + `histogram_quantile(0.95, sum by (le) (rate(db_client_operation_duration_seconds_bucket{${SERVICE_FILTER}}[5m])))`, + 'p95', + )], + { unit: 's', steps: [{ color: 'green', value: 0 }, { color: 'yellow', value: 0.05 }, { color: 'red', value: 0.5 }], decimals: 3 }, +) + +elements['panel-51'] = timeseriesPanel( + 51, + 'DB Pool Connections by Instance', + 'Open PostgreSQL connections, per replica (`service_instance_id`). Each instance has its own pool sized by env `DB_POOL_MAX`. One instance with a permanently-high count = pool leak on that pod.', + [query( + `sum by (service_instance_id) (db_client_connection_count{${SERVICE_FILTER}})`, + '{{service_instance_id}}', + )], + { unit: 'short' }, +) + +elements['panel-52'] = timeseriesPanel( + 52, + 'Heap Used % by Instance', + 'V8 heap used ÷ heap limit, per replica (`service_instance_id`). A single replica trending up while others stay flat = leak on that pod.', + [query( + `100 * sum by (service_instance_id) (v8js_memory_heap_used_bytes{${SERVICE_FILTER}}) / clamp_min(sum by (service_instance_id) (v8js_memory_heap_limit_bytes{${SERVICE_FILTER}}), 1)`, + '{{service_instance_id}}', + )], + { unit: 'percent' }, +) + +elements['panel-53'] = timeseriesPanel( + 53, + 'Event Loop Delay P99 by Instance', + 'P99 event-loop delay per replica. One replica climbing while others stay flat = CPU-bound work pinning that pod. >50ms sustained is bad anywhere.', + [query( + `max by (service_instance_id) (nodejs_eventloop_delay_p99_seconds{${SERVICE_FILTER}})`, + '{{service_instance_id}}', + )], + { unit: 's' }, +) + // --- Row 8: Logs ------------------------------------------------------------ elements['panel-91'] = logsPanel( 91, @@ -975,53 +950,68 @@ elements['panel-90'] = logsPanel( // --------------------------------------------------------------------------- const rows = [ - // Row 1: Service Health — dense single-screen operations layout from the - // latest Grafana Cloud edit. Logs stay docked on the right while traffic, - // latency, LLM, token, and provider panels fill the left. + // Row 1: Service Health — two rows of glance stats + the status-code heatmap + // standing tall on the right, with the live WS-connections trend full-width + // underneath. counts (New Users / Active Sessions / WS Online) read blue with + // a trend delta; req-rate + 5xx stay traffic-light. row('Service Health', [ - item('panel-3', 0, 0, 4, 5), - item('panel-5', 4, 0, 4, 5), - item('panel-40', 8, 0, 10, 6), - item('panel-90', 18, 0, 6, 38), - item('panel-93', 0, 5, 4, 5), - item('panel-4', 4, 5, 4, 5), - item('panel-20', 8, 6, 10, 15), - item('panel-92', 0, 10, 4, 10), - item('panel-73', 4, 10, 2, 10), - item('panel-11', 6, 10, 2, 10), - item('panel-16', 0, 20, 8, 18), - item('panel-71', 8, 21, 5, 5), - item('panel-21', 13, 21, 5, 5), - item('panel-69', 8, 26, 5, 6), - item('panel-67', 13, 26, 5, 6), - item('panel-66', 8, 32, 5, 6), - item('panel-68', 13, 32, 5, 6), + item('panel-1', 0, 0, 6, 4), + item('panel-3', 6, 0, 6, 4), + item('panel-4', 12, 0, 6, 4), + item('panel-40', 18, 0, 6, 8), + item('panel-15', 0, 4, 6, 4), + item('panel-5', 6, 4, 6, 4), + item('panel-93', 12, 4, 6, 4), + item('panel-92', 0, 8, 24, 5), ]), - // Row 2: User Engagement — rolling-window active users and Prom-safe product - // analytics. The hand-tuned "日活" trend gives the row a visual engagement - // anchor while compact stats keep user/session totals nearby. + // Row 2: User Engagement — rolling-window active users (DAU/WAU/MAU) from + // user.last_seen_at. Kept its own row so it can grow (retention, cohorts) + // without crowding the health glance above. row('User Engagement', [ - item('panel-80', 0, 0, 3, 5), - item('panel-1', 3, 0, 3, 5), - item('panel-99', 6, 0, 12, 11), - item('panel-98', 18, 0, 6, 11), - item('panel-82', 0, 5, 3, 3), - item('panel-15', 3, 5, 3, 6), - item('panel-81', 0, 8, 3, 3), - item('panel-95', 0, 11, 6, 9), - item('panel-96', 6, 11, 6, 9), - item('panel-97', 12, 11, 12, 9), + item('panel-80', 0, 0, 8, 4), + item('panel-81', 8, 0, 8, 4), + item('panel-82', 16, 0, 8, 4), ]), - row('Product Analytics', []), - // Row 3: HTTP — full-width error breakdown; traffic ranking and latency moved - // into the dense Service Health screen above. + // Row 3: Product Analytics — Prom-safe event volume and server TTS health. + // Distinct-user analytics stay in Postgres `product_events`; this row + // intentionally never uses user_id/session/request labels. + row('Product Analytics', [ + item('panel-95', 0, 0, 6, 5), + item('panel-96', 6, 0, 6, 5), + item('panel-99', 12, 0, 6, 5), + item('panel-100', 18, 0, 6, 5), + item('panel-97', 0, 5, 12, 8), + item('panel-98', 12, 5, 12, 4), + item('panel-101', 12, 9, 12, 4), + item('panel-102', 0, 13, 12, 5), + item('panel-103', 12, 13, 12, 5), + ]), + // Row 3: HTTP — full-width error breakdown on top, then traffic ranking + + // latency trend side by side. row('HTTP', [ item('panel-94', 0, 0, 24, 8), + item('panel-16', 0, 8, 7, 11), + item('panel-20', 7, 8, 17, 11), + ]), + // Row 4: LLM gateway — request mix + latency side by side. + row('LLM Gateway', [ + item('panel-11', 0, 0, 12, 8), + item('panel-21', 12, 0, 12, 8), + ]), + // Row 5: Provider Upstreams — per-provider rollup so the vendor consoles + // don't have to be opened one by one. Four wide, one screen line. + row('Provider Upstreams', [ + item('panel-66', 0, 0, 6, 7), + item('panel-67', 6, 0, 6, 7), + item('panel-68', 12, 0, 6, 7), + item('panel-69', 18, 0, 6, 7), ]), // Row 4: token totals + throughput + the two revenue/quality alert stats. row('LLM Tokens & Quality', [ - item('panel-43', 0, 0, 6, 7), - item('panel-41', 6, 0, 6, 7), + item('panel-73', 0, 0, 6, 7), + item('panel-71', 6, 0, 10, 7), + item('panel-43', 16, 0, 4, 7), + item('panel-41', 20, 0, 4, 7), ]), // Row 5: router health — three "wake someone up" stats/gauge + upstream errors. row('LLM Router Health', [ @@ -1036,10 +1026,17 @@ const rows = [ item('panel-31', 8, 0, 8, 7), item('panel-32', 16, 0, 8, 7), ]), - // Row 8: focused error logs; the live application firehose is docked in - // Service Health for the latest cloud layout. + // Row 7: infra (collapsed) — by-instance breakdowns catch single-replica issues. + row('Infrastructure', [ + item('panel-50', 0, 0, 6, 6), + item('panel-51', 6, 0, 6, 6), + item('panel-52', 12, 0, 6, 6), + item('panel-53', 18, 0, 6, 6), + ], { collapse: true }), + // Row 8: full-width logs — errors on top (triage focus), firehose below. row('Logs', [ - item('panel-91', 0, 0, 24, 8), + item('panel-91', 0, 0, 24, 10), + item('panel-90', 0, 10, 24, 10), ]), ] @@ -1108,15 +1105,18 @@ const variables = [ * AIRI Server Overview dashboard. * * Reading order: - * 1. Service Health — dense operations screen with request, LLM, provider, - * token, WebSocket, status, and live application-log signals. - * 2. User Engagement — rolling DAU/WAU/MAU, total users, sessions, product - * event health, and the hand-tuned daily-active-user trend. - * 3. HTTP — full-width route error breakdown. - * 4. LLM Tokens & Quality — revenue-leak and stream-interruption alerts. - * 5. LLM Router Health — key/decrypt/fallback "wake someone up" signals. - * 6. Business — Stripe / Flux money flow. - * 7. Logs — Loki warning/error logs for live debugging. + * 1. Service Health — signup/sessions/WS counts, req-rate, 5xx, status-code + * heatmap, live WS trend: "is anything broken right now?" + * 2. User Engagement — rolling DAU/WAU/MAU from user.last_seen_at + * 3. Product Analytics — Prom-safe product event volume + server TTS health + * 4. HTTP — error breakdown by route, request ranking, latency by route + * 5. LLM Gateway — per-model request rate + latency (TTFB + end-to-end) + * 6. Provider Upstreams — per-provider rate/latency/failure + TTS chars + * 7. LLM Tokens & Quality — token totals/throughput, revenue-leak alerts + * 8. LLM Router Health — key/decrypt/fallback "wake someone up" signals + * 9. Business — Stripe / Flux money flow + * 10. Infrastructure (collapsed) — DB / runtime health for triage + * 11. Logs — Loki for live debugging * * One metric, one panel: we deliberately do not duplicate a metric across * stat/trend/bar/pie forms. Counter conventions: rate() for "now" trends, @@ -1154,10 +1154,10 @@ export const dashboard = { preload: false, tags: ['airi', 'observability', 'grafana-cloud'], timeSettings: { - autoRefresh: '', + autoRefresh: '30s', autoRefreshIntervals: ['5s', '10s', '30s', '1m', '5m', '15m', '30m', '1h', '2h', '1d'], fiscalYearStartMonth: 0, - from: 'now-6h', + from: 'now-1h', hideTimepicker: false, timezone: 'browser', to: 'now', diff --git a/apps/server/package.json b/apps/server/package.json index c322d7728..8df53757d 100644 --- a/apps/server/package.json +++ b/apps/server/package.json @@ -1,7 +1,7 @@ { "name": "@proj-airi/server", "type": "module", - "version": "0.11.1", + "version": "0.11.0", "private": true, "scripts": { "apply:env": "dotenvx run -f .env.local --overload --ignore=MISSING_ENV_FILE", @@ -61,7 +61,6 @@ "nanoid": "catalog:", "ofetch": "catalog:", "pg": "catalog:", - "posthog-node": "catalog:", "resend": "catalog:", "stripe": "catalog:", "unspeech": "catalog:xsai", diff --git a/apps/server/src/app.test.ts b/apps/server/src/app.test.ts index c16e1f641..3ea053a25 100644 --- a/apps/server/src/app.test.ts +++ b/apps/server/src/app.test.ts @@ -55,7 +55,6 @@ function createTestDeps() { providerCatalogService: {} as any, productEventService: { track: vi.fn(async () => undefined), - trackGeneration: vi.fn(async () => undefined), countDistinctUsersByFeature: vi.fn(async () => []), }, configKV: { diff --git a/apps/server/src/app.ts b/apps/server/src/app.ts index c47f6819d..bdde38f65 100644 --- a/apps/server/src/app.ts +++ b/apps/server/src/app.ts @@ -75,7 +75,6 @@ import { createStripeRoutes } from './routes/stripe' import { createVoicePackRoutes } from './routes/voice-packs' import { createConfigKVService } from './services/adapters/config-kv' import { createEmailService } from './services/adapters/email' -import { createPosthogSink } from './services/adapters/posthog' import { createAdminFluxGrantsService } from './services/domain/admin/flux-grants' import { createAdminRouterConfigService } from './services/domain/admin/router-config' import { createAdminUsersService } from './services/domain/admin/users' @@ -594,27 +593,9 @@ export async function createApp() { }, undefined, dependsOn.otel?.email), }) - const posthogSink = injeca.provide('services:posthogSink', { - dependsOn: { env: parsedEnv, lifecycle }, - // POSTHOG_PROJECT_KEY defaults to the shared project key, so the falsy - // branch is only reachable via the documented off-switch: setting the - // env var to an empty string (valibot defaults don't apply to ''). - build: ({ dependsOn }) => { - if (!dependsOn.env.POSTHOG_PROJECT_KEY) - return null - - const sink = createPosthogSink({ - projectKey: dependsOn.env.POSTHOG_PROJECT_KEY, - host: dependsOn.env.POSTHOG_API_HOST, - }) - dependsOn.lifecycle.appHooks.onStop(() => sink.shutdown()) - return sink - }, - }) - const productEventService = injeca.provide('services:productEvents', { - dependsOn: { db, otel, posthogSink }, - build: ({ dependsOn }) => createProductEventService(dependsOn.db, dependsOn.otel?.product, dependsOn.posthogSink), + dependsOn: { db, otel }, + build: ({ dependsOn }) => createProductEventService(dependsOn.db, dependsOn.otel?.product), }) const characterService = injeca.provide('services:characters', { diff --git a/apps/server/src/libs/env.ts b/apps/server/src/libs/env.ts index 381d26d42..3de03c850 100644 --- a/apps/server/src/libs/env.ts +++ b/apps/server/src/libs/env.ts @@ -175,14 +175,6 @@ const EnvSchema = object({ DB_POOL_CONNECTION_TIMEOUT_MS: optionalIntegerFromString(5000, 'DB_POOL_CONNECTION_TIMEOUT_MS', 1), DB_POOL_KEEPALIVE_INITIAL_DELAY_MS: optionalIntegerFromString(10000, 'DB_POOL_KEEPALIVE_INITIAL_DELAY_MS', 1), - // PostHog product-event forwarding (signup / payment / subscription facts). - // Defaults to the shared AIRI project key (same browser-safe phc_* key the - // client surfaces embed in posthog.config.ts), so forwarding is on out of - // the box. Set to an empty string to disable; Postgres `product_events` - // stays the source of truth either way. - POSTHOG_PROJECT_KEY: optional(string(), 'phc_pzjziJjrVZpa9SqnQqq0QEKvkmuCPH7GDTA6TbRTEf9'), // cspell:disable-line - POSTHOG_API_HOST: optional(string(), 'https://t.airi.build'), - // OpenTelemetry OTEL_SERVICE_NAMESPACE: optional(string(), 'airi'), OTEL_SERVICE_NAME: optional(string(), 'server'), diff --git a/apps/server/src/routes/admin/voice-packs/route.test.ts b/apps/server/src/routes/admin/voice-packs/route.test.ts index f7aa7560d..e692b7e99 100644 --- a/apps/server/src/routes/admin/voice-packs/route.test.ts +++ b/apps/server/src/routes/admin/voice-packs/route.test.ts @@ -49,7 +49,6 @@ function createService() { function createProductEventService(): ProductEventService { return { track: vi.fn(async () => undefined), - trackGeneration: vi.fn(async () => undefined), countDistinctUsersByFeature: vi.fn(async () => []), } } diff --git a/apps/server/src/routes/audio-speech-ws/route.test.ts b/apps/server/src/routes/audio-speech-ws/route.test.ts index 16bfc0429..9523fa6a2 100644 --- a/apps/server/src/routes/audio-speech-ws/route.test.ts +++ b/apps/server/src/routes/audio-speech-ws/route.test.ts @@ -186,7 +186,6 @@ function makeFakeDeps(overrides: { } const productEventService = { track: vi.fn(async () => undefined), - trackGeneration: vi.fn(async () => undefined), countDistinctUsersByFeature: vi.fn(async () => []), } const configKV = { diff --git a/apps/server/src/routes/openai/v1/analytics.ts b/apps/server/src/routes/openai/v1/analytics.ts deleted file mode 100644 index 85aca5290..000000000 --- a/apps/server/src/routes/openai/v1/analytics.ts +++ /dev/null @@ -1,21 +0,0 @@ -import type { AiGenerationAppSurface } from '../../../services/domain/product-events' - -export const AIRI_CHAT_SESSION_ID_HEADER = 'x-airi-session-id' -export const AIRI_CHAT_ROUND_ID_HEADER = 'x-airi-round-id' -export const AIRI_CHAT_APP_SURFACE_HEADER = 'x-airi-app-surface' - -const CLIENT_CHAT_ANALYTICS_SURFACES = new Set(['web', 'mobile', 'electron']) - -/** - * Resolves the product runtime from a trusted client hint. - * - * Unknown values are not coerced to `server`: `$ai_generation` uses - * `capture_surface` for the process that emitted the event, while - * `app_surface` stays reserved for the user's actual product runtime. - */ -export function resolveChatAnalyticsSurface(value: string | undefined): AiGenerationAppSurface | undefined { - if (CLIENT_CHAT_ANALYTICS_SURFACES.has(value as AiGenerationAppSurface)) - return value as AiGenerationAppSurface - - return undefined -} diff --git a/apps/server/src/routes/openai/v1/index.ts b/apps/server/src/routes/openai/v1/index.ts index e9bd2a46f..ddaeaadbc 100644 --- a/apps/server/src/routes/openai/v1/index.ts +++ b/apps/server/src/routes/openai/v1/index.ts @@ -5,12 +5,6 @@ import type { LlmTracingDeps, V1RouteDeps } from './types' import { authGuard } from '../../../middlewares/auth' import { configGuard } from '../../../middlewares/config-guard' -import { - AIRI_CHAT_APP_SURFACE_HEADER, - AIRI_CHAT_ROUND_ID_HEADER, - AIRI_CHAT_SESSION_ID_HEADER, - resolveChatAnalyticsSurface, -} from './analytics' import { createV1Gateway } from './gateway' import { chatCompletionsRateLimit } from './middlewares' import { chatCompletions } from './operations/chat-completions' @@ -47,9 +41,7 @@ export function createV1Routes(input: CreateV1RoutesDeps) { return { userId: user.id, body, - sessionId: c.req.header(AIRI_CHAT_SESSION_ID_HEADER), - roundId: c.req.header(AIRI_CHAT_ROUND_ID_HEADER), - appSurface: resolveChatAnalyticsSurface(c.req.header(AIRI_CHAT_APP_SURFACE_HEADER)), + sessionId: c.req.header('x-airi-session-id'), abortSignal: c.req.raw.signal, } }, @@ -74,7 +66,7 @@ export function createV1Routes(input: CreateV1RoutesDeps) { return { userId: user.id, body, - sessionId: c.req.header(AIRI_CHAT_SESSION_ID_HEADER), + sessionId: c.req.header('x-airi-session-id'), abortSignal: c.req.raw.signal, } }, diff --git a/apps/server/src/routes/openai/v1/operations/chat-completions/index.ts b/apps/server/src/routes/openai/v1/operations/chat-completions/index.ts index 103da8569..a30642bd2 100644 --- a/apps/server/src/routes/openai/v1/operations/chat-completions/index.ts +++ b/apps/server/src/routes/openai/v1/operations/chat-completions/index.ts @@ -1,6 +1,5 @@ import type { CapabilityAliasRoute } from '../../../../../schemas/provider-catalog' import type { UsageInfo } from '../../../../../services/domain/billing/billing' -import type { AiGenerationAppSurface } from '../../../../../services/domain/product-events' import type { GatewayCallback } from '../../gateway' import type { V1RouteDeps } from '../../types' @@ -21,25 +20,9 @@ export interface ChatCompletionsOperationRequest { userId: string body: Record sessionId?: string - roundId?: string - appSurface?: AiGenerationAppSurface abortSignal?: AbortSignal } -interface GenerationCaptureInput { - deps: V1RouteDeps - userId: string - requestId: string - sessionId?: string - roundId?: string - appSurface?: AiGenerationAppSurface - generationModel: string - routeCtxProvider: string - usage: UsageInfo - durationMs: number - stream: boolean -} - export function chatCompletions(deps: V1RouteDeps): GatewayCallback<'chat.completions'> { const logger = useLogger('v1-completions').useGlobalConfig() const telemetry = createRouteTelemetry({ @@ -199,11 +182,7 @@ export function chatCompletions(deps: V1RouteDeps): GatewayCallback<'chat.comple durationMs, requestId, userId: input.userId, - sessionId: input.sessionId, - roundId: input.roundId, - appSurface: input.appSurface, requestModel, - generationModel: langfuseModel, routeCtxProvider: routeCtx.provider, billing, billingPolicy, @@ -220,11 +199,7 @@ export function chatCompletions(deps: V1RouteDeps): GatewayCallback<'chat.comple durationMs, requestId, userId: input.userId, - sessionId: input.sessionId, - roundId: input.roundId, - appSurface: input.appSurface, requestModel, - generationModel: langfuseModel, routeCtxProvider: routeCtx.provider, billing, billingPolicy, @@ -238,37 +213,6 @@ interface ChatModelAliasPlan { modelIds: string[] } -function captureGeneration(input: GenerationCaptureInput): void { - const generationId = input.roundId ?? input.requestId - const conversationId = input.sessionId ?? input.requestId - const totalTokens = input.usage.promptTokens != null && input.usage.completionTokens != null - ? input.usage.promptTokens + input.usage.completionTokens - : undefined - - input.deps.productEventService.trackGeneration({ - userId: input.userId, - traceId: conversationId, - generationId, - model: input.generationModel, - provider: input.routeCtxProvider || 'unknown', - providerType: 'official', - usageSource: input.usage.promptTokens != null || input.usage.completionTokens != null - ? 'reported' - : 'unavailable', - inputTokens: input.usage.promptTokens, - outputTokens: input.usage.completionTokens, - totalTokens, - costUsdSource: 'unavailable', - conversationId, - conversationIdSource: input.sessionId ? 'client_header' : 'server_request', - roundId: generationId, - ...(input.appSurface && { appSurface: input.appSurface }), - captureSurface: 'server', - latencySeconds: input.durationMs / 1000, - stream: input.stream, - }) -} - async function resolveChatModelAliasPlan(deps: V1RouteDeps, aliasId: string): Promise { const alias = await deps.providerCatalogService.resolveEnabledAlias('llm', aliasId) const primaryRoutes = alias.routes.filter(route => route.pool === 'primary') @@ -358,11 +302,7 @@ function streamChatCompletion(input: { durationMs: number requestId: string userId: string - sessionId?: string - roundId?: string - appSurface?: AiGenerationAppSurface requestModel: string - generationModel: string routeCtxProvider: string billing: ChatBilling billingPolicy: ChatBillingPolicy @@ -481,20 +421,6 @@ function streamChatCompletion(input: { }) input.telemetry.recordMetrics({ model: input.requestModel, status: input.response.status, type: 'chat', provider: input.routeCtxProvider, durationMs: input.durationMs, fluxConsumed, ...usage }) - captureGeneration({ - deps: input.deps, - userId: input.userId, - requestId: input.requestId, - sessionId: input.sessionId, - roundId: input.roundId, - appSurface: input.appSurface, - generationModel: input.generationModel, - routeCtxProvider: input.routeCtxProvider, - usage, - durationMs: input.durationMs, - stream: true, - }) - // Debit flux via DB transaction (source of truth) // NOTICE: streaming response is already sent, so we cannot reject on failure. // Log at error level so unpaid usage is visible in monitoring/alerts. @@ -581,11 +507,7 @@ async function completeNonStreamingChat(input: { durationMs: number requestId: string userId: string - sessionId?: string - roundId?: string - appSurface?: AiGenerationAppSurface requestModel: string - generationModel: string routeCtxProvider: string billing: ChatBilling billingPolicy: ChatBillingPolicy @@ -634,20 +556,6 @@ async function completeNonStreamingChat(input: { }) input.telemetry.recordMetrics({ model: input.requestModel, status: input.response.status, type: 'chat', provider: input.routeCtxProvider, durationMs: input.durationMs, fluxConsumed, ...usage }) - captureGeneration({ - deps: input.deps, - userId: input.userId, - requestId: input.requestId, - sessionId: input.sessionId, - roundId: input.roundId, - appSurface: input.appSurface, - generationModel: input.generationModel, - routeCtxProvider: input.routeCtxProvider, - usage, - durationMs: input.durationMs, - stream: false, - }) - // Debit flux via DB transaction (source of truth). // The upstream call has already happened (cost incurred), so partial // debit + `fluxUnbilled` is the only sane recovery — same shape as the diff --git a/apps/server/src/routes/openai/v1/route.test.ts b/apps/server/src/routes/openai/v1/route.test.ts index 337a8e565..33b34876a 100644 --- a/apps/server/src/routes/openai/v1/route.test.ts +++ b/apps/server/src/routes/openai/v1/route.test.ts @@ -14,11 +14,6 @@ import { afterAll, beforeEach, describe, expect, it, vi } from 'vitest' import { createV1Routes } from '.' import { ApiError } from '../../../utils/error' -import { - AIRI_CHAT_APP_SURFACE_HEADER, - AIRI_CHAT_ROUND_ID_HEADER, - AIRI_CHAT_SESSION_ID_HEADER, -} from './analytics' function createMockFluxService(flux = 100): FluxService { return { @@ -148,7 +143,6 @@ function createMockLlmRouter(impl?: Partial): LlmRouterService function createMockProductEventService(): ProductEventService { return { track: vi.fn(async () => undefined), - trackGeneration: vi.fn(async () => undefined), countDistinctUsersByFeature: vi.fn(async () => []), } } @@ -913,7 +907,7 @@ describe('v1CompletionsRoutes', () => { } }) - it('records Langfuse and PostHog generations with authoritative usage and correlation', async () => { + it('records Langfuse chat generation with the router-resolved upstream model', async () => { const llmRouter = createMockLlmRouter({ route: vi.fn(async (_req, ctx) => { if (ctx) { @@ -930,27 +924,12 @@ describe('v1CompletionsRoutes', () => { }) as any, }) const llmTracing = createMockLlmTracing() - const productEventService = createMockProductEventService() - const app = createTestApp( - createMockFluxService(), - createMockConfigKV(), - undefined, - undefined, - undefined, - llmRouter, - llmTracing, - productEventService, - ) + const app = createTestApp(createMockFluxService(), createMockConfigKV(), undefined, undefined, undefined, llmRouter, llmTracing) await app.fetch( new Request('http://localhost/api/v1/openai/chat/completions', { method: 'POST', - headers: { - 'Content-Type': 'application/json', - [AIRI_CHAT_SESSION_ID_HEADER]: 'conversation-1', - [AIRI_CHAT_ROUND_ID_HEADER]: 'round-1', - [AIRI_CHAT_APP_SURFACE_HEADER]: 'electron', - }, + headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ model: 'chat-auto', messages: [{ role: 'user', content: 'hi' }] }), }), { user: testUser } as any, @@ -963,88 +942,6 @@ describe('v1CompletionsRoutes', () => { userId: 'user-1', }), ) - expect(productEventService.trackGeneration).toHaveBeenCalledWith({ - userId: 'user-1', - traceId: 'conversation-1', - generationId: 'round-1', - model: 'openai/gpt-4o-mini', - provider: 'openrouter', - providerType: 'official', - usageSource: 'reported', - inputTokens: 1, - outputTokens: 2, - totalTokens: 3, - costUsdSource: 'unavailable', - conversationId: 'conversation-1', - conversationIdSource: 'client_header', - roundId: 'round-1', - appSurface: 'electron', - captureSurface: 'server', - latencySeconds: expect.any(Number), - stream: false, - }) - }) - - it('uses request-level correlation for server-captured generations without chat headers', async () => { - const llmRouter = createMockLlmRouter({ - route: vi.fn(async (_req, ctx) => { - if (ctx) { - ctx.provider = 'openrouter' - ctx.upstreamModel = 'openai/gpt-4o-mini' - } - return new Response(JSON.stringify({ - choices: [], - usage: { prompt_tokens: 1, completion_tokens: 2, total_tokens: 3 }, - }), { - status: 200, - headers: { 'Content-Type': 'application/json' }, - }) - }) as any, - }) - const productEventService = createMockProductEventService() - const app = createTestApp( - createMockFluxService(), - createMockConfigKV(), - undefined, - undefined, - undefined, - llmRouter, - createMockLlmTracing(), - productEventService, - ) - - await app.fetch( - new Request('http://localhost/api/v1/openai/chat/completions', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ model: 'chat-auto', messages: [{ role: 'user', content: 'hi' }] }), - }), - { user: testUser } as any, - ) - - expect(productEventService.trackGeneration).toHaveBeenCalledWith({ - userId: 'user-1', - traceId: expect.any(String), - generationId: expect.any(String), - model: 'openai/gpt-4o-mini', - provider: 'openrouter', - providerType: 'official', - usageSource: 'reported', - inputTokens: 1, - outputTokens: 2, - totalTokens: 3, - costUsdSource: 'unavailable', - conversationId: expect.any(String), - conversationIdSource: 'server_request', - roundId: expect.any(String), - captureSurface: 'server', - latencySeconds: expect.any(Number), - stream: false, - }) - const generation = vi.mocked(productEventService.trackGeneration).mock.calls[0]?.[0] - expect(generation?.traceId).toBe(generation?.conversationId) - expect(generation?.roundId).toBe(generation?.generationId) - expect(generation).not.toHaveProperty('appSurface') }) it('should not charge flux when upstream returns error', async () => { diff --git a/apps/server/src/routes/stripe/operations/checkout.ts b/apps/server/src/routes/stripe/operations/checkout.ts index 3c5c3d9e8..dc8e29f94 100644 --- a/apps/server/src/routes/stripe/operations/checkout.ts +++ b/apps/server/src/routes/stripe/operations/checkout.ts @@ -33,11 +33,6 @@ export interface CheckoutOperationInput { request: Request } -interface PosthogIdentityHeaders { - distinctId?: string - sessionId?: string -} - /** * Creates Stripe checkout sessions for Flux packages. * @@ -80,7 +75,6 @@ export function createCheckoutOperation(deps: CheckoutOperationDeps) { const paymentMethods = await deps.configKV.getOptional('STRIPE_PAYMENT_METHODS') const paymentMethodOptions = await deps.configKV.getOptional('STRIPE_PAYMENT_METHOD_OPTIONS') ?? {} - const posthogIdentity = readPosthogIdentityHeaders(input.request) const sessionParams: CheckoutSessionCreateParams = { line_items: [{ price: stripePriceId, quantity: 1 }], @@ -93,8 +87,6 @@ export function createCheckoutOperation(deps: CheckoutOperationDeps) { metadata: { userId: input.user.id, fluxAmount: String(fluxAmount), - ...(posthogIdentity.distinctId && { posthogDistinctId: posthogIdentity.distinctId }), - ...(posthogIdentity.sessionId && { posthogSessionId: posthogIdentity.sessionId }), }, } @@ -141,31 +133,9 @@ export function createCheckoutOperation(deps: CheckoutOperationDeps) { flux_amount: fluxAmount, amount_total: session.amount_total, currency: session.currency, - ...(posthogIdentity.distinctId && { posthog_distinct_id: posthogIdentity.distinctId }), - ...(posthogIdentity.sessionId && { posthog_session_id: posthogIdentity.sessionId }), }, }) return { url: session.url } } } - -function readPosthogIdentityHeaders(request: Request): PosthogIdentityHeaders { - const distinctId = readStripeMetadataHeader(request, 'x-posthog-distinct-id') - const sessionId = readStripeMetadataHeader(request, 'x-posthog-session-id') - return { - ...(distinctId && { distinctId }), - ...(sessionId && { sessionId }), - } -} - -function readStripeMetadataHeader(request: Request, name: string): string | undefined { - const value = request.headers.get(name)?.trim() - if (!value) - return undefined - - // Stripe metadata values are capped and user-controlled headers can be - // oversized. Truncating keeps the checkout request valid without turning - // analytics identity into a payment blocker. - return value.slice(0, 200) -} diff --git a/apps/server/src/routes/stripe/operations/webhook.ts b/apps/server/src/routes/stripe/operations/webhook.ts index 7a1449b69..8f593454b 100644 --- a/apps/server/src/routes/stripe/operations/webhook.ts +++ b/apps/server/src/routes/stripe/operations/webhook.ts @@ -89,8 +89,6 @@ export function createWebhookOperation(deps: WebhookOperationDeps) { const userId = event.data.object.metadata?.userId if (userId) { const fluxAmount = Number(event.data.object.metadata?.fluxAmount) - const posthogDistinctId = event.data.object.metadata?.posthogDistinctId - const posthogSessionId = event.data.object.metadata?.posthogSessionId void deps.productEventService?.track({ userId, feature: 'billing', @@ -101,10 +99,6 @@ export function createWebhookOperation(deps: WebhookOperationDeps) { amount_total: event.data.object.amount_total, currency: event.data.object.currency, flux_amount: Number.isFinite(fluxAmount) ? fluxAmount : null, - stripe_checkout_session_id: event.data.object.id, - stripe_customer_id: typeof event.data.object.customer === 'string' ? event.data.object.customer : event.data.object.customer?.id ?? null, - ...(posthogDistinctId && { posthog_distinct_id: posthogDistinctId }), - ...(posthogSessionId && { posthog_session_id: posthogSessionId }), }, }) } diff --git a/apps/server/src/routes/stripe/route.test.ts b/apps/server/src/routes/stripe/route.test.ts index ebba3cb33..1dbd76ee0 100644 --- a/apps/server/src/routes/stripe/route.test.ts +++ b/apps/server/src/routes/stripe/route.test.ts @@ -10,7 +10,6 @@ import { describe, expect, it, vi } from 'vitest' import { createStripeRoutes, formatPrice } from '.' import { ApiError } from '../../utils/error' -import { createCheckoutOperation } from './operations/checkout' import { createWebhookOperation } from './operations/webhook' // --- Mock helpers --- @@ -303,77 +302,6 @@ describe('stripeRoutes', () => { ) expect(res.status).toBe(503) }) - - it('stores browser PostHog identity in Stripe checkout metadata', async () => { - const createSession = vi.fn(async input => ({ - id: 'cs_1', - url: 'https://checkout.stripe.com/cs_1', - customer: null, - mode: 'payment', - status: 'open', - payment_status: 'unpaid', - amount_total: 500, - currency: 'usd', - success_url: 'http://localhost/settings/flux?success=true', - cancel_url: 'http://localhost/settings/flux?canceled=true', - payment_intent: null, - subscription: null, - metadata: input.metadata, - expires_at: null, - })) - const productEventService = { track: vi.fn() } - const operation = createCheckoutOperation({ - stripe: { - checkout: { - sessions: { - create: createSession, - }, - }, - } as any, - priceCatalog: { - findActivePrice: vi.fn(async () => ({ - id: 'price_test_500', - currency: 'usd', - unitAmount: 500, - currencyOptions: {}, - metadata: { fluxAmount: '500' }, - })), - getActivePrices: vi.fn(), - } as any, - stripeService: createMockStripeService(), - configKV: createMockConfigKV({ STRIPE_PAYMENT_METHODS: undefined }), - env: testEnv, - productEventService: productEventService as any, - }) - - await operation({ - user: testUser as any, - body: { stripePriceId: 'price_test_500' }, - request: new Request('http://localhost/api/v1/stripe/checkout', { - headers: { - 'x-posthog-distinct-id': 'anon-browser-1', - 'x-posthog-session-id': 'ph-session-1', - }, - }), - }) - - expect(createSession).toHaveBeenCalledWith(expect.objectContaining({ - metadata: { - userId: 'user-1', - fluxAmount: '500', - posthogDistinctId: 'anon-browser-1', - posthogSessionId: 'ph-session-1', - }, - })) - expect(productEventService.track).toHaveBeenCalledWith(expect.objectContaining({ - userId: 'user-1', - action: 'checkout_started', - metadata: expect.objectContaining({ - posthog_distinct_id: 'anon-browser-1', - posthog_session_id: 'ph-session-1', - }), - })) - }) }) describe('gET /api/v1/stripe/orders', () => { @@ -543,75 +471,6 @@ describe('stripeRoutes', () => { expect(res.status).toBe(503) }) - it('records payment completion with Stripe and PostHog identity from checkout metadata', async () => { - const checkoutEvent = { - id: 'evt_checkout_completed', - type: 'checkout.session.completed', - data: { - object: { - id: 'cs_1', - customer: 'cus_1', - customer_email: 'test@example.com', - mode: 'payment', - status: 'complete', - payment_status: 'paid', - amount_total: 500, - currency: 'usd', - success_url: 'http://localhost/settings/flux?success=true', - cancel_url: 'http://localhost/settings/flux?canceled=true', - payment_intent: 'pi_1', - subscription: null, - metadata: { - userId: 'user-1', - fluxAmount: '500', - posthogDistinctId: 'anon-browser-1', - posthogSessionId: 'ph-session-1', - }, - expires_at: null, - }, - }, - } - const productEventService = { track: vi.fn() } - const billingService = createMockBillingService() - const webhook = createWebhookOperation({ - stripe: { - webhooks: { - constructEvent: vi.fn(() => checkoutEvent), - }, - } as any, - webhookSecret: 'whsec_test', - fluxService: createMockFluxService(), - stripeService: createMockStripeService(), - billingService, - productEventService: productEventService as any, - }) - - await webhook({ signature: 'test_sig', body: '{}' }) - - expect(billingService.creditFluxFromStripeCheckout).toHaveBeenCalledWith(expect.objectContaining({ - stripeEventId: 'evt_checkout_completed', - userId: 'user-1', - stripeSessionId: 'cs_1', - fluxAmount: 500, - })) - expect(productEventService.track).toHaveBeenCalledWith({ - userId: 'user-1', - feature: 'billing', - action: 'payment_completed', - status: 'succeeded', - source: 'stripe.webhook', - metadata: { - amount_total: 500, - currency: 'usd', - flux_amount: 500, - stripe_checkout_session_id: 'cs_1', - stripe_customer_id: 'cus_1', - posthog_distinct_id: 'anon-browser-1', - posthog_session_id: 'ph-session-1', - }, - }) - }) - it('records subscription lifecycle product events from Stripe webhooks', async () => { const subscriptionEvent = { id: 'evt_sub_created', diff --git a/apps/server/src/services/adapters/posthog.ts b/apps/server/src/services/adapters/posthog.ts deleted file mode 100644 index bbbe2d8b9..000000000 --- a/apps/server/src/services/adapters/posthog.ts +++ /dev/null @@ -1,78 +0,0 @@ -import { useLogger } from '@guiiai/logg' -import { PostHog } from 'posthog-node' - -const logger = useLogger('posthog') - -/** - * One product event forwarded to PostHog, keyed by the Better Auth user id - * so it merges with the browser person identified via `posthog.identify()`. - */ -export interface PosthogCaptureInput { - distinctId: string - event: string - properties: Record -} - -/** - * Minimal capture boundary the product-events service depends on. Kept as - * an interface so tests inject a fake instead of mocking the SDK. - */ -export interface PosthogSink { - /** - * Queue a high-volume analytics event without waiting for a network - * roundtrip. Use on request hot paths where occasional process-exit loss is - * preferable to user-visible latency. - */ - captureQueued?: (input: PosthogCaptureInput) => void - capture: (input: PosthogCaptureInput) => Promise - /** Flush and close the underlying client. Call on server shutdown. */ - shutdown: () => Promise -} - -/** - * PostHog sink for server-side product events. - * - * Low-frequency conversion facts use `captureImmediate` (one HTTP roundtrip - * per event) because they terminate money/auth funnels. High-frequency AI - * generation facts use `captureQueued`, which is buffered by the SDK and - * flushed on shutdown, so chat completion requests don't wait on PostHog. - * - * Capture failures are logged and swallowed — analytics forwarding must - * never fail the Stripe webhook or auth flow that triggered it. The - * Postgres `product_events` row is the source of truth either way. - */ -export function createPosthogSink(options: { projectKey: string, host: string }): PosthogSink { - const client = new PostHog(options.projectKey, { host: options.host }) - - return { - captureQueued(input: PosthogCaptureInput): void { - try { - client.capture({ - distinctId: input.distinctId, - event: input.event, - properties: input.properties, - }) - } - catch (err) { - logger.withError(err).withFields({ event: input.event }).warn('Failed to enqueue product event to PostHog') - } - }, - - async capture(input: PosthogCaptureInput): Promise { - try { - await client.captureImmediate({ - distinctId: input.distinctId, - event: input.event, - properties: input.properties, - }) - } - catch (err) { - logger.withError(err).withFields({ event: input.event }).warn('Failed to forward product event to PostHog') - } - }, - - async shutdown(): Promise { - await client.shutdown() - }, - } -} diff --git a/apps/server/src/services/domain/product-events.test.ts b/apps/server/src/services/domain/product-events.test.ts index 0c32f38b0..73bec67c6 100644 --- a/apps/server/src/services/domain/product-events.test.ts +++ b/apps/server/src/services/domain/product-events.test.ts @@ -1,7 +1,6 @@ import type { Database } from '../../libs/db' import type { ProductMetrics } from '../../otel' -import { sql } from 'drizzle-orm' import { beforeAll, beforeEach, describe, expect, it, vi } from 'vitest' import { mockDB } from '../../libs/mock-db' @@ -137,239 +136,4 @@ describe('productEventService', () => { flux_balance_bucket: 'zero', }) }) - - it('forwards allowlisted business facts to PostHog keyed by user id, mapping user_signed_up to signup_completed', async () => { - const capture = vi.fn(async () => {}) - const sink = { capture, shutdown: vi.fn(async () => {}) } - const service = createProductEventService(db, null, sink) - - await service.track({ - userId: 'user-1', - feature: 'billing', - action: 'payment_completed', - status: 'succeeded', - source: 'stripe.webhook', - metadata: { amount_minor_unit: 990, currency: 'usd' }, - }) - await service.track({ - userId: 'user-2', - feature: 'auth', - action: 'user_signed_up', - status: 'succeeded', - }) - - expect(capture).toHaveBeenNthCalledWith(1, { - distinctId: 'user-1', - event: 'payment_completed', - properties: { - app_surface: 'server', - airi_user_id: 'user-1', - feature: 'billing', - status: 'succeeded', - source: 'stripe.webhook', - amount_minor_unit: 990, - currency: 'usd', - }, - }) - expect(capture).toHaveBeenNthCalledWith(2, { - distinctId: 'user-2', - event: 'signup_completed', - properties: { - app_surface: 'server', - airi_user_id: 'user-2', - feature: 'auth', - status: 'succeeded', - }, - }) - - const rows = await db.select().from(schema.productEvents) - expect(rows).toHaveLength(2) - }) - - it('merges Stripe webhook conversions with the browser PostHog person when a distinct id is present', async () => { - const capture = vi.fn(async () => {}) - const sink = { capture, shutdown: vi.fn(async () => {}) } - const service = createProductEventService(db, null, sink) - - await service.track({ - userId: 'user-1', - feature: 'billing', - action: 'payment_completed', - status: 'succeeded', - source: 'stripe.webhook', - metadata: { - posthog_distinct_id: 'anon-browser-1', - posthog_session_id: 'ph-session-1', - stripe_checkout_session_id: 'cs_1', - }, - }) - - expect(capture).toHaveBeenNthCalledWith(1, { - distinctId: 'user-1', - event: '$identify', - properties: { - $anon_distinct_id: 'anon-browser-1', - $session_id: 'ph-session-1', - airi_user_id: 'user-1', - }, - }) - expect(capture).toHaveBeenNthCalledWith(2, { - distinctId: 'user-1', - event: 'payment_completed', - properties: { - app_surface: 'server', - airi_user_id: 'user-1', - posthog_distinct_id: 'anon-browser-1', - $session_id: 'ph-session-1', - feature: 'billing', - status: 'succeeded', - source: 'stripe.webhook', - posthog_session_id: 'ph-session-1', - stripe_checkout_session_id: 'cs_1', - }, - }) - }) - - it('does not forward high-volume per-request actions to PostHog', async () => { - const capture = vi.fn(async () => {}) - const sink = { capture, shutdown: vi.fn(async () => {}) } - const service = createProductEventService(db, null, sink) - - await service.track({ - userId: 'user-1', - feature: 'gen_ai_chat', - action: 'completion_succeeded', - status: 'succeeded', - }) - await service.track({ - userId: 'user-1', - feature: 'billing', - action: 'checkout_started', - status: 'started', - }) - - expect(capture).not.toHaveBeenCalled() - const rows = await db.select().from(schema.productEvents) - expect(rows).toHaveLength(2) - }) - - it('captures an LLM generation as a PostHog AI fact without storing prompts or responses', async () => { - const capture = vi.fn(async () => {}) - const captureQueued = vi.fn() - const sink = { capture, captureQueued, shutdown: vi.fn(async () => {}) } - const service = createProductEventService(db, null, sink) - - service.trackGeneration({ - userId: 'user-1', - traceId: 'session-1', - generationId: 'round-1', - model: 'openai/gpt-5-mini', - provider: 'openai', - providerType: 'official', - usageSource: 'reported', - inputTokens: 12, - outputTokens: 8, - totalTokens: 20, - conversationId: 'session-1', - conversationIdSource: 'client_header', - roundId: 'round-1', - appSurface: 'electron', - captureSurface: 'server', - }) - - expect(capture).not.toHaveBeenCalled() - expect(captureQueued).toHaveBeenCalledWith({ - distinctId: 'user-1', - event: '$ai_generation', - properties: { - $ai_trace_id: 'session-1', - $ai_session_id: 'session-1', - $ai_span_id: 'round-1', - $ai_model: 'openai/gpt-5-mini', - $ai_provider: 'openai', - $ai_input_tokens: 12, - $ai_output_tokens: 8, - $ai_total_tokens: 20, - $insert_id: 'ai-generation:round-1', - airi_user_id: 'user-1', - provider_type: 'official', - usage_source: 'reported', - token_usage_available: true, - cost_usd_source: 'unavailable', - cost_usd_known: false, - conversation_id: 'session-1', - conversation_id_source: 'client_header', - round_id: 'round-1', - app_surface: 'electron', - capture_surface: 'server', - }, - }) - - const rows = await db.select().from(schema.productEvents) - expect(rows).toHaveLength(0) - }) - - it('does not forward to PostHog when the DB write fails', async () => { - // ROOT CAUSE: - // - // track() swallows DB insert errors to protect the caller, but the - // PostHog forwarding block ran unconditionally afterwards — a Postgres - // outage during a Stripe webhook would mint `payment_completed` in - // PostHog with no `product_events` row backing it, breaking the - // "Postgres is the fact of record" invariant and later reconciliation. - // Found by PR #2038 review. - // - // Fixed by gating forwarding on a `persisted` flag set only after the - // insert resolves. - const capture = vi.fn(async () => {}) - const sink = { capture, shutdown: vi.fn(async () => {}) } - const service = createProductEventService(db, null, sink) - - // Simulate a DB outage by renaming the table out from under the insert. - await db.execute(sql`ALTER TABLE product_events RENAME TO product_events_outage`) - try { - await expect(service.track({ - userId: 'user-1', - feature: 'billing', - action: 'payment_completed', - status: 'succeeded', - })).resolves.toBeUndefined() - } - finally { - await db.execute(sql`ALTER TABLE product_events_outage RENAME TO product_events`) - } - - expect(capture).not.toHaveBeenCalled() - const rows = await db.select().from(schema.productEvents) - expect(rows).toHaveLength(0) - }) - - it('persists the product event even when a misbehaving sink throws', async () => { - // ROOT CAUSE: - // - // The PosthogSink contract says implementations swallow transport - // errors, but the forwarding call sits on the Stripe webhook path — - // if a sink ever throws, an unguarded `await` would fail the webhook - // after the fact was already persisted, causing Stripe to retry and - // (before the idempotency guard) double-process the payment. - // - // track() therefore wraps forwarding in its own try/catch: the DB row - // must survive and track() must resolve regardless of sink behavior. - const capture = vi.fn(async () => { - throw new Error('posthog exploded') - }) - const sink = { capture, shutdown: vi.fn(async () => {}) } - const service = createProductEventService(db, null, sink) - - await expect(service.track({ - userId: 'user-1', - feature: 'billing', - action: 'payment_completed', - status: 'succeeded', - })).resolves.toBeUndefined() - - const rows = await db.select().from(schema.productEvents) - expect(rows).toHaveLength(1) - expect(rows[0]).toMatchObject({ action: 'payment_completed', status: 'succeeded' }) - }) }) diff --git a/apps/server/src/services/domain/product-events.ts b/apps/server/src/services/domain/product-events.ts index 56a6bf0bc..a1f4cef91 100644 --- a/apps/server/src/services/domain/product-events.ts +++ b/apps/server/src/services/domain/product-events.ts @@ -1,7 +1,6 @@ import type { Database } from '../../libs/db' import type { ProductMetrics } from '../../otel' import type { ProductEventMetadata } from '../../schemas/product-events' -import type { PosthogSink } from '../adapters/posthog' import { useLogger } from '@guiiai/logg' import { and, asc, count, gte, lt, sql } from 'drizzle-orm' @@ -76,63 +75,6 @@ export interface ProductEventAggregateRow { distinctUsers: number } -/** Product runtime where the user initiated the AI generation. */ -export type AiGenerationAppSurface = 'web' | 'mobile' | 'electron' - -/** Runtime that captured the `$ai_generation` fact. */ -export type AiGenerationCaptureSurface = 'server' | 'client' - -/** Explains whether `conversation_id` is an app conversation or a server fallback. */ -export type AiGenerationConversationIdSource = 'client_header' | 'server_request' - -/** Explains whether AIRI supplied a trustworthy USD cost for this generation. */ -export type AiGenerationCostUsdSource = 'reported' | 'estimated' | 'unavailable' - -/** Content-free PostHog AI generation fact keyed to the authenticated user. */ -export interface AiGenerationEventInput { - userId: string - traceId: string - generationId: string - model: string - provider: string - providerType: 'official' | 'custom' | 'unknown' - usageSource: 'reported' | 'estimated' | 'unavailable' - inputTokens?: number - outputTokens?: number - totalTokens?: number - totalCostUsd?: number - costUsdSource?: AiGenerationCostUsdSource - /** Always present for joins; `conversationIdSource` tells whether it is request-level fallback. */ - conversationId: string - /** Distinguishes real client conversation ids from server-generated request fallbacks. */ - conversationIdSource: AiGenerationConversationIdSource - roundId?: string - /** Omitted when the server cannot determine the user's product runtime. */ - appSurface?: AiGenerationAppSurface - /** Defaults to `server` because this service runs in the API process. */ - captureSurface?: AiGenerationCaptureSurface - latencySeconds?: number - stream?: boolean -} - -/** - * Server-side actions worth a PostHog copy, mapped to the event name the - * client-side funnels expect. Only business facts that terminate or anchor - * a funnel are forwarded — per-request LLM/TTS volume stays in Postgres and - * Grafana where it belongs (see `docs/ai-context/metrics-ownership.md`). - * - * `user_signed_up` maps to `signup_completed` because the identified server - * hook is the canonical registration fact for every signup method. Anonymous - * auth UI progress uses `signup_form_completed` and never reuses this name. - */ -const POSTHOG_FORWARDED_ACTIONS: Partial> = { - user_signed_up: 'signup_completed', - payment_completed: 'payment_completed', - subscription_started: 'subscription_started', - subscription_renewed: 'subscription_renewed', - subscription_cancelled: 'subscription_cancelled', -} - /** * Builds bounded Prometheus labels from product event inputs. */ @@ -154,11 +96,6 @@ function metricLabels(input: ProductEventInput): Record { return attrs } -function stringMetadata(input: ProductEventInput, key: string): string | undefined { - const value = input.metadata?.[key] - return typeof value === 'string' && value.length > 0 ? value : undefined -} - /** * Creates AIRI's first-party product analytics event writer. * @@ -176,57 +113,9 @@ function stringMetadata(input: ProductEventInput, key: string): string | undefin * Returns: * - Best-effort event writer plus a DB aggregation helper for analytics jobs. */ -export function createProductEventService(db: Database, metrics?: ProductMetrics | null, posthog?: PosthogSink | null) { +export function createProductEventService(db: Database, metrics?: ProductMetrics | null) { return { - trackGeneration(input: AiGenerationEventInput): void { - if (!posthog) - return - - const event = { - distinctId: input.userId, - event: '$ai_generation', - properties: { - $ai_trace_id: input.traceId, - $ai_session_id: input.conversationId, - $ai_span_id: input.generationId, - $ai_model: input.model, - $ai_provider: input.provider, - ...(input.inputTokens != null && { $ai_input_tokens: input.inputTokens }), - ...(input.outputTokens != null && { $ai_output_tokens: input.outputTokens }), - ...(input.totalTokens != null && { $ai_total_tokens: input.totalTokens }), - ...(input.totalCostUsd != null && { $ai_total_cost_usd: input.totalCostUsd }), - ...(input.latencySeconds != null && { $ai_latency: input.latencySeconds }), - ...(input.stream != null && { $ai_stream: input.stream }), - $insert_id: `ai-generation:${input.generationId}`, - airi_user_id: input.userId, - provider_type: input.providerType, - usage_source: input.usageSource, - token_usage_available: input.usageSource !== 'unavailable', - cost_usd_source: input.costUsdSource ?? 'unavailable', - cost_usd_known: input.totalCostUsd != null, - conversation_id: input.conversationId, - conversation_id_source: input.conversationIdSource, - ...(input.roundId && { round_id: input.roundId }), - ...(input.appSurface && { app_surface: input.appSurface }), - capture_surface: input.captureSurface ?? 'server', - }, - } - - if (posthog.captureQueued) { - posthog.captureQueued(event) - return - } - - void posthog.capture(event) - .catch(err => logger.withError(err).withFields({ generationId: input.generationId }).warn('Failed to capture PostHog AI generation')) - }, - async track(input: ProductEventInput): Promise { - // Postgres is the fact of record, so forwarding is gated both ways: - // the DB write comes first (a PostHog outage can't lose the row) and - // forwarding only runs when the row actually landed (a DB outage - // can't mint PostHog events with no DB backing). - let persisted = false try { await db.insert(schema.productEvents).values({ userId: input.userId, @@ -240,7 +129,6 @@ export function createProductEventService(db: Database, metrics?: ProductMetrics metadata: input.metadata, createdAt: input.createdAt, }) - persisted = true metrics?.events.add(1, metricLabels(input)) } @@ -252,49 +140,6 @@ export function createProductEventService(db: Database, metrics?: ProductMetrics status: input.status, }).warn('Failed to write product event; swallowing to protect caller') } - - // PostHog copy so browser funnels (identified by the same Better Auth - // user id) get their server-side terminator events. - const forwardedEvent = POSTHOG_FORWARDED_ACTIONS[input.action] - if (persisted && posthog && forwardedEvent) { - try { - const posthogDistinctId = stringMetadata(input, 'posthog_distinct_id') - const posthogSessionId = stringMetadata(input, 'posthog_session_id') - if (posthogDistinctId && posthogDistinctId !== input.userId) { - await posthog.capture({ - distinctId: input.userId, - event: '$identify', - properties: { - $anon_distinct_id: posthogDistinctId, - airi_user_id: input.userId, - ...(posthogSessionId && { $session_id: posthogSessionId }), - }, - }) - } - - await posthog.capture({ - distinctId: input.userId, - event: forwardedEvent, - properties: { - app_surface: 'server', - airi_user_id: input.userId, - ...(posthogDistinctId && { posthog_distinct_id: posthogDistinctId }), - ...(posthogSessionId && { $session_id: posthogSessionId }), - feature: input.feature, - status: input.status, - ...(input.source && { source: input.source }), - ...(input.reason && { reason: input.reason }), - ...input.metadata, - }, - }) - } - catch (err) { - // The sink contract already swallows transport errors; this guard - // is the last line so a misbehaving sink can never fail the - // webhook/auth flow that produced the business fact. - logger.withError(err).withFields({ action: input.action }).warn('PostHog forwarding threw; product event already persisted') - } - } }, async countDistinctUsersByFeature(input: ProductEventAggregateInput): Promise { diff --git a/apps/stage-pocket/android/app-version.properties b/apps/stage-pocket/android/app-version.properties index 3c3976ac7..69705c5c7 100644 --- a/apps/stage-pocket/android/app-version.properties +++ b/apps/stage-pocket/android/app-version.properties @@ -1,2 +1,2 @@ -AIRI_VERSION_NAME=0.11.1 -AIRI_VERSION_CODE=14 +AIRI_VERSION_NAME=0.11.0 +AIRI_VERSION_CODE=13 diff --git a/apps/stage-pocket/ios/App/App.xcodeproj/project.pbxproj b/apps/stage-pocket/ios/App/App.xcodeproj/project.pbxproj index 241ce4a7d..aca8c9b25 100644 --- a/apps/stage-pocket/ios/App/App.xcodeproj/project.pbxproj +++ b/apps/stage-pocket/ios/App/App.xcodeproj/project.pbxproj @@ -317,7 +317,7 @@ ASSETCATALOG_COMPILER_INCLUDE_ALL_APPICON_ASSETS = NO; CODE_SIGN_IDENTITY = "Apple Development"; CODE_SIGN_STYLE = Automatic; - CURRENT_PROJECT_VERSION = 18; + CURRENT_PROJECT_VERSION = 17; DEVELOPMENT_TEAM = 433DLLA855; INFOPLIST_FILE = App/Info.plist; INFOPLIST_KEY_CFBundleDisplayName = AIRI; @@ -327,7 +327,7 @@ "$(inherited)", "@executable_path/Frameworks", ); - MARKETING_VERSION = 0.11.1; + MARKETING_VERSION = 0.11.0; OTHER_SWIFT_FLAGS = "$(inherited) \"-D\" \"COCOAPODS\" \"-DDEBUG\""; PRODUCT_BUNDLE_IDENTIFIER = "ai.moeru.airi-pocket"; PRODUCT_NAME = "$(TARGET_NAME)"; @@ -345,7 +345,7 @@ ASSETCATALOG_COMPILER_INCLUDE_ALL_APPICON_ASSETS = NO; CODE_SIGN_IDENTITY = "Apple Distribution"; CODE_SIGN_STYLE = Manual; - CURRENT_PROJECT_VERSION = 18; + CURRENT_PROJECT_VERSION = 17; DEVELOPMENT_TEAM = 433DLLA855; INFOPLIST_FILE = App/Info.plist; INFOPLIST_KEY_CFBundleDisplayName = AIRI; @@ -355,7 +355,7 @@ "$(inherited)", "@executable_path/Frameworks", ); - MARKETING_VERSION = 0.11.1; + MARKETING_VERSION = 0.11.0; PRODUCT_BUNDLE_IDENTIFIER = "ai.moeru.airi-pocket"; PRODUCT_NAME = "$(TARGET_NAME)"; PROVISIONING_PROFILE_SPECIFIER = "AIRI CI"; diff --git a/apps/stage-pocket/src/App.vue b/apps/stage-pocket/src/App.vue index afecb30c4..10498f4ba 100644 --- a/apps/stage-pocket/src/App.vue +++ b/apps/stage-pocket/src/App.vue @@ -1,6 +1,5 @@ - - -
- - - - diff --git a/apps/stage-tamagotchi/src/renderer/pages/about.vue b/apps/stage-tamagotchi/src/renderer/pages/about.vue index b7272dd16..904100ac2 100644 --- a/apps/stage-tamagotchi/src/renderer/pages/about.vue +++ b/apps/stage-tamagotchi/src/renderer/pages/about.vue @@ -7,14 +7,14 @@ import semver from 'semver' import { useElectronAutoUpdater, useElectronEventaInvoke } from '@proj-airi/electron-vueuse' import { AboutContent, BugReportDialog, createBugReportPageContext, MarkdownRenderer } from '@proj-airi/stage-ui/components' -import { useAnalytics, useBreakpoints } from '@proj-airi/stage-ui/composables' +import { useBreakpoints } from '@proj-airi/stage-ui/composables' import { useSharedAnalyticsStore } from '@proj-airi/stage-ui/stores/analytics' import { Button, ContainerError, DoubleCheckButton, FieldSelect, Progress } from '@proj-airi/ui' import { useClipboard } from '@vueuse/core' import { storeToRefs } from 'pinia' import { DialogContent, DialogDescription, DialogOverlay, DialogPortal, DialogRoot, DialogTitle } from 'reka-ui' import { DrawerContent, DrawerDescription, DrawerHandle, DrawerOverlay, DrawerPortal, DrawerRoot, DrawerTitle } from 'vaul-vue' -import { computed, onMounted, ref, watch } from 'vue' +import { computed, onMounted, ref } from 'vue' import { useI18n } from 'vue-i18n' import { electronGetUpdaterPreferences, electronSetUpdaterPreferences } from '../../shared/eventa' @@ -32,13 +32,6 @@ const { quitAndInstall, } = useElectronAutoUpdater() -const { - trackUpdateCheckClicked, - trackUpdateDownloaded, - trackUpdateInstallClicked, - trackSettingsChanged, -} = useAnalytics() - const isDisabled = computed(() => updateState.value.status === 'disabled') const isLatestVersion = computed(() => { return updateState.value.status === 'not-available' && !isDisabled.value @@ -126,23 +119,6 @@ const isDowngradeUpdate = computed(() => { const getUpdaterPreferences = useElectronEventaInvoke(electronGetUpdaterPreferences) const setUpdaterPreferences = useElectronEventaInvoke(electronSetUpdaterPreferences) -function handleCheckForUpdates() { - trackUpdateCheckClicked({ channel: selectedUpdateChannel.value }) - return checkForUpdates() -} - -function handleQuitAndInstall() { - trackUpdateInstallClicked({ channel: selectedUpdateChannel.value, version: updateState.value.info?.version }) - quitAndInstall() -} - -// Fires on the state transition (not the download click) so the event means -// "an update binary actually landed on this machine". -watch(() => updateState.value.status, (status) => { - if (status === 'downloaded') - trackUpdateDownloaded({ channel: selectedUpdateChannel.value, version: updateState.value.info?.version }) -}) - function handleDownloadClick() { if (updateState.value.info?.releaseNotes) showChangelog.value = true @@ -205,16 +181,9 @@ async function setUpdateChannelPreference(channel: UpdateChannelOption) { isUpdateChannelUpdating.value = true try { - const previousChannel = selectedUpdateChannel.value const nextChannel = channel === 'auto' ? undefined : channel as ElectronUpdaterChannel const preferences = await setUpdaterPreferences({ channel: nextChannel }) selectedUpdateChannel.value = preferences?.channel ?? 'auto' - trackSettingsChanged({ - setting_name: 'update_channel', - previous_value: previousChannel, - new_value: selectedUpdateChannel.value, - source: 'settings', - }) await checkForUpdates() } finally { @@ -356,7 +325,7 @@ onMounted(() => {
{{ restartButtonLabel }}