2 commits

Author	SHA1	Message	Date
rcourtman	c409e7a05e	feat(eval): add patrol-specific eval scenarios and assertions ... Add comprehensive patrol evaluation framework: - patrol.go: Runner for patrol scenarios with streaming support - patrol_assertions.go: Assertions for tool usage, findings, timing - patrol_scenarios.go: Scenarios for basic, investigation, finding quality - eval_test.go: Unit tests for patrol eval runner Scenarios: - patrol-basic: Verifies patrol completes with tools and findings - patrol-investigation: Ensures investigation before reporting - patrol-finding-quality: Validates finding structure and evidence Run with: go run ./cmd/eval -scenario patrol	2026-01-28 23:19:11 +00:00
rcourtman	a04d41ce2c	Add end-to-end evaluation framework for AI assistant testing ... Implement comprehensive eval framework for testing Pulse Assistant: Core components: - Runner: Executes scenarios against live API with SSE stream parsing - Assertions: Reusable checks (tool usage, content, duration, errors) - Scenarios: Multi-step test workflows with configurable assertions Basic scenarios: - QuickSmokeTest: Minimal functionality verification - ReadOnlyInfrastructure: List, logs, status operations - RoutingValidation: Command routing to correct targets - LogTailing: Bounded log commands complete properly - Discovery: Infrastructure discovery capabilities Advanced scenarios: - TroubleshootingScenario: Multi-step investigation workflow - DeepDiveScenario: Thorough single-service investigation - ConfigInspectionScenario: Reading configuration files - ResourceAnalysisScenario: Cross-container resource comparison - MultiNodeScenario: Operations across Proxmox nodes - DockerInDockerScenario: Docker containers inside LXCs - ContextChainScenario: Context retention across turns Usage: go test ./internal/ai/eval -live -run TestQuickSmokeTest	2026-01-28 16:49:24 +00:00