mirror of
https://github.com/block/goose.git
synced 2026-04-28 19:49:51 +00:00
Add MLflow integration guide (#7563)
Some checks are pending
Canary / Upload Install Script (push) Blocked by required conditions
Canary / bundle-desktop (push) Blocked by required conditions
Canary / bundle-desktop-linux (push) Blocked by required conditions
Canary / bundle-desktop-windows (push) Blocked by required conditions
Canary / Prepare Version (push) Waiting to run
Canary / build-cli (push) Blocked by required conditions
Canary / Release (push) Blocked by required conditions
CI / changes (push) Waiting to run
CI / Check Rust Code Format (push) Blocked by required conditions
CI / Build and Test Rust Project (push) Blocked by required conditions
CI / Lint Rust Code (push) Blocked by required conditions
CI / Check OpenAPI Schema is Up-to-Date (push) Blocked by required conditions
CI / Test and Lint Electron Desktop App (push) Blocked by required conditions
Deploy Documentation / deploy (push) Waiting to run
Live Provider Tests / check-fork (push) Waiting to run
Live Provider Tests / changes (push) Blocked by required conditions
Live Provider Tests / Build Binary (push) Blocked by required conditions
Live Provider Tests / Smoke Tests (push) Blocked by required conditions
Live Provider Tests / Smoke Tests (Code Execution) (push) Blocked by required conditions
Live Provider Tests / Compaction Tests (push) Blocked by required conditions
Live Provider Tests / goose server HTTP integration tests (push) Blocked by required conditions
Publish Ask AI Bot Docker Image / docker (push) Waiting to run
Publish Docker Image / docker (push) Waiting to run
Scorecard supply-chain security / Scorecard analysis (push) Waiting to run
Some checks are pending
Canary / Upload Install Script (push) Blocked by required conditions
Canary / bundle-desktop (push) Blocked by required conditions
Canary / bundle-desktop-linux (push) Blocked by required conditions
Canary / bundle-desktop-windows (push) Blocked by required conditions
Canary / Prepare Version (push) Waiting to run
Canary / build-cli (push) Blocked by required conditions
Canary / Release (push) Blocked by required conditions
CI / changes (push) Waiting to run
CI / Check Rust Code Format (push) Blocked by required conditions
CI / Build and Test Rust Project (push) Blocked by required conditions
CI / Lint Rust Code (push) Blocked by required conditions
CI / Check OpenAPI Schema is Up-to-Date (push) Blocked by required conditions
CI / Test and Lint Electron Desktop App (push) Blocked by required conditions
Deploy Documentation / deploy (push) Waiting to run
Live Provider Tests / check-fork (push) Waiting to run
Live Provider Tests / changes (push) Blocked by required conditions
Live Provider Tests / Build Binary (push) Blocked by required conditions
Live Provider Tests / Smoke Tests (push) Blocked by required conditions
Live Provider Tests / Smoke Tests (Code Execution) (push) Blocked by required conditions
Live Provider Tests / Compaction Tests (push) Blocked by required conditions
Live Provider Tests / goose server HTTP integration tests (push) Blocked by required conditions
Publish Ask AI Bot Docker Image / docker (push) Waiting to run
Publish Docker Image / docker (push) Waiting to run
Scorecard supply-chain security / Scorecard analysis (push) Waiting to run
Signed-off-by: B-Step62 <yuki.watanabe@databricks.com> Co-authored-by: Rizel Scarlett <rizel@tbd.email>
This commit is contained in:
parent
bbb36bb3cd
commit
0f2f32b806
2 changed files with 77 additions and 0 deletions
77
documentation/docs/tutorials/mlflow.md
Normal file
77
documentation/docs/tutorials/mlflow.md
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
---
|
||||
description: Integrate goose with MLflow to observe and evaluate agent performance
|
||||
---
|
||||
|
||||
# Observability with MLflow
|
||||
|
||||
This tutorial covers how to integrate goose with MLflow to trace your goose sessions and understand how the agent is performing.
|
||||
|
||||
## What is MLflow
|
||||
|
||||
[MLflow](https://mlflow.org/) is an [open-source](https://github.com/mlflow/mlflow) platform for managing the end-to-end machine learning and AI lifecycle. MLflow Tracing provides detailed observability into AI agent execution, capturing LLM calls, tool usage, and agent decisions with a rich visualization UI.
|
||||
|
||||
## Why MLflow for goose
|
||||
|
||||
- **Detailed trace visualization**: Inspect every LLM call, tool execution, and agent decision in a hierarchical trace view.
|
||||
- **Token usage tracking**: Monitor input/output token counts and costs across sessions.
|
||||
- **Evaluation framework**: Evaluate agent outputs using built-in LLM judges and custom scorers.
|
||||
- **Prompt management**: Version and manage prompts used across your AI applications.
|
||||
- **Open source**: Fully open-source with no vendor lock-in, self-host anywhere.
|
||||
|
||||
## Set up MLflow
|
||||
|
||||
Install MLflow and start the tracking server:
|
||||
|
||||
```bash
|
||||
pip install mlflow
|
||||
mlflow server --port 5000
|
||||
```
|
||||
|
||||
The MLflow UI will be available at `http://localhost:5000`.
|
||||
|
||||
:::tip
|
||||
For production use, configure a SQL backend store (PostgreSQL, MySQL) instead of the default SQLite. See the [MLflow documentation](https://mlflow.org/docs/latest/self-hosting/architecture/backend-store.html) for details.
|
||||
:::
|
||||
|
||||
## Configure goose to export OTLP to MLflow
|
||||
|
||||
goose exports OpenTelemetry data over OTLP/HTTP. Point the exporter to MLflow's OTLP endpoint:
|
||||
|
||||
```bash
|
||||
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:5000"
|
||||
export OTEL_EXPORTER_OTLP_HEADERS="x-mlflow-experiment-id=0"
|
||||
```
|
||||
|
||||
The `x-mlflow-experiment-id` header specifies which MLflow experiment to log traces to. Use `0` for the default experiment, or create a dedicated experiment:
|
||||
|
||||
```bash
|
||||
pip install mlflow
|
||||
mlflow experiments create --experiment-name "goose-traces"
|
||||
# Use the returned experiment ID in the header
|
||||
```
|
||||
|
||||
To export only traces (disable metrics and logs export):
|
||||
|
||||
```bash
|
||||
export OTEL_TRACES_EXPORTER=otlp
|
||||
export OTEL_METRICS_EXPORTER=none
|
||||
export OTEL_LOGS_EXPORTER=none
|
||||
```
|
||||
|
||||
## Run goose with MLflow enabled
|
||||
|
||||
Start goose normally. With the OTLP environment variables set, goose will automatically export traces to MLflow:
|
||||
|
||||
```bash
|
||||
goose session
|
||||
```
|
||||
|
||||
Open the MLflow UI at `http://localhost:5000` and navigate to the **Traces** tab to see detailed traces of your goose session, including LLM calls, tool executions, and token usage.
|
||||
|
||||

|
||||
|
||||
## Learn more
|
||||
|
||||
- [MLflow Tracing documentation](https://mlflow.org/docs/latest/genai/tracing/)
|
||||
- [MLflow OpenTelemetry integration](https://mlflow.org/docs/latest/genai/tracing/app-instrumentation/opentelemetry.html)
|
||||
- [MLflow goose integration guide](https://mlflow.org/docs/latest/genai/tracing/integrations/listing/goose.html)
|
||||
Loading…
Add table
Add a link
Reference in a new issue