Add MLflow integration guide (#7563)
Some checks are pending
Canary / Upload Install Script (push) Blocked by required conditions
Canary / bundle-desktop (push) Blocked by required conditions
Canary / bundle-desktop-linux (push) Blocked by required conditions
Canary / bundle-desktop-windows (push) Blocked by required conditions
Canary / Prepare Version (push) Waiting to run
Canary / build-cli (push) Blocked by required conditions
Canary / Release (push) Blocked by required conditions
CI / changes (push) Waiting to run
CI / Check Rust Code Format (push) Blocked by required conditions
CI / Build and Test Rust Project (push) Blocked by required conditions
CI / Lint Rust Code (push) Blocked by required conditions
CI / Check OpenAPI Schema is Up-to-Date (push) Blocked by required conditions
CI / Test and Lint Electron Desktop App (push) Blocked by required conditions
Deploy Documentation / deploy (push) Waiting to run
Live Provider Tests / check-fork (push) Waiting to run
Live Provider Tests / changes (push) Blocked by required conditions
Live Provider Tests / Build Binary (push) Blocked by required conditions
Live Provider Tests / Smoke Tests (push) Blocked by required conditions
Live Provider Tests / Smoke Tests (Code Execution) (push) Blocked by required conditions
Live Provider Tests / Compaction Tests (push) Blocked by required conditions
Live Provider Tests / goose server HTTP integration tests (push) Blocked by required conditions
Publish Ask AI Bot Docker Image / docker (push) Waiting to run
Publish Docker Image / docker (push) Waiting to run
Scorecard supply-chain security / Scorecard analysis (push) Waiting to run

Signed-off-by: B-Step62 <yuki.watanabe@databricks.com>
Co-authored-by: Rizel Scarlett <rizel@tbd.email>
This commit is contained in:
Yuki Watanabe 2026-03-07 19:06:07 +09:00 committed by GitHub
parent bbb36bb3cd
commit 0f2f32b806
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 77 additions and 0 deletions

Binary file not shown.

After

Width:  |  Height:  |  Size: 428 KiB

View file

@ -0,0 +1,77 @@
---
description: Integrate goose with MLflow to observe and evaluate agent performance
---
# Observability with MLflow
This tutorial covers how to integrate goose with MLflow to trace your goose sessions and understand how the agent is performing.
## What is MLflow
[MLflow](https://mlflow.org/) is an [open-source](https://github.com/mlflow/mlflow) platform for managing the end-to-end machine learning and AI lifecycle. MLflow Tracing provides detailed observability into AI agent execution, capturing LLM calls, tool usage, and agent decisions with a rich visualization UI.
## Why MLflow for goose
- **Detailed trace visualization**: Inspect every LLM call, tool execution, and agent decision in a hierarchical trace view.
- **Token usage tracking**: Monitor input/output token counts and costs across sessions.
- **Evaluation framework**: Evaluate agent outputs using built-in LLM judges and custom scorers.
- **Prompt management**: Version and manage prompts used across your AI applications.
- **Open source**: Fully open-source with no vendor lock-in, self-host anywhere.
## Set up MLflow
Install MLflow and start the tracking server:
```bash
pip install mlflow
mlflow server --port 5000
```
The MLflow UI will be available at `http://localhost:5000`.
:::tip
For production use, configure a SQL backend store (PostgreSQL, MySQL) instead of the default SQLite. See the [MLflow documentation](https://mlflow.org/docs/latest/self-hosting/architecture/backend-store.html) for details.
:::
## Configure goose to export OTLP to MLflow
goose exports OpenTelemetry data over OTLP/HTTP. Point the exporter to MLflow's OTLP endpoint:
```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:5000"
export OTEL_EXPORTER_OTLP_HEADERS="x-mlflow-experiment-id=0"
```
The `x-mlflow-experiment-id` header specifies which MLflow experiment to log traces to. Use `0` for the default experiment, or create a dedicated experiment:
```bash
pip install mlflow
mlflow experiments create --experiment-name "goose-traces"
# Use the returned experiment ID in the header
```
To export only traces (disable metrics and logs export):
```bash
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=none
export OTEL_LOGS_EXPORTER=none
```
## Run goose with MLflow enabled
Start goose normally. With the OTLP environment variables set, goose will automatically export traces to MLflow:
```bash
goose session
```
Open the MLflow UI at `http://localhost:5000` and navigate to the **Traces** tab to see detailed traces of your goose session, including LLM calls, tool executions, and token usage.
![goose trace in MLflow](../assets/guides/mlflow-goose-tracing.png)
## Learn more
- [MLflow Tracing documentation](https://mlflow.org/docs/latest/genai/tracing/)
- [MLflow OpenTelemetry integration](https://mlflow.org/docs/latest/genai/tracing/app-instrumentation/opentelemetry.html)
- [MLflow goose integration guide](https://mlflow.org/docs/latest/genai/tracing/integrations/listing/goose.html)