> ## Documentation Index
> Fetch the complete documentation index at: https://grandcentral.backbase.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluate your agent

> Langfuse-based evaluation framework for testing and measuring agent performance

Building AI agents without evaluations is like shipping software without tests. You don't know whether they work correctly until users complain. An **evaluation framework** provides a structured way to test your agents against datasets and measure their performance using custom evaluators.

The SDK includes a Langfuse-based evaluation framework that lets you:

* **Test agent behavior**: Run agents against predefined test cases
* **Measure performance**: Use custom evaluators to score responses
* **Track experiments**: All results are logged to Langfuse for analysis
* **Automate CI/CD**: Integrate evaluations into your deployment pipeline

<Info>
  Built on Langfuse experiments. Your evaluation results are automatically visualized in your Langfuse dashboard for comparison and analysis.
</Info>

## Prerequisites

Before using the evaluation framework, ensure you have:

1. **Langfuse credentials** configured:

   ```bash .env theme={"system"}
   LANGFUSE_PUBLIC_KEY=pk-xxx
   LANGFUSE_SECRET_KEY=sk-xxx
   LANGFUSE_HOST=https://cloud.langfuse.com  # Optional, defaults to cloud
   ```

2. **BB AI SDK** installed in your project

## Quick start

Get evaluations running in 5 steps:

<Steps>
  <Step title="Initialize evals folder">
    Run the CLI command to scaffold the evals structure:

    ```bash theme={"system"}
    bb-ai-sdk evals init
    ```

    This creates an `evals/` folder with:

    | File                | Purpose                                      |
    | ------------------- | -------------------------------------------- |
    | `__init__.py`       | Initializes observability for your framework |
    | `agents.py`         | Registers your task functions                |
    | `evaluators.py`     | Defines custom evaluators                    |
    | `evals_config.yaml` | Experiment configuration                     |
    | `datasets/`         | CSV dataset files                            |

    Configure observability in `evals/__init__.py` (match your FastAPI app’s `service_name` / `environment`; pin **`bb-ai-sdk==0.1.9`**). The CLI scaffold may name this helper `init_observability()`:

    ```python theme={"system"}
    import os
    from bb_ai_sdk.observability import configure_observability


    def init_observability() -> None:
        # No fastapi_app in the eval runner process — same service_name as app.py
        configure_observability(
            service_name="my-agent-api",
            environment=os.getenv("APP_ENV", "development"),
            framework="agno",  # or "langchain" / "langgraph"
        )


    init_observability()
    ```

    Install the matching `[instrument-agno]` (or langchain/langgraph) extra. LangChain/LangGraph can also use callback handlers — see [Observability](/agentic-ai/bb-ai-sdk/observability).

    <Tip>
      For more information about configuring observability, see the [Observability documentation](/agentic-ai/bb-ai-sdk/observability).
    </Tip>
  </Step>

  <Step title="Register your agent task">
    Edit `evals/agents.py` to register your agent as a task function:

    ```python agents.py theme={"system"}
    from evals import register_task
    from src.agents.my_agent import create_agent

    # Create your agent instance
    agent = create_agent()

    @register_task("my_agent")
    def my_agent_task(*, item, **kwargs):
        """
        Task function for Langfuse experiments.
        
        Args:
            item: Langfuse dataset item with .input attribute
            **kwargs: Additional keyword arguments
            
        Returns:
            String result from agent execution
        """
        result = agent.run(item.input)
        return result.content
    ```

    <Note>
      The task name in `@register_task("my_agent")` must match the `name` field in your config file.
    </Note>
  </Step>

  <Step title="Create a custom evaluator">
    Edit `evals/evaluators.py` to define how responses are scored:

    ```python evaluators.py theme={"system"}
    from langfuse.experiment import Evaluation
    from evals import register_evaluator

    @register_evaluator("accuracy_evaluator")
    def accuracy_evaluator(
        *,
        input: str,
        output: str | None,
        expected_output: str | None = None,
        metadata: dict | None = None,
        **kwargs,
    ) -> Evaluation:
        """Check if output matches expected output."""
        if output is None or expected_output is None:
            return Evaluation(name="accuracy", value=0.0, comment="Missing output or expected")
        
        is_match = output.strip().lower() == expected_output.strip().lower()
        return Evaluation(
            name="accuracy",
            value=1.0 if is_match else 0.0,
            comment="Match" if is_match else "No match",
        )
    ```

    <Note>
      The custom evaluator name in `@register_evaluator("accuracy_evaluator")` must match the name used in your config file.
    </Note>
  </Step>

  <Step title="Configure the experiment">
    Edit `evals/evals_config.yaml` to define your evaluation:

    ```yaml evals_config.yaml theme={"system"}
    agents:
      - name: "my_agent"              # Must match @register_task name
        skipEval: false               # Set to true to skip this task
        dataset:
          name: "my_dataset"          # CSV file at evals/datasets/my_dataset.csv
        evaluators:
          - "accuracy_evaluator"      # Must match @register_evaluator name
    ```
  </Step>

  <Step title="Create a dataset">
    Add a CSV file at `evals/datasets/my_dataset.csv`:

    ```csv my_dataset.csv theme={"system"}
    input,expected_output
    "What is 2+2?","4"
    "Hello, how are you?","I'm doing well, thank you!"
    "What is the capital of France?","Paris"
    ```

    <Check>
      Run evaluations with `bb-ai-sdk evals run` and view results in your Langfuse dashboard!
    </Check>
  </Step>
</Steps>

## Registering task functions

Task functions connect your agents to the evaluation framework. They define how to invoke your agent and return results.

### Task function signature

All task functions must:

* Accept keyword arguments including `item` (with `.input` attribute)
* Return a string result

  ```python theme={"system"}
  @register_task("task_name")
  def task_function(*, item, **kwargs) -> str:
      # item.input contains the test case input
      result = your_agent.invoke(item.input)
      return str(result)
  ```

## Creating custom evaluators

Evaluators score agent responses against expected outputs or custom criteria.

### Evaluator function signature

Evaluators must:

* Accept keyword arguments: `input`, `output`, `expected_output`, `metadata`
* Return a Langfuse `Evaluation` object with `name`, `value` (score), and optional `comment`

  ```python theme={"system"}
  from langfuse.experiment import Evaluation
  from evals import register_evaluator

  @register_evaluator("evaluator_name")
  def my_evaluator(
      *,
      input: str,
      output: str | None,
      expected_output: str | None = None,
      metadata: dict | None = None,
      **kwargs,
  ) -> Evaluation:
      # Calculate score (0.0 to 1.0)
      score = calculate_score(output, expected_output)
      
      return Evaluation(
          name="evaluator_name",
          value=score,
          comment="Optional explanation"
      )
  ```

## Configuration

The `evals_config.yaml` file defines which agents to evaluate, their datasets, and evaluators.

### Configuration structure

```yaml evals_config.yaml theme={"system"}
agents:
  - name: "agent_name"            # Required: matches @register_task name
    skipEval: false               # Optional: skip this agent (default: false)
    dataset:
      name: "dataset_name"        # Required: CSV filename (without .csv)
    evaluators:                   # Optional: list of evaluator names
      - "evaluator_1"
      - "evaluator_2"
```

## Dataset format

Datasets are CSV files stored in `evals/datasets/`.

### CSV structure

| Column            | Required | Description                                   |
| ----------------- | -------- | --------------------------------------------- |
| `input`           | Yes      | The input prompt/question                     |
| `expected_output` | No       | Expected response (for comparison evaluators) |
| `metadata`        | No       | Added to the `metadata` dictionary            |

### Example datasets

<Tabs>
  <Tab title="Basic Q&A">
    ```csv qa_dataset.csv theme={"system"}
    input,expected_output
    "What is 2+2?","4"
    "What color is the sky?","blue"
    "How many days in a week?","7"
    ```
  </Tab>

  <Tab title="With Metadata">
    ```csv support_dataset.csv theme={"system"}
    input,expected_output,metadata
    "Tell me about Langfuse","Langfuse is an open source LLM ops platform.","{\"lang\":\"en\"}"
    "How do I reset my password?","Navigate to Settings > Security > Reset Password","{\"category\":\"account\",\"priority\":\"high\"}"
    "What are your business hours?","We're open Monday-Friday, 9am-5pm","{\"category\":\"general\",\"priority\":\"low\"}"
    ```
  </Tab>

  <Tab title="Input Only">
    ```csv generation_dataset.csv theme={"system"}
    input
    "Write a haiku about programming"
    "Explain quantum computing in simple terms"
    "Generate a product description for wireless headphones"
    ```
  </Tab>
</Tabs>

## Running evaluations

### Using the CLI

```bash theme={"system"}
# Run with default config (evals/evals_config.yaml)
bb-ai-sdk evals run

# Run with custom config path
bb-ai-sdk evals run --config path/to/config.yaml
```

### Using Python

```python theme={"system"}
# Run with default config
python -m evals

# Run with custom config
python -m evals --config path/to/config.yaml
```

## How it works

The following diagram shows the evaluation framework flow:

```mermaid theme={"system"}
%%{init: {
  'theme': 'base',
  'themeVariables': {
    'primaryColor': '#ffffff',
    'primaryBorderColor': '#295eff',
    'primaryTextColor': '#091c35',
    'lineColor': '#091c35',
    'secondaryColor': '#f3f6f9',
    'tertiaryColor': '#ebf0f5',
    'fontFamily': 'Libre Franklin, sans-serif'
  }
}}%%
flowchart TD
    Start(( )) --> A[Load config]
    A --> B[Auto-discover tasks<br/>and evaluators]
    B --> C{For each agent}
    C --> D[Check dataset in Langfuse]
    D --> E{Dataset exists?}
    E -->|No| F[Upload from CSV]
    E -->|Yes| G[Use existing]
    F --> G
    G --> H[Run experiment]
    H --> I[Execute task function]
    I --> J[Run evaluators]
    J --> K[Log results to Langfuse]
    K --> C
    C -->|Done| Stop(( ))

    style Start fill:#295eff,stroke:#295eff,color:#295eff
    style Stop fill:#091c35,stroke:#091c35,color:#091c35
```

<Steps>
  <Step title="Auto-discovery">
    The framework automatically imports `evals.agents` and `evals.evaluators` modules to discover registered functions.
  </Step>

  <Step title="Dataset management">
    For each agent, the framework checks if the dataset exists in Langfuse. If not, it uploads the CSV file automatically.
  </Step>

  <Step title="Experiment execution">
    The framework calls the task function for each dataset item and captures the results as Langfuse traces.
  </Step>

  <Step title="Evaluation">
    Each evaluator runs on the task output, and the framework logs scores to Langfuse.
  </Step>
</Steps>

## Troubleshooting

<AccordionGroup>
  <Accordion title="Task not found error">
    **Error**: `ValueError: Task 'my_agent' is not registered`

    **Cause**: Task name in config doesn't match the `@register_task` decorator.

    **Solution**: Ensure names match exactly:

    ```yaml theme={"system"}
    # evals_config.yaml
    agents:
      - name: "my_agent"  # Must match decorator
    ```

    ```python theme={"system"}
    # agents.py
    @register_task("my_agent")  # Must match config
    def my_task(*, item, **kwargs):
        ...
    ```
  </Accordion>

  <Accordion title="Dataset CSV not found">
    **Error**: `FileNotFoundError: CSV file not found at default path`

    **Cause**: The CSV file doesn't exist at the expected location.

    **Solution**: Ensure the CSV file exists at `evals/datasets/{dataset_name}.csv`:

    ```bash theme={"system"}
    evals/
    └── datasets/
        └── my_dataset.csv  # Must match config dataset.name
    ```
  </Accordion>

  <Accordion title="Evaluator not found">
    **Error**: `ValueError: Evaluator 'my_evaluator' is not registered`

    **Cause**: Evaluator name in config doesn't match the `@register_evaluator` decorator.

    **Solution**: Ensure names match exactly in `evaluators.py` and the configuration file.
  </Accordion>

  <Accordion title="Langfuse credentials error">
    **Error**: `ValueError: Langfuse credentials not configured`

    **Solution**: Set environment variables:

    ```bash theme={"system"}
    export LANGFUSE_PUBLIC_KEY=pk-xxx
    export LANGFUSE_SECRET_KEY=sk-xxx
    ```

    Or in `.env` file:

    ```bash .env theme={"system"}
    LANGFUSE_PUBLIC_KEY=pk-xxx
    LANGFUSE_SECRET_KEY=sk-xxx
    ```
  </Accordion>

  <Accordion title="Empty results in Langfuse">
    **Cause**: Task function returning `None` or empty string.

    **Solution**: Ensure your task function returns a valid string:

    ```python theme={"system"}
    @register_task("my_agent")
    def my_task(*, item, **kwargs):
        result = agent.run(item.input)
        # Ensure we return a string
        return str(result.content) if result.content else "No response"
    ```
  </Accordion>
</AccordionGroup>

## Next steps

<CardGroup cols={2}>
  <Card title="Observability" icon="chart-line" href="/agentic-ai/bb-ai-sdk/observability">
    Learn about tracing and monitoring your agents
  </Card>

  <Card title="AI Gateway" icon="plug" href="/agentic-ai/bb-ai-sdk/ai-gateway">
    Connect to AI models through the gateway
  </Card>

  <Card title="CI/CD workflows" icon="rotate" href="/agentic-ai/ci-cd-workflows/overview">
    Integrate evals into your deployment pipeline
  </Card>

  <Card title="Starter kits" icon="rocket" href="/agentic-ai/starter-kits/overview">
    See evals integrated in production-ready templates
  </Card>
</CardGroup>