Skip to main content
The bb_ai_sdk.observability module instruments your agent application with OpenTelemetry. When you call configure_observability() at startup, the SDK registers a tracer, attaches framework and HTTP instrumentors, and exports spans to Langfuse, Grafana, or an OTLP endpoint you configure. When you call bb_ai_sdk.logging.init(), the SDK adds trace and span IDs to log lines and applies the same redaction rules to logs and span attributes. On each run, instrumentors record model calls, tool invocations, and request boundaries. You can stamp runs with scope() and bb.* attributes for filtering. The same setup works for local development, evaluation runs, and deployed services—you change export settings with environment variables, not application code.
The SDK builds on OpenTelemetry. You can switch export backends without rewriting agent code.

What the SDK provides

The observability module delivers three capabilities:

Instrumentation

The SDK hooks into HTTP (FastAPI), agent frameworks (Agno, LangChain, LangGraph), LLM clients, outbound HTTP, and threaded work—so spans cover the full run without hand-written instrumentation for each call.

Context

The SDK defines bb.* attributes (bb.project, bb.trace.name, bb.agent.name, and others) and scope() so you can filter traces by deployable, team, environment, and tenant.

Export and correlation

The SDK batches and exports spans via OTLP. With logging.init(), log lines carry trace_id and span_id, and one redaction engine covers logs and span attributes.
Use exported traces to monitor latency and token usage, review runs for audit, debug failures, and feed evaluation experiments—not only for incident response.

What you call at startup

The SDK exposes two init() functions in different modules. Only one observability entry point belongs in your application code.
ModuleFunctionCall from your app?
bb_ai_sdk.logginginit()Yes — after logging.basicConfig, for redaction and log trace correlation
bb_ai_sdk.observabilityconfigure_observability()Yes — once at startup, for tracing, instrumentors, and (optionally) FastAPI middleware
bb_ai_sdk.observabilityinit()Only for advanced export settings — call init() first with backend, otlp_*, org fields, or batch tuning, then configure_observability(..., init_observability=False). Do not call init() after configure_observability() with default settings.
Do not call observability.init() after configure_observability() with default settings — that double-initializes the tracer. For Langfuse or custom OTLP, configure_observability() alone is enough (it calls init(service_name=..., environment=...) internally). For Grafana presets, batch tuning, or org fields on the Resource, call init(...) first, then configure_observability(..., init_observability=False) — see Advanced export settings.
Typical FastAPI + Agno startup (load .env before SDK imports when you use a local env file):
import logging
from dotenv import load_dotenv
from fastapi import FastAPI

load_dotenv()

from bb_ai_sdk.logging import DEFAULT_DATEFMT, STANDARD_FORMAT, init as init_logging
from bb_ai_sdk.observability import configure_observability

app = FastAPI(title="my-agent-api")

logging.basicConfig(
    level=logging.INFO,
    format=STANDARD_FORMAT,
    datefmt=DEFAULT_DATEFMT,
)
init_logging(capture_warnings=True)

configure_observability(
    fastapi_app=app,
    framework="agno",          # match your [instrument-*] extras
    service_name="my-agent-api",
    environment="development",
)
Create AIGateway and agent instances after configure_observability(). Set export destination with environment variables—see Where traces go. The following diagram shows the observability pipeline from installation to trace export:
PhaseWhat you doWhat the SDK does
WireInstall [instrument-*] extras; set env vars; call logging.init() and configure_observability() once at startupRegisters TracerProvider, instrumentors, middleware, and redaction
RunServe traffic; optional scope(...) per request; use await team.arun(...) in multi-agent HTTP appsCreates a span tree; applies bb.* at span start
CaptureOptional @trace or trace_context for custom stepsRecords LLM, tool, and framework spans; tags guardrail blocks
ExportConfigure proxy/NO_PROXY when neededBatches spans and sends them to your configured backend
UseOpen Langfuse, Grafana, or your OTLP UI; run evals; filter by service.name and bb.*Same wiring in app and eval processes when service_name matches
Never commit API keys to version control. Add .env to your .gitignore.

Choose your path

Pick the setup that matches your application. Each path uses configure_observability() only—not observability.init().

FastAPI + Agno

HTTP agent with Agno teams or single agents. Install [instrument-fastapi,instrument-agno].

Script or eval runner

No FastAPI—local scripts, batch jobs, or bb-ai-sdk evals. Omit fastapi_app; set framework= to match your extras.

LangChain or LangGraph

Pass framework="langchain" or framework="langgraph" to configure_observability().

Gateway-only script

configure_observability(service_name=...) only—no framework= if you have no agent framework. See Get started.
Follow these steps to wire tracing for most agent applications.
1

Install instrumentation extras

Install bb-ai-sdk 0.1.9 with the extras that match your app. See Installation — observability extras.
framework= valueExtra
"agno"[instrument-agno]
"langchain"[instrument-langchain]
"langgraph"[instrument-langgraph]
FastAPI HTTP spans[instrument-fastapi]
Typical FastAPI + Agno agent:
uv add "bb-ai-sdk[instrument-fastapi,instrument-agno]==0.1.9" --index backbase
2

Configure export destination

The SDK reads export settings from environment variables. Choose Langfuse, Grafana, or a custom OTLP endpoint—one export path per deployment unless your operations team instructs you otherwise.See Where traces go for the full variable list and path-specific notes.
3

Initialize logging and observability

Use the startup sequence: logging.basicConfigbb_ai_sdk.logging.init()configure_observability(...).For logging format presets, redaction patterns, and coverage details, see Logging and redaction.
4

Run your agent

Create gateway and agent instances after configure_observability(). Instrumentors capture LLM and tool spans automatically.For multi-agent HTTP services, wrap each team run in scope(...) and use await team.arun(...)—not sync team.run(...). See Multi-agent HTTP services.
5

Verify traces

  1. Send one request or run one script invocation.
  2. Open Langfuse, Grafana, or your OTLP backend (traces may take up to 5 seconds to appear while the SDK batches export).
  3. Confirm you see a request boundary span, framework spans, and LLM spans in one trace.
Don’t have Langfuse credentials? See the Onboarding guide.

Scripts and evals

For eval runners or scripts without FastAPI, omit fastapi_app:
configure_observability(
    service_name="my-agent-api",
    environment="development",
    framework="agno",
)
Use the same service_name in evals/__init__.py as in your app so eval and production traces align. See Evaluate your agent.

LangChain and LangGraph

Install the matching extra, then pass framework="langchain" or framework="langgraph" to configure_observability()—same startup sequence as Agno. For per-chain callback control only, see Manual assembly — LangChain and LangGraph.

Where traces go

The SDK exports spans over OTLP. You choose the destination—Langfuse, Grafana, or a custom OTLP endpoint—with environment variables. Application code stays the same.

Export paths

PathWhen to useRequired env varsCommon optional vars
LangfuseLocal development, evals, or when the app holds Langfuse keysLANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEYLANGFUSE_HOST, OBSERVABILITY_ENABLED
GrafanaTeams using Grafana Cloud for tracesGRAFANA_BEARER_TOKEN, OTEL_EXPORTER_OTLP_ENDPOINT (or pass otlp_endpoint= to init(backend="grafana", ...))OBSERVABILITY_ENABLED
Custom OTLPDeployed runtimes where operations owns trace exportOTEL_EXPORTER_OTLP_ENDPOINT (or OTLP_ENDPOINT)OTEL_EXPORTER_OTLP_HEADERS, OTEL_RESOURCE_ATTRIBUTES (often project=<your-project-identifier>), OTEL_SERVICE_NAME, OBSERVABILITY_ENABLED
On the custom OTLP path, your operations team may require OTEL_RESOURCE_ATTRIBUTES=project=<your-project-identifier> so traces route to the right project in the collector UI. The SDK merges this env var into the Resource—it does not require a specific project= value. Confirm the identifier with your operations team. Do not mix LANGFUSE_*, GRAFANA_*, and OTEL_EXPORTER_OTLP_* unless your operations team instructs you to.

Canonical environment block

.env
# Option A: Langfuse (typical for local dev or direct export)
LANGFUSE_PUBLIC_KEY=pk-lf-xxx
LANGFUSE_SECRET_KEY=sk-lf-xxx
# Optional: LANGFUSE_HOST=https://cloud.langfuse.com

# Option B: Grafana Cloud (omit LANGFUSE_*)
# GRAFANA_BEARER_TOKEN=your-grafana-token
# OTEL_EXPORTER_OTLP_ENDPOINT=https://your-grafana.com/otlp
# Or pass backend="grafana" and otlp_endpoint= to init() — see Grafana tab below

# Option C: Custom OTLP (typical on deployed runtimes — omit LANGFUSE_*)
# OTEL_EXPORTER_OTLP_ENDPOINT=https://your-otlp-endpoint
# OTLP_ENDPOINT=https://your-otlp-endpoint
# OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer your-token
# OTEL_RESOURCE_ATTRIBUTES=project=<your-project-identifier>
# OTEL_SERVICE_NAME=my-agent-api

# Tracing is on by default. Set OBSERVABILITY_ENABLED=false to disable.
# Only true or false are accepted (case-insensitive).
Without OTEL_EXPORTER_OTLP_ENDPOINT, the SDK targets Langfuse Cloud (honoring LANGFUSE_HOST when set). Set LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY so export is authenticated—not to choose the endpoint URL.In Langfuse you can inspect LLM and tool spans, token usage, latency, and session groupings. Langfuse features include cost tracking, trace hierarchies, and experiment integration via bb-ai-sdk evals.
from bb_ai_sdk.observability import configure_observability

configure_observability(service_name="my-agent", framework="agno")
If Langfuse keys are missing, init() still succeeds: the endpoint defaults to Langfuse Cloud, auth headers are unset, spans are created locally, and export is rejected by the backend. The agent does not crash—the SDK fails safe on observability errors.

Proxy configuration

On the corporate network, configure proxy settings so OTLP export reaches your backend:
.env
HTTP_PROXY=http://webproxy.infra.backbase.cloud:8888
HTTPS_PROXY=http://webproxy.infra.backbase.cloud:8888
NO_PROXY=localhost,127.0.0.1,cloud.langfuse.com,*.langfuse.com,langfuse
NO_PROXY must include your Langfuse, Grafana, or OTLP host. The corporate web proxy can block trace export if this is missing.

Customize trace context

After basic tracing works, add SDK context so you can filter and compare runs.

bb.* attribute keys

The SDK defines vendor-neutral names in bb_ai_sdk.observability.attributes. Use the constants—not string literals—so attributes stay consistent across backends:
from bb_ai_sdk.observability import scope
from bb_ai_sdk.observability.attributes import (
    BB_PROJECT,
    BB_TRACE_NAME,
    BB_TRACE_INPUT,
    BB_AGENT_NAME,
)
KeyWhere it appearsHow you set it
service.nameResource (OTel)configure_observability(..., service_name=...) — one per deployable
deployment.environment / bb.environmentResourceenvironment= on configure_observability
bb.agent.idResource and/or spanscope(agent_id=...) per request or team
bb.agent.nameResource and/or spanscope(agent_name=...) per run when multiple agents share one service
bb.organization.idResourceinit(..., organization_id=...) before configure_observability(..., init_observability=False), or scope() when needed
bb.organization.nameResourceinit(..., organization_name=...) before configure_observability(..., init_observability=False)
bb.projectEvery span in contextHeader X-Observability-Project or scope(project=...)
bb.trace.nameEvery span in contextscope(trace_name=...)
bb.trace.inputEvery span in contextscope(trace_input=...) — optional input preview
guardrails.blockedSpanSet by the SDK when NeMo Guardrails blocks a call
The SDK installs ScopeAttributeSpanProcessor when you call configure_observability(). It reads scope() context and applies bb.project, bb.trace.*, and optional bb.agent.* on span start.
Do not set vendor-specific names (for example langfuse.trace.name) in application code. Use scope(trace_name=..., trace_input=...); your backend maps bb.* keys as needed.
Optional routing header: send X-Observability-Project on API requests. The SDK middleware maps it to bb.project on nested spans.

Multi-agent HTTP services

For one deployable that hosts multiple Agno teams, see also the Multi agent starter.
  • Call configure_observability(fastapi_app=app, framework="agno", service_name="...", environment=...) once at import time.
  • Install [instrument-fastapi,instrument-agno] so the SDK wires FastAPI, Agno, outbound HTTPX, and threading instrumentation.
  • Use await team.arun(...), not sync team.run(...). Sync run can create a second root trace disconnected from the FastAPI span.
  • Use one service_name per HTTP service; stamp each run with scope(agent_name=..., trace_name=..., trace_input=...).
with scope(
    agent_name=team_name,
    trace_name=team_name,
    trace_input=query,
):
    result = await team.arun(query)
Symptom: two trace IDs for one API call, or tool spans with no parent. Check: arun vs run, both instrument extras installed, and NO_PROXY includes your export host.

Custom spans

Instrumentors capture LLM and framework steps. Add your own spans when you need visibility into business logic.

The @trace decorator

from bb_ai_sdk.observability import trace

@trace()
def process_user_request(user_input: str) -> str:
    return "processed result"

@trace(name="validate-user-input")
def validate_input(data: dict) -> bool:
    return True

Custom attributes

@trace(attributes={
    "prompt.version": "v1.2.3",
    "prompt.name": "customer-support-prompt",
    "user.id": "user-123",
})
def run_experiment():
    pass

The trace_context context manager

from bb_ai_sdk.observability import trace_context

def complex_operation():
    with trace_context("multi-step-operation") as span:
        span.add_event("Step 1: Validating input")
        validate_input()
        span.set_attribute("result.count", len(result))
        return result

Context utilities

from bb_ai_sdk.observability import (
    get_current_trace_id,
    get_current_span,
    get_tracer_provider,
)

trace_id = get_current_trace_id()
span = get_current_span()
provider = get_tracer_provider()
Use get_current_trace_id() with standard loggers after bb_ai_sdk.logging.init()—log format presets already include trace_id and span_id.

Manual assembly (without configure_observability)

Use this path when you assemble OpenAIInstrumentor, LangChain callback handlers, or instrument() yourself instead of configure_observability(). For Grafana presets or batch tuning while still using configure_observability() for instrumentors, see Advanced export settings instead.
configure_observability() already calls init(), registers framework instrumentors, and adds FastAPI middleware. Prefer it for all new agents.

Gateway-only script

If you use configure_observability(service_name=...) without a framework=, the SDK wires HTTPX and threading instrumentors and exports spans when env vars are set—the same pattern as Get started. The sync AIGateway client does not auto-attach OpenAIInstrumentor in the SDK today; use AsyncAIGateway, or attach OpenAIInstrumentor manually as below. Manual assembly without configure_observability():
from dotenv import load_dotenv

load_dotenv()

from bb_ai_sdk.observability import init, get_tracer_provider
from openinference.instrumentation.openai import OpenAIInstrumentor
from bb_ai_sdk.ai_gateway import AIGateway

init(service_name="my-agent")
OpenAIInstrumentor().instrument(tracer_provider=get_tracer_provider())

gateway = AIGateway.create(
    model_id="gpt-4o",
    agent_id="550e8400-e29b-41d4-a716-446655440000",
)
OpenAIInstrumentor monkey-patches the OpenAI client so LLM calls through AIGateway appear as spans.

LangChain and LangGraph (manual callbacks)

When you use callback handlers instead of configure_observability(framework=...), call init() once, then attach handlers per invocation:
from bb_ai_sdk.observability import init, LangChainOpenTelemetryCallbackHandler
from bb_ai_sdk.ai_gateway import AIGateway
from bb_ai_sdk.ai_gateway.adapters.langchain import to_langchain
from langchain.prompts import ChatPromptTemplate
from langchain.schema.output_parser import StrOutputParser

init(service_name="langchain-agent")
callback = LangChainOpenTelemetryCallbackHandler()

gateway = AIGateway.create(model_id="gpt-4o", agent_id="...")
model = to_langchain(gateway)
chain = ChatPromptTemplate.from_template("Tell me about {topic}") | model | StrOutputParser()

result = chain.invoke(
    {"topic": "AI observability"},
    config={"callbacks": [callback]},
)

Advanced export settings

configure_observability() forwards only service_name and environment to init(). Export backend, OTLP tuning, organization fields, and custom Resource attributes are set on init() — then call configure_observability(..., init_observability=False) so instrumentors and FastAPI middleware still wire up without a second init().

Batch export

The SDK batches spans before export. By default, traces may take up to 5 seconds to appear. Pass tuning parameters to init():
from bb_ai_sdk.observability import configure_observability, init

init(
    service_name="my-agent",
    environment="development",
    otlp_batch_size=512,
    otlp_batch_timeout=5.0,
    otlp_max_queue_size=10000,
)
configure_observability(
    service_name="my-agent",
    framework="agno",
    init_observability=False,
)
For faster feedback during development, set otlp_batch_timeout=1.0. Avoid low values in production—they increase network overhead.

Custom resource attributes and tenancy

init(
    service_name="my-agent",
    environment="production",
    organization_id="org-123",
    organization_name="Acme Corp",
    resource_attributes={
        "service.version": "1.2.3",
        "deployment.region": "us-east-1",
    },
)
configure_observability(
    service_name="my-agent",
    framework="agno",
    init_observability=False,
)
You can also merge Resource attributes with OTEL_RESOURCE_ATTRIBUTES in .env.

Disable tracing

Set OBSERVABILITY_ENABLED=false, or pass enabled=False to init(). Load environment variables before SDK imports when you use a .env file.

Best practices

Initialize once at startup

Call configure_observability() once when the process starts—before routes or agents handle traffic. Call bb_ai_sdk.logging.init() once after logging.basicConfig.
With default settings, do not call observability.init() after configure_observability() — the SDK already invoked init() internally. For advanced export settings, call init() first, then configure_observability(..., init_observability=False).

Set environment and tenancy early

Use environment= on configure_observability (or on init() when you use the advanced path) to separate dev, staging, and production traces. Pass organization_id and organization_name on init() for multi-tenant cost attribution on the Resource.

Use meaningful span names

Prefer @trace(name="validate-user-input") over generic names like process or step1.

Keep credentials in environment variables

Do not hardcode Langfuse keys, Grafana tokens, or OTLP credentials in source code.

Debug trace export

To troubleshoot export issues, enable OpenTelemetry debug logging after bb_ai_sdk.logging.init() so output stays redacted:
import logging

logging.getLogger("opentelemetry").setLevel(logging.DEBUG)
logging.getLogger("opentelemetry.sdk").setLevel(logging.DEBUG)
logging.getLogger("opentelemetry.exporter").setLevel(logging.DEBUG)
Disable verbose OTel logging in production. For log format, redaction coverage, and operator patterns, see Logging and redaction.

Troubleshooting

Cause: Missing or invalid export credentials, or wrong export path (Langfuse, Grafana, or OTLP).Solution: Verify environment variables for your chosen path. See Export paths. Confirm traces in the Langfuse, Grafana, or OTLP UI.
Cause: LANGFUSE_PUBLIC_KEY or LANGFUSE_SECRET_KEY is missing or invalid. The SDK still initializes: the OTLP endpoint defaults to Langfuse Cloud, auth headers are unset, spans are buffered locally, and export is rejected. The agent keeps running.Solution: Set both LANGFUSE_* keys in .env (load before SDK imports). Enable OpenTelemetry debug logging—see Debug trace export—to confirm export errors.
Cause: Missing instrumentation extras, configure_observability() called after gateway or agent creation, or sync AIGateway without OpenAIInstrumentor.Solution: Install the matching [instrument-*] extra. Call configure_observability() before creating AIGateway or agent instances. For gateway-only sync scripts, use AsyncAIGateway (auto-instruments when the tracer is up) or attach OpenAIInstrumentor as in Gateway-only script.
Cause: Sync team.run() in a FastAPI handler, or missing FastAPI/Agno extras.Solution: Use await team.arun(...). Install [instrument-fastapi,instrument-agno]. See Multi-agent HTTP services.
Cause: Corporate web proxy intercepting OTLP export.Solution: Set NO_PROXY to include your Langfuse, Grafana, or OTLP host. See Proxy configuration.
Cause: Span queue growing when export fails.Solution: Fix network connectivity to the export endpoint. Lower otlp_max_queue_size if needed.
Cause: Environment variables loaded after SDK import.Solution:
from dotenv import load_dotenv

load_dotenv()
from bb_ai_sdk.observability import configure_observability

API reference

Configure_observability()

configure_observability
function
Application entry point for instrumentation. When the tracer is not initialized yet and init_observability=True (default), the SDK calls init(service_name=..., environment=...), then instrument() for HTTPX, threading, framework OTel, and (when fastapi_app is set) FastAPI route spans and X-Observability-Project middleware.Export backend, OTLP batch tuning, organization fields, and custom Resource attributes are not parameters on this function — set them on init() and pass init_observability=False. See Advanced export settings.

Scope()

scope
context manager
Sets bb.* context for the current async or sync block. The SDK applies values on span start via ScopeAttributeSpanProcessor.

Init() (observability)

init
function
Low-level TracerProvider and OTLP export setup. configure_observability() calls this with service_name and environment only when init_observability=True and the tracer is not yet initialized.Call init() yourself when you need parameters configure_observability() does not expose — then pass init_observability=False to configure_observability(). For full manual control without configure_observability(), call init() and instrument() — see Manual assembly.

Trace()

trace
decorator
Decorator that creates OpenTelemetry spans for functions.

Trace_context()

trace_context
context manager
Context manager for manual span control.

Get_tracer_provider()

get_tracer_provider
function
Returns the TracerProvider the SDK created—use with manual instrumentors such as OpenAIInstrumentor.

Callback handlers

LangChainOpenTelemetryCallbackHandler
class
Callback handler for LangChain operations when not using configure_observability(framework="langchain").
LangGraphOpenTelemetryCallbackHandler
class
Callback handler for LangGraph operations. Optional graph_name constructor argument.

Next steps

Logging and redaction

Log formats, redaction patterns, and correlation with traces

Evaluate your agent

Run evals with the same service_name and export setup

Multi agent starter

Full FastAPI + Agno + observability reference implementation

Get started

Minimal gateway + tracing example