bb_ai_sdk.observability module instruments your agent application with OpenTelemetry. When you call configure_observability() at startup, the SDK registers a tracer, attaches framework and HTTP instrumentors, and exports spans to Langfuse, Grafana, or an OTLP endpoint you configure. When you call bb_ai_sdk.logging.init(), the SDK adds trace and span IDs to log lines and applies the same redaction rules to logs and span attributes.
On each run, instrumentors record model calls, tool invocations, and request boundaries. You can stamp runs with scope() and bb.* attributes for filtering. The same setup works for local development, evaluation runs, and deployed services—you change export settings with environment variables, not application code.
The SDK builds on OpenTelemetry. You can switch export backends without rewriting agent code.
What the SDK provides
The observability module delivers three capabilities:Instrumentation
The SDK hooks into HTTP (FastAPI), agent frameworks (Agno, LangChain, LangGraph), LLM clients, outbound HTTP, and threaded work—so spans cover the full run without hand-written instrumentation for each call.
Context
The SDK defines
bb.* attributes (bb.project, bb.trace.name, bb.agent.name, and others) and scope() so you can filter traces by deployable, team, environment, and tenant.Export and correlation
The SDK batches and exports spans via OTLP. With
logging.init(), log lines carry trace_id and span_id, and one redaction engine covers logs and span attributes.What you call at startup
The SDK exposes twoinit() functions in different modules. Only one observability entry point belongs in your application code.
| Module | Function | Call from your app? |
|---|---|---|
bb_ai_sdk.logging | init() | Yes — after logging.basicConfig, for redaction and log trace correlation |
bb_ai_sdk.observability | configure_observability() | Yes — once at startup, for tracing, instrumentors, and (optionally) FastAPI middleware |
bb_ai_sdk.observability | init() | Only for advanced export settings — call init() first with backend, otlp_*, org fields, or batch tuning, then configure_observability(..., init_observability=False). Do not call init() after configure_observability() with default settings. |
.env before SDK imports when you use a local env file):
AIGateway and agent instances after configure_observability(). Set export destination with environment variables—see Where traces go.
The following diagram shows the observability pipeline from installation to trace export:
| Phase | What you do | What the SDK does |
|---|---|---|
| Wire | Install [instrument-*] extras; set env vars; call logging.init() and configure_observability() once at startup | Registers TracerProvider, instrumentors, middleware, and redaction |
| Run | Serve traffic; optional scope(...) per request; use await team.arun(...) in multi-agent HTTP apps | Creates a span tree; applies bb.* at span start |
| Capture | Optional @trace or trace_context for custom steps | Records LLM, tool, and framework spans; tags guardrail blocks |
| Export | Configure proxy/NO_PROXY when needed | Batches spans and sends them to your configured backend |
| Use | Open Langfuse, Grafana, or your OTLP UI; run evals; filter by service.name and bb.* | Same wiring in app and eval processes when service_name matches |
Choose your path
Pick the setup that matches your application. Each path usesconfigure_observability() only—not observability.init().
FastAPI + Agno
HTTP agent with Agno teams or single agents. Install
[instrument-fastapi,instrument-agno].Script or eval runner
No FastAPI—local scripts, batch jobs, or
bb-ai-sdk evals. Omit fastapi_app; set framework= to match your extras.LangChain or LangGraph
Pass
framework="langchain" or framework="langgraph" to configure_observability().Gateway-only script
configure_observability(service_name=...) only—no framework= if you have no agent framework. See Get started.Set up the SDK
Follow these steps to wire tracing for most agent applications.Install instrumentation extras
Install
Typical FastAPI + Agno agent:
bb-ai-sdk 0.1.9 with the extras that match your app. See Installation — observability extras.framework= value | Extra |
|---|---|
"agno" | [instrument-agno] |
"langchain" | [instrument-langchain] |
"langgraph" | [instrument-langgraph] |
| FastAPI HTTP spans | [instrument-fastapi] |
Configure export destination
The SDK reads export settings from environment variables. Choose Langfuse, Grafana, or a custom OTLP endpoint—one export path per deployment unless your operations team instructs you otherwise.See Where traces go for the full variable list and path-specific notes.
Initialize logging and observability
Use the startup sequence:
logging.basicConfig → bb_ai_sdk.logging.init() → configure_observability(...).For logging format presets, redaction patterns, and coverage details, see Logging and redaction.Run your agent
Create gateway and agent instances after
configure_observability(). Instrumentors capture LLM and tool spans automatically.For multi-agent HTTP services, wrap each team run in scope(...) and use await team.arun(...)—not sync team.run(...). See Multi-agent HTTP services.Scripts and evals
For eval runners or scripts without FastAPI, omitfastapi_app:
service_name in evals/__init__.py as in your app so eval and production traces align. See Evaluate your agent.
LangChain and LangGraph
Install the matching extra, then passframework="langchain" or framework="langgraph" to configure_observability()—same startup sequence as Agno.
For per-chain callback control only, see Manual assembly — LangChain and LangGraph.
Where traces go
The SDK exports spans over OTLP. You choose the destination—Langfuse, Grafana, or a custom OTLP endpoint—with environment variables. Application code stays the same.Export paths
| Path | When to use | Required env vars | Common optional vars |
|---|---|---|---|
| Langfuse | Local development, evals, or when the app holds Langfuse keys | LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY | LANGFUSE_HOST, OBSERVABILITY_ENABLED |
| Grafana | Teams using Grafana Cloud for traces | GRAFANA_BEARER_TOKEN, OTEL_EXPORTER_OTLP_ENDPOINT (or pass otlp_endpoint= to init(backend="grafana", ...)) | OBSERVABILITY_ENABLED |
| Custom OTLP | Deployed runtimes where operations owns trace export | OTEL_EXPORTER_OTLP_ENDPOINT (or OTLP_ENDPOINT) | OTEL_EXPORTER_OTLP_HEADERS, OTEL_RESOURCE_ATTRIBUTES (often project=<your-project-identifier>), OTEL_SERVICE_NAME, OBSERVABILITY_ENABLED |
OTEL_RESOURCE_ATTRIBUTES=project=<your-project-identifier> so traces route to the right project in the collector UI. The SDK merges this env var into the Resource—it does not require a specific project= value. Confirm the identifier with your operations team.
Do not mix LANGFUSE_*, GRAFANA_*, and OTEL_EXPORTER_OTLP_* unless your operations team instructs you to.
Canonical environment block
.env
- Langfuse
- Grafana
- Custom OTLP
Without
OTEL_EXPORTER_OTLP_ENDPOINT, the SDK targets Langfuse Cloud (honoring LANGFUSE_HOST when set). Set LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY so export is authenticated—not to choose the endpoint URL.In Langfuse you can inspect LLM and tool spans, token usage, latency, and session groupings. Langfuse features include cost tracking, trace hierarchies, and experiment integration via bb-ai-sdk evals.If Langfuse keys are missing,
init() still succeeds: the endpoint defaults to Langfuse Cloud, auth headers are unset, spans are created locally, and export is rejected by the backend. The agent does not crash—the SDK fails safe on observability errors.Proxy configuration
On the corporate network, configure proxy settings so OTLP export reaches your backend:.env
Customize trace context
After basic tracing works, add SDK context so you can filter and compare runs.bb.* attribute keys
The SDK defines vendor-neutral names in bb_ai_sdk.observability.attributes. Use the constants—not string literals—so attributes stay consistent across backends:
| Key | Where it appears | How you set it |
|---|---|---|
service.name | Resource (OTel) | configure_observability(..., service_name=...) — one per deployable |
deployment.environment / bb.environment | Resource | environment= on configure_observability |
bb.agent.id | Resource and/or span | scope(agent_id=...) per request or team |
bb.agent.name | Resource and/or span | scope(agent_name=...) per run when multiple agents share one service |
bb.organization.id | Resource | init(..., organization_id=...) before configure_observability(..., init_observability=False), or scope() when needed |
bb.organization.name | Resource | init(..., organization_name=...) before configure_observability(..., init_observability=False) |
bb.project | Every span in context | Header X-Observability-Project or scope(project=...) |
bb.trace.name | Every span in context | scope(trace_name=...) |
bb.trace.input | Every span in context | scope(trace_input=...) — optional input preview |
guardrails.blocked | Span | Set by the SDK when NeMo Guardrails blocks a call |
ScopeAttributeSpanProcessor when you call configure_observability(). It reads scope() context and applies bb.project, bb.trace.*, and optional bb.agent.* on span start.
Optional routing header: send X-Observability-Project on API requests. The SDK middleware maps it to bb.project on nested spans.
Multi-agent HTTP services
For one deployable that hosts multiple Agno teams, see also the Multi agent starter.- Call
configure_observability(fastapi_app=app, framework="agno", service_name="...", environment=...)once at import time. - Install
[instrument-fastapi,instrument-agno]so the SDK wires FastAPI, Agno, outbound HTTPX, and threading instrumentation. - Use
await team.arun(...), not syncteam.run(...). Syncruncan create a second root trace disconnected from the FastAPI span. - Use one
service_nameper HTTP service; stamp each run withscope(agent_name=..., trace_name=..., trace_input=...).
Custom spans
Instrumentors capture LLM and framework steps. Add your own spans when you need visibility into business logic.The @trace decorator
Custom attributes
The trace_context context manager
Context utilities
get_current_trace_id() with standard loggers after bb_ai_sdk.logging.init()—log format presets already include trace_id and span_id.
Manual assembly (without configure_observability)
Use this path when you assembleOpenAIInstrumentor, LangChain callback handlers, or instrument() yourself instead of configure_observability(). For Grafana presets or batch tuning while still using configure_observability() for instrumentors, see Advanced export settings instead.
configure_observability() already calls init(), registers framework instrumentors, and adds FastAPI middleware. Prefer it for all new agents.Gateway-only script
If you useconfigure_observability(service_name=...) without a framework=, the SDK wires HTTPX and threading instrumentors and exports spans when env vars are set—the same pattern as Get started. The sync AIGateway client does not auto-attach OpenAIInstrumentor in the SDK today; use AsyncAIGateway, or attach OpenAIInstrumentor manually as below. Manual assembly without configure_observability():
OpenAIInstrumentor monkey-patches the OpenAI client so LLM calls through AIGateway appear as spans.LangChain and LangGraph (manual callbacks)
When you use callback handlers instead ofconfigure_observability(framework=...), call init() once, then attach handlers per invocation:
- LangChain
- LangGraph
Advanced export settings
configure_observability() forwards only service_name and environment to init(). Export backend, OTLP tuning, organization fields, and custom Resource attributes are set on init() — then call configure_observability(..., init_observability=False) so instrumentors and FastAPI middleware still wire up without a second init().
Batch export
The SDK batches spans before export. By default, traces may take up to 5 seconds to appear. Pass tuning parameters toinit():
Custom resource attributes and tenancy
OTEL_RESOURCE_ATTRIBUTES in .env.
Disable tracing
SetOBSERVABILITY_ENABLED=false, or pass enabled=False to init(). Load environment variables before SDK imports when you use a .env file.
Best practices
Initialize once at startup
Callconfigure_observability() once when the process starts—before routes or agents handle traffic. Call bb_ai_sdk.logging.init() once after logging.basicConfig.
Set environment and tenancy early
Useenvironment= on configure_observability (or on init() when you use the advanced path) to separate dev, staging, and production traces. Pass organization_id and organization_name on init() for multi-tenant cost attribution on the Resource.
Use meaningful span names
Prefer@trace(name="validate-user-input") over generic names like process or step1.
Keep credentials in environment variables
Do not hardcode Langfuse keys, Grafana tokens, or OTLP credentials in source code.Debug trace export
To troubleshoot export issues, enable OpenTelemetry debug logging afterbb_ai_sdk.logging.init() so output stays redacted:
Troubleshooting
Traces not appearing in the backend
Traces not appearing in the backend
Cause: Missing or invalid export credentials, or wrong export path (Langfuse, Grafana, or OTLP).Solution: Verify environment variables for your chosen path. See Export paths. Confirm traces in the Langfuse, Grafana, or OTLP UI.
Init succeeds but Langfuse shows no traces
Init succeeds but Langfuse shows no traces
Cause:
LANGFUSE_PUBLIC_KEY or LANGFUSE_SECRET_KEY is missing or invalid. The SDK still initializes: the OTLP endpoint defaults to Langfuse Cloud, auth headers are unset, spans are buffered locally, and export is rejected. The agent keeps running.Solution: Set both LANGFUSE_* keys in .env (load before SDK imports). Enable OpenTelemetry debug logging—see Debug trace export—to confirm export errors.No LLM or tool spans in the trace
No LLM or tool spans in the trace
Cause: Missing instrumentation extras,
configure_observability() called after gateway or agent creation, or sync AIGateway without OpenAIInstrumentor.Solution: Install the matching [instrument-*] extra. Call configure_observability() before creating AIGateway or agent instances. For gateway-only sync scripts, use AsyncAIGateway (auto-instruments when the tracer is up) or attach OpenAIInstrumentor as in Gateway-only script.Two trace IDs for one HTTP request
Two trace IDs for one HTTP request
Cause: Sync
team.run() in a FastAPI handler, or missing FastAPI/Agno extras.Solution: Use await team.arun(...). Install [instrument-fastapi,instrument-agno]. See Multi-agent HTTP services.Traces not reaching the backend (proxy)
Traces not reaching the backend (proxy)
Cause: Corporate web proxy intercepting OTLP export.Solution: Set
NO_PROXY to include your Langfuse, Grafana, or OTLP host. See Proxy configuration.High memory usage
High memory usage
Cause: Span queue growing when export fails.Solution: Fix network connectivity to the export endpoint. Lower
otlp_max_queue_size if needed.OBSERVABILITY_ENABLED has no effect
OBSERVABILITY_ENABLED has no effect
Cause: Environment variables loaded after SDK import.Solution:
API reference
Configure_observability()
Application entry point for instrumentation. When the tracer is not initialized yet and
init_observability=True (default), the SDK calls init(service_name=..., environment=...), then instrument() for HTTPX, threading, framework OTel, and (when fastapi_app is set) FastAPI route spans and X-Observability-Project middleware.Export backend, OTLP batch tuning, organization fields, and custom Resource attributes are not parameters on this function — set them on init() and pass init_observability=False. See Advanced export settings.Scope()
Sets
bb.* context for the current async or sync block. The SDK applies values on span start via ScopeAttributeSpanProcessor.Init() (observability)
Low-level TracerProvider and OTLP export setup.
configure_observability() calls this with service_name and environment only when init_observability=True and the tracer is not yet initialized.Call init() yourself when you need parameters configure_observability() does not expose — then pass init_observability=False to configure_observability(). For full manual control without configure_observability(), call init() and instrument() — see Manual assembly.Trace()
Decorator that creates OpenTelemetry spans for functions.
Trace_context()
Context manager for manual span control.
Get_tracer_provider()
Returns the
TracerProvider the SDK created—use with manual instrumentors such as OpenAIInstrumentor.Callback handlers
Callback handler for LangChain operations when not using
configure_observability(framework="langchain").Callback handler for LangGraph operations. Optional
graph_name constructor argument.Next steps
Logging and redaction
Log formats, redaction patterns, and correlation with traces
Evaluate your agent
Run evals with the same
service_name and export setupMulti agent starter
Full FastAPI + Agno + observability reference implementation
Get started
Minimal gateway + tracing example