5.6 KiB
OpenTelemetry Implementation Notes
This document outlines best practices and implementation details for OpenTelemetry in AgentOps instrumentations.
Key Concepts
Context Propagation
OpenTelemetry relies on proper context propagation to maintain parent-child relationships between spans. This is essential for:
- Creating accurate trace waterfalls in visualizations
- Ensuring all spans from the same logical operation share a trace ID
- Allowing proper querying and filtering of related operations
Core Patterns
When implementing instrumentations that need to maintain context across different execution contexts:
-
Store span contexts in dictionaries:
# Use weakref dictionaries to avoid memory leaks self._span_contexts = weakref.WeakKeyDictionary() self._trace_root_contexts = weakref.WeakKeyDictionary() -
Create spans with explicit parent contexts:
parent_context = self._get_parent_context(trace_obj) with trace.start_as_current_span( name=span_name, context=parent_context, kind=trace.SpanKind.CLIENT, attributes=attributes, ) as span: # Span operations here # Store the span's context for future reference context = trace.set_span_in_context(span) self._span_contexts[span_obj] = context -
Implement helper methods to retrieve appropriate parent contexts:
def _get_parent_context(self, trace_obj): # Try to get the trace's root context if it exists if trace_obj in self._trace_root_contexts: return self._trace_root_contexts[trace_obj] # Otherwise, use the current context return context_api.context.get_current() -
Debug trace continuity:
current_span = trace.get_current_span() span_context = current_span.get_span_context() trace_id = format_trace_id(span_context.trace_id) logging.debug(f"Current span trace ID: {trace_id}")
Common Pitfalls
-
Naming conflicts: Avoid using
traceas a parameter name when you're also importing the OpenTelemetrytracemodule# Bad def on_trace_start(self, trace): # This will cause conflicts with the imported trace module # Good def on_trace_start(self, trace_obj): # No conflicts with OpenTelemetry's trace module -
Missing parent contexts: Always explicitly provide parent contexts when available, don't rely on current context alone
-
Memory leaks: Use
weakref.WeakKeyDictionary()for storing spans to allow garbage collection -
Lost context: When calling async or callback functions, be sure to preserve and pass the context
Testing Context Propagation
To verify proper context propagation:
- Enable debug logging for trace IDs
- Run a simple end-to-end test that generates multiple spans
- Verify all spans share the same trace ID
- Check that parent-child relationships are correctly established
# Example debug logging
logging.debug(f"Span {span.name} has trace ID: {format_trace_id(span.get_span_context().trace_id)}")
Timestamp Handling in OpenTelemetry
When working with OpenTelemetry spans and timestamps:
-
Automatic Timestamp Tracking: OpenTelemetry automatically tracks timestamps for spans. When a span is created with
tracer.start_span()ortracer.start_as_current_span(), the start time is captured automatically. Whenspan.end()is called, the end time is recorded. -
No Manual Timestamp Setting Required: The standard instrumentation pattern does not require manually setting timestamp attributes on spans. Instead, OpenTelemetry handles this internally through the SpanProcessor and Exporter classes.
-
Timestamp Representation: In the OpenTelemetry data model, timestamps are stored as nanoseconds since the Unix epoch (January 1, 1970).
-
Serialization Responsibility: The serialization of timestamps from OTel spans to output formats like JSON is handled by the Exporter components. If timestamps aren't appearing correctly in output APIs, the issue is likely in the API exporter, not in the span creation code.
-
Debugging Timestamps: To debug timestamp issues, verify that spans are properly starting and ending, rather than manually setting timestamp attributes:
# Good pattern - timestamps handled by OpenTelemetry automatically
with tracer.start_as_current_span("my_operation") as span:
# Do work
pass # span.end() is called automatically
Note: If timestamps are missing in API output (e.g., empty "start_time" fields), focus on fixes in the exporter and serialization layer, not by manually tracking timestamps in instrumentation code.
Attributes in OpenTelemetry
When working with span attributes in OpenTelemetry:
-
Root Attributes Node: The root
attributesobject in the API output JSON should always be empty. This is by design. All attribute data should be stored in thespan_attributesobject. -
Span Attributes: The
span_attributesobject is where all user-defined and semantic attribute data should be stored. This allows for a structured, hierarchical representation of attributes. -
Structure Difference: While the root
attributesappears as an empty object in the API output, this is normal and expected. Do not attempt to populate this object directly or duplicate data fromspan_attributesinto it. -
Setting Attributes: Always set span attributes using the semantic conventions defined in the
agentops/semconvmodule:
from agentops.semconv import agent
# Good pattern - using semantic conventions
span.set_attribute(agent.AGENT_NAME, "My Agent")