953 lines
40 KiB
Plaintext
953 lines
40 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a7cd4a28",
|
|
"metadata": {},
|
|
"source": [
|
|
"# OpenAI Agents Patterns Demonstration\n",
|
|
"\n",
|
|
"This notebook demonstrates various common agentic patterns using the Agents SDK and how one can observe them using the AgentOps platform.\n",
|
|
"\n",
|
|
"**Note: This notebook was edited using Claude MCP NotebookEdit tool!**\n",
|
|
"\n",
|
|
"## General Flow\n",
|
|
"\n",
|
|
"This notebook will walk you through several key agent patterns:\n",
|
|
"\n",
|
|
"1. **Agents as Tools** - Using agents as callable tools within other agents\n",
|
|
"2. **Deterministic Flows** - Breaking down tasks into sequential steps\n",
|
|
"3. **Forcing Tool Use** - Controlling when and how agents use tools\n",
|
|
"4. **Input Guardrails** - Validating inputs before agent execution\n",
|
|
"5. **LLM as a Judge** - Using LLMs to evaluate and improve outputs\n",
|
|
"6. **Output Guardrails** - Validating outputs after agent execution\n",
|
|
"7. **Parallelization** - Running multiple agents concurrently\n",
|
|
"8. **Routing** - Directing requests to specialized agents\n",
|
|
"9. **Streaming Guardrails** - Real-time validation during streaming\n",
|
|
"\n",
|
|
"Each pattern demonstrates how AgentOps automatically tracks and monitors your agent interactions, providing valuable insights into performance, costs, and behavior."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7fb27b941602401d91542211134fc71a",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Prerequisites\n",
|
|
"\n",
|
|
"Before running this notebook, you'll need:\n",
|
|
"\n",
|
|
"1. **AgentOps Account**: Create a free account at [app.agentops.ai](https://app.agentops.ai)\n",
|
|
"2. **AgentOps API Key**: Obtain your API key from your AgentOps dashboard\n",
|
|
"3. **OpenAI API Key**: Get your API key from [platform.openai.com](https://platform.openai.com)\n",
|
|
"\n",
|
|
"Make sure to set these as environment variables or create a `.env` file in your project root with:\n",
|
|
"\n",
|
|
"```\n",
|
|
"AGENTOPS_API_KEY=your_agentops_api_key_here\n",
|
|
"OPENAI_API_KEY=your_openai_api_key_here\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "66be7a30",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Install required packages\n",
|
|
"%pip install -q agentops\n",
|
|
"%pip install -q openai-agents\n",
|
|
"%pip install -q pydotenv"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "acae54e37e7d407bbb7b55eff062a284",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Required imports - Note: agentops must be imported before agents\n",
|
|
"import asyncio\n",
|
|
"import os\n",
|
|
"import uuid\n",
|
|
"from typing import Any, Literal\n",
|
|
"from dataclasses import dataclass\n",
|
|
"from dotenv import load_dotenv\n",
|
|
"from pydantic import BaseModel, Field\n",
|
|
"\n",
|
|
"# Import agentops FIRST\n",
|
|
"import agentops\n",
|
|
"\n",
|
|
"# Then import agents\n",
|
|
"from agents import (\n",
|
|
" Agent,\n",
|
|
" ItemHelpers,\n",
|
|
" MessageOutputItem,\n",
|
|
" Runner,\n",
|
|
" trace,\n",
|
|
" TResponseInputItem,\n",
|
|
" FunctionToolResult,\n",
|
|
" ModelSettings,\n",
|
|
" RunContextWrapper,\n",
|
|
" ToolsToFinalOutputFunction,\n",
|
|
" ToolsToFinalOutputResult,\n",
|
|
" function_tool,\n",
|
|
" GuardrailFunctionOutput,\n",
|
|
" InputGuardrailTripwireTriggered,\n",
|
|
" OutputGuardrailTripwireTriggered,\n",
|
|
" input_guardrail,\n",
|
|
" output_guardrail,\n",
|
|
")\n",
|
|
"\n",
|
|
"from openai.types.responses import ResponseTextDeltaEvent"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "780a20ee",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Load environment variables and set API keys\n",
|
|
"load_dotenv()\n",
|
|
"os.environ[\"AGENTOPS_API_KEY\"] = os.getenv(\"AGENTOPS_API_KEY\", \"your_api_key_here\")\n",
|
|
"os.environ[\"OPENAI_API_KEY\"] = os.getenv(\"OPENAI_API_KEY\", \"your_openai_api_key_here\")\n",
|
|
"\n",
|
|
"# Initialize AgentOps\n",
|
|
"agentops.init(auto_start_session=False)\n",
|
|
"# Note: tracer will be defined in each section's cell for clarity, using the specific tags for that pattern."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e37212a8",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 1. Agents as Tools Pattern\n",
|
|
"\n",
|
|
"The mental model for handoffs is that the new agent \"takes over\". It sees the previous conversation history, and owns the conversation from that point onwards. However, this is not the only way to use agents. You can also use agents as a tool - the tool agent goes off and runs on its own, and then returns the result to the original agent.\n",
|
|
"\n",
|
|
"For example, you could model the translation task above as tool calls instead: rather than handing over to the language-specific agent, you could call the agent as a tool, and then use the result in the next step. This enables things like translating multiple languages at once.\n",
|
|
"\n",
|
|
"This pattern demonstrates using agents as callable tools within other agents. The orchestrator agent receives a user message and then picks which specialized agents to call as tools."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "52377535",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Start the AgentOps trace session\n",
|
|
"tracer = agentops.start_trace(trace_name=\"Agents as Tools Pattern\", tags=[\"agentops-example\", \"openai-agents\"])\n",
|
|
"\n",
|
|
"# Define specialized translation agents\n",
|
|
"spanish_agent = Agent(\n",
|
|
" name=\"spanish_agent\",\n",
|
|
" instructions=\"You translate the user's message to Spanish\",\n",
|
|
" handoff_description=\"An english to spanish translator\",\n",
|
|
")\n",
|
|
"\n",
|
|
"french_agent = Agent(\n",
|
|
" name=\"french_agent\",\n",
|
|
" instructions=\"You translate the user's message to French\",\n",
|
|
" handoff_description=\"An english to french translator\",\n",
|
|
")\n",
|
|
"\n",
|
|
"italian_agent = Agent(\n",
|
|
" name=\"italian_agent\",\n",
|
|
" instructions=\"You translate the user's message to Italian\",\n",
|
|
" handoff_description=\"An english to italian translator\",\n",
|
|
")\n",
|
|
"\n",
|
|
"# Orchestrator agent that uses other agents as tools\n",
|
|
"orchestrator_agent = Agent(\n",
|
|
" name=\"orchestrator_agent\",\n",
|
|
" instructions=(\n",
|
|
" \"You are a translation agent. You use the tools given to you to translate.\"\n",
|
|
" \"If asked for multiple translations, you call the relevant tools in order.\"\n",
|
|
" \"You never translate on your own, you always use the provided tools.\"\n",
|
|
" ),\n",
|
|
" tools=[\n",
|
|
" spanish_agent.as_tool(\n",
|
|
" tool_name=\"translate_to_spanish\",\n",
|
|
" tool_description=\"Translate the user's message to Spanish\",\n",
|
|
" ),\n",
|
|
" french_agent.as_tool(\n",
|
|
" tool_name=\"translate_to_french\",\n",
|
|
" tool_description=\"Translate the user's message to French\",\n",
|
|
" ),\n",
|
|
" italian_agent.as_tool(\n",
|
|
" tool_name=\"translate_to_italian\",\n",
|
|
" tool_description=\"Translate the user's message to Italian\",\n",
|
|
" ),\n",
|
|
" ],\n",
|
|
")\n",
|
|
"\n",
|
|
"synthesizer_agent = Agent(\n",
|
|
" name=\"synthesizer_agent\",\n",
|
|
" instructions=\"You inspect translations, correct them if needed, and produce a final concatenated response.\",\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"async def run_agents_as_tools_demo():\n",
|
|
" msg = \"Hello, how are you today?\"\n",
|
|
" print(f\"Input: {msg}\")\n",
|
|
"\n",
|
|
" with trace(\"Orchestrator evaluator\"):\n",
|
|
" orchestrator_result = await Runner.run(orchestrator_agent, msg)\n",
|
|
"\n",
|
|
" for item in orchestrator_result.new_items:\n",
|
|
" if isinstance(item, MessageOutputItem):\n",
|
|
" text = ItemHelpers.text_message_output(item)\n",
|
|
" if text:\n",
|
|
" print(f\" - Translation step: {text}\")\n",
|
|
"\n",
|
|
" synthesizer_result = await Runner.run(synthesizer_agent, orchestrator_result.to_input_list())\n",
|
|
"\n",
|
|
" print(f\"Final response: {synthesizer_result.final_output}\")\n",
|
|
"\n",
|
|
"\n",
|
|
"# Run the demo\n",
|
|
"await run_agents_as_tools_demo()\n",
|
|
"\n",
|
|
"# End the AgentOps trace session\n",
|
|
"agentops.end_trace(tracer, end_state=\"Success\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "98e54a31",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 2. Deterministic Flow Pattern\n",
|
|
"\n",
|
|
"A common tactic is to break down a task into a series of smaller steps. Each task can be performed by an agent, and the output of one agent is used as input to the next. For example, if your task was to generate a story, you could break it down into the following steps:\n",
|
|
"\n",
|
|
"1. Generate an outline\n",
|
|
"2. Generate the story\n",
|
|
"3. Generate the ending\n",
|
|
"\n",
|
|
"Each of these steps can be performed by an agent. The output of one agent is used as input to the next.\n",
|
|
"\n",
|
|
"This pattern demonstrates breaking down a complex task into a series of smaller, sequential steps. Each step is performed by an agent, and the output of one agent is used as input to the next."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "3428aa44",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Start the AgentOps trace session\n",
|
|
"tracer = agentops.start_trace(trace_name=\"Deterministic Flow Pattern\", tags=[\"agentops-example\", \"openai-agents\"])\n",
|
|
"\n",
|
|
"# Define the story generation workflow\n",
|
|
"story_outline_agent = Agent(\n",
|
|
" name=\"story_outline_agent\",\n",
|
|
" instructions=\"Generate a very short story outline based on the user's input.\",\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"class OutlineCheckerOutput(BaseModel):\n",
|
|
" good_quality: bool\n",
|
|
" is_scifi: bool\n",
|
|
"\n",
|
|
"\n",
|
|
"outline_checker_agent = Agent(\n",
|
|
" name=\"outline_checker_agent\",\n",
|
|
" instructions=\"Read the given story outline, and judge the quality. Also, determine if it is a scifi story.\",\n",
|
|
" output_type=OutlineCheckerOutput,\n",
|
|
")\n",
|
|
"\n",
|
|
"story_agent = Agent(\n",
|
|
" name=\"story_agent\",\n",
|
|
" instructions=\"Write a short story based on the given outline.\",\n",
|
|
" output_type=str,\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"async def run_deterministic_flow_demo():\n",
|
|
" input_prompt = \"A story about robots exploring space\"\n",
|
|
" print(f\"Input: {input_prompt}\")\n",
|
|
"\n",
|
|
" with trace(\"Deterministic story flow\"):\n",
|
|
" # 1. Generate an outline\n",
|
|
" outline_result = await Runner.run(story_outline_agent, input_prompt)\n",
|
|
" print(\"Outline generated\")\n",
|
|
"\n",
|
|
" # 2. Check the outline\n",
|
|
" outline_checker_result = await Runner.run(outline_checker_agent, outline_result.final_output)\n",
|
|
"\n",
|
|
" # 3. Add a gate to stop if the outline is not good quality or not a scifi story\n",
|
|
" assert isinstance(outline_checker_result.final_output, OutlineCheckerOutput)\n",
|
|
" if not outline_checker_result.final_output.good_quality:\n",
|
|
" print(\"Outline is not good quality, so we stop here.\")\n",
|
|
" return\n",
|
|
"\n",
|
|
" if not outline_checker_result.final_output.is_scifi:\n",
|
|
" print(\"Outline is not a scifi story, so we stop here.\")\n",
|
|
" return\n",
|
|
"\n",
|
|
" print(\"Outline is good quality and a scifi story, so we continue to write the story.\")\n",
|
|
"\n",
|
|
" # 4. Write the story\n",
|
|
" story_result = await Runner.run(story_agent, outline_result.final_output)\n",
|
|
" print(f\"Story: {story_result.final_output}\")\n",
|
|
"\n",
|
|
"\n",
|
|
"# Run the demo\n",
|
|
"await run_deterministic_flow_demo()\n",
|
|
"\n",
|
|
"# End the AgentOps trace session\n",
|
|
"agentops.end_trace(tracer, end_state=\"Success\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d9095df7",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 3. Forcing Tool Use Pattern\n",
|
|
"\n",
|
|
"This pattern shows how to force an agent to use a tool using `ModelSettings(tool_choice=\"required\")`. This is useful when you want to ensure the agent always uses a specific tool rather than generating a response directly.\n",
|
|
"\n",
|
|
"You can run it with 3 options:\n",
|
|
"1. `default`: The default behavior, which is to send the tool output to the LLM. In this case, `tool_choice` is not set, because otherwise it would result in an infinite loop - the LLM would call the tool, the tool would run and send the results to the LLM, and that would repeat (because the model is forced to use a tool every time.)\n",
|
|
"2. `first_tool_result`: The first tool result is used as the final output.\n",
|
|
"3. `custom`: A custom tool use behavior function is used. The custom function receives all the tool results, and chooses to use the first tool result to generate the final output.\n",
|
|
"\n",
|
|
"For this demo, we'll allow the user to choose which tool use behavior to test:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9a63283cbaf04dbcab1f6479b197f3a8",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Start the AgentOps trace session\n",
|
|
"tracer = agentops.start_trace(trace_name=\"Forcing Tool Use Pattern\", tags=[\"agentops-example\", \"openai-agents\"])\n",
|
|
"\n",
|
|
"\n",
|
|
"# Define the weather tool and agent\n",
|
|
"class Weather(BaseModel):\n",
|
|
" city: str\n",
|
|
" temperature_range: str\n",
|
|
" conditions: str\n",
|
|
"\n",
|
|
"\n",
|
|
"@function_tool\n",
|
|
"def get_weather(city: str) -> Weather:\n",
|
|
" print(\"[debug] get_weather called\")\n",
|
|
" return Weather(city=city, temperature_range=\"14-20C\", conditions=\"Sunny with wind\")\n",
|
|
"\n",
|
|
"\n",
|
|
"async def custom_tool_use_behavior(\n",
|
|
" context: RunContextWrapper[Any], results: list[FunctionToolResult]\n",
|
|
") -> ToolsToFinalOutputResult:\n",
|
|
" weather: Weather = results[0].output\n",
|
|
" return ToolsToFinalOutputResult(is_final_output=True, final_output=f\"{weather.city} is {weather.conditions}.\")\n",
|
|
"\n",
|
|
"\n",
|
|
"# User can choose which behavior to test\n",
|
|
"print(\"Choose tool use behavior:\")\n",
|
|
"print(\"1. default - Send tool output to LLM\")\n",
|
|
"print(\"2. first_tool - Use first tool result as final output\")\n",
|
|
"print(\"3. custom - Use custom tool behavior function\")\n",
|
|
"\n",
|
|
"choice = input(\"Enter choice (1, 2, or 3): \").strip()\n",
|
|
"\n",
|
|
"if choice == \"1\":\n",
|
|
" tool_use_behavior = \"default\"\n",
|
|
"elif choice == \"2\":\n",
|
|
" tool_use_behavior = \"first_tool\"\n",
|
|
"elif choice == \"3\":\n",
|
|
" tool_use_behavior = \"custom\"\n",
|
|
"else:\n",
|
|
" tool_use_behavior = \"default\"\n",
|
|
" print(\"Invalid choice, using default\")\n",
|
|
"\n",
|
|
"\n",
|
|
"async def run_forcing_tool_use_demo(tool_use_behavior: str):\n",
|
|
" print(f\"Testing {tool_use_behavior} behavior:\")\n",
|
|
"\n",
|
|
" if tool_use_behavior == \"default\":\n",
|
|
" behavior: Literal[\"run_llm_again\", \"stop_on_first_tool\"] | ToolsToFinalOutputFunction = \"run_llm_again\"\n",
|
|
" elif tool_use_behavior == \"first_tool\":\n",
|
|
" behavior = \"stop_on_first_tool\"\n",
|
|
" elif tool_use_behavior == \"custom\":\n",
|
|
" behavior = custom_tool_use_behavior\n",
|
|
"\n",
|
|
" agent = Agent(\n",
|
|
" name=\"Weather agent\",\n",
|
|
" instructions=\"You are a helpful agent.\",\n",
|
|
" tools=[get_weather],\n",
|
|
" tool_use_behavior=behavior,\n",
|
|
" model_settings=ModelSettings(tool_choice=\"required\" if tool_use_behavior != \"default\" else None),\n",
|
|
" )\n",
|
|
"\n",
|
|
" result = await Runner.run(agent, input=\"What's the weather in Tokyo?\")\n",
|
|
" print(f\"Result: {result.final_output}\")\n",
|
|
"\n",
|
|
"\n",
|
|
"# Run the demo\n",
|
|
"await run_forcing_tool_use_demo(tool_use_behavior)\n",
|
|
"\n",
|
|
"# End the AgentOps trace session\n",
|
|
"agentops.end_trace(tracer, end_state=\"Success\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "eaada293",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 4. Input Guardrails Pattern\n",
|
|
"\n",
|
|
"Related to parallelization, you often want to run input guardrails to make sure the inputs to your agents are valid. For example, if you have a customer support agent, you might want to make sure that the user isn't trying to ask for help with a math problem.\n",
|
|
"\n",
|
|
"You can definitely do this without any special Agents SDK features by using parallelization, but we support a special guardrail primitive. Guardrails can have a \"tripwire\" - if the tripwire is triggered, the agent execution will immediately stop and a `GuardrailTripwireTriggered` exception will be raised.\n",
|
|
"\n",
|
|
"This is really useful for latency: for example, you might have a very fast model that runs the guardrail and a slow model that runs the actual agent. You wouldn't want to wait for the slow model to finish, so guardrails let you quickly reject invalid inputs.\n",
|
|
"\n",
|
|
"This pattern demonstrates how to use input guardrails to validate user inputs before they reach the main agent. Guardrails can prevent inappropriate or off-topic requests from being processed."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "0a597f58",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Start the AgentOps trace session\n",
|
|
"tracer = agentops.start_trace(trace_name=\"Input Guardrails Pattern\", tags=[\"agentops-example\", \"openai-agents\"])\n",
|
|
"\n",
|
|
"\n",
|
|
"# Define the guardrail\n",
|
|
"class MathHomeworkOutput(BaseModel):\n",
|
|
" reasoning: str\n",
|
|
" is_math_homework: bool\n",
|
|
"\n",
|
|
"\n",
|
|
"guardrail_agent = Agent(\n",
|
|
" name=\"Guardrail check\",\n",
|
|
" instructions=\"Check if the user is asking you to do their math homework.\",\n",
|
|
" output_type=MathHomeworkOutput,\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"@input_guardrail\n",
|
|
"async def math_guardrail(\n",
|
|
" context: RunContextWrapper[None], agent: Agent, input: str | list[TResponseInputItem]\n",
|
|
") -> GuardrailFunctionOutput:\n",
|
|
" result = await Runner.run(guardrail_agent, input, context=context.context)\n",
|
|
" final_output = result.final_output_as(MathHomeworkOutput)\n",
|
|
"\n",
|
|
" return GuardrailFunctionOutput(\n",
|
|
" output_info=final_output,\n",
|
|
" tripwire_triggered=final_output.is_math_homework,\n",
|
|
" )\n",
|
|
"\n",
|
|
"\n",
|
|
"async def run_input_guardrails_demo():\n",
|
|
" agent = Agent(\n",
|
|
" name=\"Customer support agent\",\n",
|
|
" instructions=\"You are a customer support agent. You help customers with their questions.\",\n",
|
|
" input_guardrails=[math_guardrail],\n",
|
|
" )\n",
|
|
"\n",
|
|
" # Test with different inputs\n",
|
|
" test_inputs = [\"What's the capital of California?\", \"Can you help me solve for x: 2x + 5 = 11\"]\n",
|
|
"\n",
|
|
" for user_input in test_inputs:\n",
|
|
" print(f\"Testing input: {user_input}\")\n",
|
|
" try:\n",
|
|
" result = await Runner.run(agent, user_input)\n",
|
|
" print(f\"Response: {result.final_output}\")\n",
|
|
" except InputGuardrailTripwireTriggered:\n",
|
|
" message = \"Sorry, I can't help you with your math homework.\"\n",
|
|
" print(f\"Guardrail triggered: {message}\")\n",
|
|
" print()\n",
|
|
"\n",
|
|
"\n",
|
|
"# Run the demo\n",
|
|
"await run_input_guardrails_demo()\n",
|
|
"\n",
|
|
"# End the AgentOps trace session\n",
|
|
"agentops.end_trace(tracer, end_state=\"Success\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1aa0b993",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 5. LLM as a Judge Pattern\n",
|
|
"\n",
|
|
"LLMs can often improve the quality of their output if given feedback. A common pattern is to generate a response using a model, and then use a second model to provide feedback. You can even use a small model for the initial generation and a larger model for the feedback, to optimize cost.\n",
|
|
"\n",
|
|
"For example, you could use an LLM to generate an outline for a story, and then use a second LLM to evaluate the outline and provide feedback. You can then use the feedback to improve the outline, and repeat until the LLM is satisfied with the outline.\n",
|
|
"\n",
|
|
"This pattern shows how to use one LLM to evaluate and improve the output of another. The first agent generates content, and the second agent judges the quality and provides feedback for improvement."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9bbb7b94",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Start the AgentOps trace session\n",
|
|
"tracer = agentops.start_trace(trace_name=\"LLM as a Judge Pattern\", tags=[\"agentops-example\", \"openai-agents\"])\n",
|
|
"\n",
|
|
"# Define the story generation and evaluation agents\n",
|
|
"story_outline_generator = Agent(\n",
|
|
" name=\"story_outline_generator\",\n",
|
|
" instructions=(\n",
|
|
" \"You generate a very short story outline based on the user's input.\"\n",
|
|
" \"If there is any feedback provided, use it to improve the outline.\"\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"@dataclass\n",
|
|
"class EvaluationFeedback:\n",
|
|
" feedback: str\n",
|
|
" score: Literal[\"pass\", \"needs_improvement\", \"fail\"]\n",
|
|
"\n",
|
|
"\n",
|
|
"evaluator = Agent[None](\n",
|
|
" name=\"evaluator\",\n",
|
|
" instructions=(\n",
|
|
" \"You evaluate a story outline and decide if it's good enough.\"\n",
|
|
" \"If it's not good enough, you provide feedback on what needs to be improved.\"\n",
|
|
" \"Never give it a pass on the first try.\"\n",
|
|
" ),\n",
|
|
" output_type=EvaluationFeedback,\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"async def run_llm_as_judge_demo():\n",
|
|
" msg = \"A story about time travel\"\n",
|
|
" print(f\"Input: {msg}\")\n",
|
|
" input_items: list[TResponseInputItem] = [{\"content\": msg, \"role\": \"user\"}]\n",
|
|
"\n",
|
|
" latest_outline: str | None = None\n",
|
|
" iteration = 0\n",
|
|
" max_iterations = 3 # Limit iterations for demo\n",
|
|
"\n",
|
|
" with trace(\"LLM as a judge\"):\n",
|
|
" while iteration < max_iterations:\n",
|
|
" iteration += 1\n",
|
|
" print(f\"Iteration {iteration}:\")\n",
|
|
"\n",
|
|
" story_outline_result = await Runner.run(\n",
|
|
" story_outline_generator,\n",
|
|
" input_items,\n",
|
|
" )\n",
|
|
"\n",
|
|
" input_items = story_outline_result.to_input_list()\n",
|
|
" latest_outline = ItemHelpers.text_message_outputs(story_outline_result.new_items)\n",
|
|
" print(\"Story outline generated\")\n",
|
|
"\n",
|
|
" evaluator_result = await Runner.run(evaluator, input_items)\n",
|
|
" result: EvaluationFeedback = evaluator_result.final_output\n",
|
|
"\n",
|
|
" print(f\"Evaluator score: {result.score}\")\n",
|
|
"\n",
|
|
" if result.score == \"pass\":\n",
|
|
" print(\"Story outline is good enough, exiting.\")\n",
|
|
" break\n",
|
|
"\n",
|
|
" print(\"Re-running with feedback\")\n",
|
|
" input_items.append({\"content\": f\"Feedback: {result.feedback}\", \"role\": \"user\"})\n",
|
|
"\n",
|
|
" print(f\"Final story outline: {latest_outline}\")\n",
|
|
"\n",
|
|
"\n",
|
|
"# Run the demo\n",
|
|
"await run_llm_as_judge_demo()\n",
|
|
"\n",
|
|
"# End the AgentOps trace session\n",
|
|
"agentops.end_trace(tracer, end_state=\"Success\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3e2acc90",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 6. Output Guardrails Pattern\n",
|
|
"\n",
|
|
"Related to parallelization, you often want to run output guardrails to make sure the outputs from your agents are valid. Guardrails can have a \"tripwire\" - if the tripwire is triggered, the agent execution will immediately stop and a `GuardrailTripwireTriggered` exception will be raised.\n",
|
|
"\n",
|
|
"This is really useful for latency: for example, you might have a very fast model that runs the guardrail and a slow model that runs the actual agent. You wouldn't want to wait for the slow model to finish, so guardrails let you quickly reject invalid outputs.\n",
|
|
"\n",
|
|
"This pattern demonstrates how to use output guardrails to validate agent outputs after they are generated. This can help prevent sensitive information from being shared or ensure outputs meet quality standards."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "71ccbecc",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Start the AgentOps trace session\n",
|
|
"tracer = agentops.start_trace(trace_name=\"Output Guardrails Pattern\", tags=[\"agentops-example\", \"openai-agents\"])\n",
|
|
"\n",
|
|
"\n",
|
|
"# The agent's output type\n",
|
|
"class MessageOutput(BaseModel):\n",
|
|
" reasoning: str = Field(description=\"Thoughts on how to respond to the user's message\")\n",
|
|
" response: str = Field(description=\"The response to the user's message\")\n",
|
|
" user_name: str | None = Field(description=\"The name of the user who sent the message, if known\")\n",
|
|
"\n",
|
|
"\n",
|
|
"@output_guardrail\n",
|
|
"async def sensitive_data_check(\n",
|
|
" context: RunContextWrapper, agent: Agent, output: MessageOutput\n",
|
|
") -> GuardrailFunctionOutput:\n",
|
|
" phone_number_in_response = \"650\" in output.response\n",
|
|
" phone_number_in_reasoning = \"650\" in output.reasoning\n",
|
|
"\n",
|
|
" return GuardrailFunctionOutput(\n",
|
|
" output_info={\n",
|
|
" \"phone_number_in_response\": phone_number_in_response,\n",
|
|
" \"phone_number_in_reasoning\": phone_number_in_reasoning,\n",
|
|
" },\n",
|
|
" tripwire_triggered=phone_number_in_response or phone_number_in_reasoning,\n",
|
|
" )\n",
|
|
"\n",
|
|
"\n",
|
|
"output_guardrail_agent = Agent(\n",
|
|
" name=\"Assistant\",\n",
|
|
" instructions=\"You are a helpful assistant.\",\n",
|
|
" output_type=MessageOutput,\n",
|
|
" output_guardrails=[sensitive_data_check],\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"async def run_output_guardrails_demo():\n",
|
|
" # Test with safe input\n",
|
|
" print(\"Testing with safe input:\")\n",
|
|
" try:\n",
|
|
" result = await Runner.run(output_guardrail_agent, \"What's the capital of California?\")\n",
|
|
" print(\"Safe message passed\")\n",
|
|
" print(f\"Response: {result.final_output.response}\")\n",
|
|
" except OutputGuardrailTripwireTriggered as e:\n",
|
|
" print(f\"Unexpected guardrail trigger: {e.guardrail_result.output.output_info}\")\n",
|
|
"\n",
|
|
" print()\n",
|
|
" # Test with potentially sensitive input\n",
|
|
" print(\"Testing with potentially sensitive input:\")\n",
|
|
" try:\n",
|
|
" result = await Runner.run(output_guardrail_agent, \"My phone number is 650-123-4567. Where do you think I live?\")\n",
|
|
" print(f\"Guardrail didn't trip - this is unexpected. Output: {result.final_output.response}\")\n",
|
|
" except OutputGuardrailTripwireTriggered as e:\n",
|
|
" print(f\"Guardrail tripped as expected. Info: {e.guardrail_result.output.output_info}\")\n",
|
|
"\n",
|
|
"\n",
|
|
"# Run the demo\n",
|
|
"await run_output_guardrails_demo()\n",
|
|
"\n",
|
|
"# End the AgentOps trace session\n",
|
|
"agentops.end_trace(tracer, end_state=\"Success\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0f91df27",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 7. Parallelization Pattern\n",
|
|
"\n",
|
|
"Running multiple agents in parallel is a common pattern. This can be useful for both latency (e.g. if you have multiple steps that don't depend on each other) and also for other reasons e.g. generating multiple responses and picking the best one.\n",
|
|
"\n",
|
|
"This example runs a translation agent multiple times in parallel, and then picks the best translation.\n",
|
|
"\n",
|
|
"This pattern shows how to run multiple agents in parallel to improve latency or generate multiple options to choose from. In this example, we run translation agents multiple times and pick the best result."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c40082e4",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Start the AgentOps trace session\n",
|
|
"tracer = agentops.start_trace(trace_name=\"Output Guardrails Pattern\", tags=[\"agentops-example\", \"openai-agents\"])\n",
|
|
"\n",
|
|
"# Define agents for parallelization\n",
|
|
"spanish_translation_agent = Agent(\n",
|
|
" name=\"spanish_agent\",\n",
|
|
" instructions=\"You translate the user's message to Spanish\",\n",
|
|
")\n",
|
|
"\n",
|
|
"translation_picker = Agent(\n",
|
|
" name=\"translation_picker\",\n",
|
|
" instructions=\"You pick the best Spanish translation from the given options.\",\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"async def run_parallelization_demo():\n",
|
|
" msg = \"Good morning, I hope you have a wonderful day!\"\n",
|
|
" print(f\"Input: {msg}\")\n",
|
|
"\n",
|
|
" with trace(\"Parallel translation\"):\n",
|
|
" # Run three translation agents in parallel\n",
|
|
" res_1, res_2, res_3 = await asyncio.gather(\n",
|
|
" Runner.run(spanish_translation_agent, msg),\n",
|
|
" Runner.run(spanish_translation_agent, msg),\n",
|
|
" Runner.run(spanish_translation_agent, msg),\n",
|
|
" )\n",
|
|
"\n",
|
|
" outputs = [\n",
|
|
" ItemHelpers.text_message_outputs(res_1.new_items),\n",
|
|
" ItemHelpers.text_message_outputs(res_2.new_items),\n",
|
|
" ItemHelpers.text_message_outputs(res_3.new_items),\n",
|
|
" ]\n",
|
|
"\n",
|
|
" translations = \"\\n\\n\".join(outputs)\n",
|
|
" print(f\"\\n\\nTranslations:\\n\\n{translations}\")\n",
|
|
"\n",
|
|
" best_translation = await Runner.run(\n",
|
|
" translation_picker,\n",
|
|
" f\"Input: {msg}\\n\\nTranslations:\\n{translations}\",\n",
|
|
" )\n",
|
|
"\n",
|
|
" print(\"\\n\\n-----\")\n",
|
|
" print(f\"Best translation: {best_translation.final_output}\")\n",
|
|
"\n",
|
|
"\n",
|
|
"# Run the demo\n",
|
|
"await run_parallelization_demo()\n",
|
|
"\n",
|
|
"# End the AgentOps trace session\n",
|
|
"agentops.end_trace(tracer, end_state=\"Success\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9b60d8d2",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 8. Routing Pattern\n",
|
|
"\n",
|
|
"In many situations, you have specialized sub-agents that handle specific tasks. You can use handoffs to route the task to the right agent.\n",
|
|
"\n",
|
|
"For example, you might have a frontline agent that receives a request, and then hands off to a specialized agent based on the language of the request.\n",
|
|
"\n",
|
|
"This pattern demonstrates handoffs and routing between specialized agents. The triage agent receives the first message and hands off to the appropriate agent based on the language of the request."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "698b1251",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Start the AgentOps trace session\n",
|
|
"tracer = agentops.start_trace(trace_name=\"Routing Pattern\", tags=[\"agentops-example\", \"openai-agents\"])\n",
|
|
"\n",
|
|
"# Define language-specific agents\n",
|
|
"french_routing_agent = Agent(\n",
|
|
" name=\"french_agent\",\n",
|
|
" instructions=\"You only speak French\",\n",
|
|
")\n",
|
|
"\n",
|
|
"spanish_routing_agent = Agent(\n",
|
|
" name=\"spanish_agent\",\n",
|
|
" instructions=\"You only speak Spanish\",\n",
|
|
")\n",
|
|
"\n",
|
|
"english_routing_agent = Agent(\n",
|
|
" name=\"english_agent\",\n",
|
|
" instructions=\"You only speak English\",\n",
|
|
")\n",
|
|
"\n",
|
|
"triage_agent = Agent(\n",
|
|
" name=\"triage_agent\",\n",
|
|
" instructions=\"Handoff to the appropriate agent based on the language of the request.\",\n",
|
|
" handoffs=[french_routing_agent, spanish_routing_agent, english_routing_agent],\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"async def run_routing_demo():\n",
|
|
" # Create an ID for this conversation\n",
|
|
" conversation_id = str(uuid.uuid4().hex[:16])\n",
|
|
"\n",
|
|
" test_messages = [\"Hello, how can you help me?\", \"Bonjour, comment allez-vous?\", \"Hola, ¿cómo estás?\"]\n",
|
|
"\n",
|
|
" for msg in test_messages:\n",
|
|
" print(f\"\\nTesting message: {msg}\")\n",
|
|
"\n",
|
|
" with trace(\"Routing example\", group_id=conversation_id):\n",
|
|
" inputs = [{\"content\": msg, \"role\": \"user\"}]\n",
|
|
" result = await Runner.run(triage_agent, input=inputs)\n",
|
|
" print(f\"Response: {result.final_output}\")\n",
|
|
"\n",
|
|
"\n",
|
|
"# Run the demo\n",
|
|
"await run_routing_demo()\n",
|
|
"\n",
|
|
"# End the AgentOps trace session\n",
|
|
"agentops.end_trace(tracer, end_state=\"Success\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6554dbbf",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 9. Streaming Guardrails Pattern\n",
|
|
"\n",
|
|
"This example shows how to use guardrails as the model is streaming. Output guardrails run after the final output has been generated; this example runs guardrails every N tokens, allowing for early termination if bad output is detected.\n",
|
|
"\n",
|
|
"The expected output is that you'll see a bunch of tokens stream in, then the guardrail will trigger and stop the streaming.\n",
|
|
"\n",
|
|
"This pattern shows how to use guardrails during streaming to provide real-time validation. Unlike output guardrails that run after completion, streaming guardrails can interrupt the generation process early."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "11a383f1",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Start the AgentOps trace session\n",
|
|
"tracer = agentops.start_trace(trace_name=\"Streaming Guardrails Pattern\", tags=[\"agentops-example\", \"openai-agents\"])\n",
|
|
"\n",
|
|
"# Define streaming guardrail agent\n",
|
|
"streaming_agent = Agent(\n",
|
|
" name=\"Assistant\",\n",
|
|
" instructions=(\n",
|
|
" \"You are a helpful assistant. You ALWAYS write long responses, making sure to be verbose and detailed.\"\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"class GuardrailOutput(BaseModel):\n",
|
|
" reasoning: str = Field(description=\"Reasoning about whether the response could be understood by a ten year old.\")\n",
|
|
" is_readable_by_ten_year_old: bool = Field(description=\"Whether the response is understandable by a ten year old.\")\n",
|
|
"\n",
|
|
"\n",
|
|
"guardrail_streaming_agent = Agent(\n",
|
|
" name=\"Checker\",\n",
|
|
" instructions=(\n",
|
|
" \"You will be given a question and a response. Your goal is to judge whether the response \"\n",
|
|
" \"is simple enough to be understood by a ten year old.\"\n",
|
|
" ),\n",
|
|
" output_type=GuardrailOutput,\n",
|
|
" model=\"gpt-4o-mini\",\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"async def check_guardrail(text: str) -> GuardrailOutput:\n",
|
|
" result = await Runner.run(guardrail_streaming_agent, text)\n",
|
|
" return result.final_output_as(GuardrailOutput)\n",
|
|
"\n",
|
|
"\n",
|
|
"async def run_streaming_guardrails_demo():\n",
|
|
" question = \"What is a black hole, and how does it behave?\"\n",
|
|
" print(f\"Question: {question}\")\n",
|
|
"\n",
|
|
" result = Runner.run_streamed(streaming_agent, question)\n",
|
|
" current_text = \"\"\n",
|
|
"\n",
|
|
" # We will check the guardrail every N characters\n",
|
|
" next_guardrail_check_len = 300\n",
|
|
" guardrail_task = None\n",
|
|
"\n",
|
|
" async for event in result.stream_events():\n",
|
|
" if event.type == \"raw_response_event\" and isinstance(event.data, ResponseTextDeltaEvent):\n",
|
|
" print(event.data.delta, end=\"\", flush=True)\n",
|
|
" current_text += event.data.delta\n",
|
|
"\n",
|
|
" # Check if it's time to run the guardrail check\n",
|
|
" if len(current_text) >= next_guardrail_check_len and not guardrail_task:\n",
|
|
" print(\"\\n[Running guardrail check]\")\n",
|
|
" guardrail_task = asyncio.create_task(check_guardrail(current_text))\n",
|
|
" next_guardrail_check_len += 300\n",
|
|
"\n",
|
|
" # Every iteration of the loop, check if the guardrail has been triggered\n",
|
|
" if guardrail_task and guardrail_task.done():\n",
|
|
" guardrail_result = guardrail_task.result()\n",
|
|
" if not guardrail_result.is_readable_by_ten_year_old:\n",
|
|
" print(\"\\n\\n================\\n\\n\")\n",
|
|
" print(f\"Guardrail triggered. Reasoning:\\n{guardrail_result.reasoning}\")\n",
|
|
" break\n",
|
|
" guardrail_task = None\n",
|
|
"\n",
|
|
" # Do one final check on the final output\n",
|
|
" if current_text:\n",
|
|
" guardrail_result = await check_guardrail(current_text)\n",
|
|
" if not guardrail_result.is_readable_by_ten_year_old:\n",
|
|
" print(\"\\n\\n================\\n\\n\")\n",
|
|
" print(f\"Final guardrail triggered. Reasoning:\\n{guardrail_result.reasoning}\")\n",
|
|
"\n",
|
|
"\n",
|
|
"# Run the demo\n",
|
|
"await run_streaming_guardrails_demo()\n",
|
|
"\n",
|
|
"# End the AgentOps trace session\n",
|
|
"agentops.end_trace(tracer, end_state=\"Success\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1f7f8dda",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Conclusion\n",
|
|
"\n",
|
|
"This notebook has demonstrated 9 key agent patterns that are commonly used in production AI applications. Each pattern showcases how agents can be orchestrated to perform complex tasks, validate inputs and outputs, and improve overall application performance. \n",
|
|
"\n",
|
|
"**AgentOps** provides comprehensive observability for AI agents, automatically tracking all these interactions and providing valuable insights into:\n",
|
|
"\n",
|
|
"- **Performance metrics** - Latency, throughput, success rates across all agent patterns\n",
|
|
"- **Cost analysis** - Token usage, model costs, and optimization opportunities for complex workflows\n",
|
|
"- **Quality monitoring** - Output quality assessment, guardrail effectiveness, and pattern success rates\n",
|
|
"- **Debugging support** - Detailed trace visualization, error tracking, and workflow analysis\n",
|
|
"- **Agent behavior insights** - Understanding how different patterns perform in production environments\n",
|
|
"- **Workflow optimization** - Identifying bottlenecks and improving agent coordination\n",
|
|
"\n",
|
|
"Visit [app.agentops.ai](https://app.agentops.ai) to explore your agent sessions and gain deeper insights into your AI application's behavior."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "agentops (3.11.11)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.11"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
} |