agentops/examples/xai/grok_vision_examples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XAI Vision Example\n",
    "This notebook demonstrates how to use XAI with AgentOps via the OpenAI python client. \n",
    "\n",
    "We are going to use the latest Grok model from XAI to create a program that will capture the text in an image and explain it. We will use AgentOps to track the program's performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's install the required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -U openai\n",
    "%pip install -U agentops"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then import them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "import agentops\n",
    "import os\n",
    "from dotenv import load_dotenv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll grab our API keys. You can use dotenv like below or however else you like to load environment variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "load_dotenv()\n",
    "os.environ[\"AGENTOPS_API_KEY\"] = os.getenv(\"AGENTOPS_API_KEY\", \"your_api_key_here\")\n",
    "os.environ[\"XAI_API_KEY\"] = os.getenv(\"XAI_API_KEY\", \"your_xai_api_key_here\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we initialize the AgentOps client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "agentops.init(auto_start_session=False)\n",
    "tracer = agentops.start_trace(trace_name=\"XAI Vision Example\", tags=[\"xai-example\", \"grok-vision\", \"agentops-example\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we are all set! Note the seesion url above. We will use it to track the program's performance.\n",
    "\n",
    "Let's initialize the OpenAI client with the XAI API key and base url."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = OpenAI(\n",
    "    base_url=\"https://api.x.ai/v1\",\n",
    "    api_key=os.getenv(\"XAI_API_KEY\", \"your_xai_api_key_here\"),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will set the system and instruction prompts for the program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SYSTEM_PROMPT = \"\"\"You are an expert image analysis assistant. When presented with an image, carefully examine and describe its contents in detail. \n",
    "\n",
    "For this task, your goal is to:\n",
    "1. Identify all key elements, objects, people, or text in the image\n",
    "2. Provide a comprehensive description of what you observe\n",
    "3. Explain the context or historical significance if applicable\n",
    "4. Describe the image in a clear, objective, and informative manner\n",
    "\n",
    "Please be precise, thorough, and focus on providing meaningful insights about the visual content.\"\"\"\n",
    "\n",
    "USER_PROMPT = [\n",
    "    {\"type\": \"text\", \"text\": \"Analyze the image and provide a detailed description of what you see.\"},\n",
    "    {\n",
    "        \"type\": \"image_url\",\n",
    "        \"image_url\": {\"url\": \"https://upload.wikimedia.org/wikipedia/commons/f/ff/First_Computer_Bug%2C_1945.jpg\"},\n",
    "    },\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will use the OpenAI client to process the image and generate a response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = client.chat.completions.create(\n",
    "    model=\"grok-2-vision-1212\",\n",
    "    messages=[{\"role\": \"system\", \"content\": SYSTEM_PROMPT}, {\"role\": \"user\", \"content\": USER_PROMPT}],\n",
    "    max_tokens=4096,\n",
    ")\n",
    "\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Awesome! It returns a fascinating response explaining the image and also deciphering the text content. All of this can be tracked with AgentOps by going to the session url above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "agentops.end_trace(tracer, end_state=\"Success\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We end the session with a success state and a success reason. This is useful if you want to track the success or failure of the chatbot. In that case you can set the end state to failure and provide a reason. By default the session will have an indeterminate end state."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "agentops (3.11.11)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}