diff --git a/docs/notebooks/small-object-detection-with-supervision.ipynb b/docs/notebooks/small-object-detection-with-supervision.ipynb index 382ab469..6d60f94e 100644 --- a/docs/notebooks/small-object-detection-with-supervision.ipynb +++ b/docs/notebooks/small-object-detection-with-supervision.ipynb @@ -1,877 +1,877 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "eXrIs38rQnO3" - }, - "source": [ - "# Detect Small Objects with `InferenceSlicer`\n", - "\n", - "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/roboflow/supervision/blob/develop/docs/notebooks/small-object-detection-with-sahi.ipynb)\n", - "[![Roboflow](https://raw.githubusercontent.com/roboflow-ai/notebooks/main/assets/badges/roboflow-blogpost.svg)](https://blog.roboflow.com/detect-small-objects/)\n", - "[![arXiv](https://img.shields.io/badge/arXiv-2401.17270-b31b1b.svg)](https://arxiv.org/abs/2202.06934)\n", - "\n", - "\n", - "This cookbook shows how to use [Slicing Aided Hyper Inference (SAHI) ](https://arxiv.org/abs/2202.06934) for small object detection with `supervision`.\n", - "\n", - "![\"Small Object Detection\"](https://raw.githubusercontent.com/ediardo/notebooks/main/sahi/animation.gif \"Small Object Detection\")\n", - "\n", - "Click the Open in Colab button to run the cookbook on Google Colab.\n", - "\n", - "### Before you start\n", - "\n", - "You'll need:\n", - "\n", - "- A free Roboflow account. Don't have one? [Create one here](https://app.roboflow.com/login).\n", - "- An API key from Roboflow. Need help getting one? [Learn more here](https://docs.roboflow.com/api-reference/authentication).\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "yrXlck1mSIu7" - }, - "source": [ - "## Install required packages\n", - "\n", - "Let's install the dependencies for this project. Here's a list of what\n", - "\n", - "\n", - "\n", - "* `inference`: a package by Roboflow for easy deployment of computer vision models.\n", - "* `supervision`: a package by Roboflow that provides utilities for building and managing computer vision applications. The `[assets]` keyword adds the optional dependencies.\n", - "* `opencv-python`: A library for computer vision tasks, including image processing and object detection.\n", - "* `numpy`: a core library for numerical computing with powerful array and matrix operations.\n", - "* `leafmap`: a tool for creating interactive visualizations.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# TODO: change to pip install supervision when https://github.com/roboflow/supervision/pull/1434 is released on 0.23.0\n", - "%pip install git+https://github.com/roboflow/supervision.git@develop" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "p74YnNv8vh8D", - "outputId": "a1edb3c4-fce2-4371-d1fa-7105b494738d", - "scrolled": true - }, - "outputs": [], - "source": [ - "%pip install inference opencv-python numpy leafmap" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "S3r-s602Eu0G" - }, - "source": [ - "## Crowd counting with Computer Vision\n", - "\n", - "How would you go about solving the problem of counting people in crowds? After some tests, I found that the best approach is to detect people\u2019s heads. Other body parts are likely occluded by other people, but heads are usually exposed, especially in aerial or high-level shots.\n", - "\n", - "### Using an Open-Source Public Model for People Detection\n", - "\n", - "Detecting people (or their heads) is a common problem that has been addressed by many researchers in the past. In this project, we\u2019ll use an open-source public dataset and a fine-tuned model to perform inference on images.\n", - "\n", - "![Roboflow Universe](https://raw.githubusercontent.com/ediardo/notebooks/main/sahi/roboflow_universe.png \"Open source model for counting people's heads\")\n", - "\n", - "Some details about the project [\"people_counterv0 Computer Vision Project\"](https://universe.roboflow.com/sit-cx0ng/people_counterv0) hosted on Roboflow Universe:\n", - "\n", - "- Dataset of 4,574 images\n", - "- mAP=49.2% / Precision=74.5% / Recall=39.2\n", - "- Model: Roboflow 2.0 Object Detection (fast)\n", - "- Checkpoint: COCOv6n\n", - "- Created by: [SIT](https://universe.roboflow.com/sit-cx0ng)\n", - "\n", - "### Download the image\n", - "\n", - "Run the code below to download the image we'll use in this cookbook." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import requests\n", - "import matplotlib.pyplot as plt\n", - "import cv2\n", - "import supervision as sv\n", - "\n", - "def download_image(url):\n", - " target_name = \"human_tower.jpg\" \n", - " try:\n", - " print(\"Downloading image from wikimedia.org...\")\n", - " response = requests.get(url, headers={'User-Agent': 'YourCustomUserAgent/1.0'})\n", - " response.raise_for_status() # Raises an error for bad responses\n", - " \n", - " with open(target_name, \"wb\") as file:\n", - " file.write(response.content)\n", - " print(\"Download complete\")\n", - " return target_name\n", - " \n", - " except requests.exceptions.RequestException as e:\n", - " print(f\"Failed to download the image: {e}\")\n", - " return None\n", - "\n", - "image_url = \"https://upload.wikimedia.org/wikipedia/commons/thumb/d/d0/4_de_8_amb_l%27agulla_carregat_Castellers_de_Barcelona_%2821937141066%29.jpg/2560px-4_de_8_amb_l%27agulla_carregat_Castellers_de_Barcelona_%2821937141066%29.jpg\"\n", - "image_path = download_image(image_url)\n", - "\n", - "if image_path is None:\n", - " print(\"Could not download the image...\")\n", - "else:\n", - " image = cv2.imread(image_path)\n", - " image_wh = (image.shape[1], image.shape[0])\n", - " print(f\"Image shape: {image_wh[0]}w x {image_wh[1]}h\")\n", - " sv.plot_image(image)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You're looking at a Castell, a human tower traditionally built at festivals in parts of Catalonia, Spain, and has since spread to the Balearic Islands and the Valencian Community. The source of the image is [here](https://commons.wikimedia.org/wiki/File:4_de_8_amb_l%27agulla_carregat_Castellers_de_Barcelona_(21937141066).jpg), and you could learn more about these human towers in [Wikipedia](https://en.wikipedia.org/wiki/Castell)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "vrXJTlUXxXFN" - }, - "source": [ - "\n", - "## Let's try our model's performance\n", - "\n", - "Before we dive into the SAHI technique for small object detection, it\u2019s useful to see how a fine-tuned model performs with the image as is\u2014without any pre-processing or slicing. The goal is to understand when the model starts to fail so that we can progressively move towards an efficient slicing strategy.\n", - "\n", - "Let\u2019s run the model!" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 670 - }, - "id": "gCYNvD8syhxE", - "outputId": "aa2632c6-6a6d-47f3-874d-692080d9afa6" - }, - "outputs": [], - "source": [ - "from inference_sdk import InferenceHTTPClient\n", - "import supervision as sv\n", - "import os\n", - "\n", - "MODEL_ID = \"people_counterv0/1\"\n", - "API_KEY=os.environ[\"ROBOFLOW_API_KEY\"] # <---Set you API key here\n", - "\n", - " # <-- Set your API key here. For more information on how to retrieve your API key, visit: https://docs.roboflow.com/api-reference/authentication\n", - "# from google.colab import userdata\n", - "# API_KEY = userdata.get(\"ROBOFLOW_API_KEY\") \n", - "\n", - "client = InferenceHTTPClient(\n", - " api_url=\"https://detect.roboflow.com\",\n", - " api_key=API_KEY,\n", - ")\n", - "\n", - "# Run inference\n", - "results = client.infer(image_path, model_id=MODEL_ID)\n", - "detections = sv.Detections.from_inference(results)\n", - "\n", - "print(f\"Inference time: {results['time']} seconds\")\n", - "print(f\"Found {len(detections)} people\")\n", - "\n", - "COLOR_BBOX_PEOPLE=sv.ColorPalette.DEFAULT.colors[6]\n", - "\n", - "bbox_annotator = sv.BoxAnnotator(\n", - " color=COLOR_BBOX_PEOPLE,\n", - " thickness=2\n", - ")\n", - "\n", - "# Annotate our image with detections.\n", - "image_no_sahi = bbox_annotator.annotate(scene=image.copy(), detections=detections)\n", - "\n", - "sv.plot_image(image_no_sahi)\n", - "cv2.imwrite(\"1x1.jpg\", image_no_sahi)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The model shows strong performance in detecting people in the lower half of the image, but it struggles to accurately predict boxes in the upper half. This suggests two key insights: first, the model is proficient at identifying people\u2019s heads from various angles, and second, using SAHI could effectively address the detection challenges in the upper portion of the image. Now, it\u2019s time to try SAHI!" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ufFqhE5q3GSQ" - }, - "source": [ - "## Using SAHI for small object detection\n", - "\n", - "Detecting small objects with CNNs can be tricky because small pixel groups provide fewer features for deeper layers, making details easy to miss. SAHI addresses this by:\n", - "\n", - "1. Image Slicing: Dividing the high-resolution image into smaller, overlapping tiles, allowing the model to \u201czoom in\u201d on details.\n", - "2. Hyper Inference: Aggregating results from all slices into a coherent set of detections.\n", - "\n", - "By using SAHI, models like YOLOv8 can greatly improve small object detection, overcoming the usual challenges with feature representation and resolution.\n", - "\n", - "![SAHI](https://raw.githubusercontent.com/obss/sahi/main/resources/sliced_inference.gif \"something\")\n", - "\n", - "SAHI can be seen as a framework for addressing the small object detection problem. If you\u2019re interested in learning more about the motivations behind this solution, you can read the paper [Slicing Aided Hyper Inference and Fine-tuning for Small Object Detection](https://arxiv.org/abs/2202.06934).\n", - "\n", - "## Slicing our image with `supervision`\n", - "\n", - "Let\u2019s begin by visualizing how these tiles would appear on our image. I\u2019ll start with a small set of 2x2 tiles, with a zero overlap both vertically (height) and horizontally (width) between the tiles. The final values of these parameters will ultimately depend on your use case, so trial and error is encouraged!\n", - "\n", - "Some of the methods below are for visualizing the tiles and overlapping. You'll only need the `calculate_tile_size` method in your application to calculate the size of the tiles.\n", - "\n", - "### Utility functions for visualizing tiles" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 - }, - "id": "HpVjry6m3F0-", - "outputId": "ffdb41d8-7e6b-45bb-ab5b-1cb9dfb6c377" - }, - "outputs": [], - "source": [ - "import numpy as np\n", - "from inference import get_model\n", - "import math\n", - "\n", - "def tile_image(image_shape, slice_wh, overlap_wh)-> np.ndarray:\n", - " \"\"\"\n", - " Computes the coordinates and dimensions of tiles for an image with specified slicing and overlap parameters.\n", - " \n", - " Parameters:\n", - " -----------\n", - " image_shape : tuple of int\n", - " The shape of the image to be tiled, specified as (width, height).\n", - " \n", - " slice_wh : tuple of int\n", - " The dimensions (width, height) of each tile.\n", - " \n", - " overlap_wh : tuple of int\n", - " The overlap dimensions (width, height) between adjacent tiles.\n", - " \n", - " Returns:\n", - " --------\n", - " np.ndarray\n", - " An array of offsets where each row represents a tile's bounding box \n", - " in the format (x1, y1, x2, y2). The coordinates are rounded up to the nearest integer.\n", - " \n", - " \"\"\"\n", - " offsets = sv.InferenceSlicer._generate_offset(\n", - " resolution_wh=image_shape,\n", - " slice_wh=slice_wh,\n", - " overlap_ratio_wh=None,\n", - " overlap_wh=overlap_wh\n", - " )\n", - " \n", - " offsets = np.ceil(offsets).astype(int)\n", - " \n", - " return offsets\n", - "\n", - "def draw_transparent_tiles(scene: np.ndarray, x: int, y: int, w:int, h:int, index: int = None):\n", - " \"\"\"\n", - " Draws a transparent tile with an optional index label on the given scene.\n", - "\n", - " Parameters:\n", - " -----------\n", - " scene : np.ndarray\n", - " The input image on which the transparent tile will be drawn. It should be a NumPy array representing the image.\n", - "\n", - " x : int\n", - " The x-coordinate of the top-left corner of the tile.\n", - "\n", - " y : int\n", - " The y-coordinate of the top-left corner of the tile.\n", - "\n", - " w : int\n", - " The width of the tile.\n", - "\n", - " h : int\n", - " The height of the tile.\n", - "\n", - " index : int, optional, default=None\n", - " If provided, draws the index number in the center of the tile. If None, no index number is drawn.\n", - "\n", - " Returns:\n", - " --------\n", - " np.ndarray\n", - " A new image with a transparent tile and optional index label drawn on it.\n", - " \"\"\"\n", - " alpha=0.18\n", - " overlay_image = scene.copy()\n", - " \n", - " # Generate a mask for the tile\n", - " rectangle = np.zeros((h, w, 3), dtype=np.uint8)\n", - " rectangle.fill(255)\n", - " \n", - " rect = sv.Rect(x=x, y=y, width=w, height=h)\n", - " overlay_image = sv.draw_image(scene=overlay_image, image=rectangle, opacity=alpha, rect=rect)\n", - " \n", - " # Draw a border around the edge of the mask\n", - " border_color = sv.Color.BLACK\n", - " border_thickness=2\n", - " overlay_image = sv.draw_rectangle(scene=overlay_image,rect=sv.Rect(x=x, y=y, width=w, height=h), color=border_color, thickness=border_thickness)\n", - " \n", - " # Draw index number on the center of the tile after blending\n", - " if index is not None:\n", - " text_anchor = sv.Point(x=x + w // 2, y=y + h // 2)\n", - " # Calculate the center of the rectangle\n", - " center_x = rect.x + rect.width // 2\n", - " center_y = rect.y + rect.height // 2\n", - " \n", - " # Define the text anchor point as the center of the rectangle\n", - " text_anchor = sv.Point(x=center_x, y=center_y)\n", - " text_scale=3\n", - " text_thickness=5\n", - " text=str(index + 1)\n", - " text_color=sv.Color.BLACK\n", - " # Draw the text on the scene\n", - " overlay_image = sv.draw_text(overlay_image, text, text_anchor, text_color, text_scale, text_thickness)\n", - " \n", - " return overlay_image\n", - "\n", - "def draw_tiles(scene: np.ndarray, offsets, show_index=False):\n", - " \"\"\"\n", - " Draws transparent tiles on a scene based on the given offsets.\n", - "\n", - " Parameters:\n", - " -----------\n", - " scene : np.ndarray\n", - " The input image on which tiles will be drawn. It should be a NumPy array representing the image.\n", - " \n", - " offsets : list of tuples\n", - " A list of tuples where each tuple represents the bounding box of a tile in the format (x1, y1, x2, y2).\n", - " - (x1, y1) is the top-left corner of the tile.\n", - " - (x2, y2) is the bottom-right corner of the tile.\n", - "\n", - " show_index : bool, optional, default=False\n", - " If True, draws the index number of each tile on the scene. Otherwise, the tiles are drawn without indices.\n", - "\n", - " Returns:\n", - " --------\n", - " np.ndarray\n", - " A new image with transparent tiles drawn on it.\n", - "\n", - " \"\"\"\n", - " tiled_image = scene.copy()\n", - " \n", - " for index, offset in enumerate(offsets):\n", - " x = offset[0]\n", - " y = offset[1]\n", - " width = offset[2] - x\n", - " height = offset[3] - y\n", - " \n", - " if (show_index):\n", - " tiled_image = draw_transparent_tiles(scene=tiled_image, x=x, y=y, w=width, h=height, index=index)\n", - " else:\n", - " tiled_image = draw_transparent_tiles(scene=tiled_image, x=x, y=y, w=width, h=height, index=None)\n", - " \n", - " return tiled_image\n", - "\n", - "def print_offsets(offsets):\n", - " for index, offset in enumerate(offsets):\n", - " w = (offset[2] - offset[0])\n", - " h = (offset[3] - offset[1])\n", - " print(f\"Tile {index + 1}\")\n", - " print(f\" w={w}, h={h}, x1={offset[0]}, y1={offset[1]}, x2={offset[2]}, y2={offset[3]}, area={w*h}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 - }, - "id": "HpVjry6m3F0-", - "outputId": "ffdb41d8-7e6b-45bb-ab5b-1cb9dfb6c377" - }, - "source": [ - "### Calculate Tile Size\n", - "\n", - "The `calculate_tile_size` function computes the dimensions of each tile when dividing an image into a grid, considering both the image\u2019s size and a specified tiling strategy. It takes the image dimensions (`image_shape`), the number of rows and columns for the grid (`tiles`), and an optional overlap ratio (`overlap_ratio_wh`) to determine how much adjacent tiles overlap. The function returns the adjusted tile size, including the overlap, and the specific overlap dimensions.\n", - "\n", - "**NOTE**: As of `supervision==0.22.0` you need to provide the tile size. This function calculates it for you." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 - }, - "id": "HpVjry6m3F0-", - "outputId": "ffdb41d8-7e6b-45bb-ab5b-1cb9dfb6c377" - }, - "outputs": [], - "source": [ - "def calculate_tile_size(image_shape: tuple[int, int], tiles: tuple[int, int], overlap_ratio_wh: tuple[float, float] = (0.0, 0.0)):\n", - " \"\"\"\n", - " Calculate the size of the tiles based on the image shape, the number of tiles, and the overlap ratio.\n", - "\n", - " Parameters:\n", - " ----------\n", - " image_shape : tuple[int, int]\n", - " The dimensions of the image as (width, height).\n", - " \n", - " tiles : tuple[int, int]\n", - " The tiling strategy defined as (rows, columns), specifying the number of tiles along the height and width of the image.\n", - " \n", - " overlap_ratio_wh : tuple[float, float], optional\n", - " The overlap ratio for width and height as (overlap_ratio_w, overlap_ratio_h). This defines the fraction of overlap between adjacent tiles. Default is (0.0, 0.0), meaning no overlap.\n", - "\n", - " Returns:\n", - " -------\n", - " tuple[tuple[int, int], tuple[int, int]]\n", - " A tuple containing:\n", - " - The size of each tile as (tile_width, tile_height), accounting for overlap.\n", - " - The overlap dimensions as (overlap_width, overlap_height).\n", - "\n", - " Example:\n", - " -------\n", - " >>> image_shape = (1024, 768)\n", - " >>> tiles = (4, 4)\n", - " >>> overlap_ratio_wh = (0.15, 0.15)\n", - " >>> calculate_tile_size(image_shape, tiles, overlap_ratio_wh)\n", - " ((295, 221), (39, 29))\n", - "\n", - " Notes:\n", - " -----\n", - " - The function calculates the width and height of each tile based on the number of rows and columns specified.\n", - " - It then applies the overlap ratio to adjust the tile size, ensuring that tiles overlap by a specified fraction.\n", - " - The overlap dimensions are rounded up to the nearest integer to ensure complete coverage.\n", - " \"\"\"\n", - "\n", - " w, h = image_shape\n", - " rows, columns = tiles\n", - " \n", - " tile_width = (w / columns)\n", - " tile_height = (h / rows)\n", - " \n", - " overlap_ratio_w, overlap_ratio_h = overlap_ratio_wh\n", - " overlap_wh = (math.ceil(tile_width * overlap_ratio_w), math.ceil(tile_height * overlap_ratio_h))\n", - " \n", - " tile_width = math.ceil(tile_width + overlap_wh[0])\n", - " tile_height = math.ceil(tile_height + overlap_wh[1])\n", - " tile_size = (tile_width, tile_height)\n", - " \n", - " return tile_size, overlap_wh" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 - }, - "id": "HpVjry6m3F0-", - "outputId": "ffdb41d8-7e6b-45bb-ab5b-1cb9dfb6c377" - }, - "outputs": [], - "source": [ - " \n", - "tiles = (2,2) # The number of tiles you want\n", - "overlap_ratio_wh = (0.0, 0.0) # The overlap between tiles\n", - "slice_wh, overlap_wh = calculate_tile_size(image_wh, tiles, overlap_ratio_wh)\n", - "offsets = tile_image(image_wh, slice_wh, overlap_wh)\n", - "\n", - "print(f\"Image shape: {image_wh[0]}w x {image_wh[1]}h\")\n", - "print(f\"Tiles: {tiles}\")\n", - "print(slice_wh)\n", - "print(f\"Generated {len(offsets)} tiles. These are the calculated dimensions\")\n", - "print_offsets(offsets)\n", - "\n", - "tiled_image = draw_tiles(scene=image.copy(), offsets=offsets, show_index=True)\n", - "\n", - "sv.plot_image(tiled_image)\n", - "# cv2.imwrite(\"2x2.jpg\", image_no_sahi)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "pqHWZhWUawP3" - }, - "source": [ - "You can see that the image has been sliced into four different tiles. Next, each tile will be independently processed by the model, and supervision will merge all the predictions into a coherent set of detections. Notice that we're not using overlapping in at this time (more on that later).\n", - "\n", - "## Run Inference on a Sliced Image With `supervision`\n", - "\n", - "Running inference on slices of your image is easy with the class `InferenceSlicer` from [Supervision](https://supervision.roboflow.com/latest/detection/tools/inference_slicer/#inferenceslicer). This API from Roboflow divides a larger image into smaller slices, performs inference on each slice, and then merges the detections into a single `detections` object.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 - }, - "id": "Zpd-NT-m1-ul", - "outputId": "d85c72be-edab-4e2e-e5ef-9ab02047c66e", - "scrolled": true - }, - "outputs": [], - "source": [ - "import time\n", - "import leafmap\n", - "\n", - "def callback(image_slice: np.ndarray) -> sv.Detections:\n", - " result = get_model(model_id=MODEL_ID, api_key=API_KEY).infer(image_slice )[0]\n", - " return sv.Detections.from_inference(result)\n", - "\n", - "tiles = (2,2) # The number of tiles you want\n", - "overlap_ratio_wh = (0.0, 0.0) # The overlap between tiles\n", - "slice_wh, overlap_wh = calculate_tile_size(image_wh, tiles, overlap_ratio_wh)\n", - "offsets = tile_image(image_wh, slice_wh, overlap_wh)\n", - "\n", - "slicer = sv.InferenceSlicer(\n", - " callback=callback,\n", - " slice_wh=slice_wh,\n", - " overlap_ratio_wh=None,\n", - " overlap_wh=overlap_wh,\n", - " thread_workers=4\n", - ")\n", - "\n", - "start_time = time.time()\n", - "detections = slicer(image)\n", - "end_time = time.time()\n", - "elapsed_time = end_time - start_time\n", - "\n", - "print(f\"Image shape: {image_wh[0]}w x {image_wh[1]}h\")\n", - "print(f\"Tile size: {slice_wh[0]}w x {image_wh[1]}\")\n", - "print(f\"Overlap: {overlap_wh[0]}w x {overlap_wh[1]}h. Ratio {overlap_ratio_wh}\")\n", - "print(f\"Found {len(detections)} people\")\n", - "print(f\"Inference time: {elapsed_time} seconds\")\n", - "\n", - "tiled_image_with_sahi_2x2 = draw_tiles(scene=image.copy(), offsets=offsets, show_index=True)\n", - "tiled_image_with_sahi_2x2 = bbox_annotator.annotate(scene=tiled_image_with_sahi_2x2, detections=detections)\n", - "\n", - "# For visualization\n", - "leafmap.image_comparison(\n", - " img1=cv2.cvtColor(tiled_image_with_sahi_2x2, cv2.COLOR_BGR2RGB),\n", - " img2=cv2.cvtColor(image_no_sahi, cv2.COLOR_BGR2RGB),\n", - " label1=\"Slicing 2x2\",\n", - " label2=\"No Slicing\",\n", - " starting_position=50,\n", - ")\n", - "\n", - "#cv2.imwrite(\"2x2_no_overlapping.jpg\", bbox_annotator.annotate(scene=image.copy(), detections=detections))\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "W6TvNnXpewwc" - }, - "source": [ - "Great! We\u2019ve detected 726 people, up from the 185 we initially detected without image slicing. The model is still detecting people from different angles, but it continues to struggle with detecting people located in the farther parts of the plaza. It\u2019s time to increase the number of tiles\u2014in other words, zoom in so the model can capture more details of the small heads of people.\n", - "\n", - "![Missing detections](https://raw.githubusercontent.com/ediardo/notebooks/main/sahi/detections.png)\n", - "\n", - "### Increasing Tile Density: Moving to a 5x5 Grid\n", - "\n", - "Now that we\u2019ve seen improvements with a 2x2 grid, it\u2019s time to push the model further. By increasing the number of tiles to a 5x5 grid, we effectively zoom in on the image, allowing the model to capture finer details, such as smaller and more distant features that might have been missed before. This approach will help us understand how well the model performs with even more zoomed-in images. Let\u2019s explore how this change affects our detection accuracy and overall performance." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import time\n", - "\n", - "def callback(image_slice: np.ndarray) -> sv.Detections:\n", - " result = get_model(model_id=MODEL_ID, api_key=API_KEY).infer(image_slice )[0]\n", - " return sv.Detections.from_inference(result)\n", - "\n", - "tiles = (5,5) # The number of tiles you want\n", - "overlap_ratio_wh = (0.0, 0.0) # The overlap between tiles\n", - "slice_wh, overlap_wh = calculate_tile_size(image_wh, tiles, overlap_ratio_wh)\n", - "offsets = tile_image(image_wh, slice_wh, overlap_wh)\n", - "\n", - "slicer = sv.InferenceSlicer(\n", - " callback=callback,\n", - " slice_wh=slice_wh,\n", - " overlap_wh=overlap_wh,\n", - " overlap_ratio_wh=None,\n", - " thread_workers=4\n", - ")\n", - "\n", - "start_time = time.time()\n", - "detections = slicer(image)\n", - "end_time = time.time()\n", - "elapsed_time = end_time - start_time\n", - "\n", - "print(f\"Image shape: {image_wh[0]}w x {image_wh[1]}h\")\n", - "print(f\"Tiles: {tiles}\")\n", - "print(f\"Tile size: {slice_wh[0]}w x {image_wh[1]}\")\n", - "print(f\"Overlap: {overlap_wh[0]}w x {overlap_wh[1]}h. Ratio {overlap_ratio_wh}\")\n", - "print(f\"Overlap filter: {sv.OverlapFilter.NON_MAX_SUPPRESSION}\")\n", - "print(f\"Found {len(detections)} people\")\n", - "print(f\"Inference time: {elapsed_time} seconds\")\n", - "\n", - "tiled_image_with_sahi_5x5 = draw_tiles(scene=image.copy(), offsets=offsets, show_index=True)\n", - "tiled_image_with_sahi_5x5 = bbox_annotator.annotate(scene=tiled_image_with_sahi_5x5, detections=detections)\n", - "\n", - "\n", - "# For visualization\n", - "leafmap.image_comparison(\n", - " img1=cv2.cvtColor(tiled_image_with_sahi_5x5, cv2.COLOR_BGR2RGB),\n", - " img2=cv2.cvtColor(tiled_image_with_sahi_2x2, cv2.COLOR_BGR2RGB),\n", - " label1=\"Slicing 5x5\",\n", - " label2=\"Slicing 2x2\",\n", - " starting_position=50,\n", - ")\n", - "\n", - "#cv2.imwrite(\"5x5_no_overlapping.jpg\", bbox_annotator.annotate(scene=image.copy(), detections=detections))\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "W6TvNnXpewwc" - }, - "source": [ - "We\u2019ve just detected 1,494 people using a 25-tile grid (5 rows x 5 columns), a significant increase from the 726 people detected with the 4-tile (2x2) grid. However, as we increase the number of tiles, a new challenge arises: duplicate detections or missed detections along the edges of the tiles. This issue becomes evident in these examples, where overlapping or gaps between tiles lead to inaccuracies in our model\u2019s detection.\n", - "\n", - "| Example| Observations |\n", - "|----|----| \n", - "| ![Overlapping](https://github.com/ediardo/notebooks/blob/main/sahi/overlapping_1.png?raw=true \"Overlapping\") | False Negative, Incomplete bbox | \n", - "| ![Overlapping](https://raw.githubusercontent.com/ediardo/notebooks/main/sahi/overlapping_2.png \"Overlapping\")| Double detection, Incomplete bbox|\n", - "| ![Overlapping](https://raw.githubusercontent.com/ediardo/notebooks/main/sahi/overlapping_3.png \"Overlapping\")| Incomplete bounding box|\n", - "\n", - "## Improving Object Detection Near Boundaries with Overlapping\n", - "\n", - "When objects, like people, appear at the edges of tiles, they might be detected twice or missed entirely if they span across two tiles. This can lead to inaccurate detection results. To solve this, we use overlapping tiles, allowing the model to see parts of adjacent tiles simultaneously. This overlap helps ensure that objects near the boundaries are fully captured, reducing duplicates and improving accuracy.\n", - "\n", - "We\u2019ll set the overlap ratio to 15% on the tile\u2019s width and height. This overlap helps ensure that objects near the boundaries are fully captured, reducing duplicates and improving accuracy." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 83 - }, - "id": "ip3ohW0fezQo", - "outputId": "b233a920-0413-4307-91c5-fae7d7e90017", - "scrolled": true - }, - "outputs": [], - "source": [ - "tiles = (5,5) # The number of tiles you want\n", - "overlap_ratio_wh = (0.05, 0.05) # Ratio of overlapping, width/height\n", - "slice_wh, overlap_wh = calculate_tile_size(image_wh, tiles, overlap_ratio_wh)\n", - "offsets = tile_image(image_wh, slice_wh, overlap_wh)\n", - "\n", - "slicer = sv.InferenceSlicer(\n", - " callback=callback,\n", - " #overlap_filter=sv.OverlapFilter.NON_MAX_MERGE,\n", - " overlap_filter=sv.OverlapFilter.NON_MAX_SUPPRESSION,\n", - " iou_threshold=0.75,\n", - " slice_wh=slice_wh,\n", - " overlap_ratio_wh=None,\n", - " overlap_wh=overlap_wh,\n", - " thread_workers=4\n", - ")\n", - "start_time = time.time()\n", - "detections = slicer(image)\n", - "end_time = time.time()\n", - "elapsed_time = end_time - start_time\n", - "\n", - "print(f\"Image shape: {image_wh[0]}w x {image_wh[1]}h\")\n", - "print(f\"Tiles: {tiles}\")\n", - "print(f\"Tile size: {slice_wh[0]}w x {image_wh[1]}\")\n", - "print(f\"Overlap: {overlap_wh[0]}w x {overlap_wh[1]}h. Ratio {overlap_ratio_wh}\")\n", - "print(f\"Overlap Filter: {sv.OverlapFilter.NON_MAX_SUPPRESSION}\")\n", - "print(f\"Found {len(detections)} people\")\n", - "print(f\"Inference time: {elapsed_time} seconds\")\n", - "#print_offsets(offsets)\n", - "\n", - "tiled_image_with_overlapping_nms_5x5 = draw_tiles(scene=image.copy(), offsets=offsets, show_index=True)\n", - "tiled_image_with_overlapping_nms_5x5 = bbox_annotator.annotate(scene=tiled_image_with_overlapping_nms_5x5, detections=detections)\n", - "\n", - "leafmap.image_comparison(\n", - " img1=cv2.cvtColor(tiled_image_with_overlapping_nms_5x5, cv2.COLOR_BGR2RGB),\n", - " img2=cv2.cvtColor(tiled_image_with_sahi_5x5, cv2.COLOR_BGR2RGB),\n", - " label1=\"Tiles with overlapping\",\n", - " label2=\"Tiles with no overlapping\",\n", - " starting_position=50,\n", - " make_responsive=True\n", - ")\n", - "\n", - "#cv2.imwrite(\"5x5_with_overlapping_nms.jpg\", bbox_annotator.annotate(scene=image.copy(), detections=detections))" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "_RXyuDMjmWGv" - }, - "source": [ - "## Non-Max Supression vs Non-Max Merge \n", - "\n", - "When dealing with overlapping detections, it\u2019s essential to determine which detections represent the same object and which are unique. Non-Maximum Suppression (NMS) and Non-Maximum Merging (NMM) are two techniques commonly used to address this challenge. NMS works by eliminating redundant detections based on confidence scores, while NMM combines overlapping detections to enhance the representation of objects spanning multiple tiles. Understanding the difference between these methods helps optimize object detection, particularly near tile boundaries.\n", - "\n", - "In `supervision`, the `overlap_filter` parameter allows us to specify the strategy for handling overlapping detections in slices. This parameter can take on two values:\n", - "\n", - "- `sv.OverlapFilter.NON_MAX_SUPRESSION` (default): Eliminates redundant detections by keeping the one with the highest confidence score.\n", - "- `sv.OverlapFilter.NON_MAX_MERGE`: Combines overlapping detections to create a more comprehensive representation of objects spanning multiple tiles.\n", - "\n", - "It\u2019s important to note that this method is not perfect and may require further testing and fine-tuning to achieve optimal results in various use cases. You should validate the outputs and adjust parameters as needed to handle specific scenarios effectively." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 - }, - "id": "Ou0vF3-kmYoA", - "outputId": "3367ee55-dda7-48e4-e23b-053ecda6bcff", - "scrolled": true - }, - "outputs": [], - "source": [ - "tiles = (5,5) # The number of tiles you want\n", - "overlap_ratio_wh = (0.05, 0.05)\n", - "slice_wh, overlap_wh = calculate_tile_size(image_wh, tiles, overlap_ratio_wh)\n", - "offsets = tile_image(image_wh, slice_wh, overlap_wh)\n", - "slicer = sv.InferenceSlicer(\n", - " callback=callback,\n", - " overlap_filter=sv.OverlapFilter.NON_MAX_MERGE,\n", - " #overlap_filter=sv.OverlapFilter.NON_MAX_SUPPRESSION,\n", - " iou_threshold=0.75,\n", - " slice_wh=slice_wh,\n", - " overlap_ratio_wh=None,\n", - " overlap_wh=overlap_wh,\n", - " thread_workers=4\n", - ")\n", - "start_time = time.time()\n", - "detections = slicer(image)\n", - "end_time = time.time()\n", - "elapsed_time = end_time - start_time\n", - "\n", - "print(f\"Image shape: {image_wh[0]}w x {image_wh[1]}h\")\n", - "print(f\"Tile size: {slice_wh[0]}w x {image_wh[1]}\")\n", - "print(f\"Overlap: {overlap_wh[0]}w x {overlap_wh[1]}h. Ratio {overlap_ratio_wh}\")\n", - "print(f\"Overlap Filter: {sv.OverlapFilter.NON_MAX_MERGE}\")\n", - "print(f\"Found {len(detections)} people\")\n", - "print(f\"Inference time: {elapsed_time} seconds\")\n", - "#print_offsets(offsets)\n", - "\n", - "tiled_image_with_overlapping_nmm_5x5 = draw_tiles(scene=image.copy(), offsets=offsets, show_index=True)\n", - "tiled_image_with_overlapping_nmm_5x5 = bbox_annotator.annotate(scene=tiled_image_with_overlapping_nmm_5x5, detections=detections)\n", - "\n", - "\n", - "leafmap.image_comparison(\n", - " img1=cv2.cvtColor(tiled_image_with_overlapping_nmm_5x5, cv2.COLOR_BGR2RGB),\n", - " img2=cv2.cvtColor(tiled_image_with_overlapping_nms_5x5, cv2.COLOR_BGR2RGB),\n", - " label1=\"Overlapping with Non-Max Merging\",\n", - " label2=\"Overlapping with Non-Max Supression\",\n", - " starting_position=50,\n", - " make_responsive=True\n", - ")\n", - "\n", - "# cv2.imwrite(\"5x5_with_overlapping_nmm.jpg\", bbox_annotator.annotate(scene=image.copy(), detections=detections))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Conclusion\n", - "\n", - "In this cookbook, we\u2019ve explored the advantages of using the SAHI technique for enhancing small object detection and the importance of experimenting with various tiling strategies to effectively zoom into images. By combining these approaches, we can improve the accuracy and reliability of object detection models, particularly in challenging scenarios where objects are small or located near the boundaries of tiles. These methods offer practical solutions to common challenges in computer vision, empowering developers to build more robust and precise detection systems.\n", - "\n", - "![\"Crowd Detection\"](https://raw.githubusercontent.com/ediardo/notebooks/main/sahi/5x5_with_overlapping_nmm.jpg \"Crowd Detection\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## More resources\n", - "\n", - "- `InferenceSlicer`: https://supervision.roboflow.com/detection/tools/inference_slicer/\n", - "- Detect Small Objects https://supervision.roboflow.com/latest/how_to/detect_small_objects/\n", - "- What is Non-Max Merging?: https://blog.roboflow.com/non-max-merging/\n", - "- How to Detect Small Objects: A Guide https://blog.roboflow.com/detect-small-objects/\n", - "- How to Use SAHI to Detect Small Objects: https://blog.roboflow.com/how-to-use-sahi-to-detect-small-objects/\n", - "- SAHI paper: https://arxiv.org/abs/2202.06934\n", - "- C4W3L07 Nonmax Suppression, Andrew Ng: https://www.youtube.com/watch?v=VAo84c1hQX8" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "colab": { - "machine_shape": "hm", - "provenance": [] - }, - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.11.6" - } + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "eXrIs38rQnO3" + }, + "source": [ + "# Detect Small Objects with `InferenceSlicer`\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/roboflow/supervision/blob/develop/docs/notebooks/small-object-detection-with-sahi.ipynb)\n", + "[![Roboflow](https://raw.githubusercontent.com/roboflow-ai/notebooks/main/assets/badges/roboflow-blogpost.svg)](https://blog.roboflow.com/detect-small-objects/)\n", + "[![arXiv](https://img.shields.io/badge/arXiv-2401.17270-b31b1b.svg)](https://arxiv.org/abs/2202.06934)\n", + "\n", + "\n", + "This cookbook shows how to use [Slicing Aided Hyper Inference (SAHI) ](https://arxiv.org/abs/2202.06934) for small object detection with `supervision`.\n", + "\n", + "![\"Small Object Detection\"](https://raw.githubusercontent.com/ediardo/notebooks/main/sahi/animation.gif \"Small Object Detection\")\n", + "\n", + "Click the Open in Colab button to run the cookbook on Google Colab.\n", + "\n", + "### Before you start\n", + "\n", + "You'll need:\n", + "\n", + "- A free Roboflow account. Don't have one? [Create one here](https://app.roboflow.com/login).\n", + "- An API key from Roboflow. Need help getting one? [Learn more here](https://docs.roboflow.com/api-reference/authentication).\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yrXlck1mSIu7" + }, + "source": [ + "## Install required packages\n", + "\n", + "Let's install the dependencies for this project. Here's a list of what\n", + "\n", + "\n", + "\n", + "* `inference`: a package by Roboflow for easy deployment of computer vision models.\n", + "* `supervision`: a package by Roboflow that provides utilities for building and managing computer vision applications. The `[assets]` keyword adds the optional dependencies.\n", + "* `opencv-python`: A library for computer vision tasks, including image processing and object detection.\n", + "* `numpy`: a core library for numerical computing with powerful array and matrix operations.\n", + "* `leafmap`: a tool for creating interactive visualizations.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# TODO: change to pip install supervision when https://github.com/roboflow/supervision/pull/1434 is released on 0.23.0\n", + "%pip install git+https://github.com/roboflow/supervision.git@develop" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, - "nbformat": 4, - "nbformat_minor": 4 + "id": "p74YnNv8vh8D", + "outputId": "a1edb3c4-fce2-4371-d1fa-7105b494738d", + "scrolled": true + }, + "outputs": [], + "source": [ + "%pip install inference opencv-python numpy leafmap" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "S3r-s602Eu0G" + }, + "source": [ + "## Crowd counting with Computer Vision\n", + "\n", + "How would you go about solving the problem of counting people in crowds? After some tests, I found that the best approach is to detect people’s heads. Other body parts are likely occluded by other people, but heads are usually exposed, especially in aerial or high-level shots.\n", + "\n", + "### Using an Open-Source Public Model for People Detection\n", + "\n", + "Detecting people (or their heads) is a common problem that has been addressed by many researchers in the past. In this project, we’ll use an open-source public dataset and a fine-tuned model to perform inference on images.\n", + "\n", + "![Roboflow Universe](https://raw.githubusercontent.com/ediardo/notebooks/main/sahi/roboflow_universe.png \"Open source model for counting people's heads\")\n", + "\n", + "Some details about the project [\"people_counterv0 Computer Vision Project\"](https://universe.roboflow.com/sit-cx0ng/people_counterv0) hosted on Roboflow Universe:\n", + "\n", + "- Dataset of 4,574 images\n", + "- mAP=49.2% / Precision=74.5% / Recall=39.2\n", + "- Model: Roboflow 2.0 Object Detection (fast)\n", + "- Checkpoint: COCOv6n\n", + "- Created by: [SIT](https://universe.roboflow.com/sit-cx0ng)\n", + "\n", + "### Download the image\n", + "\n", + "Run the code below to download the image we'll use in this cookbook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import requests\n", + "import matplotlib.pyplot as plt\n", + "import cv2\n", + "import supervision as sv\n", + "\n", + "def download_image(url):\n", + " target_name = \"human_tower.jpg\" \n", + " try:\n", + " print(\"Downloading image from wikimedia.org...\")\n", + " response = requests.get(url, headers={'User-Agent': 'YourCustomUserAgent/1.0'})\n", + " response.raise_for_status() # Raises an error for bad responses\n", + " \n", + " with open(target_name, \"wb\") as file:\n", + " file.write(response.content)\n", + " print(\"Download complete\")\n", + " return target_name\n", + " \n", + " except requests.exceptions.RequestException as e:\n", + " print(f\"Failed to download the image: {e}\")\n", + " return None\n", + "\n", + "image_url = \"https://upload.wikimedia.org/wikipedia/commons/thumb/d/d0/4_de_8_amb_l%27agulla_carregat_Castellers_de_Barcelona_%2821937141066%29.jpg/2560px-4_de_8_amb_l%27agulla_carregat_Castellers_de_Barcelona_%2821937141066%29.jpg\"\n", + "image_path = download_image(image_url)\n", + "\n", + "if image_path is None:\n", + " print(\"Could not download the image...\")\n", + "else:\n", + " image = cv2.imread(image_path)\n", + " image_wh = (image.shape[1], image.shape[0])\n", + " print(f\"Image shape: {image_wh[0]}w x {image_wh[1]}h\")\n", + " sv.plot_image(image)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You're looking at a Castell, a human tower traditionally built at festivals in parts of Catalonia, Spain, and has since spread to the Balearic Islands and the Valencian Community. The source of the image is [here](https://commons.wikimedia.org/wiki/File:4_de_8_amb_l%27agulla_carregat_Castellers_de_Barcelona_(21937141066).jpg), and you could learn more about these human towers in [Wikipedia](https://en.wikipedia.org/wiki/Castell)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vrXJTlUXxXFN" + }, + "source": [ + "\n", + "## Let's try our model's performance\n", + "\n", + "Before we dive into the SAHI technique for small object detection, it’s useful to see how a fine-tuned model performs with the image as is—without any pre-processing or slicing. The goal is to understand when the model starts to fail so that we can progressively move towards an efficient slicing strategy.\n", + "\n", + "Let’s run the model!" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 670 + }, + "id": "gCYNvD8syhxE", + "outputId": "aa2632c6-6a6d-47f3-874d-692080d9afa6" + }, + "outputs": [], + "source": [ + "from inference_sdk import InferenceHTTPClient\n", + "import supervision as sv\n", + "import os\n", + "\n", + "MODEL_ID = \"people_counterv0/1\"\n", + "API_KEY=os.environ[\"ROBOFLOW_API_KEY\"] # <---Set you API key here\n", + "\n", + "# Set your API key here. Retrieve your API key: https://docs.roboflow.com/api-reference/authentication\n", + "# from google.colab import userdata\n", + "# API_KEY = userdata.get(\"ROBOFLOW_API_KEY\") \n", + "\n", + "client = InferenceHTTPClient(\n", + " api_url=\"https://detect.roboflow.com\",\n", + " api_key=API_KEY,\n", + ")\n", + "\n", + "# Run inference\n", + "results = client.infer(image_path, model_id=MODEL_ID)\n", + "detections = sv.Detections.from_inference(results)\n", + "\n", + "print(f\"Inference time: {results['time']} seconds\")\n", + "print(f\"Found {len(detections)} people\")\n", + "\n", + "COLOR_BBOX_PEOPLE=sv.ColorPalette.DEFAULT.colors[6]\n", + "\n", + "bbox_annotator = sv.BoxAnnotator(\n", + " color=COLOR_BBOX_PEOPLE,\n", + " thickness=2\n", + ")\n", + "\n", + "# Annotate our image with detections.\n", + "image_no_sahi = bbox_annotator.annotate(scene=image.copy(), detections=detections)\n", + "\n", + "sv.plot_image(image_no_sahi)\n", + "cv2.imwrite(\"1x1.jpg\", image_no_sahi)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The model shows strong performance in detecting people in the lower half of the image, but it struggles to accurately predict boxes in the upper half. This suggests two key insights: first, the model is proficient at identifying people’s heads from various angles, and second, using SAHI could effectively address the detection challenges in the upper portion of the image. Now, it’s time to try SAHI!" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ufFqhE5q3GSQ" + }, + "source": [ + "## Using SAHI for small object detection\n", + "\n", + "Detecting small objects with CNNs can be tricky because small pixel groups provide fewer features for deeper layers, making details easy to miss. SAHI addresses this by:\n", + "\n", + "1. Image Slicing: Dividing the high-resolution image into smaller, overlapping tiles, allowing the model to “zoom in” on details.\n", + "2. Hyper Inference: Aggregating results from all slices into a coherent set of detections.\n", + "\n", + "By using SAHI, models like YOLOv8 can greatly improve small object detection, overcoming the usual challenges with feature representation and resolution.\n", + "\n", + "![SAHI](https://raw.githubusercontent.com/obss/sahi/main/resources/sliced_inference.gif \"something\")\n", + "\n", + "SAHI can be seen as a framework for addressing the small object detection problem. If you’re interested in learning more about the motivations behind this solution, you can read the paper [Slicing Aided Hyper Inference and Fine-tuning for Small Object Detection](https://arxiv.org/abs/2202.06934).\n", + "\n", + "## Slicing our image with `supervision`\n", + "\n", + "Let’s begin by visualizing how these tiles would appear on our image. I’ll start with a small set of 2x2 tiles, with a zero overlap both vertically (height) and horizontally (width) between the tiles. The final values of these parameters will ultimately depend on your use case, so trial and error is encouraged!\n", + "\n", + "Some of the methods below are for visualizing the tiles and overlapping. You'll only need the `calculate_tile_size` method in your application to calculate the size of the tiles.\n", + "\n", + "### Utility functions for visualizing tiles" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "id": "HpVjry6m3F0-", + "outputId": "ffdb41d8-7e6b-45bb-ab5b-1cb9dfb6c377" + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "from inference import get_model\n", + "import math\n", + "\n", + "def tile_image(image_shape, slice_wh, overlap_wh)-> np.ndarray:\n", + " \"\"\"\n", + " Computes the coordinates and dimensions of tiles for an image with specified slicing and overlap parameters.\n", + " \n", + " Parameters:\n", + " -----------\n", + " image_shape : tuple of int\n", + " The shape of the image to be tiled, specified as (width, height).\n", + " \n", + " slice_wh : tuple of int\n", + " The dimensions (width, height) of each tile.\n", + " \n", + " overlap_wh : tuple of int\n", + " The overlap dimensions (width, height) between adjacent tiles.\n", + " \n", + " Returns:\n", + " --------\n", + " np.ndarray\n", + " An array of offsets where each row represents a tile's bounding box \n", + " in the format (x1, y1, x2, y2). The coordinates are rounded up to the nearest integer.\n", + " \n", + " \"\"\"\n", + " offsets = sv.InferenceSlicer._generate_offset(\n", + " resolution_wh=image_shape,\n", + " slice_wh=slice_wh,\n", + " overlap_ratio_wh=None,\n", + " overlap_wh=overlap_wh\n", + " )\n", + " \n", + " offsets = np.ceil(offsets).astype(int)\n", + " \n", + " return offsets\n", + "\n", + "def draw_transparent_tiles(scene: np.ndarray, x: int, y: int, w:int, h:int, index: int = None):\n", + " \"\"\"\n", + " Draws a transparent tile with an optional index label on the given scene.\n", + "\n", + " Parameters:\n", + " -----------\n", + " scene : np.ndarray\n", + " The input image on which the transparent tile will be drawn. It should be a NumPy array representing the image.\n", + "\n", + " x : int\n", + " The x-coordinate of the top-left corner of the tile.\n", + "\n", + " y : int\n", + " The y-coordinate of the top-left corner of the tile.\n", + "\n", + " w : int\n", + " The width of the tile.\n", + "\n", + " h : int\n", + " The height of the tile.\n", + "\n", + " index : int, optional, default=None\n", + " If provided, draws the index number in the center of the tile. If None, no index number is drawn.\n", + "\n", + " Returns:\n", + " --------\n", + " np.ndarray\n", + " A new image with a transparent tile and optional index label drawn on it.\n", + " \"\"\"\n", + " alpha=0.18\n", + " overlay_image = scene.copy()\n", + " \n", + " # Generate a mask for the tile\n", + " rectangle = np.zeros((h, w, 3), dtype=np.uint8)\n", + " rectangle.fill(255)\n", + " \n", + " rect = sv.Rect(x=x, y=y, width=w, height=h)\n", + " overlay_image = sv.draw_image(scene=overlay_image, image=rectangle, opacity=alpha, rect=rect)\n", + " \n", + " # Draw a border around the edge of the mask\n", + " border_color = sv.Color.BLACK\n", + " border_thickness=2\n", + " overlay_image = sv.draw_rectangle(scene=overlay_image,rect=sv.Rect(x=x, y=y, width=w, height=h), color=border_color, thickness=border_thickness)\n", + " \n", + " # Draw index number on the center of the tile after blending\n", + " if index is not None:\n", + " text_anchor = sv.Point(x=x + w // 2, y=y + h // 2)\n", + " # Calculate the center of the rectangle\n", + " center_x = rect.x + rect.width // 2\n", + " center_y = rect.y + rect.height // 2\n", + " \n", + " # Define the text anchor point as the center of the rectangle\n", + " text_anchor = sv.Point(x=center_x, y=center_y)\n", + " text_scale=3\n", + " text_thickness=5\n", + " text=str(index + 1)\n", + " text_color=sv.Color.BLACK\n", + " # Draw the text on the scene\n", + " overlay_image = sv.draw_text(overlay_image, text, text_anchor, text_color, text_scale, text_thickness)\n", + " \n", + " return overlay_image\n", + "\n", + "def draw_tiles(scene: np.ndarray, offsets, show_index=False):\n", + " \"\"\"\n", + " Draws transparent tiles on a scene based on the given offsets.\n", + "\n", + " Parameters:\n", + " -----------\n", + " scene : np.ndarray\n", + " The input image on which tiles will be drawn. It should be a NumPy array representing the image.\n", + " \n", + " offsets : list of tuples\n", + " A list of tuples where each tuple represents the bounding box of a tile in the format (x1, y1, x2, y2).\n", + " - (x1, y1) is the top-left corner of the tile.\n", + " - (x2, y2) is the bottom-right corner of the tile.\n", + "\n", + " show_index : bool, optional, default=False\n", + " If True, draws the index number of each tile on the scene. Otherwise, the tiles are drawn without indices.\n", + "\n", + " Returns:\n", + " --------\n", + " np.ndarray\n", + " A new image with transparent tiles drawn on it.\n", + "\n", + " \"\"\"\n", + " tiled_image = scene.copy()\n", + " \n", + " for index, offset in enumerate(offsets):\n", + " x = offset[0]\n", + " y = offset[1]\n", + " width = offset[2] - x\n", + " height = offset[3] - y\n", + " \n", + " if (show_index):\n", + " tiled_image = draw_transparent_tiles(scene=tiled_image, x=x, y=y, w=width, h=height, index=index)\n", + " else:\n", + " tiled_image = draw_transparent_tiles(scene=tiled_image, x=x, y=y, w=width, h=height, index=None)\n", + " \n", + " return tiled_image\n", + "\n", + "def print_offsets(offsets):\n", + " for index, offset in enumerate(offsets):\n", + " w = (offset[2] - offset[0])\n", + " h = (offset[3] - offset[1])\n", + " print(f\"Tile {index + 1}\")\n", + " print(f\" w={w}, h={h}, x1={offset[0]}, y1={offset[1]}, x2={offset[2]}, y2={offset[3]}, area={w*h}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "id": "HpVjry6m3F0-", + "outputId": "ffdb41d8-7e6b-45bb-ab5b-1cb9dfb6c377" + }, + "source": [ + "### Calculate Tile Size\n", + "\n", + "The `calculate_tile_size` function computes the dimensions of each tile when dividing an image into a grid, considering both the image’s size and a specified tiling strategy. It takes the image dimensions (`image_shape`), the number of rows and columns for the grid (`tiles`), and an optional overlap ratio (`overlap_ratio_wh`) to determine how much adjacent tiles overlap. The function returns the adjusted tile size, including the overlap, and the specific overlap dimensions.\n", + "\n", + "**NOTE**: As of `supervision==0.22.0` you need to provide the tile size. This function calculates it for you." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "id": "HpVjry6m3F0-", + "outputId": "ffdb41d8-7e6b-45bb-ab5b-1cb9dfb6c377" + }, + "outputs": [], + "source": [ + "def calculate_tile_size(image_shape: tuple[int, int], tiles: tuple[int, int], overlap_ratio_wh: tuple[float, float] = (0.0, 0.0)):\n", + " \"\"\"\n", + " Calculate the size of the tiles based on the image shape, the number of tiles, and the overlap ratio.\n", + "\n", + " Parameters:\n", + " ----------\n", + " image_shape : tuple[int, int]\n", + " The dimensions of the image as (width, height).\n", + " \n", + " tiles : tuple[int, int]\n", + " The tiling strategy defined as (rows, columns), specifying the number of tiles along the height and width of the image.\n", + " \n", + " overlap_ratio_wh : tuple[float, float], optional\n", + " The overlap ratio for width and height as (overlap_ratio_w, overlap_ratio_h). This defines the fraction of overlap between adjacent tiles. Default is (0.0, 0.0), meaning no overlap.\n", + "\n", + " Returns:\n", + " -------\n", + " tuple[tuple[int, int], tuple[int, int]]\n", + " A tuple containing:\n", + " - The size of each tile as (tile_width, tile_height), accounting for overlap.\n", + " - The overlap dimensions as (overlap_width, overlap_height).\n", + "\n", + " Example:\n", + " -------\n", + " >>> image_shape = (1024, 768)\n", + " >>> tiles = (4, 4)\n", + " >>> overlap_ratio_wh = (0.15, 0.15)\n", + " >>> calculate_tile_size(image_shape, tiles, overlap_ratio_wh)\n", + " ((295, 221), (39, 29))\n", + "\n", + " Notes:\n", + " -----\n", + " - The function calculates the width and height of each tile based on the number of rows and columns specified.\n", + " - It then applies the overlap ratio to adjust the tile size, ensuring that tiles overlap by a specified fraction.\n", + " - The overlap dimensions are rounded up to the nearest integer to ensure complete coverage.\n", + " \"\"\"\n", + "\n", + " w, h = image_shape\n", + " rows, columns = tiles\n", + " \n", + " tile_width = (w / columns)\n", + " tile_height = (h / rows)\n", + " \n", + " overlap_ratio_w, overlap_ratio_h = overlap_ratio_wh\n", + " overlap_wh = (math.ceil(tile_width * overlap_ratio_w), math.ceil(tile_height * overlap_ratio_h))\n", + " \n", + " tile_width = math.ceil(tile_width + overlap_wh[0])\n", + " tile_height = math.ceil(tile_height + overlap_wh[1])\n", + " tile_size = (tile_width, tile_height)\n", + " \n", + " return tile_size, overlap_wh" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "id": "HpVjry6m3F0-", + "outputId": "ffdb41d8-7e6b-45bb-ab5b-1cb9dfb6c377" + }, + "outputs": [], + "source": [ + " \n", + "tiles = (2,2) # The number of tiles you want\n", + "overlap_ratio_wh = (0.0, 0.0) # The overlap between tiles\n", + "slice_wh, overlap_wh = calculate_tile_size(image_wh, tiles, overlap_ratio_wh)\n", + "offsets = tile_image(image_wh, slice_wh, overlap_wh)\n", + "\n", + "print(f\"Image shape: {image_wh[0]}w x {image_wh[1]}h\")\n", + "print(f\"Tiles: {tiles}\")\n", + "print(slice_wh)\n", + "print(f\"Generated {len(offsets)} tiles. These are the calculated dimensions\")\n", + "print_offsets(offsets)\n", + "\n", + "tiled_image = draw_tiles(scene=image.copy(), offsets=offsets, show_index=True)\n", + "\n", + "sv.plot_image(tiled_image)\n", + "# cv2.imwrite(\"2x2.jpg\", image_no_sahi)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "pqHWZhWUawP3" + }, + "source": [ + "You can see that the image has been sliced into four different tiles. Next, each tile will be independently processed by the model, and supervision will merge all the predictions into a coherent set of detections. Notice that we're not using overlapping in at this time (more on that later).\n", + "\n", + "## Run Inference on a Sliced Image With `supervision`\n", + "\n", + "Running inference on slices of your image is easy with the class `InferenceSlicer` from [Supervision](https://supervision.roboflow.com/latest/detection/tools/inference_slicer/#inferenceslicer). This API from Roboflow divides a larger image into smaller slices, performs inference on each slice, and then merges the detections into a single `detections` object.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "id": "Zpd-NT-m1-ul", + "outputId": "d85c72be-edab-4e2e-e5ef-9ab02047c66e", + "scrolled": true + }, + "outputs": [], + "source": [ + "import time\n", + "import leafmap\n", + "\n", + "def callback(image_slice: np.ndarray) -> sv.Detections:\n", + " result = get_model(model_id=MODEL_ID, api_key=API_KEY).infer(image_slice )[0]\n", + " return sv.Detections.from_inference(result)\n", + "\n", + "tiles = (2,2) # The number of tiles you want\n", + "overlap_ratio_wh = (0.0, 0.0) # The overlap between tiles\n", + "slice_wh, overlap_wh = calculate_tile_size(image_wh, tiles, overlap_ratio_wh)\n", + "offsets = tile_image(image_wh, slice_wh, overlap_wh)\n", + "\n", + "slicer = sv.InferenceSlicer(\n", + " callback=callback,\n", + " slice_wh=slice_wh,\n", + " overlap_ratio_wh=None,\n", + " overlap_wh=overlap_wh,\n", + " thread_workers=4\n", + ")\n", + "\n", + "start_time = time.time()\n", + "detections = slicer(image)\n", + "end_time = time.time()\n", + "elapsed_time = end_time - start_time\n", + "\n", + "print(f\"Image shape: {image_wh[0]}w x {image_wh[1]}h\")\n", + "print(f\"Tile size: {slice_wh[0]}w x {image_wh[1]}\")\n", + "print(f\"Overlap: {overlap_wh[0]}w x {overlap_wh[1]}h. Ratio {overlap_ratio_wh}\")\n", + "print(f\"Found {len(detections)} people\")\n", + "print(f\"Inference time: {elapsed_time} seconds\")\n", + "\n", + "tiled_image_with_sahi_2x2 = draw_tiles(scene=image.copy(), offsets=offsets, show_index=True)\n", + "tiled_image_with_sahi_2x2 = bbox_annotator.annotate(scene=tiled_image_with_sahi_2x2, detections=detections)\n", + "\n", + "# For visualization\n", + "leafmap.image_comparison(\n", + " img1=cv2.cvtColor(tiled_image_with_sahi_2x2, cv2.COLOR_BGR2RGB),\n", + " img2=cv2.cvtColor(image_no_sahi, cv2.COLOR_BGR2RGB),\n", + " label1=\"Slicing 2x2\",\n", + " label2=\"No Slicing\",\n", + " starting_position=50,\n", + ")\n", + "\n", + "#cv2.imwrite(\"2x2_no_overlapping.jpg\", bbox_annotator.annotate(scene=image.copy(), detections=detections))\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "W6TvNnXpewwc" + }, + "source": [ + "Great! We’ve detected 726 people, up from the 185 we initially detected without image slicing. The model is still detecting people from different angles, but it continues to struggle with detecting people located in the farther parts of the plaza. It’s time to increase the number of tiles—in other words, zoom in so the model can capture more details of the small heads of people.\n", + "\n", + "![Missing detections](https://raw.githubusercontent.com/ediardo/notebooks/main/sahi/detections.png)\n", + "\n", + "### Increasing Tile Density: Moving to a 5x5 Grid\n", + "\n", + "Now that we’ve seen improvements with a 2x2 grid, it’s time to push the model further. By increasing the number of tiles to a 5x5 grid, we effectively zoom in on the image, allowing the model to capture finer details, such as smaller and more distant features that might have been missed before. This approach will help us understand how well the model performs with even more zoomed-in images. Let’s explore how this change affects our detection accuracy and overall performance." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import time\n", + "\n", + "def callback(image_slice: np.ndarray) -> sv.Detections:\n", + " result = get_model(model_id=MODEL_ID, api_key=API_KEY).infer(image_slice )[0]\n", + " return sv.Detections.from_inference(result)\n", + "\n", + "tiles = (5,5) # The number of tiles you want\n", + "overlap_ratio_wh = (0.0, 0.0) # The overlap between tiles\n", + "slice_wh, overlap_wh = calculate_tile_size(image_wh, tiles, overlap_ratio_wh)\n", + "offsets = tile_image(image_wh, slice_wh, overlap_wh)\n", + "\n", + "slicer = sv.InferenceSlicer(\n", + " callback=callback,\n", + " slice_wh=slice_wh,\n", + " overlap_wh=overlap_wh,\n", + " overlap_ratio_wh=None,\n", + " thread_workers=4\n", + ")\n", + "\n", + "start_time = time.time()\n", + "detections = slicer(image)\n", + "end_time = time.time()\n", + "elapsed_time = end_time - start_time\n", + "\n", + "print(f\"Image shape: {image_wh[0]}w x {image_wh[1]}h\")\n", + "print(f\"Tiles: {tiles}\")\n", + "print(f\"Tile size: {slice_wh[0]}w x {image_wh[1]}\")\n", + "print(f\"Overlap: {overlap_wh[0]}w x {overlap_wh[1]}h. Ratio {overlap_ratio_wh}\")\n", + "print(f\"Overlap filter: {sv.OverlapFilter.NON_MAX_SUPPRESSION}\")\n", + "print(f\"Found {len(detections)} people\")\n", + "print(f\"Inference time: {elapsed_time} seconds\")\n", + "\n", + "tiled_image_with_sahi_5x5 = draw_tiles(scene=image.copy(), offsets=offsets, show_index=True)\n", + "tiled_image_with_sahi_5x5 = bbox_annotator.annotate(scene=tiled_image_with_sahi_5x5, detections=detections)\n", + "\n", + "\n", + "# For visualization\n", + "leafmap.image_comparison(\n", + " img1=cv2.cvtColor(tiled_image_with_sahi_5x5, cv2.COLOR_BGR2RGB),\n", + " img2=cv2.cvtColor(tiled_image_with_sahi_2x2, cv2.COLOR_BGR2RGB),\n", + " label1=\"Slicing 5x5\",\n", + " label2=\"Slicing 2x2\",\n", + " starting_position=50,\n", + ")\n", + "\n", + "#cv2.imwrite(\"5x5_no_overlapping.jpg\", bbox_annotator.annotate(scene=image.copy(), detections=detections))\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "W6TvNnXpewwc" + }, + "source": [ + "We’ve just detected 1,494 people using a 25-tile grid (5 rows x 5 columns), a significant increase from the 726 people detected with the 4-tile (2x2) grid. However, as we increase the number of tiles, a new challenge arises: duplicate detections or missed detections along the edges of the tiles. This issue becomes evident in these examples, where overlapping or gaps between tiles lead to inaccuracies in our model’s detection.\n", + "\n", + "| Example| Observations |\n", + "|----|----| \n", + "| ![Overlapping](https://github.com/ediardo/notebooks/blob/main/sahi/overlapping_1.png?raw=true \"Overlapping\") | False Negative, Incomplete bbox | \n", + "| ![Overlapping](https://raw.githubusercontent.com/ediardo/notebooks/main/sahi/overlapping_2.png \"Overlapping\")| Double detection, Incomplete bbox|\n", + "| ![Overlapping](https://raw.githubusercontent.com/ediardo/notebooks/main/sahi/overlapping_3.png \"Overlapping\")| Incomplete bounding box|\n", + "\n", + "## Improving Object Detection Near Boundaries with Overlapping\n", + "\n", + "When objects, like people, appear at the edges of tiles, they might be detected twice or missed entirely if they span across two tiles. This can lead to inaccurate detection results. To solve this, we use overlapping tiles, allowing the model to see parts of adjacent tiles simultaneously. This overlap helps ensure that objects near the boundaries are fully captured, reducing duplicates and improving accuracy.\n", + "\n", + "We’ll set the overlap ratio to 15% on the tile’s width and height. This overlap helps ensure that objects near the boundaries are fully captured, reducing duplicates and improving accuracy." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 83 + }, + "id": "ip3ohW0fezQo", + "outputId": "b233a920-0413-4307-91c5-fae7d7e90017", + "scrolled": true + }, + "outputs": [], + "source": [ + "tiles = (5,5) # The number of tiles you want\n", + "overlap_ratio_wh = (0.05, 0.05) # Ratio of overlapping, width/height\n", + "slice_wh, overlap_wh = calculate_tile_size(image_wh, tiles, overlap_ratio_wh)\n", + "offsets = tile_image(image_wh, slice_wh, overlap_wh)\n", + "\n", + "slicer = sv.InferenceSlicer(\n", + " callback=callback,\n", + " #overlap_filter=sv.OverlapFilter.NON_MAX_MERGE,\n", + " overlap_filter=sv.OverlapFilter.NON_MAX_SUPPRESSION,\n", + " iou_threshold=0.75,\n", + " slice_wh=slice_wh,\n", + " overlap_ratio_wh=None,\n", + " overlap_wh=overlap_wh,\n", + " thread_workers=4\n", + ")\n", + "start_time = time.time()\n", + "detections = slicer(image)\n", + "end_time = time.time()\n", + "elapsed_time = end_time - start_time\n", + "\n", + "print(f\"Image shape: {image_wh[0]}w x {image_wh[1]}h\")\n", + "print(f\"Tiles: {tiles}\")\n", + "print(f\"Tile size: {slice_wh[0]}w x {image_wh[1]}\")\n", + "print(f\"Overlap: {overlap_wh[0]}w x {overlap_wh[1]}h. Ratio {overlap_ratio_wh}\")\n", + "print(f\"Overlap Filter: {sv.OverlapFilter.NON_MAX_SUPPRESSION}\")\n", + "print(f\"Found {len(detections)} people\")\n", + "print(f\"Inference time: {elapsed_time} seconds\")\n", + "#print_offsets(offsets)\n", + "\n", + "tiled_image_with_overlapping_nms_5x5 = draw_tiles(scene=image.copy(), offsets=offsets, show_index=True)\n", + "tiled_image_with_overlapping_nms_5x5 = bbox_annotator.annotate(scene=tiled_image_with_overlapping_nms_5x5, detections=detections)\n", + "\n", + "leafmap.image_comparison(\n", + " img1=cv2.cvtColor(tiled_image_with_overlapping_nms_5x5, cv2.COLOR_BGR2RGB),\n", + " img2=cv2.cvtColor(tiled_image_with_sahi_5x5, cv2.COLOR_BGR2RGB),\n", + " label1=\"Tiles with overlapping\",\n", + " label2=\"Tiles with no overlapping\",\n", + " starting_position=50,\n", + " make_responsive=True\n", + ")\n", + "\n", + "#cv2.imwrite(\"5x5_with_overlapping_nms.jpg\", bbox_annotator.annotate(scene=image.copy(), detections=detections))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_RXyuDMjmWGv" + }, + "source": [ + "## Non-Max Supression vs Non-Max Merge \n", + "\n", + "When dealing with overlapping detections, it’s essential to determine which detections represent the same object and which are unique. Non-Maximum Suppression (NMS) and Non-Maximum Merging (NMM) are two techniques commonly used to address this challenge. NMS works by eliminating redundant detections based on confidence scores, while NMM combines overlapping detections to enhance the representation of objects spanning multiple tiles. Understanding the difference between these methods helps optimize object detection, particularly near tile boundaries.\n", + "\n", + "In `supervision`, the `overlap_filter` parameter allows us to specify the strategy for handling overlapping detections in slices. This parameter can take on two values:\n", + "\n", + "- `sv.OverlapFilter.NON_MAX_SUPRESSION` (default): Eliminates redundant detections by keeping the one with the highest confidence score.\n", + "- `sv.OverlapFilter.NON_MAX_MERGE`: Combines overlapping detections to create a more comprehensive representation of objects spanning multiple tiles.\n", + "\n", + "It’s important to note that this method is not perfect and may require further testing and fine-tuning to achieve optimal results in various use cases. You should validate the outputs and adjust parameters as needed to handle specific scenarios effectively." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "id": "Ou0vF3-kmYoA", + "outputId": "3367ee55-dda7-48e4-e23b-053ecda6bcff", + "scrolled": true + }, + "outputs": [], + "source": [ + "tiles = (5,5) # The number of tiles you want\n", + "overlap_ratio_wh = (0.05, 0.05)\n", + "slice_wh, overlap_wh = calculate_tile_size(image_wh, tiles, overlap_ratio_wh)\n", + "offsets = tile_image(image_wh, slice_wh, overlap_wh)\n", + "slicer = sv.InferenceSlicer(\n", + " callback=callback,\n", + " overlap_filter=sv.OverlapFilter.NON_MAX_MERGE,\n", + " #overlap_filter=sv.OverlapFilter.NON_MAX_SUPPRESSION,\n", + " iou_threshold=0.75,\n", + " slice_wh=slice_wh,\n", + " overlap_ratio_wh=None,\n", + " overlap_wh=overlap_wh,\n", + " thread_workers=4\n", + ")\n", + "start_time = time.time()\n", + "detections = slicer(image)\n", + "end_time = time.time()\n", + "elapsed_time = end_time - start_time\n", + "\n", + "print(f\"Image shape: {image_wh[0]}w x {image_wh[1]}h\")\n", + "print(f\"Tile size: {slice_wh[0]}w x {image_wh[1]}\")\n", + "print(f\"Overlap: {overlap_wh[0]}w x {overlap_wh[1]}h. Ratio {overlap_ratio_wh}\")\n", + "print(f\"Overlap Filter: {sv.OverlapFilter.NON_MAX_MERGE}\")\n", + "print(f\"Found {len(detections)} people\")\n", + "print(f\"Inference time: {elapsed_time} seconds\")\n", + "#print_offsets(offsets)\n", + "\n", + "tiled_image_with_overlapping_nmm_5x5 = draw_tiles(scene=image.copy(), offsets=offsets, show_index=True)\n", + "tiled_image_with_overlapping_nmm_5x5 = bbox_annotator.annotate(scene=tiled_image_with_overlapping_nmm_5x5, detections=detections)\n", + "\n", + "\n", + "leafmap.image_comparison(\n", + " img1=cv2.cvtColor(tiled_image_with_overlapping_nmm_5x5, cv2.COLOR_BGR2RGB),\n", + " img2=cv2.cvtColor(tiled_image_with_overlapping_nms_5x5, cv2.COLOR_BGR2RGB),\n", + " label1=\"Overlapping with Non-Max Merging\",\n", + " label2=\"Overlapping with Non-Max Supression\",\n", + " starting_position=50,\n", + " make_responsive=True\n", + ")\n", + "\n", + "# cv2.imwrite(\"5x5_with_overlapping_nmm.jpg\", bbox_annotator.annotate(scene=image.copy(), detections=detections))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Conclusion\n", + "\n", + "In this cookbook, we’ve explored the advantages of using the SAHI technique for enhancing small object detection and the importance of experimenting with various tiling strategies to effectively zoom into images. By combining these approaches, we can improve the accuracy and reliability of object detection models, particularly in challenging scenarios where objects are small or located near the boundaries of tiles. These methods offer practical solutions to common challenges in computer vision, empowering developers to build more robust and precise detection systems.\n", + "\n", + "![\"Crowd Detection\"](https://raw.githubusercontent.com/ediardo/notebooks/main/sahi/5x5_with_overlapping_nmm.jpg \"Crowd Detection\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## More resources\n", + "\n", + "- `InferenceSlicer`: https://supervision.roboflow.com/detection/tools/inference_slicer/\n", + "- Detect Small Objects https://supervision.roboflow.com/latest/how_to/detect_small_objects/\n", + "- What is Non-Max Merging?: https://blog.roboflow.com/non-max-merging/\n", + "- How to Detect Small Objects: A Guide https://blog.roboflow.com/detect-small-objects/\n", + "- How to Use SAHI to Detect Small Objects: https://blog.roboflow.com/how-to-use-sahi-to-detect-small-objects/\n", + "- SAHI paper: https://arxiv.org/abs/2202.06934\n", + "- C4W3L07 Nonmax Suppression, Andrew Ng: https://www.youtube.com/watch?v=VAo84c1hQX8" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "colab": { + "machine_shape": "hm", + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.6" + } + }, + "nbformat": 4, + "nbformat_minor": 4 }