diff --git a/docs/how_to/detect_and_annotate.md b/docs/how_to/detect_and_annotate.md index ca0c5666..b068a923 100644 --- a/docs/how_to/detect_and_annotate.md +++ b/docs/how_to/detect_and_annotate.md @@ -4,17 +4,29 @@ comments: true # Detect and Annotate -Supervision offers a streamlined solution to effortlessly annotate predictions from a -range of object detection and segmentation models. This guide demonstrates how to -execute inference using the YOLOv8 model with either the -[Inference](https://github.com/roboflow/inference) or -[Ultralytics](https://github.com/ultralytics/ultralytics) packages. Following this, -you'll learn how to import these predictions into Supervision for image annotation -purposes. +Supervision provides a seamless process for annotating predictions generated by various +object detection and segmentation models. This guide shows how to perform inference +with the [Inference](https://github.com/roboflow/inference), +[Ultralytics](https://github.com/ultralytics/ultralytics) or +[Transformers](https://github.com/huggingface/transformers) packages. Following this, +you'll learn how to import these predictions into Supervision and use them to annotate +source image. ## Run Inference -First, you'll need to obtain predictions from your object detection or segmentation model. +First, you'll need to obtain predictions from your object detection or segmentation +model. + +=== "Inference" + + ```python + import cv2 + from inference import get_model + + model = get_model(model_id="yolov8n-640") + image = cv2.imread() + results = model.infer(image)[0] + ``` === "Ultralytics" @@ -27,26 +39,52 @@ First, you'll need to obtain predictions from your object detection or segmentat results = model(image)[0] ``` -=== "Inference" +=== "Transformers" ```python - import cv2 - from inference.models.utils import get_roboflow_model + import torch + from PIL import Image + from transformers import DetrImageProcessor, DetrForObjectDetection - model = get_roboflow_model(model_id="yolov8n-640", api_key=) - image = cv2.imread() - results = model.infer(image)[0] + processor = DetrImageProcessor.from_pretrained("facebook/detr-resnet-50") + model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50") + + image = Image.open() + inputs = processor(images=image, return_tensors="pt") + + with torch.no_grad(): + outputs = model(**inputs) + + width, height = image.size + target_size = torch.tensor([[height, width]]) + results = processor.post_process_object_detection( + outputs=outputs, target_sizes=target_size)[0] ``` ## Load Predictions into Supervision Now that we have predictions from a model, we can load them into Supervision. +=== "Inference" + + We can do so using the [`sv.Detections.from_inference`](detection/core/#supervision.detection.core.Detections.from_inference) method, which accepts model results from both detection and segmentation models. + + ```{ .py hl_lines="2 8" } + import cv2 + import supervision as sv + from inference import get_model + + model = get_model(model_id="yolov8n-640") + image = cv2.imread() + results = model.infer(image)[0] + detections = sv.Detections.from_inference(results) + ``` + === "Ultralytics" We can do so using the [`sv.Detections.from_ultralytics`](detection/core/#supervision.detection.core.Detections.from_ultralytics) method, which accepts model results from both detection and segmentation models. - ```python + ```{ .py hl_lines="2 8" } import cv2 import supervision as sv from ultralytics import YOLO @@ -57,34 +95,123 @@ Now that we have predictions from a model, we can load them into Supervision. detections = sv.Detections.from_ultralytics(results) ``` -=== "Inference" +=== "Transformers" - We can do so using the [`sv.Detections.from_inference`](detection/core/#supervision.detection.core.Detections.from_inference) method, which accepts model results from both detection and segmentation models. + We can do so using the [`sv.Detections.from_transformers`](detection/core/#supervision.detection.core.Detections.from_transformers) method, which accepts model results from both detection and segmentation models. - ```python - import cv2 + ```{ .py hl_lines="2 19" } + import torch import supervision as sv - from inference.models.utils import get_roboflow_model + from PIL import Image + from transformers import DetrImageProcessor, DetrForObjectDetection - model = get_roboflow_model(model_id="yolov8n-640", api_key= - image = cv2.imread() - results = model.infer(image)[0] - detections = sv.Detections.from_inference(results) + processor = DetrImageProcessor.from_pretrained("facebook/detr-resnet-50") + model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50") + + image = Image.open() + inputs = processor(images=image, return_tensors="pt") + + with torch.no_grad(): + outputs = model(**inputs) + + width, height = image.size + target_size = torch.tensor([[height, width]]) + results = processor.post_process_object_detection( + outputs=outputs, target_sizes=target_size)[0] + detections = sv.Detections.from_transformers(results) ``` -You can conveniently load predictions from other computer vision frameworks and libraries using: +You can load predictions from other computer vision frameworks and libraries using: - [`from_deepsparse`](detection/core/#supervision.detection.core.Detections.from_deepsparse) ([Deepsparse](https://github.com/neuralmagic/deepsparse)) - [`from_detectron2`](detection/core/#supervision.detection.core.Detections.from_detectron2) ([Detectron2](https://github.com/facebookresearch/detectron2)) - [`from_mmdetection`](detection/core/#supervision.detection.core.Detections.from_mmdetection) ([MMDetection](https://github.com/open-mmlab/mmdetection)) -- [`from_inference`](detection/core/#supervision.detection.core.Detections.from_inference) ([Roboflow Inference](https://github.com/roboflow/inference)) - [`from_sam`](detection/core/#supervision.detection.core.Detections.from_sam) ([Segment Anything Model](https://github.com/facebookresearch/segment-anything)) -- [`from_transformers`](detection/core/#supervision.detection.core.Detections.from_transformers) ([HuggingFace Transformers](https://github.com/huggingface/transformers)) - [`from_yolo_nas`](detection/core/#supervision.detection.core.Detections.from_yolo_nas) ([YOLO-NAS](https://github.com/Deci-AI/super-gradients/blob/master/YOLONAS.md)) -## Annotate Image +## Annotate Image with Detections -Finally, we can annotate the image with the predictions. Since we are working with an object detection model, we will use the [`sv.BoundingBoxAnnotator`](annotators/#supervision.annotators.core.BoundingBoxAnnotator) and [`sv.LabelAnnotator`](annotators/#supervision.annotators.core.LabelAnnotator) classes. If you are running the segmentation model [`sv.MaskAnnotator`](annotators/#supervision.annotators.core.MaskAnnotator) is a drop-in replacement for [`sv.BoundingBoxAnnotator`](annotators/#supervision.annotators.core.BoundingBoxAnnotator) that will allow you to draw masks instead of boxes. +Finally, we can annotate the image with the predictions. Since we are working with an object detection model, we will use the [`sv.BoundingBoxAnnotator`](annotators/#supervision.annotators.core.BoundingBoxAnnotator) and [`sv.LabelAnnotator`](annotators/#supervision.annotators.core.LabelAnnotator) classes. + +=== "Inference" + + ```{ .py hl_lines="10-16" } + import cv2 + import supervision as sv + from inference import get_model + + model = get_model(model_id="yolov8n-640") + image = cv2.imread() + results = model.infer(image)[0] + detections = sv.Detections.from_inference(results) + + bounding_box_annotator = sv.BoundingBoxAnnotator() + label_annotator = sv.LabelAnnotator() + + annotated_image = bounding_box_annotator.annotate( + scene=image, detections=detections) + annotated_image = label_annotator.annotate( + scene=annotated_image, detections=detections) + ``` + +=== "Ultralytics" + + ```{ .py hl_lines="10-16" } + import cv2 + import supervision as sv + from ultralytics import YOLO + + model = YOLO("yolov8n.pt") + image = cv2.imread() + results = model(image)[0] + detections = sv.Detections.from_ultralytics(results) + + bounding_box_annotator = sv.BoundingBoxAnnotator() + label_annotator = sv.LabelAnnotator() + + annotated_image = bounding_box_annotator.annotate( + scene=image, detections=detections) + annotated_image = label_annotator.annotate( + scene=annotated_image, detections=detections) + ``` + +=== "Transformers" + + ```{ .py hl_lines="21-27" } + import torch + import supervision as sv + from PIL import Image + from transformers import DetrImageProcessor, DetrForObjectDetection + + processor = DetrImageProcessor.from_pretrained("facebook/detr-resnet-50") + model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50") + + image = Image.open() + inputs = processor(images=image, return_tensors="pt") + + with torch.no_grad(): + outputs = model(**inputs) + + width, height = image.size + target_size = torch.tensor([[height, width]]) + results = processor.post_process_object_detection( + outputs=outputs, target_sizes=target_size)[0] + detections = sv.Detections.from_transformers(results) + + bounding_box_annotator = sv.BoundingBoxAnnotator() + label_annotator = sv.LabelAnnotator() + + annotated_image = bounding_box_annotator.annotate( + scene=image, detections=detections) + annotated_image = label_annotator.annotate( + scene=annotated_image, detections=detections) + ``` + +![basic-annotation](https://media.roboflow.com/supervision_annotate_example.png) + +## Display Custom Labels + + === "Ultralytics" @@ -102,9 +229,9 @@ Finally, we can annotate the image with the predictions. Since we are working wi label_annotator = sv.LabelAnnotator() labels = [ - model.model.names[class_id] - for class_id - in detections.class_id + f"{class_name} {confidence:.2f}" + for class_name, confidence + in zip(detections['class_name'], detections.confidence) ] annotated_image = bounding_box_annotator.annotate( @@ -118,9 +245,9 @@ Finally, we can annotate the image with the predictions. Since we are working wi ```python import cv2 import supervision as sv - from inference.models.utils import get_roboflow_model + from inference import get_model - model = get_roboflow_model(model_id="yolov8n-640", api_key= + model = get_model(model_id="yolov8n-640") image = cv2.imread() results = model.infer(image)[0] detections = sv.Detections.from_inference(results) @@ -128,18 +255,20 @@ Finally, we can annotate the image with the predictions. Since we are working wi bounding_box_annotator = sv.BoundingBoxAnnotator() label_annotator = sv.LabelAnnotator() + labels = [ + f"{class_name} {confidence:.2f}" + for class_name, confidence + in zip(detections['class_name'], detections.confidence) + ] + annotated_image = bounding_box_annotator.annotate( scene=image, detections=detections) annotated_image = label_annotator.annotate( scene=annotated_image, detections=detections) ``` -![Predictions plotted on an image](https://media.roboflow.com/supervision_annotate_example.png) +![custom-label-annotation](https://media.roboflow.com/supervision-annotator-examples/label-annotator-example-purple.png) -## Display Annotated Image +## Annotate Image with Segmentations -To display the annotated image in Jupyter Notebook or Google Colab, use the [`sv.plot_image`](utils/notebook/#supervision.utils.notebook.plot_image) function. - -```python -sv.plot_image(annotated_image) -``` +If you are running the segmentation model [`sv.MaskAnnotator`](annotators/#supervision.annotators.core.MaskAnnotator) is a drop-in replacement for [`sv.BoundingBoxAnnotator`](annotators/#supervision.annotators.core.BoundingBoxAnnotator) that will allow you to draw masks instead of boxes. \ No newline at end of file