26 Commits
Author SHA1 Message Date
Daniiiil1 6b143e4e41 test(metrics): verify live sklearn parity (#2470)
* add sklearn parity goldens
* compute sklearn parity live
* separate empty input semantics
2026-08-05 19:24:44 +02:00
7fc91180fe fix(metrics): track prediction-only classes in Recall (#2468)
#2331 made Precision and F1Score include classes that appear only in
predictions, and added regression tests to both. Recall was not touched, so
the line #2331 replaced is still there and the three metrics disagree about
which classes exist for identical input:

    precision.matched_classes -> [0 1]   precision_per_class (2, 10)
    recall.matched_classes    -> [0]     recall_per_class    (1, 10)
    f1.matched_classes        -> [0 1]

These read as parallel outputs, so zipping them silently truncates rather
than raising.

Recall for a class with no ground-truth instances is 0.0 rather than
undefined, which is what sklearn reports (it infers labels from the union of
y_true and y_pred) and what #2331 cited as its own standard. MICRO is
unchanged because an absent class contributes no false negatives, and
WEIGHTED is unchanged because its ground-truth support is zero. MACRO does
change, and the changelog says so.

Also of note: recall.py already carried #2331's WEIGHTED zero-support guard,
whose comment refers to 'only false-positive classes'. That state could not
arise in recall.py, because unique_classes came from ground truth alone. The
guard was propagated; the union that gives it meaning was not.

Addresses the review on #2468. Building the class union inside
_compute_recall_for_classes only covers samples that reach it, and samples with
predictions but no targets are skipped earlier in _compute. So matched_classes
could still disagree with Precision and F1Score for list inputs containing a
background image, which is the exact invariant the new test asserts.

Before, for one normal sample plus one background image predicting class 2:

    precision.matched_classes -> [0 2]
    recall.matched_classes    -> [0]

Recall now handles len(targets) == 0 and len(predictions) > 0 the way Precision
does. No recall value changes, since a background image produces no false
negatives; only the tracked class set does.

* test: cover Recall bg-image size-bucket, dup & non-contiguous ids
* docs: strengthen Recall changelog migration note
* docs+perf: Recall doctest example; dedupe-then-union micro-opt

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-08-03 20:57:05 +02:00
9aaf7fdbd6 fix(metrics): avoid division by zero RuntimeWarning in F1Score using np.divide (#2437)
* fix(metrics): avoid division by zero RuntimeWarning in F1Score using np.divide
* test+changelog: add F1Score zero-denom regression test; add changelog entry
* follow-up cleanup: mirror F1Score np.divide fix in mean_average_recall

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-07-17 17:11:45 +02:00
1efa5b8eaa feat(cv2): complete fallback integration (#2439)
* feat(cv2): complete fallback integration
* fallback-fixes: reject invalid addWeighted dtype; O(N) approxPolyDP anchor seeding
* tests: copyMakeBorder sequence parity; drop non-empty facade-import assert; fix Windows path separator in boundary check
* fix(cv2): copyMakeBorder scalar value only fills channel 0 on multichannel images

---------

Co-authored-by: Codex <codex@openai.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
2026-07-17 07:50:45 +02:00
94cfb7f290 fix(metrics): ignore out-of-bucket detections in size-bucketed sco… (#2428)
* fix: ignore out-of-bucket   detections in size-bucketed scoring
* fix: honor area metadata in buckets

- Prefer stored COCO area metadata before geometry, mask, or OBB fallbacks.
- Add explicit-area, mask, and OBB regression coverage.
- Align COCO, mAP, and changelog area semantics.

---------

Co-authored-by: jirka <6035284+Borda@users.noreply.github.com>
Co-authored-by: Codex <codex@openai.com>
2026-07-15 12:48:19 +02:00
5344cb99dd fix: close out remaining review findings (#2418)
Re-verify remaining supervision review backlog against develop HEAD; most items were already resolved by an intervening commit, only genuinely-open gaps got new fixes.
Fix float32 precision loss in box_iou_batch for large coordinates (GeoTIFF-scale) by accumulating in float64.
Raise ValueError instead of a strippable assert in EvaluationDataset.load_predictions for unknown image ids.
Add HeatMapAnnotator.reset() to clear accumulated heat for annotator reuse.
Add missing coverage: labelme export basename collisions, _greedy_match matcher, metrics.core ABC/enum contracts, metrics.utils.utils pandas guard; remove a global RNG-seed pollution site in a metrics test.
Document the last two undocumented public exports (calculate_masks_centroids, is_compressed_rle) and add usage examples to 17 previously-example-less public functions/classes (NMS/NMM helpers, draw utils, PolygonZoneAnnotator, mask/polygon converters).

* tests: load_predictions ValueError branch + empty-dataset coverage
* fix: box_iou_batch int-dtype overflow, narrow float32 precision claim
* feat: add reset() to TraceAnnotator/DetectionsSmoother, fix docstrings
* docs: fix temp file leak in coco.py docstring, rename misnamed test

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-07-09 17:53:52 +02:00
75023c5f2f fix: remaining review findings in dataset, docs, and tests (#2416)
- Added `sv.mask_to_roi` as an explicit migration path for exclusive mask bounds
- Fixed COCO, CreateML, and Pascal VOC export validation to reject ambiguous or colliding dataset paths before writing
- Fixed in-memory `DetectionDataset` split and merge behavior
- Fixed `supervision` imports to avoid loading ByteTrack until it is used
- Fixed detection conversion helpers to support coordinate-convention migration while preserving legacy inclusive defaults
- Fixed Azure tag mapping, anchor rounding, and line-zone smoothing to avoid incorrect or ghost detections
- Fixed video processing shutdown handling for timeout and full-queue cases
- Improved downloader, validator, documentation, and regression coverage for the shipped dataset, detection, annotator, image, and video behavior

---------

Co-authored-by: Codex <codex@openai.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-07-08 23:00:32 +02:00
Jirka BorovecandCodex 072f78471c fix: resolve medium dataset findings (#2408)
- Reinstated NumPy-safe `Classifications` equality and ordered class-list comparisons in dataset equality.
- Restored greedy matching plus size-bucket scoring for Precision, Recall, F1, and MeanAverageRecall, with regression coverage for the medium-object boundary case.
- Filter size-bucket precision, recall, and F1 against target boxes so predictions no longer claim the bucket.
- Preserve confidence order for bucketed mAR@K scoring and return zero when a bucket has no support.
- Add regression coverage for bucket matching, empty-support mAR, top-K limits, and missing-mask errors.

---------

Co-authored-by: Codex <codex@openai.com>
2026-07-08 08:56:34 +02:00
Jirka BorovecandCodex 814a226eba fix(metrics): harden scoring edge cases (#2411)
- Use COCO 101-point AP averaging in the legacy mAP path so perfect and imperfect curves score consistently.
- Validate confusion-matrix class ids before indexing and preserve target ignore flags in the COCO-style evaluator.
- Keep mAR per-class recall for each max-detection cutoff and cover the scoring fixes with focused regressions.
- Return empty mAR scores with the same max-detection axis as non-empty results.
- Add an empty-input regression covering recall score and per-class result shapes.
- Update the public mAR docstring to describe per-image detection limits.

---------

Co-authored-by: Codex <codex@openai.com>
2026-07-07 18:46:05 +02:00
Jirka Borovecandclaude[bot] bd0f44fcfd fix: resolve remaining High findings from deep codebase review (#2389)
- Fixed crop annotation so overlapping detections sample from the original scene
- Fixed dataset exports to reject basename collisions, including case-insensitive collisions
- Fixed LMM connector mapping to support mirror enum aliases without a hand-maintained dispatch table
- Updated benchmark documentation to install the released inference package with metrics support

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
2026-07-03 22:58:07 +02:00
Jirka Borovecandclaude[bot] 0e1056df19 fix(metrics): count false positives on empty-GT images (#2397)
- Fixed mAP calculation to count predictions on background-only images as false positives
- Fixed all-background mAP inputs to return 0.0 instead of NaN when no ground-truth classes exist
- Updated `from_tensors` documentation to define empty-target background images and their false-positive behavior

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
2026-07-03 16:17:08 +02:00
Jirka Borovecandclaude[bot] 99049d84e1 Fix: resolve major complex review (#2388)
- Fixed in-memory dict-form `DetectionDataset` image access, iteration, equality, and merge behavior, with deprecation messaging retained
- Fixed mAP to honor `metric_target` for mask and oriented-bounding-box evaluation, including correct IoU routing, area handling, crowd semantics, and missing-content errors
- Fixed `ConfusionMatrix.plot()` when plotting raw counts with default normalization disabled
- Improved mask mAP crowd handling performance and memory usage
- Updated the count-in-zone guide to use current APIs

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
2026-07-02 18:14:04 +02:00
058d8fd990 perf(detection): count mask pixels with count_nonzero (#2361)
Mask pixel-area counting used `np.sum` — `np.array([np.sum(m) for m in
masks])` in `Detections.area` and `np.sum(mask, axis=(1, 2))` in the metrics
`get_mask_size_category`. For boolean masks `np.count_nonzero` (with no axis)
dispatches to NumPy's SIMD popcount over the raw byte buffer, whereas every
axis-reduction form — `np.sum(..., axis=...)` and even `np.count_nonzero(...,
axis=...)` — falls back to a slower generic reduction. So counting per mask
with `np.count_nonzero` is several times faster than the "obvious" vectorized
sum, while producing bit-identical integer counts.

Route both sites through `np.fromiter((np.count_nonzero(m) for m in masks),
dtype=np.int64, count=len(masks))`. `dtype=np.int64` preserves the documented
`Detections.area` mask-branch dtype on every platform (a bare
`np.array([...])` of Python ints would be int32 on Windows).

Measured ~5x on 640x640 masks (e.g. `Detections.area`, N=300: ~24ms -> ~4ms),
faster across densities. `get_mask_size_category` feeds the size-bucketed
F1/Precision/Recall/mAP/mAR metrics, where it is invoked repeatedly per
dataset. Counts are integer-exact (verified over 400 randomized trials plus
empty / all-true / all-false / 1x1 edge cases).

Adds parity tests for `Detections.area` (dense mask) and
`get_mask_size_category` against an `np.sum` reference.

---------

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
2026-07-01 23:22:46 +02:00
beb047095f feat: add compact RLE mask ingestion (#2367)
- Added compact COCO RLE mask ingestion with a `CompactMask` representation and optional compact mask parsing during inference for substantially lower memory usage on sparse segmentation results.
- Added `Detections.to_compact_masks()` to convert existing dense masks into compact masks while preserving detection and collection metadata.
- Improved compact mask decoding performance with cropped RLE processing, batched decoding on the fast path, vectorized decoding for small images, optimized RLE traversal, and faster delta decoding.
- Improved mask metrics to operate directly on `CompactMask` instances, preserving the compact representation while producing results equivalent to dense masks.
- Fixed mixed-modality inference handling by keeping detections and masks aligned, isolating malformed RLE failures to individual predictions where possible, and falling back safely when decoding cannot be completed.
- Fixed compact mask conversion and parsing to preserve dense-mask pixel content across public parsing and slicing paths, while correctly documenting and applying the intended bbox-cropping behavior for compact COCO RLE masks.
- Improved COCO RLE validation with checks for malformed payloads, invalid dimensions, count overflows, image size limits, count-sum mismatches, bounding-box mismatches, and safe fallback behavior for incompatible mask sizes.
- Added inference benchmarks and documentation demonstrating the memory and inference-time characteristics of compact masks, including guidance on their performance tradeoffs and behavior.

---------

Co-authored-by: Codex <codex@openai.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-07-01 12:50:40 +02:00
Jirka Borovecandclaude[bot] 4624dde770 fix(metrics): replace np.unique matching with greedy algorithm (#2380)
The two-pass np.unique deduplication in _match_detection_batch dropped
valid TP assignments when a prediction's best-IoU target was already
claimed by a higher-confidence prediction. The greedy one-pass algorithm
(sort by IoU desc, assign if neither target nor pred already matched)
was already used in _split_detections_by_outcome and
ConfusionMatrix.evaluate_detection_batch but missing from Recall,
F1Score, Precision, MeanAverageRecall, and MeanAveragePrecision.

- Fix all five _match_detection_batch implementations
- Add regression tests reproducing the issue #2378 example in each
  affected metric class (IoU matrix [[1.0, 0.667], [0.333, 0.538]])
- Add kind='stable' to np.argsort in _match_detection_batch across all 5 metric
  implementations (Recall, Precision, F1Score, MeanAverageRecall, deprecated
  MeanAveragePrecision) to ensure deterministic TP assignment when IoU values tie
- Add test_greedy_matching_two_valid_pairs to TestDetectionMetrics covering the
  deprecated MeanAveragePrecision._match_detection_batch with the issue #2378 IoU
  matrix, closing the one missing regression test flagged by /review

* refactor(metrics): extract _greedy_match helper to eliminate 5x duplication

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
2026-07-01 12:30:09 +02:00
0a95bae8a8 chore: bump minimum Python to 3.10 (#2260)
- Drop Python 3.9 from CI test matrix
- requires-python = ">=3.10" in pyproject.toml
- ruff target-version py39 → py310
- mypy python_version 3.9 → 3.10
- Remove Python 3.9 classifier

---------

Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-06-29 14:45:30 +02:00
Jirka BorovecandCodex 10b538373b chore: postpone annotations and add validation refinements (#2357)
Add postponed annotations to test modules and modernize one test helper annotation for Python 3.9-compatible collection.

---------

Co-authored-by: Codex <codex@openai.com>
2026-06-27 08:36:26 +02:00
57bb5e7e8b Add adaptive TP/FP/FN validation mosaic export (#2271)
- remove top-level cv2/annotator imports; lazy-load inside rendering functions
- remove save_result_images bool; save_directory_path is now keyword-only after metric_target
- drop hardcoded result/ subdirectory from benchmark output path
- propagate metric_target into _split_detections_by_outcome for correct OBB IoU dispatch
- add filename collision UserWarning in benchmark loop
- remove dead/unreachable combined None-check in _split_detections_by_outcome
- add Google-style docstrings to all 5 new private visualization functions
- add TestSplitDetectionsByOutcome covering 7 edge cases (empty inputs, cross-class, confidence-None)
- fix FP/FN pixel assertions to check interior box pixels rather than border/title regions
- fix benchmark_a_model.md: full panel names, add Visual Benchmarking section, update API examples

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: jirka <6035284+Borda@users.noreply.github.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
2026-06-26 21:33:49 +02:00
2854932965 fix(metrics): count false positives on background images and absent classes (#2331)
- Fixed Precision and F1Score metrics to correctly count false positives on images with no ground-truth objects
- Fixed Precision and F1Score metrics to include prediction-only classes when computing class statistics and confusion-matrix aggregates
- Corrected MICRO and MACRO averaging so false positives from absent classes affect the score as expected
- Preserved WEIGHTED averaging behavior by weighting only classes with ground-truth support
- Added safeguards for all-background evaluation batches, returning stable zero-valued weighted scores when no support exists
- Fixed handling of predictions with `class_id=None` in background-only evaluation paths
- Added regression coverage for background-image false positives, absent-class predictions, averaging modes, and zero-support edge cases
- Updated metric documentation and result metadata to reflect that tracked classes now include prediction-only classes

---------

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
2026-06-17 19:46:10 +02:00
a549f44792 Add OBB support to ConfusionMatrix via MetricTarget (#2247)
* feat(metrics): OBB support in ConfusionMatrix, MASKS guard, shape validation
* refactor(tests): extract reusable ConfusionMatrix MASKS test cases
* refine(metrics): exclude metric_target from ConfusionMatrix.__eq__
* refactor(metrics): consolidate _assert_supported_target call sites
* fix(metrics): normalise OBB (N,4,2) shape explicitly in detections_to_tensor
* test(metrics): add OBB wrong-target-cols case to validate_input_tensors
* test(metrics): add OBB edge-case parametrize cases to detections_to_tensor
* test(metrics): replace validate_input_tensors with _validate_input_tensors in test_detection
* refactor(metrics): make ConfusionMatrix unhashable
* fix(metrics): add shape validation to evaluate_detection_batch
* fix(metrics): cast class_id to float32 in detections_to_tensor
* test(metrics): fix test_evaluate_detection_batch targets shape

---------

Co-authored-by: jirka <6035284+Borda@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-06-10 13:29:33 +02:00
7d2259669b Fix OBB IoU for non-square canvases (#2282)
* Fix OBB IoU canvas dimensions
* test(metrics): fix OBB metric parametrize — remove MAP, add smoke test
* test(metrics): clarify MAP OBB test is smoke test only
* test(iou): parametrize OBB scaling invariance test with y-dominant case
* fix(iou): add empty-input guard and x/y axis docs to oriented_box_iou_batch

---------

Co-authored-by: jirka <6035284+Borda@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-06-01 09:20:38 +02:00
acbaf4ff25 Correct confusion matrix calculation-function evaluate_detection_batch (#1853)
* Correct confusion matrix calculation-function evaluate_detection_batch
* Correct confusion matrix computation
* Add 3 more tests for empty Detections-GTs
* Minor changes in too-long lines corrected
* Replace deprecated mock_detections with _create_detections
* Fix indentation in test assertions and add missing `self` parameter in `test_confusion_matrix`.
* Update metric computations to improve numerical stability and replace deprecated NumPy functions
* Add IoU+class matching tests and synthetic dataset fixtures for detection metrics
* Use `Optional` for type hinting `classes` parameter in `_yolo_dataset_factory`. Import `Optional` from `typing`.
* Refactor detection metric tests to simplify confusion matrix assertions and enhance IoU+class matching validation.
* Apply suggestions from code review
* fix(pre_commit): 🎨 auto format pre-commit hooks

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-12 18:54:00 +09:00
124be19928 Fix MeanAverageRecall compute mAR@K using top-K detections per image [COCO-compliant] (#2136)
* Supersedes #1967
* fix: COCO-compliant mAR calculation
* Add complex test of mAP
* fix(metrics): cast optional detections fields in mAR metric for mypy
* Add `create_yolo_dataset` utility and refactor tests with fixtures for reusable scenarios
* Refine docstrings for clarity and consistency, adding inline formatting and fixing typos in test helpers and metrics.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: jirka <6035284+Borda@users.noreply.github.com>
2026-02-12 17:59:11 +09:00
Jirka Borovec 8e4fedf3f1 Add match arguments to all pytest.raises for more precise error validation in tests (#2145) 2026-02-10 21:22:55 +09:00
Jirka Borovecandpre-commit-ci[bot] 2d0404e67c Extend docstrings to test cases for enhanced readability (#2130)
* Extend docstrings to test cases for enhanced readability and consistency across testing modules
* Extend test case docstrings across multiple modules for improved clarity and consistency
* Refactor test case docstrings for clarity and consistency in `test_annotators.py`

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-02-03 17:28:33 +01:00
Jirka Borovec b8a2a6996c refactor: reorganize project structure and update module imports (#2112)
* refactor: reorganize project structure and update module imports
* configure setuptools to use 'src' layout and adjust package discovery
* refactor: move tests to `tests` directory and update imports accordingly
* update lint and mypy configurations to align with `src` and `tests` structure
* update test imports to use corrected `tests.helpers` module path
* fix(pre_commit): 🎨 auto format pre-commit hooks
2026-02-03 15:24:47 +01:00