Evaluate object detection

Overview

Scores a detector’s predictions against ground truth, publishes a report with overall, per-class, and per-image results, and records the scores on the model version that made the predictions. The node’s badge shows the run’s mAP 50:95.

Use it after Predict YOLO, or another object detection node, has created predictions for a validation or test dataset.

Prerequisites

Requirement	What you need
Ground-truth dataset	Existing Dataset with the reference object annotations.
Prediction dataset	Existing Dataset with model annotations, for example from Predict YOLO. Prediction labels should have a numeric `confidence` or `confidence-model` tag.

The node evaluates only asset IDs found in both datasets. It skips assets found on one side, so check dataset membership when you expect a complete comparison.

How it works

Reads the annotation of every image the two datasets share, as bounding boxes.

Applies the class filter, confidence and IoU operating points, and the per-image detection limit. With auto confidence, it selects the score with the highest aggregate F1 value.

Calculates metrics for the full dataset intersection and each selected ground-truth tag scope. The input datasets stay unchanged.

Uploads the report and records the scores as an evaluation of the picked model version, so they appear wherever that model is listed.

FAQ

What can this node evaluate?

The evaluator works with image bounding boxes. It does not evaluate videos, masks, keypoints, or tracking results. Other supported geometries are reduced to bounding boxes. A run fails when the two datasets together hold more than 600 classes, because the report charts cannot show more.

Why does the node report No common assets with annotations?

The datasets do not share any loaded image asset IDs. Build the prediction dataset from the same source assets instead of uploading copies as new assets.

Why does the node report missing classes, annotations, or predictions?

Class filtering may have removed all supported boxes, or one annotation set may have no valid boxes. Check the class names and make sure both datasets contain valid object annotations.

Why is confidence rejected or the threshold analysis wrong?

The value may be outside 0-1, or prediction labels may lack numeric confidence tags. Missing confidence is treated as 1.0. Use a value from 0 through 1, or add numeric confidence or confidence-model tags.

What happens to images without a recorded size?

They are still evaluated, but their objects are left out of the object-size views (AP by size and error types by size), since relative size cannot be computed. The run log says how many images this affected.

Why is there no report scope for a selected tag value?

No shared ground-truth asset has that tag value. Select a value present on at least one shared image.

What happens when I run it again?

Each run uploads a new report and records a new evaluation on the model version. Earlier reports and evaluations stay, so the model keeps the history of every run.

Why is the Evaluation output empty?

The registry could not be reached when the run finished. The report is already uploaded by then, so a failure here costs the evaluation row, not the run — rerun the node to record it.

Inputs

Ground truth dataset

datasetRequired

Dataset with ground-truth object annotations. Key: GT_DATASET.

Predictions

datasetRequired

Dataset holding the model predictions for the same assets as the ground truth, usually the output of a predict node. Only assets that appear in both datasets are evaluated. Key: PRED_DATASET.

Model

modelRequired

Model version the predictions were made with. This run is recorded as an evaluation of that version, so its scores and a link to this report appear wherever the model is listed. Key: MODEL.

Classes

ontology

Classes to include in the evaluation. Objects of any other class are ignored in both datasets, and an empty selection scores every class. Values come from GT_DATASET. Key: CLASSES.

IoU threshold

selectDefaults to 0.5Required

Overlap a prediction needs with a ground truth object to count as the same object. Drives the confusion matrix and the true, false positive and missed counts. Key: IOU_THRESHOLD.

Options:

0.50 (0.5)
0.55 (0.55)
0.60 (0.6)
0.65 (0.65)
0.70 (0.7)
0.75 (0.75)
0.80 (0.8)
0.85 (0.85)
0.90 (0.9)
0.95 (0.95)

Confidence threshold

stringDefaults to autoRequired

Predictions scoring below this confidence are left out of the metrics. Keep auto to let the run pick the confidence with the best F1 score. Key: CONFIDENCE_THRESHOLD.

Max detections

integerDefaults to 100Required

Most predictions kept per image, highest confidence first. Anything beyond that count is ignored. Minimum: 1. Key: MAX_DETECTIONS.

Split by tags

asset_tags

Ground truth asset tags to break the report down by, for example weather or camera. Each selected tag value gets its own view of the report, and an empty selection reports on the whole dataset. Values come from GT_DATASET. Key: TAGS.

Outputs

Object detection evaluation report

report

Machine-readable object detection evaluation report for preview and compatible comparison nodes. Shown as an artifact. Key: REPORT_ID.

Evaluation

model_evaluation

The evaluation row this run recorded on the model version. Empty when the registry could not be reached. Key: EVALUATION_ID.

mAP

string

mAP score shown as a badge on the node. Shown on the node as a badge. Key: MAP_SCORE.

Confidence threshold

number

Confidence threshold the reported metrics were computed at - the F1-optimal score when Confidence Threshold is auto, otherwise the value you set. Optimal over all classes pooled, not per class. Key: CONF_THRESHOLD.

Models and configuration

This node evaluates existing predictions. It does not load or run a model. Keep the defaults for a standard evaluation, then change only the settings you need.

Optional settings

Goal	Setting
Evaluate all classes	Leave Classes empty.
Test a fixed operating point	Set IoU Threshold and Confidence Threshold instead of using automatic confidence selection.
Compare a class subset	Select the same class names used by both annotation sets.
Compare tagged cohorts	Use Split by Tags to add scopes from ground-truth asset tags.
Attach results to a model	Pick the Model version the predictions came from. Recording the evaluation is best-effort and does not affect report creation.

The Model picker lists your workspace’s own models, filtered to object detection. Each run records one evaluation on the picked version: mAP 50:95, AP50, precision, recall, and F1 at the evaluated operating point, the ground-truth dataset, the list of classes the run scored, and a link to the report. Scores are only comparable between evaluations that used the same ground truth and the same classes.

Report

Start with these five signals. Then open the per-class and per-image views to find the errors.

Metric	What it tells you
mAP 50:95	Overall detection quality across classes and increasingly strict box-overlap thresholds.
AP50 / AP75	Whether errors come mainly from finding objects or placing boxes accurately. A large drop at AP75 suggests localization problems.
Precision	How often predicted objects are correct. Low precision means more false positives.
Recall	How many ground-truth objects are found. Low recall means more missed objects.
F1	The balance between precision and recall at the active confidence threshold.

Metric glossary

Metric	Definition and interpretation
mAP 50:95	Mean Average Precision across classes and COCO IoU thresholds from 0.50 through 0.95.
AP50	Average Precision at IoU 0.50, a permissive measure of whether objects are found.
AP75	Average Precision at IoU 0.75, which places more weight on accurate box localization.
AP at the IoU threshold	Average Precision at the configured IoU threshold. Shown as an extra row when the threshold is neither 0.50 nor 0.75, and as a column of the per-class table.
Precision	The share of predictions counted as correct at the active operating point.
Recall	The share of ground-truth objects found at the active operating point.
F1	The harmonic mean of precision and recall and the criterion used by `auto` confidence selection.
IoU	Mean Intersection over Union for matched true-positive boxes after confidence filtering.
Classification accuracy	The share of localized matches assigned the correct class.
Calibration	`1 - expected calibration error` over ten confidence bins. Values closer to `1` indicate better agreement between confidence and observed correctness.
AP small / medium / large	Average Precision restricted to objects whose box covers under 0.5% of the image area, up to 5%, and above 5%. Relative rather than COCO’s absolute pixel bins, so the same object is binned the same way whatever resolution it was captured at.
AP gain	How much Average Precision the model would recover if one category of error were corrected. Reported both for each error type on its own and for the types corrected cumulatively.

Error types

Every detection that is not a true positive is placed in exactly one category, and every ground-truth object that no correction could recover is counted as a miss. The categories are evaluated in order, so the first one that applies wins.

Error type	Definition
Classification	The box overlaps a ground-truth object closely enough, but carries the wrong class.
Localization	The class is right, but the box overlaps the object more loosely than the IoU threshold requires.
Both	Wrong class and a loose box at the same time.
Duplicate	The box would have been correct, but a higher-scoring detection already claimed that object.
Background	The box overlaps no ground-truth object at all.
Missed	A ground-truth object that no correction to any detection could recover.

Error counts cover every detection the model produced, including those below the confidence threshold, because Average Precision ranks all detections. They therefore do not match the operating-point counts in Outcome counts.

Report views

The report covers the full dataset intersection and each non-empty tag scope. The scope selector switches every view to the selected subset.

Section	What it shows
Overview	A verdict, headline metric cards, every detection metric at the operating point, and the model, dataset, and settings the run was given.
Per-class breakdown	Average Precision by class, and a table with AP, AP at the configured IoU, TP, FP, FN, support, precision, recall, and F1 for every class.
Error decomposition	The share of each error type, the AP each type would add if it were fixed, alone and cumulatively, and the error mix per class.
Confusion and errors	A confusion matrix normalized by ground-truth class and the most frequently confused class pairs.
Outcome counts	TP, FP, and FN totals at the operating point, overall and by class.
Recall and precision	Recall and precision by class, and precision-recall curves by class.
Confidence and calibration	Precision, recall, and F1 across confidence scores and IoU thresholds, the confidence distribution, mean IoU, calibration, and a reliability diagram.
Object size and crowding	Average Precision and error mix by relative object size, and recall against objects per image and overlap with the nearest object.
Suspected annotation issues	Confident predictions that suggest a missing label, a wrong class, or an imprecise box in the ground truth, each opening the ground truth beside the prediction.
Explore predictions	An image gallery and a per-image table of TP, FP, FN, precision, recall, and F1.

Compare object detectors reads the model and dataset identity, categories, per-class metrics, and per-image metrics from this report. Compare reports only when they use the same ground-truth dataset, image set, annotations, categories, and evaluation settings.

Runtime

The node runs on a standard OvalBee worker. It does not need a GPU, CUDA, or a specific RTX model. Runtime grows with the number of shared images, detections, classes, and tag scopes.

JSON config

Machine-readable node interface for automation and advanced usage.

Show JSON

{
  "name": "Evaluate object detection",
  "description": "Calculate object detection metrics by comparing predictions against ground truth, both supplied as separate datasets with their annotations.",
  "category": "Evaluate",
  "namespace": null,
  "templateKey": "evaluate/evaluate_object_detection",
  "version": "v1",
  "inputs": [
    {
      "key": "GT_DATASET",
      "label": "Ground truth dataset",
      "type": "dataset",
      "description": "Dataset with ground-truth object annotations.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "PRED_DATASET",
      "label": "Predictions",
      "type": "dataset",
      "description": "Dataset holding the model predictions for the same assets as the ground truth, usually the output of a predict node. Only assets that appear in both datasets are evaluated.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "MODEL",
      "label": "Model",
      "type": "model",
      "description": "Model version the predictions were made with. This run is recorded as an evaluation of that version, so its scores and a link to this report appear wherever the model is listed.\n",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "scope": "my",
        "task_type": "object_detection"
      }
    },
    {
      "key": "CLASSES",
      "label": "Classes",
      "type": "ontology",
      "description": "Classes to include in the evaluation. Objects of any other class are ignored in both datasets, and an empty selection scores every class.",
      "required": false,
      "default": null,
      "visibleWhen": null,
      "options": {
        "ref": "GT_DATASET",
        "selectable": true
      }
    },
    {
      "key": "IOU_THRESHOLD",
      "label": "IoU threshold",
      "type": "select",
      "description": "Overlap a prediction needs with a ground truth object to count as the same object. Drives the confusion matrix and the true, false positive and missed counts.",
      "required": true,
      "default": "0.5",
      "visibleWhen": null,
      "options": {
        "options": [
          {
            "label": "0.50",
            "value": "0.5"
          },
          {
            "label": "0.55",
            "value": "0.55"
          },
          {
            "label": "0.60",
            "value": "0.6"
          },
          {
            "label": "0.65",
            "value": "0.65"
          },
          {
            "label": "0.70",
            "value": "0.7"
          },
          {
            "label": "0.75",
            "value": "0.75"
          },
          {
            "label": "0.80",
            "value": "0.8"
          },
          {
            "label": "0.85",
            "value": "0.85"
          },
          {
            "label": "0.90",
            "value": "0.9"
          },
          {
            "label": "0.95",
            "value": "0.95"
          }
        ]
      }
    },
    {
      "key": "CONFIDENCE_THRESHOLD",
      "label": "Confidence threshold",
      "type": "string",
      "description": "Predictions scoring below this confidence are left out of the metrics. Keep auto to let the run pick the confidence with the best F1 score.",
      "required": true,
      "default": "auto",
      "visibleWhen": null
    },
    {
      "key": "MAX_DETECTIONS",
      "label": "Max detections",
      "type": "number",
      "description": "Most predictions kept per image, highest confidence first. Anything beyond that count is ignored.",
      "required": true,
      "default": 100,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1
      }
    },
    {
      "key": "TAGS",
      "label": "Split by tags",
      "type": "asset_tags",
      "description": "Ground truth asset tags to break the report down by, for example weather or camera. Each selected tag value gets its own view of the report, and an empty selection reports on the whole dataset.",
      "required": false,
      "default": null,
      "visibleWhen": null,
      "options": {
        "ref": "GT_DATASET",
        "selectable": true
      }
    }
  ],
  "outputs": [
    {
      "key": "REPORT_ID",
      "label": "Object detection evaluation report",
      "type": "report",
      "kind": "object_detection_evaluation_report_v1",
      "description": "Machine-readable object detection evaluation report for preview and compatible comparison nodes.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "EVALUATION_ID",
      "label": "Evaluation",
      "type": "model_evaluation",
      "description": "The evaluation row this run recorded on the model version. Empty when the registry could not be reached.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "MAP_SCORE",
      "label": "mAP",
      "type": "string",
      "description": "mAP score shown as a badge on the node.",
      "artifact": false,
      "badge": true,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "CONF_THRESHOLD",
      "label": "Confidence threshold",
      "type": "number",
      "description": "Confidence threshold the reported metrics were computed at - the F1-optimal score when Confidence Threshold is auto, otherwise the value you set. Optimal over all classes pooled, not per class.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    }
  ],
  "widgets": {
    "widget": {
      "id": "report-preview",
      "settings": {
        "reportAsset": {
          "type": "variable",
          "value": "self.outputs.REPORT_ID"
        },
        "name": {
          "type": "input",
          "value": "Open evaluation"
        }
      }
    }
  }
}