Compare object detectors

Overview

Ranks two to five object detection evaluations of the same ground truth, or measures how revising the ground truth changes the score of one fixed model.

In a model comparison, the report puts every model on the same metrics and breaks the difference down by class, by image, and, when every report has the data, by error type and object size. The model with the highest mAP 50:95 becomes the winner: its model version, evaluation report, and evaluation are published as outputs so the flow can continue with it, and the badge names it.

To compare annotation revisions instead, connect the outputs of two Evaluate object detection nodes, one run on the original ground truth and one on the revised copy, and set Evaluation dataset to the original. That report shows original, revised, and changed metrics, per-class changes, and per-image comparisons against each version’s own ground truth. The winner outputs stay empty and the badge shows the mAP change.

Prerequisites

Requirement	What you need
Ground-truth dataset	The Dataset every selected evaluation was measured against.
Two to five evaluations	Evaluations recorded by Evaluate object detection with the same IoU threshold, max detections, and confidence setting, each with its report still available.
Revised ground truth (optional)	For an annotation revision comparison only: a second dataset with the same images and classes, evaluated with the same model version, the same prediction annotations, and the same manual confidence threshold.

How it works

Reads the report of every selected evaluation and checks that they can be compared.

Treats evaluations of one dataset as a model comparison, and two evaluations of different datasets as an annotation revision comparison.

In a model comparison, ranks the models by mAP 50:95 and picks the single best, or reports a tie. In a revision comparison, computes the change in each metric.

Publishes the comparison report and sets the outputs for that kind of comparison.

FAQ

How many models can I compare at once?

Between two and five, and the same evaluation cannot be picked twice. Selection order does not matter, because the ranking comes from the metrics rather than from the order you picked.

Why were my evaluations rejected?

In a model comparison, every evaluation must be of the Evaluation dataset, cover the same images with the same ground-truth annotations and classes, and use the same IoU threshold, max detections, and confidence setting: auto on all of them, or the same manual value. An evaluation whose report was deleted cannot be compared. Re-run the evaluations with matching settings before comparing.

Why was my annotation revision comparison rejected?

It takes exactly two evaluations, and exactly one of them must be of the Evaluation dataset. Both must use the same model version, the same images and classes, the same prediction annotations, and the same IoU threshold, max detections, and manual confidence threshold. Set a fixed confidence on both Evaluate nodes, because auto can pick a different threshold for each ground truth.

What happens when two models tie for the top?

The winner outputs stay empty and the badge reads Tie, rather than picking one arbitrarily. Break the tie on a metric you care about by reading the comparison report.

What does the error decomposition compare?

Every prediction that is not a true positive is assigned exactly one cause (wrong class, loose box, both, duplicate, or a detection where nothing is annotated), and every unrecoverable ground-truth object counts as a miss. The report charts each model’s mix of those causes, and the AP each model would recover if one cause were fixed on its own. Two models with the same mAP often miss it for different reasons, and that is the number that says which.

Why are the error decomposition and object size sections missing?

They appear only when every selected report carries that data, and reports made by older runs of Evaluate object detection don’t. Re-run the evaluations on the same dataset and compare the new ones. The rest of the comparison works either way.

Does it pick up a new evaluation after I rerun Evaluate?

Only when the input is bound to the Evaluate node’s Evaluation output. An evaluation picked from the list stays that exact evaluation, however many times the Evaluate node runs again.

Why didn't it run when the evaluations finished?

Require configuration is on for this node, so the flow skips it until the evaluations are set. Pick or bind them, then run the node or the flow again.

Does it change anything on the models?

No. It records no evaluation and changes nothing in the model registry. The winner exists only as the node’s outputs and badge.

Inputs

Evaluation dataset

datasetRequired

Ground-truth dataset for detector comparison, or the original dataset for annotation revision comparison. Key: GT_DATASET.

Object detection evaluations

model_evaluationRequired

Two to five evaluations of this dataset to compare, in any order. To compare annotation revisions instead, connect the outputs of the original and the revised Evaluate node. Values come from GT_DATASET. Key: EVALUATIONS.

Outputs

Comparison report

report

Asset ID containing the comparison report. Shown as an artifact. Key: REPORT_ID.

Best Model ID

model

Registry reference of the unique highest-mAP model. Empty when the highest mAP is tied or when comparing annotation revisions. Shown as an artifact. Key: BEST_MODEL_ID.

Best Report ID

report

Unique highest-mAP evaluation report asset ID. Empty when the highest mAP is tied or when comparing annotation revisions. Key: BEST_REPORT_ID.

Best Evaluation ID

model_evaluation

The winning evaluation, ready to wire into another comparison. Empty when the highest mAP is tied or when comparing annotation revisions. Key: BEST_EVALUATION_ID.

Winner

string

Name of the unique winner, or Tie when reports share the highest mAP. An annotation revision comparison shows the mAP change instead. Shown on the node as a badge. Key: WINNER.

Models and configuration

There is nothing to tune. The Object detection evaluations picker offers evaluations of the selected Evaluation dataset only, and the comparison inherits its IoU threshold, max detections, and confidence setting from the evaluations. Every report must come from Evaluate object detection, because a generic report has no metrics this node can line up.

Report

The comparison report opens from the node’s preview widget. In a model comparison it names the winner and shows where each model leads, by class and by image, and which images separate them.

Report sections

Section	What it shows
Header cards	The dataset, the shared evaluation settings, the winner or tie, and a card per model.
Key metrics	mAP, AP50, AP75, precision, recall, F1, and confidence threshold for every model, with each model’s distance from the best mAP.
Average precision, precision, recall, and F1-score by class	One grouped chart per metric, every model side by side.
Error decomposition	Each model’s error mix, error counts by type, and the AP each model would gain by fixing one error type. Shown only when every report carries it.
Object size and crowding	AP by object size, and recall against objects per image and overlap with the nearest object. Shown only when every report carries it.
Per-image comparison	Every image, ordered by the widest F1 spread between models, opening the ground truth beside each model’s predictions.
Outcome counts	TP, FP, and FN for each model at its operating point.
Per-class comparison	How many classes each model wins by AP, and a class-by-model matrix.
Per-image analysis	How many images each model wins by F1, and each model’s per-image F1 distribution.
Images to inspect	Galleries of the images with the biggest disagreement, the images every model finds hard, and the images where the winner falls behind another model.

An annotation revision report is shorter: a table of original, revised, and changed metrics and object counts, per-class changes in AP, precision, recall, and F1, and a per-image table that opens both ground truths beside the same predictions.

Runtime

Runs on an ordinary worker. It reads the selected reports and writes one, so its cost does not depend on dataset size.

JSON config

Machine-readable node interface for automation and advanced usage.

Show JSON

{
  "name": "Compare object detectors",
  "description": "Compare detector evaluations on the same ground truth, or two annotation versions using the same model predictions and evaluation settings.",
  "category": "Evaluate",
  "namespace": null,
  "templateKey": "evaluate/compare_object_detectors",
  "version": "v1",
  "inputs": [
    {
      "key": "GT_DATASET",
      "label": "Evaluation dataset",
      "type": "dataset",
      "description": "Ground-truth dataset for detector comparison, or the original dataset for annotation revision comparison.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "EVALUATIONS",
      "label": "Object detection evaluations",
      "type": "model_evaluation",
      "description": "Two to five evaluations of this dataset to compare, in any order. To compare annotation revisions instead, connect the outputs of the original and the revised Evaluate node.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "ref": "GT_DATASET",
        "task_type": "object_detection",
        "minItems": 2,
        "maxItems": 5
      }
    }
  ],
  "outputs": [
    {
      "key": "REPORT_ID",
      "label": "Comparison report",
      "type": "report",
      "description": "Asset ID containing the comparison report.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "BEST_MODEL_ID",
      "label": "Best Model ID",
      "type": "model",
      "description": "Registry reference of the unique highest-mAP model. Empty when the highest mAP is tied or when comparing annotation revisions.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "BEST_REPORT_ID",
      "label": "Best Report ID",
      "type": "report",
      "kind": "object_detection_evaluation_report_v1",
      "description": "Unique highest-mAP evaluation report asset ID. Empty when the highest mAP is tied or when comparing annotation revisions.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "BEST_EVALUATION_ID",
      "label": "Best Evaluation ID",
      "type": "model_evaluation",
      "description": "The winning evaluation, ready to wire into another comparison. Empty when the highest mAP is tied or when comparing annotation revisions.",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "WINNER",
      "label": "Winner",
      "type": "string",
      "description": "Name of the unique winner, or Tie when reports share the highest mAP. An annotation revision comparison shows the mAP change instead.",
      "artifact": false,
      "badge": true,
      "badgeOpens": null,
      "hidden": false
    }
  ],
  "automation": {
    "requires_configuration": true
  },
  "widgets": {
    "widget": {
      "id": "report-preview",
      "settings": {
        "reportAsset": {
          "type": "variable",
          "value": "self.outputs.REPORT_ID"
        },
        "name": {
          "type": "input",
          "value": "Open comparison"
        }
      }
    }
  }
}