Node LibraryEvaluate

Evaluate prelabeling efficiency

Overview

Measures how much labeling work an object detection model’s predictions would save, by comparing them with ground truth on the same images.

Standard detection metrics answer whether a model is accurate. This one answers a different question: given these predictions, how much correcting is left? Every prediction and every ground-truth object is sorted into correct, shifted, class-confused, false positive, or missed, and the result is an assistance score with an efficiency level shown on the badge. Nothing is recorded on the model in the Models registry.

Predict DEIM Evaluate prelabeling efficiency Report

Prerequisites

RequirementWhat you need
Ground-truth datasetAn existing Dataset whose annotations you trust as correct.
Prediction datasetThe output of a prediction node such as Predict YOLO run on the same image assets, so every ground-truth asset has a prediction to compare against.

How it works

1

Drops predictions below the confidence threshold, then matches the rest to ground-truth boxes one to one on each shared image, highest overlap first.

2

Sorts every pair by how much work it leaves: a good enough box, one that needs dragging, one with the wrong class, an invented box, or a missed object.

3

Turns those counts into an assistance score and writes the report with the full breakdown.

FAQ

mAP asks how accurate the model is. This node asks how much of the labeling it does for you, which is not the same thing: a model that draws slightly loose boxes scores poorly on mAP but still saves most of the work, because dragging a corner is faster than drawing a box.

Perfect IoU is the overlap above which a box is accepted as-is. Minimal IoU is the overlap below which a prediction is not considered the same object at all. Between them sit the shifted boxes, the ones a labeler nudges rather than redraws.

They encode your own tolerance, so tie them to how the labels get used. Default to the shipped values, then raise Perfect IoU if boxes that pass still need fixing in practice.

It grades the assistance score: Production-grade from 0.9, Strong assist from 0.6, Partial assist from 0.3, and Minimal assist below that. Read the report for the counts behind it before acting on a borderline result.

It trades invented boxes against missed objects, and those cost a labeler differently: deleting a wrong box is quick, spotting a missing one is not. Raise it when the report shows many false positives.

Only assets present in both are scored. An image that exists in one dataset and not the other contributes nothing, so a partial prediction run quietly shrinks the evaluation set rather than failing.

The two datasets share no image assets, or no boxes were left on the shared images after the confidence filter. The run still succeeds, uploads a report that states the reason, and sets the badge to Minimal assist, so check the report before trusting a low grade.

A prediction box without a numeric confidence or confidence-model tag counts as fully confident and is always kept. A Confidence threshold that is not a number is treated as 0, which keeps every prediction.

No. Every class in both datasets is scored. Matching ignores class, so a well-placed box with the wrong class counts as class-confused rather than as a false positive plus a miss.

Inputs

Ground truth dataset
datasetRequired

Dataset with ground truth object annotations. Key: GT_DATASET.

Prediction dataset
datasetRequired

Dataset with prediction annotations (e.g. output of a YOLO predict node). Key: PRED_DATASET.

Minimal IoU
selectDefaults to 0.3Required

Smallest overlap at which a prediction still counts as matching a ground truth object. Predictions that match nothing count as false positives. Key: IOU_MIN.

Options:

  • 0.10 (0.1)
  • 0.20 (0.2)
  • 0.30 (0.3)
  • 0.40 (0.4)
  • 0.50 (0.5)
Perfect IoU
selectDefaults to 0.8Required

Overlap at which a prediction of the right class needs no correction. Matches between the two bounds count as shifted boxes, which a labeler only nudges instead of drawing. Key: IOU_PERFECT.

Options:

  • 0.60 (0.6)
  • 0.70 (0.7)
  • 0.80 (0.8)
  • 0.90 (0.9)
  • 0.95 (0.95)
Confidence threshold
stringDefaults to 0.5Required

Predictions scoring below this confidence are left out of the score. Enter a number from 0 to 1, for example 0.5. Key: CONFIDENCE_THRESHOLD.

Outputs

Report
report

Asset ID containing the prelabeling efficiency report. Shown as an artifact. Key: REPORT_ID.

Efficiency Level
string

Prelabeling efficiency level shown as a badge on the node. Shown on the node as a badge. Key: EFFICIENCY_LEVEL.

Models and configuration

The node compares two datasets, so there is no model to select. What you configure is the definition of “good enough”.

Confidence threshold drops weak predictions before anything is matched. Minimal IoU and Perfect IoU split the surviving predictions into accepted, needs-nudging, and not-the-same-object.

OutcomeDefinition
CorrectMatched a ground-truth box of the same class with overlap at or above Perfect IoU.
ShiftedSame class, overlap between Minimal IoU and Perfect IoU.
Class-confusedOverlap at or above Minimal IoU, but a different class.
False positiveA prediction that matched no ground-truth box.
False negativeA ground-truth box that no prediction matched.

The assistance score is (correct + 0.5 x shifted + 0.5 x class-confused) divided by the total of all five outcomes. Shifted and class-confused boxes count as half the work saved, because a labeler only adjusts them. The percentages in the report are shares of that same total. Mean IoU averages the overlap of matched boxes of the same class.

Report

The report opens in the preview widget.

It is a single page. A verdict sentence says how much of the work the predictions save and whether the rest leans towards drawing or deleting. Then come the assistance score gauge with its grade, and cards for correct, shifted, class-confused, false positive, and false negative shares plus mean IoU. A collapsed block lists the number of images, ground-truth objects, and predictions, and the confidence threshold used.

Runtime

Runs on an ordinary worker. Cost scales with the number of annotations compared, not with image size, because no model is executed.

JSON config

Machine-readable node interface for automation and advanced usage.

{
  "name": "Evaluate prelabeling efficiency",
  "description": "Evaluate how well an object detection model works as a labeling assistant. Reports an assistance score and a breakdown of correct, shifted, class-confused, false positive, and false negative predictions.",
  "category": "Evaluate",
  "namespace": null,
  "templateKey": "evaluate/evaluate_prelabeling_efficiency",
  "version": "v1",
  "inputs": [
    {
      "key": "GT_DATASET",
      "label": "Ground truth dataset",
      "type": "dataset",
      "description": "Dataset with ground truth object annotations.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "PRED_DATASET",
      "label": "Prediction dataset",
      "type": "dataset",
      "description": "Dataset with prediction annotations (e.g. output of a YOLO predict node).",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "IOU_MIN",
      "label": "Minimal IoU",
      "type": "select",
      "description": "Smallest overlap at which a prediction still counts as matching a ground truth object. Predictions that match nothing count as false positives.",
      "required": true,
      "default": "0.3",
      "visibleWhen": null,
      "options": {
        "options": [
          {
            "label": "0.10",
            "value": "0.1"
          },
          {
            "label": "0.20",
            "value": "0.2"
          },
          {
            "label": "0.30",
            "value": "0.3"
          },
          {
            "label": "0.40",
            "value": "0.4"
          },
          {
            "label": "0.50",
            "value": "0.5"
          }
        ]
      }
    },
    {
      "key": "IOU_PERFECT",
      "label": "Perfect IoU",
      "type": "select",
      "description": "Overlap at which a prediction of the right class needs no correction. Matches between the two bounds count as shifted boxes, which a labeler only nudges instead of drawing.",
      "required": true,
      "default": "0.8",
      "visibleWhen": null,
      "options": {
        "options": [
          {
            "label": "0.60",
            "value": "0.6"
          },
          {
            "label": "0.70",
            "value": "0.7"
          },
          {
            "label": "0.80",
            "value": "0.8"
          },
          {
            "label": "0.90",
            "value": "0.9"
          },
          {
            "label": "0.95",
            "value": "0.95"
          }
        ]
      }
    },
    {
      "key": "CONFIDENCE_THRESHOLD",
      "label": "Confidence threshold",
      "type": "string",
      "description": "Predictions scoring below this confidence are left out of the score. Enter a number from 0 to 1, for example 0.5.",
      "required": true,
      "default": "0.5",
      "visibleWhen": null
    }
  ],
  "outputs": [
    {
      "key": "REPORT_ID",
      "label": "Report",
      "type": "report",
      "kind": "dataset_intelligence_findings_v1",
      "description": "Asset ID containing the prelabeling efficiency report.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "EFFICIENCY_LEVEL",
      "label": "Efficiency Level",
      "type": "string",
      "description": "Prelabeling efficiency level shown as a badge on the node.",
      "artifact": false,
      "badge": true,
      "badgeOpens": null,
      "hidden": false
    }
  ],
  "widgets": {
    "widget": {
      "id": "report-preview",
      "settings": {
        "reportAsset": {
          "type": "variable",
          "value": "self.outputs.REPORT_ID"
        },
        "name": {
          "type": "input",
          "value": "Open efficiency report"
        }
      }
    }
  }
}

References