Evaluate prelabeling efficiency
Overview
Measures how much labeling work an object detection model’s predictions would save, by comparing them with ground truth on the same images.
Standard detection metrics answer whether a model is accurate. This one answers a different question: given these predictions, how much correcting is left? Every prediction and every ground-truth object is sorted into correct, shifted, class-confused, false positive, or missed, and the result is an assistance score with an efficiency level shown on the badge. Nothing is recorded on the model in the Models registry.
Prerequisites
How it works
Drops predictions below the confidence threshold, then matches the rest to ground-truth boxes one to one on each shared image, highest overlap first.
Sorts every pair by how much work it leaves: a good enough box, one that needs dragging, one with the wrong class, an invented box, or a missed object.
FAQ
Should I use this node for a live training session's prelabels?
No - use Evaluate live training prelabeling. A live session keeps training, so its prelabels come from different weights, and labeling replaces them on the images. That node finds them in the prelabel node’s prediction history and charts the score along the session. This node suits a fixed model, such as the output of a Predict node.
How is this different from normal detection metrics?
mAP asks how accurate the model is. This node asks how much of the labeling it does for you, which is not the same thing: a model that draws slightly loose boxes scores poorly on mAP but still saves most of the work, because dragging a corner is faster than drawing a box.
What do the two IoU thresholds do?
Perfect IoU is the overlap above which a box is accepted as-is. Minimal IoU is the overlap below which a prediction is not considered the same object at all. Between them sit the shifted boxes, the ones a labeler nudges rather than redraws.
How should I set the thresholds?
They encode your own tolerance, so tie them to how the labels get used. Default to the shipped values, then raise Perfect IoU if boxes that pass still need fixing in practice.
What does the efficiency level on the badge mean?
It grades the assistance score: Production-grade from 0.9, Strong assist from 0.6, Partial assist from 0.3, and Minimal assist below that. Read the report for the counts behind it before acting on a borderline result.
Why does the confidence threshold matter so much here?
It trades invented boxes against missed objects, and those cost a labeler differently: deleting a wrong box is quick, spotting a missing one is not. Raise it when the report shows many false positives.
What happens when the two datasets do not line up?
Only assets present in both are scored. An image that exists in one dataset and not the other contributes nothing, so a partial prediction run quietly shrinks the evaluation set rather than failing.
Why does the report say no metrics were computed?
The two datasets share no image assets, or no boxes were left on the shared images after the confidence filter. The run still succeeds, uploads a report that states the reason, and sets the badge to Minimal assist, so check the report before trusting a low grade.
How are predictions without a confidence tag handled?
A prediction box without a numeric confidence or confidence-model tag counts as fully confident and is always kept. A Confidence threshold that is not a number is treated as 0, which keeps every prediction.
Can I limit the score to some classes?
No. Every class in both datasets is scored. Matching ignores class, so a well-placed box with the wrong class counts as class-confused rather than as a false positive plus a miss.
How does the node track efficiency across cycles?
Every run adds a point for its Prediction dataset version, so in a loop that prelabels a new batch each cycle the report charts one point per batch. A run on a version it already scored replaces that point, so scoring a batch again as more of it is labeled keeps the latest numbers. A run that scores nothing keeps the history as it was. The history belongs to the node: re-adding it or copying the flow starts a new one.
Inputs
Dataset with ground truth object annotations. Key: GT_DATASET.
Dataset with prediction annotations (e.g. output of a YOLO predict node). Key: PRED_DATASET.
Images labeled from scratch in a Labeling node, with no prelabels - its Labeled dataset output. The report takes their time in the labeling tool as the time one image takes by hand, and estimates how much of it the prelabels save. Leave empty to report the scores only. Key: MANUAL_DATASET.
Images still to label, such as the rest of the dataset. The report estimates the labeling time the prelabels would save on them, at the assistance score. Used only with Labeled by hand. Key: REMAINING_DATASET.
Smallest overlap at which a prediction still counts as matching a ground truth object. Predictions that match nothing count as false positives. Key: IOU_MIN.
Options:
- 0.10 (
0.1) - 0.20 (
0.2) - 0.30 (
0.3) - 0.40 (
0.4) - 0.50 (
0.5)
Overlap at which a prediction of the right class needs no correction. Matches between the two bounds count as shifted boxes, which a labeler only nudges instead of drawing. Key: IOU_PERFECT.
Options:
- 0.60 (
0.6) - 0.70 (
0.7) - 0.80 (
0.8) - 0.90 (
0.9) - 0.95 (
0.95)
Predictions scoring below this confidence are left out of the score. Enter a number from 0 to 1, for example 0.5. Key: CONFIDENCE_THRESHOLD.
Outputs
Asset ID containing the prelabeling efficiency report. Shown as an artifact. Key: REPORT_ID.
Prelabeling efficiency level shown as a badge on the node. Shown on the node as a badge. Key: EFFICIENCY_LEVEL.
Table the node card shows - the time saved, the assistance score and what labelers did with the boxes. Hidden from the flow editor. Key: SUMMARY.
Score and corrections of every batch scored so far, kept so each run can chart them over cycles. Hidden from the flow editor. Key: HISTORY.
Models and configuration
The node compares two datasets, so there is no model to select. What you configure is the definition of “good enough”.
Confidence threshold drops weak predictions before anything is matched. Minimal IoU and Perfect IoU split the surviving predictions into accepted, needs-nudging, and not-the-same-object.
Outcomes and the assistance score
The assistance score is (correct + 0.5 x shifted + 0.5 x class-confused) divided by the total of all five outcomes. Shifted and class-confused boxes count as half the work saved, because a labeler only adjusts them. The report counts boxes per outcome, each with its share of that same total. With Labeled by hand, the time saved is the time one image takes by hand, from the labeling tool, times the score and the number of images.
Report
The report opens in the preview widget.
The Overview tab comes first. A verdict sentence says how much of the work the predictions save and whether the rest leans towards drawing or deleting. Then come the assistance score gauge with its grade, and cards for the score, the time saved and the number of boxes kept as is, adjusted, with a changed class, deleted and missed. With Labeled by hand, a Labeling time section shows the time one image takes by hand and with prelabels, and the time saved on Images left to label - or, without it, on the compared images. A collapsed block lists the number of images, ground-truth objects, and predictions, and the confidence threshold used.
From the second cycle on, the report gets a second tab, Over cycles: a sentence on how far the score moved since the first and the previous cycle, a line chart of the assistance score per cycle with the efficiency levels marked, and a stacked bar chart of the share of boxes labelers kept, adjusted, deleted and drew by hand in each cycle.
Runtime
Runs on an ordinary worker. Cost scales with the number of annotations compared, not with image size, because no model is executed.
JSON config
Machine-readable node interface for automation and advanced usage.