Evaluate live training prelabeling
Overview
Measures how much of the labelers’ work a live training session’s prelabels saved, and how that changed as the session trained. Each image a labeler finished is compared with the prediction it was opened with, and the scores are charted along the session’s training iterations.
Use it instead of Evaluate prelabeling efficiency when the prelabels come from Prelabel with live training. A live session’s weights change all the time, so one number for the whole run says little; this node shows whether the session takes on more of the drawing as labelers keep submitting.
Prerequisites
How it works
Leaves out images the prelabel node accepted on its own, since nobody corrected them, and images that reached labeling without a prediction.
Matches every labeled box with the predicted boxes above the confidence threshold and sorts each into kept as is, box adjusted, class changed, deleted or missed.
Orders the images by session and by the training iteration of the weights that predicted them, and averages them into points of Images per point images each.
FAQ
Which prediction is each image compared with?
The one the labeler opened it with. The prediction history keeps each image’s newest prediction, and pre-labeling never replaces the boxes of an image a labeler has saved, so the newest prediction is the one the labeler started from.
Why are some labeled images left out?
Two kinds are left out, and the report counts both. An image the prelabel node accepted on its own still carries the session’s boxes untouched, so comparing it would compare a prediction with itself. An image that went to labeling raw, while the session was not yet good enough to prelabel, has no prediction to compare.
How is this different from Evaluate prelabeling efficiency?
Evaluate prelabeling efficiency scores a fixed model’s predictions on a set of images and gives one number per run. This node follows a model that keeps training: each image is scored against the weights that predicted it, and the report shows the score along the session. Both grade boxes the same way, so their assistance scores compare.
What happens when the session is restarted?
A restarted session counts its iterations from zero again. Its points come after the earlier session’s, labeled with the session’s number, so the curve still reads oldest to newest.
Which dataset should I pick for Labeled images?
The dataset accepted images end up in. When a later step moves them on - into a dataset that collects every accepted image - pick that one instead, so the score covers everything labeled so far rather than only the images waiting in Accepted.
How is the labeling time saved estimated?
The assistance score is the share of a labeler’s work the prelabels take over, so an image keeps 1 - score of the time it takes by hand. The time by hand is the average time in the labeling tool of the images in Labeled by hand. It is an estimate: deleting a wrong box or fixing a shifted one also takes a moment, and a few images give only a rough time per image. Label the images of Labeled by hand without any prelabels, or the time comes out too short.
Why does the badge say Not scored yet?
No labeled image had a prediction in the history to compare with. Check that Prediction history is the prelabel node’s output and that Labeled images is where its images end up once labeled. The run still succeeds and uploads a report that says so.
Inputs
Images labelers finished, with their annotations - Accepted, or a dataset that collects accepted images. Images the prelabel node accepted on its own, with nobody labeling them, are left out. Key: GT_DATASET.
The Prediction history output of Prelabel with live training - every image it pre-labeled, with the prediction a labeler opened it with. Key: PRED_DATASET.
How many labeled images one point of the chart averages, taken in the order the session predicted them. Fewer gives a finer curve, more a steadier one. Minimum: 1. Maximum: 1000. Step: 1. Key: IMAGES_PER_POINT.
Images labeled from scratch in a Labeling node, with no prelabels - its Labeled dataset output. The report takes their time in the labeling tool as the time one image takes by hand, and estimates how much of it the prelabels save. Leave empty to report the scores only. Key: MANUAL_DATASET.
Images still to label, such as the rest of the dataset. The report estimates the labeling time the prelabels would save on them, at the assistance score. Used only with Labeled by hand. Key: REMAINING_DATASET.
Smallest overlap at which a prediction still counts as matching a labeled object. Predictions that match nothing count as false positives. Key: IOU_MIN.
Options:
- 0.10 (
0.1) - 0.20 (
0.2) - 0.30 (
0.3) - 0.40 (
0.4) - 0.50 (
0.5)
Overlap at which a prediction of the right class needs no correction. Matches between the two bounds count as shifted boxes, which a labeler only nudges instead of drawing. Key: IOU_PERFECT.
Options:
- 0.60 (
0.6) - 0.70 (
0.7) - 0.80 (
0.8) - 0.90 (
0.9) - 0.95 (
0.95)
Predictions scoring below this confidence are left out of the score. Enter a number from 0 to 1, for example 0.3. Key: CONFIDENCE_THRESHOLD.
Outputs
Asset ID of the report with the overall score and its curve over the session. Shown as an artifact. Key: REPORT_ID.
Efficiency level of the assistance score, shown as a badge on the node. Shown on the node as a badge. Key: EFFICIENCY_LEVEL.
Table the node card shows - the time saved, the assistance score and what labelers did with the boxes. Hidden from the flow editor. Key: SUMMARY.
Models and configuration
The node compares two datasets, so there is no model to select. Confidence threshold drops weak predicted boxes before matching, and Minimal IoU and Perfect IoU split the rest into kept as is, box adjusted and not the same object, as in Evaluate prelabeling efficiency. The assistance score is (kept as is + 0.5 x box adjusted + 0.5 x class changed) divided by all outcomes. Images per point sets how many images one point of the curve averages.
Report
The report helps decide whether the session’s prelabels are worth keeping and whether more labeling still improves them.
Report sections
- Overview - a verdict with the assistance score, the boxes kept and drawn by hand and, with Labeled by hand, the time saved; a gauge of the score with its efficiency level; cards for the score, the time saved and the number of boxes kept as is, adjusted, with a changed class, deleted and missed; a note on any labeled images left out of the comparison; and the values this run was given.
- Labeling time - only with Labeled by hand: the time one image takes by hand and with prelabels, and the time saved on the images left to label - or, without Images left to label, on the compared images.
- Over the session - a line chart of the assistance score per point with dashed lines where each efficiency level starts, and a stacked bar chart of the share of boxes labelers kept, adjusted, deleted and drew by hand per point.
Runtime
Runs on an ordinary worker. Cost scales with the number of annotations compared, not with image size, because no model is executed.
JSON config
Machine-readable node interface for automation and advanced usage.