> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.ovalbee.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.ovalbee.com/_mcp/server. # Evaluate live training prelabeling > Measure how much of the labelers' work a live training session's prelabels saved, and how that changed as the session trained. Compares each labeled image with the prediction the labeler started from and charts the assistance score over training iterations. ## Overview Measures how much of the labelers' work a live training session's prelabels saved, and how that changed as the session trained. Each image a labeler finished is compared with the prediction it was opened with, and the scores are charted along the session's training iterations. Use it instead of [Evaluate prelabeling efficiency](/node-library/nodes/evaluate-evaluate-prelabeling-efficiency) when the prelabels come from [Prelabel with live training](/node-library/nodes/models-general-prelabel-with-live-training). A live session's weights change all the time, so one number for the whole run says little; this node shows whether the session takes on more of the drawing as labelers keep submitting. ```mermaid flowchart LR n1["Prelabel with live training"] -->|Prediction history| n3["Evaluate live training prelabeling"] n2["Labeling and Review"] -->|Accepted| n3 n3 --> n4["Report"] ``` ## Prerequisites | Requirement | What you need | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Prediction history** | The **Prediction history** output of [Prelabel with live training](/node-library/nodes/models-general-prelabel-with-live-training), which keeps every image it pre-labeled with the prediction a labeler opened it with. | | **Labeled images** | Images labelers finished, from a [Labeling](/node-library/nodes/label-labeling) step and usually a [Review](/node-library/nodes/label-review) step after it - the accepted dataset, or one that collects accepted images. | | **Labeled by hand** (optional) | A few images labeled from scratch, with no prelabels, in a [Labeling](/node-library/nodes/label-labeling) node - its **Labeled dataset** output. Their time in the labeling tool gives the time one image takes by hand. | ## How it works Reads the labeled images and the prediction history, each image with its annotation's boxes. Leaves out images the prelabel node accepted on its own, since nobody corrected them, and images that reached labeling without a prediction. Matches every labeled box with the predicted boxes above the confidence threshold and sorts each into kept as is, box adjusted, class changed, deleted or missed. Orders the images by session and by the training iteration of the weights that predicted them, and averages them into points of **Images per point** images each. When **Labeled by hand** is given, adds up the time the labeling tool measured on its images, idle time left out, and estimates the labeling time the prelabels save at the overall assistance score: on the compared images, and on **Images left to label**. Writes the assistance score, the time saved, the boxes per outcome, the labeling time and the curves along the session into the report, and puts the score's efficiency level on the badge. The node card shows the time saved, the score and the number of boxes kept as is, adjusted, deleted and missed. ## FAQ #### Which prediction is each image compared with? The one the labeler opened it with. The prediction history keeps each image's newest prediction, and pre-labeling never replaces the boxes of an image a labeler has saved, so the newest prediction is the one the labeler started from. #### Why are some labeled images left out? Two kinds are left out, and the report counts both. An image the prelabel node accepted on its own still carries the session's boxes untouched, so comparing it would compare a prediction with itself. An image that went to labeling raw, while the session was not yet good enough to prelabel, has no prediction to compare. #### How is this different from Evaluate prelabeling efficiency? [Evaluate prelabeling efficiency](/node-library/nodes/evaluate-evaluate-prelabeling-efficiency) scores a fixed model's predictions on a set of images and gives one number per run. This node follows a model that keeps training: each image is scored against the weights that predicted it, and the report shows the score along the session. Both grade boxes the same way, so their assistance scores compare. #### What happens when the session is restarted? A restarted session counts its iterations from zero again. Its points come after the earlier session's, labeled with the session's number, so the curve still reads oldest to newest. #### Which dataset should I pick for Labeled images? The dataset accepted images end up in. When a later step moves them on - into a dataset that collects every accepted image - pick that one instead, so the score covers everything labeled so far rather than only the images waiting in Accepted. #### How is the labeling time saved estimated? The assistance score is the share of a labeler's work the prelabels take over, so an image keeps 1 - score of the time it takes by hand. The time by hand is the average time in the labeling tool of the images in **Labeled by hand**. It is an estimate: deleting a wrong box or fixing a shifted one also takes a moment, and a few images give only a rough time per image. Label the images of **Labeled by hand** without any prelabels, or the time comes out too short. #### Why does the badge say Not scored yet? No labeled image had a prediction in the history to compare with. Check that **Prediction history** is the prelabel node's output and that **Labeled images** is where its images end up once labeled. The run still succeeds and uploads a report that says so. ## Inputs **`Labeled images`** `dataset` — required Images labelers finished, with their annotations - Accepted, or a dataset that collects accepted images. Images the prelabel node accepted on its own, with nobody labeling them, are left out. Key: `GT_DATASET`. --- **`Prediction history`** `dataset` — required The Prediction history output of Prelabel with live training - every image it pre-labeled, with the prediction a labeler opened it with. Key: `PRED_DATASET`. --- **`Images per point`** `number` — required, default: 20 How many labeled images one point of the chart averages, taken in the order the session predicted them. Fewer gives a finer curve, more a steadier one. Minimum: `1`. Maximum: `1000`. Step: `1`. Key: `IMAGES_PER_POINT`. --- **`Labeled by hand`** `dataset` Images labeled from scratch in a Labeling node, with no prelabels - its Labeled dataset output. The report takes their time in the labeling tool as the time one image takes by hand, and estimates how much of it the prelabels save. Leave empty to report the scores only. Key: `MANUAL_DATASET`. --- **`Images left to label`** `dataset` Images still to label, such as the rest of the dataset. The report estimates the labeling time the prelabels would save on them, at the assistance score. Used only with Labeled by hand. Key: `REMAINING_DATASET`. --- **`Minimal IoU`** `select` — required, default: 0.3 Smallest overlap at which a prediction still counts as matching a labeled object. Predictions that match nothing count as false positives. Key: `IOU_MIN`. Options: * 0.10 (`0.1`) * 0.20 (`0.2`) * 0.30 (`0.3`) * 0.40 (`0.4`) * 0.50 (`0.5`) --- **`Perfect IoU`** `select` — required, default: 0.8 Overlap at which a prediction of the right class needs no correction. Matches between the two bounds count as shifted boxes, which a labeler only nudges instead of drawing. Key: `IOU_PERFECT`. Options: * 0.60 (`0.6`) * 0.70 (`0.7`) * 0.80 (`0.8`) * 0.90 (`0.9`) * 0.95 (`0.95`) --- **`Confidence threshold`** `string` — required, default: 0.3 Predictions scoring below this confidence are left out of the score. Enter a number from 0 to 1, for example 0.3. Key: `CONFIDENCE_THRESHOLD`. --- ## Outputs **`Report`** `report` Asset ID of the report with the overall score and its curve over the session. Shown as an artifact. Key: `REPORT_ID`. --- **`Efficiency Level`** `string` Efficiency level of the assistance score, shown as a badge on the node. Shown on the node as a badge. Key: `EFFICIENCY_LEVEL`. --- **`Summary`** `object` Table the node card shows - the time saved, the assistance score and what labelers did with the boxes. Hidden from the flow editor. Key: `SUMMARY`. --- ## Models and configuration The node compares two datasets, so there is no model to select. **Confidence threshold** drops weak predicted boxes before matching, and **Minimal IoU** and **Perfect IoU** split the rest into kept as is, box adjusted and not the same object, as in [Evaluate prelabeling efficiency](/node-library/nodes/evaluate-evaluate-prelabeling-efficiency). The assistance score is (kept as is + 0.5 x box adjusted + 0.5 x class changed) divided by all outcomes. **Images per point** sets how many images one point of the curve averages. ## Report The report helps decide whether the session's prelabels are worth keeping and whether more labeling still improves them. #### Report sections * **Overview** - a verdict with the assistance score, the boxes kept and drawn by hand and, with **Labeled by hand**, the time saved; a gauge of the score with its efficiency level; cards for the score, the time saved and the number of boxes kept as is, adjusted, with a changed class, deleted and missed; a note on any labeled images left out of the comparison; and the values this run was given. * **Labeling time** - only with **Labeled by hand**: the time one image takes by hand and with prelabels, and the time saved on the images left to label - or, without **Images left to label**, on the compared images. * **Over the session** - a line chart of the assistance score per point with dashed lines where each efficiency level starts, and a stacked bar chart of the share of boxes labelers kept, adjusted, deleted and drew by hand per point. ## Runtime Runs on an ordinary worker. Cost scales with the number of annotations compared, not with image size, because no model is executed. ## JSON config Machine-readable node interface for automation and advanced usage. #### Show JSON ```json { "name": "Evaluate live training prelabeling", "description": "Measure how much of the labelers' work a live training session's prelabels saved, and how that changed as the session trained. Compares each labeled image with the prediction the labeler started from and charts the assistance score over training iterations.", "category": "Evaluate", "namespace": null, "templateKey": "evaluate/evaluate_live_training_prelabeling", "version": "v1", "inputs": [ { "key": "GT_DATASET", "label": "Labeled images", "type": "dataset", "description": "Images labelers finished, with their annotations - Accepted, or a dataset that collects accepted images. Images the prelabel node accepted on its own, with nobody labeling them, are left out.", "required": true, "default": null, "visibleWhen": null, "options": { "creatable": false } }, { "key": "PRED_DATASET", "label": "Prediction history", "type": "dataset", "description": "The Prediction history output of Prelabel with live training - every image it pre-labeled, with the prediction a labeler opened it with.", "required": true, "default": null, "visibleWhen": null, "options": { "creatable": false } }, { "key": "IMAGES_PER_POINT", "label": "Images per point", "type": "number", "description": "How many labeled images one point of the chart averages, taken in the order the session predicted them. Fewer gives a finer curve, more a steadier one.", "required": true, "default": 20, "visibleWhen": null, "options": { "min": 1, "max": 1000, "step": 1 } }, { "key": "MANUAL_DATASET", "label": "Labeled by hand", "type": "dataset", "description": "Images labeled from scratch in a Labeling node, with no prelabels - its Labeled dataset output. The report takes their time in the labeling tool as the time one image takes by hand, and estimates how much of it the prelabels save. Leave empty to report the scores only.", "required": false, "default": null, "visibleWhen": null, "options": { "creatable": false } }, { "key": "REMAINING_DATASET", "label": "Images left to label", "type": "dataset", "description": "Images still to label, such as the rest of the dataset. The report estimates the labeling time the prelabels would save on them, at the assistance score. Used only with Labeled by hand.", "required": false, "default": null, "visibleWhen": null, "options": { "creatable": false } }, { "key": "IOU_MIN", "label": "Minimal IoU", "type": "select", "description": "Smallest overlap at which a prediction still counts as matching a labeled object. Predictions that match nothing count as false positives.", "required": true, "default": "0.3", "visibleWhen": null, "options": { "options": [ { "label": "0.10", "value": "0.1" }, { "label": "0.20", "value": "0.2" }, { "label": "0.30", "value": "0.3" }, { "label": "0.40", "value": "0.4" }, { "label": "0.50", "value": "0.5" } ] } }, { "key": "IOU_PERFECT", "label": "Perfect IoU", "type": "select", "description": "Overlap at which a prediction of the right class needs no correction. Matches between the two bounds count as shifted boxes, which a labeler only nudges instead of drawing.", "required": true, "default": "0.8", "visibleWhen": null, "options": { "options": [ { "label": "0.60", "value": "0.6" }, { "label": "0.70", "value": "0.7" }, { "label": "0.80", "value": "0.8" }, { "label": "0.90", "value": "0.9" }, { "label": "0.95", "value": "0.95" } ] } }, { "key": "CONFIDENCE_THRESHOLD", "label": "Confidence threshold", "type": "string", "description": "Predictions scoring below this confidence are left out of the score. Enter a number from 0 to 1, for example 0.3.", "required": true, "default": "0.3", "visibleWhen": null } ], "outputs": [ { "key": "REPORT_ID", "label": "Report", "type": "report", "description": "Asset ID of the report with the overall score and its curve over the session.", "artifact": true, "badge": false, "badgeOpens": null, "hidden": false }, { "key": "EFFICIENCY_LEVEL", "label": "Efficiency Level", "type": "string", "description": "Efficiency level of the assistance score, shown as a badge on the node.", "artifact": false, "badge": true, "badgeOpens": null, "hidden": false }, { "key": "SUMMARY", "label": "Summary", "type": "object", "description": "Table the node card shows - the time saved, the assistance score and what labelers did with the boxes.", "artifact": false, "badge": false, "badgeOpens": null, "hidden": true } ], "widgets": { "widget": { "id": "table", "settings": { "data": { "type": "variable", "value": "self.outputs.SUMMARY" } } }, "secondaryWidget": { "id": "report-preview", "settings": { "reportAsset": { "type": "variable", "value": "self.outputs.REPORT_ID" }, "name": { "type": "input", "value": "Open prelabeling report" } } } } } ``` ## References #### Sources and related pages * [Prelabel with live training](/node-library/nodes/models-general-prelabel-with-live-training) writes the prediction history this node reads. * [Evaluate prelabeling efficiency](/node-library/nodes/evaluate-evaluate-prelabeling-efficiency) scores a fixed model's prelabels the same way.