> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.ovalbee.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.ovalbee.com/_mcp/server. # Evaluate prelabeling efficiency > Evaluate how well an object detection model works as a labeling assistant. Reports an assistance score and a breakdown of correct, shifted, class-confused, false positive, and false negative predictions. ## Overview Measures how much labeling work an object detection model's predictions would save, by comparing them with ground truth on the same images. Standard detection metrics answer whether a model is accurate. This one answers a different question: given these predictions, how much correcting is left? Every prediction and every ground-truth object is sorted into correct, shifted, class-confused, false positive, or missed, and the result is an assistance score with an efficiency level shown on the badge. Nothing is recorded on the model in the Models registry. ```mermaid flowchart LR n1["Predict DEIM"] --> n2["Evaluate prelabeling efficiency"] --> n3["Report"] ``` ## Prerequisites | Requirement | What you need | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Ground-truth dataset** | An existing [Dataset](/node-library/nodes/dataset-ops-dataset) whose annotations you trust as correct. | | **Prediction dataset** | The output of a prediction node such as [Predict YOLO](/node-library/nodes/models-yolo-predict-yolo) run on the same image assets, so every ground-truth asset has a prediction to compare against. | | **Labeled by hand** (optional) | A few images labeled from scratch, with no prelabels, in a [Labeling](/node-library/nodes/label-labeling) node - its **Labeled dataset** output. Their time in the labeling tool gives the time one image takes by hand. | ## How it works Drops predictions below the confidence threshold, then matches the rest to ground-truth boxes one to one on each shared image, highest overlap first. Sorts every pair by how much work it leaves: a good enough box, one that needs dragging, one with the wrong class, an invented box, or a missed object. Turns those counts into an assistance score. Adds the score and the breakdown as one point to the history earlier runs kept, one point per prediction dataset version. Scoring the same prelabels again replaces their point. Writes the report with the full breakdown and, from the second point on, how the score changed from cycle to cycle. ## FAQ #### Should I use this node for a live training session's prelabels? No - use [Evaluate live training prelabeling](/node-library/nodes/evaluate-evaluate-live-training-prelabeling). A live session keeps training, so its prelabels come from different weights, and labeling replaces them on the images. That node finds them in the prelabel node's prediction history and charts the score along the session. This node suits a fixed model, such as the output of a Predict node. #### How is this different from normal detection metrics? mAP asks how accurate the model is. This node asks how much of the labeling it does for you, which is not the same thing: a model that draws slightly loose boxes scores poorly on mAP but still saves most of the work, because dragging a corner is faster than drawing a box. #### What do the two IoU thresholds do? **Perfect IoU** is the overlap above which a box is accepted as-is. **Minimal IoU** is the overlap below which a prediction is not considered the same object at all. Between them sit the shifted boxes, the ones a labeler nudges rather than redraws. #### How should I set the thresholds? They encode your own tolerance, so tie them to how the labels get used. Default to the shipped values, then raise **Perfect IoU** if boxes that pass still need fixing in practice. #### What does the efficiency level on the badge mean? It grades the assistance score: **Production-grade** from 0.9, **Strong assist** from 0.6, **Partial assist** from 0.3, and **Minimal assist** below that. Read the report for the counts behind it before acting on a borderline result. #### Why does the confidence threshold matter so much here? It trades invented boxes against missed objects, and those cost a labeler differently: deleting a wrong box is quick, spotting a missing one is not. Raise it when the report shows many false positives. #### What happens when the two datasets do not line up? Only assets present in both are scored. An image that exists in one dataset and not the other contributes nothing, so a partial prediction run quietly shrinks the evaluation set rather than failing. #### Why does the report say no metrics were computed? The two datasets share no image assets, or no boxes were left on the shared images after the confidence filter. The run still succeeds, uploads a report that states the reason, and sets the badge to **Minimal assist**, so check the report before trusting a low grade. #### How are predictions without a confidence tag handled? A prediction box without a numeric `confidence` or `confidence-model` tag counts as fully confident and is always kept. A **Confidence threshold** that is not a number is treated as 0, which keeps every prediction. #### Can I limit the score to some classes? No. Every class in both datasets is scored. Matching ignores class, so a well-placed box with the wrong class counts as class-confused rather than as a false positive plus a miss. #### How does the node track efficiency across cycles? Every run adds a point for its **Prediction dataset** version, so in a loop that prelabels a new batch each cycle the report charts one point per batch. A run on a version it already scored replaces that point, so scoring a batch again as more of it is labeled keeps the latest numbers. A run that scores nothing keeps the history as it was. The history belongs to the node: re-adding it or copying the flow starts a new one. ## Inputs **`Ground truth dataset`** `dataset` — required Dataset with ground truth object annotations. Key: `GT_DATASET`. --- **`Prediction dataset`** `dataset` — required Dataset with prediction annotations (e.g. output of a YOLO predict node). Key: `PRED_DATASET`. --- **`Labeled by hand`** `dataset` Images labeled from scratch in a Labeling node, with no prelabels - its Labeled dataset output. The report takes their time in the labeling tool as the time one image takes by hand, and estimates how much of it the prelabels save. Leave empty to report the scores only. Key: `MANUAL_DATASET`. --- **`Images left to label`** `dataset` Images still to label, such as the rest of the dataset. The report estimates the labeling time the prelabels would save on them, at the assistance score. Used only with Labeled by hand. Key: `REMAINING_DATASET`. --- **`Minimal IoU`** `select` — required, default: 0.3 Smallest overlap at which a prediction still counts as matching a ground truth object. Predictions that match nothing count as false positives. Key: `IOU_MIN`. Options: * 0.10 (`0.1`) * 0.20 (`0.2`) * 0.30 (`0.3`) * 0.40 (`0.4`) * 0.50 (`0.5`) --- **`Perfect IoU`** `select` — required, default: 0.8 Overlap at which a prediction of the right class needs no correction. Matches between the two bounds count as shifted boxes, which a labeler only nudges instead of drawing. Key: `IOU_PERFECT`. Options: * 0.60 (`0.6`) * 0.70 (`0.7`) * 0.80 (`0.8`) * 0.90 (`0.9`) * 0.95 (`0.95`) --- **`Confidence threshold`** `string` — required, default: 0.5 Predictions scoring below this confidence are left out of the score. Enter a number from 0 to 1, for example 0.5. Key: `CONFIDENCE_THRESHOLD`. --- ## Outputs **`Report`** `report` Asset ID containing the prelabeling efficiency report. Shown as an artifact. Key: `REPORT_ID`. --- **`Efficiency Level`** `string` Prelabeling efficiency level shown as a badge on the node. Shown on the node as a badge. Key: `EFFICIENCY_LEVEL`. --- **`Summary`** `object` Table the node card shows - the time saved, the assistance score and what labelers did with the boxes. Hidden from the flow editor. Key: `SUMMARY`. --- **`History`** `object` Score and corrections of every batch scored so far, kept so each run can chart them over cycles. Hidden from the flow editor. Key: `HISTORY`. --- ## Models and configuration The node compares two datasets, so there is no model to select. What you configure is the definition of "good enough". **Confidence threshold** drops weak predictions before anything is matched. **Minimal IoU** and **Perfect IoU** split the surviving predictions into accepted, needs-nudging, and not-the-same-object. #### Outcomes and the assistance score | Outcome | Definition | | ------------------ | -------------------------------------------------------------------------------------- | | **Correct** | Matched a ground-truth box of the same class with overlap at or above **Perfect IoU**. | | **Shifted** | Same class, overlap between **Minimal IoU** and **Perfect IoU**. | | **Class-confused** | Overlap at or above **Minimal IoU**, but a different class. | | **False positive** | A prediction that matched no ground-truth box. | | **False negative** | A ground-truth box that no prediction matched. | The assistance score is (correct + 0.5 x shifted + 0.5 x class-confused) divided by the total of all five outcomes. Shifted and class-confused boxes count as half the work saved, because a labeler only adjusts them. The report counts boxes per outcome, each with its share of that same total. With **Labeled by hand**, the time saved is the time one image takes by hand, from the labeling tool, times the score and the number of images. ## Report The report opens in the preview widget. The **Overview** tab comes first. A verdict sentence says how much of the work the predictions save and whether the rest leans towards drawing or deleting. Then come the assistance score gauge with its grade, and cards for the score, the time saved and the number of boxes kept as is, adjusted, with a changed class, deleted and missed. With **Labeled by hand**, a **Labeling time** section shows the time one image takes by hand and with prelabels, and the time saved on **Images left to label** - or, without it, on the compared images. A collapsed block lists the number of images, ground-truth objects, and predictions, and the confidence threshold used. From the second cycle on, the report gets a second tab, **Over cycles**: a sentence on how far the score moved since the first and the previous cycle, a line chart of the assistance score per cycle with the efficiency levels marked, and a stacked bar chart of the share of boxes labelers kept, adjusted, deleted and drew by hand in each cycle. ## Runtime Runs on an ordinary worker. Cost scales with the number of annotations compared, not with image size, because no model is executed. ## JSON config Machine-readable node interface for automation and advanced usage. #### Show JSON ```json { "name": "Evaluate prelabeling efficiency", "description": "Evaluate how well an object detection model works as a labeling assistant. Reports an assistance score and a breakdown of correct, shifted, class-confused, false positive, and false negative predictions.", "category": "Evaluate", "namespace": null, "templateKey": "evaluate/evaluate_prelabeling_efficiency", "version": "v1", "inputs": [ { "key": "GT_DATASET", "label": "Ground truth dataset", "type": "dataset", "description": "Dataset with ground truth object annotations.", "required": true, "default": null, "visibleWhen": null, "options": { "creatable": false } }, { "key": "PRED_DATASET", "label": "Prediction dataset", "type": "dataset", "description": "Dataset with prediction annotations (e.g. output of a YOLO predict node).", "required": true, "default": null, "visibleWhen": null, "options": { "creatable": false } }, { "key": "MANUAL_DATASET", "label": "Labeled by hand", "type": "dataset", "description": "Images labeled from scratch in a Labeling node, with no prelabels - its Labeled dataset output. The report takes their time in the labeling tool as the time one image takes by hand, and estimates how much of it the prelabels save. Leave empty to report the scores only.", "required": false, "default": null, "visibleWhen": null, "options": { "creatable": false } }, { "key": "REMAINING_DATASET", "label": "Images left to label", "type": "dataset", "description": "Images still to label, such as the rest of the dataset. The report estimates the labeling time the prelabels would save on them, at the assistance score. Used only with Labeled by hand.", "required": false, "default": null, "visibleWhen": null, "options": { "creatable": false } }, { "key": "IOU_MIN", "label": "Minimal IoU", "type": "select", "description": "Smallest overlap at which a prediction still counts as matching a ground truth object. Predictions that match nothing count as false positives.", "required": true, "default": "0.3", "visibleWhen": null, "options": { "options": [ { "label": "0.10", "value": "0.1" }, { "label": "0.20", "value": "0.2" }, { "label": "0.30", "value": "0.3" }, { "label": "0.40", "value": "0.4" }, { "label": "0.50", "value": "0.5" } ] } }, { "key": "IOU_PERFECT", "label": "Perfect IoU", "type": "select", "description": "Overlap at which a prediction of the right class needs no correction. Matches between the two bounds count as shifted boxes, which a labeler only nudges instead of drawing.", "required": true, "default": "0.8", "visibleWhen": null, "options": { "options": [ { "label": "0.60", "value": "0.6" }, { "label": "0.70", "value": "0.7" }, { "label": "0.80", "value": "0.8" }, { "label": "0.90", "value": "0.9" }, { "label": "0.95", "value": "0.95" } ] } }, { "key": "CONFIDENCE_THRESHOLD", "label": "Confidence threshold", "type": "string", "description": "Predictions scoring below this confidence are left out of the score. Enter a number from 0 to 1, for example 0.5.", "required": true, "default": "0.5", "visibleWhen": null } ], "outputs": [ { "key": "REPORT_ID", "label": "Report", "type": "report", "kind": "dataset_intelligence_findings_v1", "description": "Asset ID containing the prelabeling efficiency report.", "artifact": true, "badge": false, "badgeOpens": null, "hidden": false }, { "key": "EFFICIENCY_LEVEL", "label": "Efficiency Level", "type": "string", "description": "Prelabeling efficiency level shown as a badge on the node.", "artifact": false, "badge": true, "badgeOpens": null, "hidden": false }, { "key": "SUMMARY", "label": "Summary", "type": "object", "description": "Table the node card shows - the time saved, the assistance score and what labelers did with the boxes.", "artifact": false, "badge": false, "badgeOpens": null, "hidden": true }, { "key": "HISTORY", "label": "History", "type": "object", "description": "Score and corrections of every batch scored so far, kept so each run can chart them over cycles.", "artifact": false, "badge": false, "badgeOpens": null, "hidden": true } ], "widgets": { "widget": { "id": "table", "settings": { "data": { "type": "variable", "value": "self.outputs.SUMMARY" } } }, "secondaryWidget": { "id": "report-preview", "settings": { "reportAsset": { "type": "variable", "value": "self.outputs.REPORT_ID" }, "name": { "type": "input", "value": "Open efficiency report" } } } } } ``` ## References #### Sources and related pages * [Evaluate](/node-library/nodes/evaluate-evaluate-object-detection) computes standard detection metrics for the same pair of datasets.