> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.ovalbee.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.ovalbee.com/_mcp/server. # Evaluate object detection > Calculate object detection metrics by comparing predictions against ground truth, both supplied as separate datasets with their annotations. ## Overview Scores a detector's predictions against ground truth, publishes a report with overall, per-class, and per-image results, and records the scores on the model version that made the predictions. The node's badge shows the run's mAP 50:95. Use it after [Predict YOLO](/node-library/nodes/models-yolo-predict-yolo), or another object detection node, has created predictions for a validation or test dataset. ```mermaid flowchart LR n1["Predict YOLO"] --> n2["Evaluate object detection"] ``` ## Prerequisites | Requirement | What you need | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Ground-truth dataset** | Existing [Dataset](/node-library/nodes/dataset-ops-dataset) with the reference object annotations. | | **Prediction dataset** | Existing [Dataset](/node-library/nodes/dataset-ops-dataset) with model annotations, for example from [Predict YOLO](/node-library/nodes/models-yolo-predict-yolo). Prediction labels should have a numeric `confidence` or `confidence-model` tag. | > **Warning** > > The node evaluates only asset IDs found in both datasets. It skips assets found on one side, so check dataset membership when you expect a complete comparison. ## How it works Reads the annotation of every image the two datasets share, as bounding boxes. Applies the class filter, confidence and IoU operating points, and the per-image detection limit. With `auto` confidence, it selects the score with the highest aggregate F1 value. Calculates metrics for the full dataset intersection and each selected ground-truth tag scope. The input datasets stay unchanged. Uploads the report and records the scores as an evaluation of the picked model version, so they appear wherever that model is listed. ## FAQ #### What can this node evaluate? The evaluator works with image bounding boxes. It does not evaluate videos, masks, keypoints, or tracking results. Other supported geometries are reduced to bounding boxes. A run fails when the two datasets together hold more than 600 classes, because the report charts cannot show more. #### Why does the node report \`No common assets with annotations\`? The datasets do not share any loaded image asset IDs. Build the prediction dataset from the same source assets instead of uploading copies as new assets. #### Why does the node report missing classes, annotations, or predictions? Class filtering may have removed all supported boxes, or one annotation set may have no valid boxes. Check the class names and make sure both datasets contain valid object annotations. #### Why is confidence rejected or the threshold analysis wrong? The value may be outside `0`-`1`, or prediction labels may lack numeric confidence tags. Missing confidence is treated as `1.0`. Use a value from `0` through `1`, or add numeric `confidence` or `confidence-model` tags. #### What happens to images without a recorded size? They are still evaluated, but their objects are left out of the object-size views (AP by size and error types by size), since relative size cannot be computed. The run log says how many images this affected. #### Why is there no report scope for a selected tag value? No shared ground-truth asset has that tag value. Select a value present on at least one shared image. #### What happens when I run it again? Each run uploads a new report and records a new evaluation on the model version. Earlier reports and evaluations stay, so the model keeps the history of every run. #### Why is the Evaluation output empty? The registry could not be reached when the run finished. The report is already uploaded by then, so a failure here costs the evaluation row, not the run — rerun the node to record it. ## Inputs **`Ground truth dataset`** `dataset` — required Dataset with ground-truth object annotations. Key: `GT_DATASET`. --- **`Predictions`** `dataset` — required Dataset holding the model predictions for the same assets as the ground truth, usually the output of a predict node. Only assets that appear in both datasets are evaluated. Key: `PRED_DATASET`. --- **`Model`** `model` — required Model version the predictions were made with. This run is recorded as an evaluation of that version, so its scores and a link to this report appear wherever the model is listed. Key: `MODEL`. --- **`Classes`** `ontology` Classes to include in the evaluation. Objects of any other class are ignored in both datasets, and an empty selection scores every class. Values come from `GT_DATASET`. Key: `CLASSES`. --- **`IoU threshold`** `select` — required, default: 0.5 Overlap a prediction needs with a ground truth object to count as the same object. Drives the confusion matrix and the true, false positive and missed counts. Key: `IOU_THRESHOLD`. Options: * 0.50 (`0.5`) * 0.55 (`0.55`) * 0.60 (`0.6`) * 0.65 (`0.65`) * 0.70 (`0.7`) * 0.75 (`0.75`) * 0.80 (`0.8`) * 0.85 (`0.85`) * 0.90 (`0.9`) * 0.95 (`0.95`) --- **`Confidence threshold`** `string` — required, default: auto Predictions scoring below this confidence are left out of the metrics. Keep auto to let the run pick the confidence with the best F1 score. Key: `CONFIDENCE_THRESHOLD`. --- **`Max detections`** `integer` — required, default: 100 Most predictions kept per image, highest confidence first. Anything beyond that count is ignored. Minimum: `1`. Key: `MAX_DETECTIONS`. --- **`Split by tags`** `asset_tags` Ground truth asset tags to break the report down by, for example weather or camera. Each selected tag value gets its own view of the report, and an empty selection reports on the whole dataset. Values come from `GT_DATASET`. Key: `TAGS`. --- ## Outputs **`Object detection evaluation report`** `report` Machine-readable object detection evaluation report for preview, compatible comparison nodes and Dataset insights. Shown as an artifact. Key: `REPORT_ID`. --- **`Evaluation`** `model_evaluation` The evaluation row this run recorded on the model version. Empty when the registry could not be reached. Key: `EVALUATION_ID`. --- **`mAP`** `string` mAP score shown as a badge on the node. Shown on the node as a badge. Key: `MAP_SCORE`. --- **`Confidence threshold`** `number` Confidence threshold the reported metrics were computed at - the F1-optimal score when Confidence Threshold is auto, otherwise the value you set. Optimal over all classes pooled, not per class. Key: `CONF_THRESHOLD`. --- **`Summary`** `object` Headline metrics table the node card shows. Hidden from the flow editor. Key: `SUMMARY`. --- ## Models and configuration This node evaluates existing predictions. It does not load or run a model. Keep the defaults for a standard evaluation, then change only the settings you need. #### Optional settings | Goal | Setting | | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | **Evaluate all classes** | Leave **Classes** empty. | | **Test a fixed operating point** | Set **IoU Threshold** and **Confidence Threshold** instead of using automatic confidence selection. | | **Compare a class subset** | Select the same class names used by both annotation sets. | | **Compare tagged cohorts** | Use **Split by Tags** to add scopes from ground-truth asset tags. | | **Attach results to a model** | Pick the **Model** version the predictions came from. Recording the evaluation is best-effort and does not affect report creation. | The **Model** picker lists your workspace's own models, filtered to object detection. Each run records one evaluation on the picked version: mAP 50:95, AP50, precision, recall, and F1 at the evaluated operating point, the ground-truth dataset, the list of classes the run scored, and a link to the report. Scores are only comparable between evaluations that used the same ground truth and the same classes. ## Report Start with these five signals. Then open the per-class and per-image views to find the errors. | Metric | What it tells you | | --------------- | --------------------------------------------------------------------------------------------------------------------------------- | | **mAP 50:95** | Overall detection quality across classes and increasingly strict box-overlap thresholds. | | **AP50 / AP75** | Whether errors come mainly from finding objects or placing boxes accurately. A large drop at AP75 suggests localization problems. | | **Precision** | How often predicted objects are correct. Low precision means more false positives. | | **Recall** | How many ground-truth objects are found. Low recall means more missed objects. | | **F1** | The balance between precision and recall at the active confidence threshold. | #### Metric glossary | Metric | Definition and interpretation | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **mAP 50:95** | Mean Average Precision across classes and COCO IoU thresholds from 0.50 through 0.95. | | **AP50** | Average Precision at IoU 0.50, a permissive measure of whether objects are found. | | **AP75** | Average Precision at IoU 0.75, which places more weight on accurate box localization. | | **AP at the IoU threshold** | Average Precision at the configured **IoU threshold**. Shown as an extra row when the threshold is neither 0.50 nor 0.75, and as a column of the per-class table. | | **Precision** | The share of predictions counted as correct at the active operating point. | | **Recall** | The share of ground-truth objects found at the active operating point. | | **F1** | The harmonic mean of precision and recall and the criterion used by `auto` confidence selection. | | **IoU** | Mean Intersection over Union for matched true-positive boxes after confidence filtering. | | **Classification accuracy** | The share of localized matches assigned the correct class. | | **Calibration** | `1 - expected calibration error` over ten confidence bins. Values closer to `1` indicate better agreement between confidence and observed correctness. | | **AP small / medium / large** | Average Precision restricted to objects whose box covers under 0.5% of the image area, up to 5%, and above 5%. Relative rather than COCO's absolute pixel bins, so the same object is binned the same way whatever resolution it was captured at. | | **AP gain** | How much Average Precision the model would recover if one category of error were corrected. Reported both for each error type on its own and for the types corrected cumulatively. | #### Error types Every detection that is not a true positive is placed in exactly one category, and every ground-truth object that no correction could recover is counted as a miss. The categories are evaluated in order, so the first one that applies wins. | Error type | Definition | | ------------------ | ------------------------------------------------------------------------------------------------- | | **Classification** | The box overlaps a ground-truth object closely enough, but carries the wrong class. | | **Localization** | The class is right, but the box overlaps the object more loosely than the IoU threshold requires. | | **Both** | Wrong class and a loose box at the same time. | | **Duplicate** | The box would have been correct, but a higher-scoring detection already claimed that object. | | **Background** | The box overlaps no ground-truth object at all. | | **Missed** | A ground-truth object that no correction to any detection could recover. | Error counts cover every detection the model produced, including those below the confidence threshold, because Average Precision ranks all detections. They therefore do not match the operating-point counts in **Outcome counts**. #### Report views The report covers the full dataset intersection and each non-empty tag scope. The scope selector switches every view to the selected subset. | Section | What it shows | | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Overview** | A verdict, headline metric cards, every detection metric at the operating point, and the model, dataset, and settings the run was given. | | **Per-class breakdown** | Average Precision by class, and a table with AP, AP at the configured IoU, TP, FP, FN, support, precision, recall, and F1 for every class. | | **Error decomposition** | The share of each error type, the AP each type would add if it were fixed, alone and cumulatively, and the error mix per class. | | **Confusion and errors** | A confusion matrix normalized by ground-truth class and the most frequently confused class pairs. | | **Outcome counts** | TP, FP, and FN totals at the operating point, overall and by class. | | **Recall and precision** | Recall and precision by class, and precision-recall curves by class. | | **Confidence and calibration** | Precision, recall, and F1 across confidence scores and IoU thresholds, the confidence distribution, mean IoU, calibration, and a reliability diagram. | | **Object size and crowding** | Average Precision and error mix by relative object size, and recall against objects per image and overlap with the nearest object. | | **Suspected annotation issues** | Confident predictions that suggest a missing label, a wrong class, or an imprecise box in the ground truth, each opening the ground truth beside the prediction. | | **Hardest images** | The images with the lowest F1, worst first, each opening the ground truth beside the predictions. [Sample images similar to failures](/node-library/nodes/sample-sample-images-similar-to-failures) starts its search from these images. | | **Explore predictions** | An image gallery and a per-image table of TP, FP, FN, precision, recall, and F1. | [Compare object detectors](/node-library/nodes/evaluate-compare-object-detectors) reads the model and dataset identity, categories, per-class metrics, and per-image metrics from this report. Compare reports only when they use the same ground-truth dataset, image set, annotations, categories, and evaluation settings. ## Runtime The node runs on a standard OvalBee worker. It does not need a GPU, CUDA, or a specific RTX model. Runtime grows with the number of shared images, detections, classes, and tag scopes. ## JSON config Machine-readable node interface for automation and advanced usage. #### Show JSON ```json { "name": "Evaluate object detection", "description": "Calculate object detection metrics by comparing predictions against ground truth, both supplied as separate datasets with their annotations.", "category": "Evaluate", "namespace": null, "templateKey": "evaluate/evaluate_object_detection", "version": "v1", "inputs": [ { "key": "GT_DATASET", "label": "Ground truth dataset", "type": "dataset", "description": "Dataset with ground-truth object annotations.", "required": true, "default": null, "visibleWhen": null, "options": { "creatable": false } }, { "key": "PRED_DATASET", "label": "Predictions", "type": "dataset", "description": "Dataset holding the model predictions for the same assets as the ground truth, usually the output of a predict node. Only assets that appear in both datasets are evaluated.", "required": true, "default": null, "visibleWhen": null, "options": { "creatable": false } }, { "key": "MODEL", "label": "Model", "type": "model", "description": "Model version the predictions were made with. This run is recorded as an evaluation of that version, so its scores and a link to this report appear wherever the model is listed.\n", "required": true, "default": null, "visibleWhen": null, "options": { "scope": "my", "task_type": "object_detection" } }, { "key": "CLASSES", "label": "Classes", "type": "ontology", "description": "Classes to include in the evaluation. Objects of any other class are ignored in both datasets, and an empty selection scores every class.", "required": false, "default": null, "visibleWhen": null, "options": { "ref": "GT_DATASET", "selectable": true, "bound": true } }, { "key": "IOU_THRESHOLD", "label": "IoU threshold", "type": "select", "description": "Overlap a prediction needs with a ground truth object to count as the same object. Drives the confusion matrix and the true, false positive and missed counts.", "required": true, "default": "0.5", "visibleWhen": null, "options": { "options": [ { "label": "0.50", "value": "0.5" }, { "label": "0.55", "value": "0.55" }, { "label": "0.60", "value": "0.6" }, { "label": "0.65", "value": "0.65" }, { "label": "0.70", "value": "0.7" }, { "label": "0.75", "value": "0.75" }, { "label": "0.80", "value": "0.8" }, { "label": "0.85", "value": "0.85" }, { "label": "0.90", "value": "0.9" }, { "label": "0.95", "value": "0.95" } ] } }, { "key": "CONFIDENCE_THRESHOLD", "label": "Confidence threshold", "type": "string", "description": "Predictions scoring below this confidence are left out of the metrics. Keep auto to let the run pick the confidence with the best F1 score.", "required": true, "default": "auto", "visibleWhen": null }, { "key": "MAX_DETECTIONS", "label": "Max detections", "type": "number", "description": "Most predictions kept per image, highest confidence first. Anything beyond that count is ignored.", "required": true, "default": 100, "visibleWhen": null, "options": { "type": "integer", "min": 1 } }, { "key": "TAGS", "label": "Split by tags", "type": "asset_tags", "description": "Ground truth asset tags to break the report down by, for example weather or camera. Each selected tag value gets its own view of the report, and an empty selection reports on the whole dataset.", "required": false, "default": null, "visibleWhen": null, "options": { "ref": "GT_DATASET", "selectable": true } } ], "outputs": [ { "key": "REPORT_ID", "label": "Object detection evaluation report", "type": "report", "kind": [ "object_detection_evaluation_report_v1", "dataset_intelligence_findings_v1" ], "description": "Machine-readable object detection evaluation report for preview, compatible comparison nodes and Dataset insights.", "artifact": true, "badge": false, "badgeOpens": null, "hidden": false }, { "key": "EVALUATION_ID", "label": "Evaluation", "type": "model_evaluation", "description": "The evaluation row this run recorded on the model version. Empty when the registry could not be reached.", "artifact": false, "badge": false, "badgeOpens": null, "hidden": false }, { "key": "MAP_SCORE", "label": "mAP", "type": "string", "description": "mAP score shown as a badge on the node.", "artifact": false, "badge": true, "badgeOpens": null, "hidden": false }, { "key": "CONF_THRESHOLD", "label": "Confidence threshold", "type": "number", "description": "Confidence threshold the reported metrics were computed at - the F1-optimal score when Confidence Threshold is auto, otherwise the value you set. Optimal over all classes pooled, not per class.", "artifact": false, "badge": false, "badgeOpens": null, "hidden": false }, { "key": "SUMMARY", "label": "Summary", "type": "object", "description": "Headline metrics table the node card shows.", "artifact": false, "badge": false, "badgeOpens": null, "hidden": true } ], "widgets": { "widget": { "id": "table", "settings": { "data": { "type": "variable", "value": "self.outputs.SUMMARY" } } }, "secondaryWidget": { "id": "report-preview", "settings": { "reportAsset": { "type": "variable", "value": "self.outputs.REPORT_ID" }, "name": { "type": "input", "value": "Open evaluation" } } } } } ``` ## References #### Sources and related pages These references define the evaluator and the compatible prediction workflow described above: * [pycocotools 2.0.10](https://pypi.org/project/pycocotools/2.0.10/) — the pinned COCO evaluation dependency used by the OvalBee nodes runtime. * [Microsoft COCO: Common Objects in Context](https://arxiv.org/abs/1405.0312) — the dataset and evaluation framework behind COCO-style object detection metrics. * [Predict YOLO](/node-library/nodes/models-yolo-predict-yolo) — creates a compatible prediction dataset while preserving source image asset IDs.