Node LibraryDatasets

Find annotation gaps

Overview

Compares ground-truth boxes with confident predictions from an external detector to find objects the annotators may have missed, and objects that may carry the wrong class. Nothing is changed automatically: missing-object candidates go into a review dataset, and class conflicts are listed in the report as suggestions.

A class is flagged when a large share of its confident predictions falls outside every existing ground-truth box. Only the uncovered predictions of flagged classes become candidates. The badge shows the number of candidate annotations, for example 1 annotation or 25 annotations, and opens the candidate dataset. The report can be passed to Dataset insights through its Findings from other nodes input.

A second prediction dataset, from a different model family, can cross-check the candidates and conflicts. Agreement is a review priority, not proof: even different detector families can share errors.

Ground-truth dataset Find annotation gaps Gap candidates

Prerequisites

RequirementWhat you need
Prediction datasetPredictions from an external pretrained detector, not one trained on the dataset under review, over exactly the same images as the ground truth. The node cannot verify training history.
Matching vocabularyClass names that match the ground-truth ones exactly. Map synonyms before running the node, and make sure the detector predicts the relevant classes: person-only predictions cannot reveal head or helmet confusion.
Rectangle annotationsBoxes only, in both datasets, in the same image coordinate system. Other geometries and mismatched annotation sizes stop the run.
Second prediction dataset (optional)Predictions from another detector, for the cross-check. It may cover only part of the images.

How it works

1

Reads the annotation of every image in each dataset, without downloading image pixels. An image with no ground-truth annotation counts as having no objects.

2

Checks whether each prediction at or above the confidence threshold is covered. A prediction is covered when any GT box has IoU strictly above Covered IoU, regardless of its class.

3

Divides each class’s uncovered predictions by the sum of its GT objects and uncovered predictions. A class is flagged when it has at least one candidate and its share reaches Gap share threshold.

4

Matches confident predictions one-to-one to GT boxes above Class match IoU, highest IoU first, then confidence, and marks matched pairs whose classes differ.

5

Writes new annotations containing only the uncovered, confident predictions of flagged classes into the candidate dataset. Images without those objects are excluded, and the input datasets stay unchanged.

FAQ

The ground-truth and prediction datasets must hold exactly the same images. Run the detector on the complete ground-truth dataset, then retry. The same applies to annotation sizes, which must match per image.

Dataset A ignores them, and the report and log say how many. Dataset B filters by its own Confidence threshold, 0.7 by default, and keeps every detection, scores or not, when that filter is off.

No. The candidate dataset holds missing-object additions only. Class conflicts are reviewed against the existing ground truth. The cross-check does not change which candidates are published either.

The first run creates it, and every rerun adds a new version to it, so earlier results stay in its history.

No. False detections and duplicates can inflate counts, and a class the detector does not predict can never be flagged.

Inputs

Prediction dataset A
datasetRequired

Predictions only, from an external pretrained detector, over the same image assets. Class names must match the ground-truth ones, and a prediction with no confidence score is ignored. Key: PRED_DATASET.

Confidence threshold
numberDefaults to 0.7Required

Predictions at or above this confidence are checked for missing annotations. Minimum: 0. Maximum: 1. Step: 0.01. Key: CONFIDENCE_THRESHOLD.

Prediction dataset B
dataset

Optional second source of predictions used to cross-check the candidates. Only shared image assets are checked, missing images remain unverified. Prefer predictions from a different model family, while recognizing that their errors may still correlate. Key: PRED_DATASET_2.

Confidence threshold
numberDefaults to 0.7

Confidence filter for dataset B, defaulting to 0.7 independently of dataset A. Disable it to retain all detections, including those without a score. Minimum: 0. Maximum: 1. Step: 0.01. Key: CONFIDENCE_THRESHOLD_2.

Ground-truth dataset
datasetRequired

Original object detection annotations. Must contain exactly the same image assets as the prediction dataset. Key: GT_DATASET.

Covered IoU
numberDefaults to 0.1Required

Overlap at which a prediction counts as already annotated, whatever class the existing annotation has. Minimum: 0. Maximum: 1. Step: 0.01. Key: IOU_COVERED.

Class match IoU
numberDefaults to 0.7Required

Overlap above which a prediction and a ground-truth annotation are treated as the same object, so a difference between their classes is reported as a class conflict. Minimum: 0. Maximum: 1. Step: 0.01. Key: IOU_MATCH.

Gap share threshold
numberDefaults to 0.3Required

Flag a class when this share of its objects looks missing. The share is uncovered predictions out of all its objects, annotated and uncovered together. Minimum: 0. Maximum: 1. Step: 0.01. Key: GAP_SHARE_THRESHOLD.

Outputs

Annotation gaps report
report

Per-class gap statistics, GT-to-prediction class conflicts above the configured class match IoU, and clickable examples. Includes findings.json for analytical nodes. Shown as an artifact. Key: REPORT_ID.

Gap candidates
dataset

Original images with only confident uncovered predictions from flagged classes. Images without candidates are excluded. Shown as an artifact. Key: GAP_DATASET.

Gap annotations
string

Number of candidate annotations published for review. Click to open the candidate dataset. Shown on the node as a badge. Selecting the badge opens GAP_DATASET. Key: GAP_CLASSES_COUNT.

Report

The report has tabs for Overview, Missing objects, Class conflicts, and, when a second prediction dataset is set, Cross-check with dataset B. The overview summarizes the run, each review tab explains its results and opens annotation comparisons, and the selection rules and limitations sit in a collapsed explanation at the end.

Missing objects has the class table: GT counts, uncovered predictions, their combined total, the gap share, distinct images with candidates, and whether the class is flagged. Classes without confident predictions show No evidence, not a zero gap rate, and Covered means every confident prediction overlaps a GT box, which may belong to another class. Click a row to inspect up to 20 images with GT and prediction overlays. The candidate dataset contains every qualifying image, whatever the preview limit. Examples mix higher and lower confidence with images holding many candidates, so they are diagnostic, not a random sample for estimating precision.

Class conflicts shows each GT class to predicted class pair with object and image counts, median IoU and confidence, and side-by-side examples. Matching happens before classes are compared, so duplicate predictions cannot flag one GT object repeatedly.

Cross-check with dataset B counts supported candidates, different alternatives, unconfirmed candidates, and unchecked candidates separately. A missing-object candidate is supported only when both datasets predict the same class and both boxes stay uncovered by GT. For class conflicts, both datasets are matched to the same GT object, and the tab separates the same alternative class from two different alternatives that both disagree with GT. Images dataset B does not cover are Not checked, not failed confirmations. A gallery shows up to four images per outcome with the GT and prediction panels side by side.

Counts are a snapshot of the run and do not follow later review decisions.

JSON config

Machine-readable node interface for automation and advanced usage.

{
  "name": "Find annotation gaps",
  "description": "Find missing annotations and possible class confusion using confident external predictions. Prepare missing-object candidates and report class conflicts for human review. Use predictions from an external model, not one trained on the same dataset.",
  "category": "Datasets",
  "namespace": null,
  "templateKey": "dataset_ops/find_annotation_gaps",
  "version": "v1",
  "inputs": [
    {
      "key": "PRED_DATASET",
      "label": "Prediction dataset A",
      "type": "dataset",
      "description": "Predictions only, from an external pretrained detector, over the same image assets. Class names must match the ground-truth ones, and a prediction with no confidence score is ignored.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "CONFIDENCE_THRESHOLD",
      "label": "Confidence threshold",
      "type": "number",
      "description": "Predictions at or above this confidence are checked for missing annotations.",
      "required": true,
      "default": 0.7,
      "visibleWhen": null,
      "options": {
        "min": 0,
        "max": 1,
        "step": 0.01
      }
    },
    {
      "key": "PRED_DATASET_2",
      "label": "Prediction dataset B",
      "type": "dataset",
      "description": "Optional second source of predictions used to cross-check the candidates. Only shared image assets are checked; missing images remain unverified. Prefer predictions from a different model family, while recognizing that their errors may still correlate.",
      "required": false,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "CONFIDENCE_THRESHOLD_2",
      "label": "Confidence threshold",
      "type": "number",
      "description": "Confidence filter for dataset B, defaulting to 0.7 independently of dataset A. Disable it to retain all detections, including those without a score.",
      "required": false,
      "default": 0.7,
      "visibleWhen": null,
      "options": {
        "min": 0,
        "max": 1,
        "step": 0.01
      }
    },
    {
      "key": "GT_DATASET",
      "label": "Ground-truth dataset",
      "type": "dataset",
      "description": "Original object detection annotations. Must contain exactly the same image assets as the prediction dataset.",
      "required": true,
      "default": null,
      "visibleWhen": null,
      "options": {
        "creatable": false
      }
    },
    {
      "key": "IOU_COVERED",
      "label": "Covered IoU",
      "type": "number",
      "description": "Overlap at which a prediction counts as already annotated, whatever class the existing annotation has.",
      "required": true,
      "default": 0.1,
      "visibleWhen": null,
      "options": {
        "min": 0,
        "max": 1,
        "step": 0.01
      }
    },
    {
      "key": "IOU_MATCH",
      "label": "Class match IoU",
      "type": "number",
      "description": "Overlap above which a prediction and a ground-truth annotation are treated as the same object, so a difference between their classes is reported as a class conflict.",
      "required": true,
      "default": 0.7,
      "visibleWhen": null,
      "options": {
        "min": 0,
        "max": 1,
        "step": 0.01
      }
    },
    {
      "key": "GAP_SHARE_THRESHOLD",
      "label": "Gap share threshold",
      "type": "number",
      "description": "Flag a class when this share of its objects looks missing. The share is uncovered predictions out of all its objects, annotated and uncovered together.",
      "required": true,
      "default": 0.3,
      "visibleWhen": null,
      "options": {
        "min": 0,
        "max": 1,
        "step": 0.01
      }
    }
  ],
  "outputs": [
    {
      "key": "REPORT_ID",
      "label": "Annotation gaps report",
      "type": "report",
      "kind": "dataset_intelligence_findings_v1",
      "description": "Per-class gap statistics, GT-to-prediction class conflicts above the configured class match IoU, and clickable examples. Includes findings.json for analytical nodes.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "GAP_DATASET",
      "label": "Gap candidates",
      "type": "dataset",
      "description": "Original images with only confident uncovered predictions from flagged classes. Images without candidates are excluded.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "GAP_CLASSES_COUNT",
      "label": "Gap annotations",
      "type": "string",
      "description": "Number of candidate annotations published for review. Click to open the candidate dataset.",
      "artifact": false,
      "badge": true,
      "badgeOpens": "GAP_DATASET",
      "hidden": false
    }
  ],
  "automation": {
    "requires_configuration": true
  },
  "widgets": {
    "widget": {
      "id": "report-preview",
      "settings": {
        "reportAsset": {
          "type": "variable",
          "value": "self.outputs.REPORT_ID"
        },
        "name": {
          "type": "input",
          "value": "Open findings"
        }
      }
    }
  }
}

References