Node LibrarySample

Review suspected label errors

Overview

Collects the images an object detection evaluation report flagged as probable ground-truth label mistakes into a dataset a labeler can open. When a model confidently disagrees with a label, one of the two is wrong, and it is not always the model. Evaluate object detection already records those disagreements in its report, and this node reads them from there.

It selects images, not objects: a reviewer opens an image once however many things were flagged on it, so an image carrying five loose boxes is ranked by its most confident one rather than by the count.

Each rerun writes a new version of the node’s own hidden LAST_RUN dataset holding that run’s selection, and the badge shows how many images it selected and opens it. Set a destination dataset to accumulate across runs as well.

Evaluate object detection Review suspected label errors Labeling and review

Prerequisites

RequirementWhat you need
An evaluation reportAn object detection evaluation report from the Evaluate object detection node. Reports produced before this node existed do not record suspected label errors, so run the evaluation again for those.
The dataset the report evaluatedImages and annotations are copied from there, so point Source dataset at the same ground-truth dataset the report was built from.

How it works

1

Reads the suspected label errors from the report — the candidates the evaluation already ranked, not a fresh analysis.

2

Keeps the issue types you asked for, drops anything the model was less sure about than the minimum confidence, and reduces the rest to one entry per image.

3

Copies up to Sample size of the most confident images, with their annotations, into the node’s own hidden dataset, and into your destination dataset if you set one.

FAQ

Issue typeWhat the model saw
Missing labelsA confident prediction where the ground truth has nothing at all. Either an object nobody labeled, or a false positive.
Wrong classesA well-placed box whose class disagrees with the label on it.
Loose boxesA prediction that overlaps a labeled object of the same class only loosely. Noisier than the other two, because a merely imprecise model produces these as well.

No. These are candidates, not verdicts. Every one of them is equally consistent with the model being wrong, which is why the node hands them to a reviewer instead of editing anything.

The report applies its own confidence floor while collecting candidates, and caps how many it records per issue type. Lowering Minimum confidence here therefore cannot widen the selection past what the report already recorded.

At least one of Missing labels, Wrong classes, or Loose boxes must be on.

The run succeeds with a count of zero, and LAST_RUN is a new, empty version. Include more issue types or lower Minimum confidence, within what the report recorded.

The flagged images are still added, but they arrive without annotations, because only annotations found in Source dataset are copied.

Inputs

Evaluation report
reportRequired

Object detection evaluation report the suspected label errors are read from. Generic reports are not supported. Key: REPORT_ID.

Source dataset
datasetRequired

Dataset the report evaluated. The selected images and their annotations are copied from here. Key: SRC_DATASET.

To dataset
dataset

Dataset that also receives the selection and keeps collecting it across runs. Leave empty to write only the node’s hidden last-run dataset. Key: DST_DATASET.

Missing labels
booleanDefaults to trueRequired

Include images where the model confidently found an object nobody labeled. Key: INCLUDE_MISSING_LABEL.

Wrong classes
booleanDefaults to trueRequired

Include images where the model disagrees with the class on a well-placed box. Key: INCLUDE_WRONG_CLASS.

Loose boxes
booleanDefaults to trueRequired

Include images where the model’s box overlaps the labeled one only loosely. Noisier than the other two. Key: INCLUDE_LOOSE_BOX.

Sample size
integerDefaults to 100Required

Maximum number of images to select. Fewer are returned when the report flagged fewer. Minimum: 1. Maximum: 10000. Step: 1. Key: SAMPLE_SIZE.

Minimum confidence
numberDefaults to 0.5Required

Ignore flagged objects the model was less sure about than this. The report already applies its own floor, so lowering this cannot widen the selection past what it recorded. Minimum: 0. Maximum: 1. Step: 0.01. Key: MIN_CONFIDENCE.

Outputs

Dataset
dataset

Key: DATASET.

Last run
dataset

Hidden dataset holding this run’s selection. A rerun writes a new version of it. Shown as an artifact. Key: LAST_RUN.

Selected images
string

Number of image assets attached to the selection. Shown on the node as a badge. Selecting the badge opens LAST_RUN. Key: SELECTED_IMAGES.

JSON config

Machine-readable node interface for automation and advanced usage.

{
  "name": "Review suspected label errors",
  "description": "Collects the images an object detection evaluation report flagged as probable ground-truth label mistakes into a dataset, ready to send for re-labeling.",
  "category": "Sample",
  "namespace": "ovalbee",
  "templateKey": "sample/review_suspected_label_errors",
  "version": "v1",
  "inputs": [
    {
      "key": "REPORT_ID",
      "label": "Evaluation report",
      "type": "report",
      "kind": "object_detection_evaluation_report_v1",
      "description": "Object detection evaluation report the suspected label errors are read from. Generic reports are not supported.",
      "required": true,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "SRC_DATASET",
      "label": "Source dataset",
      "type": "dataset",
      "description": "Dataset the report evaluated. The selected images and their annotations are copied from here.",
      "required": true,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "DST_DATASET",
      "label": "To dataset",
      "type": "dataset",
      "description": "Dataset that also receives the selection and keeps collecting it across runs. Leave empty to write only the node's hidden last-run dataset.",
      "required": false,
      "default": null,
      "visibleWhen": null
    },
    {
      "key": "INCLUDE_MISSING_LABEL",
      "label": "Missing labels",
      "type": "boolean",
      "description": "Include images where the model confidently found an object nobody labeled.",
      "required": true,
      "default": true,
      "visibleWhen": null
    },
    {
      "key": "INCLUDE_WRONG_CLASS",
      "label": "Wrong classes",
      "type": "boolean",
      "description": "Include images where the model disagrees with the class on a well-placed box.",
      "required": true,
      "default": true,
      "visibleWhen": null
    },
    {
      "key": "INCLUDE_LOOSE_BOX",
      "label": "Loose boxes",
      "type": "boolean",
      "description": "Include images where the model's box overlaps the labeled one only loosely. Noisier than the other two.",
      "required": true,
      "default": true,
      "visibleWhen": null
    },
    {
      "key": "SAMPLE_SIZE",
      "label": "Sample size",
      "type": "number",
      "description": "Maximum number of images to select. Fewer are returned when the report flagged fewer.",
      "required": true,
      "default": 100,
      "visibleWhen": null,
      "options": {
        "type": "integer",
        "min": 1,
        "max": 10000,
        "step": 1,
        "slider": false
      }
    },
    {
      "key": "MIN_CONFIDENCE",
      "label": "Minimum confidence",
      "type": "number",
      "description": "Ignore flagged objects the model was less sure about than this. The report already applies its own floor, so lowering this cannot widen the selection past what it recorded.",
      "required": true,
      "default": 0.5,
      "visibleWhen": null,
      "options": {
        "type": "float",
        "min": 0,
        "max": 1,
        "step": 0.01
      }
    }
  ],
  "outputs": [
    {
      "key": "DATASET",
      "label": "Dataset",
      "type": "dataset",
      "description": "",
      "artifact": false,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "LAST_RUN",
      "label": "Last run",
      "type": "dataset",
      "description": "Hidden dataset holding this run's selection. A rerun writes a new version of it.",
      "artifact": true,
      "badge": false,
      "badgeOpens": null,
      "hidden": false
    },
    {
      "key": "SELECTED_IMAGES",
      "label": "Selected images",
      "type": "string",
      "description": "Number of image assets attached to the selection.",
      "artifact": false,
      "badge": true,
      "badgeOpens": "LAST_RUN",
      "hidden": false
    }
  ],
  "automation": {
    "isolated": true,
    "requires_configuration": true,
    "manual_run": true,
    "discard_history": "on_success"
  }
}